CS662

Advanced Natural Language Processing

Staff

Instructor

Jonathan May

Office Hours: Mondays 12:00-1:00 pm TBD and Wednesdays 9:00–9:50 am TBD, or by appointment (at ISI on other days)

Teaching Assistant

Alexander Spangher

spangher@usc.edu

Office Hours: 12-2pm Monday, RTH 314

Lectures

Monday and Wednesday 10:00–11:50 am, DMC 261
See schedule for select days where class is canceled

Textbook

Optional: Natural Language Processing - Eisenstein (‘E’ in schedule) – or free version
Optional: Speech and Language Processing 3rd edition - Jurafsky, Martin (‘JM’ in schedule) – February 2024 pdf
Required: Selected papers from NLP literature, see (evolving) schedule

Grading

Percentage	Assessment Component
10%	In class participation
10%	Posted questions before each in-class selected paper presentation and possible quizzes
10%	In-class selected paper presentation
30%	Three Homeworks (10% each)
40%	Project, done in small groups, comprising:
	- Proposal (5%)
	- First version of report (5%)
	- In-class presentation (10%)
	- Final report (20%).

Written homeworks and project components except for final project report must be submitted on the date listed in the schedule, by 23:59:59 AoE.
Final project report is due Monday, December 16, 2024, 10:00 AM PST
A deduction of 1/5 of the total possible score will be assessed for each late day. After four late days (i.e. on the fifth), you get a 0 on the assignment (and you should come talk to us because your grade will likely suffer!)
You have four extension days, to be applied as you wish, throughout the entire class, for homeworks and project proposal / first report (NOT final report). No deduction will be assessed if an extension day is used. As an example, if an assignment is due November 10, you have two extension days remaining, you submit the assignment on November 12, and your score is 90/100. In this case you lose the extension days but your grade is not reduced; it remains 90/100. If you have one extension day, you lose it, and your grade is 70/100. If you have no extension days, your grade is 50/100.

Contact us

On Slack, or in class/office hours. Please do not email (unless notified otherwise).

Topics

(subject to change per instructor/class whim) (will not necessarily be presented in this order):

Fundamentals: Linguistic Stack (graphemes/phones - words - syntax - semantics - pragmatics - discourse; Corpora, Corpus statistics, Data cleaning, munging, and annotation; Evaluation; Linear and Nonlinear Models; Dense Representations and neural architectures (feed-forward, RNN, Transformer); Language Models; Pre-training, Fine-tuning, Prompting, Reward Alignment; Ethics; Effective written and oral communication

Applications: Multilingualism and Translation; Syntax; Information Retrieval/Question Answering; Dialogue; Information Extraction; Multimodality; Speech Recognition and Generation; Agent Interaction; Discourse

Week 1

Aug 26

Introduction, Applications: E 1, Probabilities (refresher only)

project assignment out (due 9/23)

paper selection out (due 9/9) :-

Aug 28

Data Processing. Data Resources, Evaluation, Annotation: E 4.4-4.5, JM 2, 4.7-4.9, Nathan Schneider’s unix notes, Unix for poets, sculpting text, Suleyman Reading 1, Suleyman Reading 2, Hank Green Video

Week 2

Sep 2

LABOR DAY NO CLASS

HW1 out (due 9/20)

Sep 4

Linear Classifiers: E 2.2, 2.3, 2.4. JM 4, 5, Thumbs up? Sentiment Classification using Machine Learning Techniques, Goldwater probability tutorial. The Perceptron (Rosenblatt 1958) (optional)

Week 3

Sep 9

NO CLASS

Sep 11

Non-linear Classifiers, Backprop, Gradient Descent: E 3. JM 7.2–7.5 :

Sep 13

Early Drop (no W, refund)

Week 4

Sep 16

Distributional Feature Representations: PPMI, LSA, word2vec

E 14.3, 14.5–6. JM 6, LSA via SVD, Linguistic regularities in continuous space word representations, Efficient Estimation of Word Representations in Vector Space, Distributed Representations of Words and Phrases and their Compositionality

Sep 18

N-Gram Language Models, Feed Forward and Recurrent Language Models (RNNs): E 6.1–2, 6.4. 7.5, 7.7. JM 3 Exploring the limits of language modeling LM notebook

Sep 20

HW1 due

Week 5

Sep 23

Attention, Transformer Language Models

E 6.3, JM 9, 10. Attention is all you need

project proposal due

Sep 25

Pretrained language models (ELMo, BERT, and sentence similarity): JM 11.1–11.3 ELMo paper BERT paper Zoph Fine-Tuning paper Fine-Tuning demo

Week 6

Sep 30

Prompting and Large Language Models: JM 11.4 T5 LoRA Prefix Tuning T0

HW2 out (due 10/18)

Oct 2

MEGA (Guest Lecture by Xuezhe Ma): Mega Paper Megalodon; Chumeng Liang - Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning; Questions by: Mia Sultan; Anirudh Ravi Kumar - Teaching Language Models to Self-Improve through Interactive Demonstrations; Questions by: Debaditya Pal

Week 7

Oct 7

Reinforcement Learning with Human Feedback: Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO): Ziegler RLHF Paper, DPO Paper; Emily Weiss - Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration; Questions by: Yifan Jiang; Ayush Goyal - R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’; Questions by: Wen Ye

Oct 9

Ethics (Guest Lecture by Katy Felkner): The Social Impact of Natural Language Processing, Energy and Policy Considerations for Deep Learning in NLP, Model Cards for Model Reporting; (MOVED TO 11/25) Mia Sultan - LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback; Questions by: Wilber Blas Urrutia; Eric Boxer - MacGyver: Are Large Language Models Creative Problem Solvers?; Questions by: Ryan Lee

Oct 11

Mid Drop (No W, No refund)

Week 8

Oct 14

Machine Translation (MT) slides1 slides2

JM13: Skyler Hallinan - How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs; Questions by: Joshua Robinson; Tianyi Zhang - Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models; Questions by: Hanwen Xing

Oct 16

Multilingual: Chiang, Ting-Rui - LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models; Questions by: Run Huang; Enes Burak Bilgin - Steering Llama 2 via Contrastive Activation Addition; Questions by: Skyler Hallinan

Oct 18

HW 2 due

Week 9

Oct 21

Information Retrieval (IR) and Question Answering (QA)

JM 14: Lucine Oganesian - Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense; Questions by: Anirudh Ravi Kumar; Zeyu Liu - LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression; Questions by: Fandel Lin

Oct 23

JM 15: Yifan Jiang - Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions; Questions by: Emily Weiss; Liu, Lei - Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models; Questions by: Enes Burak Bilgin

Week 10

Oct 28

Information Extraction

HW3 out (due 11/22)

JM17.3, 20: Siniukov, Maksim - An Iterative Associative Memory Model for Empathetic Response Generation; Questions by: Cheng-Han Wu; Pal, Debaditya - Answer is All You Need: Instruction-following Text Embedding via Answering the Question; Questions by: Xinyan Yu

Oct 30

Agents (Guest Lecture by Tenghao Huang)

WebArena, ToolLLM, Narrative Discourse, ReAct: Lee, Ryan - MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions; Questions by: Ayush Goyal; Yu, Xinyan - Spiral of Silence: How is Large Language Model Killing Information Retrieval? A Case Study on Open Domain Question Answering; Questions by: Maksim Siniukov

Week 11

Nov 4

Multimodal NLP (Guest Lecture by Xuezhe Ma): Huang, Run - Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models; Questions by: Eric Boxer; Robinson, Joshua - I am a Strange Dataset: Metalinguistic Tests for Language Models; Questions by: Zeyu Liu

Nov 6

Spoken Language Processing (SLP) (Guest Lecture by Sudarsana Reddy Kadiri)

JM 16: Ye, Wen - CaMML: Context-Aware Multimodal Learner for Large Models; Questions by: Lei Liu; (MOVED TO 11/25) Xing, Hanwen - Visual Grounding Helps Learn Word Meanings in Low-Data Regimes; Questions by: Ziyi Liu

Nov 8

Project Report Version 1 due

Week 12

Nov 11

VETERANS DAY NO CLASS

Nov 13

EMNLP NO CLASS: -

Nov 15

Late Drop (W, No refund)

Week 13

Nov 18

Syntax (Not actually presenting this year, in lieu of IE, which is moved. Legacy notes for background interest) POS/HMM, Constituencies, Dependencies: Liu, Ziyi - Evaluating the Deductive Competence of Large Language Models; Questions by: Lucine Oganesian; Lin, Fandel - VariErr NLI: Separating Annotation Error from Human Label Variation; Questions by: Ting-Rui Chiang

Nov 20

Discourse (Guest Lecture by Alexander Spangher) Slides: Wu, Cheng-Han - Grounding Gaps in Language Model Generations; Questions by: Zhang, Tianyi; Wilber Blas Urrutia - Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?; Questions by: Chumeng Liang

Nov 22

HW 3 due

Week 14

Nov 25

Guest Lecture by Jonathan Choi (USC Gould School of Law) Paper

(MOVED FROM 11/6) Xing, Hanwen - Visual Grounding Helps Learn Word Meanings in Low-Data Regimes

Questions by: Ziyi Liu

(MOVED FROM 10/9) Mia Sultan - LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback

Questions by: Wilber Blas Urrutia

Nov 27

THANKSGIVING BREAK; NO CLASS

Week 15

Dec 2

Project Presentations: (10:00) Ziyi Liu, Ayush Goyal - Editing Common Sense in Transformers; Questions by: Xinyan Velocity Yu, Joshua Robinson, Chumeng Liang; (10:22) Ting-Rui Chiang, Fandel Lin - Anchoring Fine-tuning of Sentence Transformer with Semantic Label Information for Efficient Truly Few-shot Classification; Questions by: Wilber Leonardo Blas Urrutia, Skyler Hallinan; (10:44) Anirudh Ravi Kumar, Maksim Siniukov, Tianyi Zhang - Learning Retrieval Augmentation for Personalized Dialogue Generation; Questions by: Eric Boxer, Yifan Jiang, Lei Liu; (11:06) Wen Ye, Cheng-Han Wu, Zeyu Liu - SMoP: Towards efficient and effective prompt tuning with sparse mixture-of-prompts; Questions by: Run Huang, Mia Sultan; (11:28) Lucine Oganesian, Enes Burak Bilgin, Debaditya Pal - Adapting Language Models to Compress Contexts; Questions by: Ryan Lee, Emily Weiss, Hanwen Xing

Dec 4

Project presentations: (10:00) Xinyan Velocity Yu, Run Huang - Visual sketchpad: Sketching as a visual chain of thought for multimodal language models.; Questions by: Enes Burak Bilgin, Ziyi Liu, Lucine Oganesian; (10:22) Joshua Robinson, Ryan Lee, Hanwen Xing - Context Compression for Auto-regressive Transformers with Sentinel Tokens; Questions by: Wen Ye, Fandel Lin; (10:44) Emily Weiss, Skyler Hallinan, Wilber Leonardo Blas Urrutia - Text embeddings reveal (almost) as much as text; Questions by: Ting-Rui Chiang, Anirudh Ravi Kumar, Cheng-Han Wu; (11:06) Eric Boxer, Mia Sultan - Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation; Questions by: Tianyi Zhang, Maksim Siniukov; (11:28) Yifan Jiang, Chumeng Liang, Lei Liu - Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration; Questions by: Zeyu Liu, Ayush Goyal, Debaditya Pal