CSCI 662 Fall 2020 course page

Jonathan May

Website	`https://www.isi.edu/~jonmay/cs662_fa20_web/`
Lectures	`https://usc.zoom.us/j/92532775397` (password on piazza) , Mondays and Wednesdays 10:00–11:50 am
Instructor & office hours	Jonathan May, Online, Mondays and Wednesdays 9:00–10:00 am or by appointment (same room as lectures)
TAs & Office hours	Mozhdeh Gheini, Tuesdays and Thursdays, 1:00–2:00 pm, Online
	Meryem M'Hamdi, Mondays and Wednesdays, 4:00–5:00 pm, Online
Textbook	Required: Natural Language Processing - Eisenstein¹
	Required: Selected papers from NLP literature, see (evolving) schedule
	Optional: Introduction to Deep Learning - Charniak ²
	Optional: Speech and Language Processing 3rd edition -Jurafsky, Martin ³
Grading	10 %: In-class participation
	10 %: Posted questions before each in-class selected paper presentation
	10 %: In-class selected paper presentation
	30 %: Three Homeworks (10% each)
	40 %: Project, comprising proposal (10%), final conference-quality paper (15%), and 15-minute in-class presentation (15%) (may be done in small groups). Final report is due December 7, 2020, 10:00 AM PST
Contact us	On Piazza or in class/office hours. Please do not email (unless notified otherwise).

Topics (subject to change per instructor/class whim) (will not be presented in this order):

Linguistic Stack (graphemes/phones - words - syntax - semantics - pragmatics - discourse)
Tools:
- Corpora, Corpus statistics, Data cleaning and munging
- Annotation and crowdwork
- Evaluation
- Models/approaches: rule-based, automata/grammars, perceptron, logistic regression, neural network models
- Effective written and oral communication
Components/Tasks/Subtasks:
- Language Models
- Syntax: POS tags, constituency tree, dependency tree, parsing
- Semantics: lexical, formal, inference tasks
- Information Extraction: Named Entities, Relations, Events
- Generation: Machine Translation, Summarization, Dialogue, Creative Generation

date material reading presentation Other TA attending

8/24 intro, applications Eisenstein 1 (not mandatory) Meryem & Mozhdeh

8/26 end of intro, probability basics Eisenstein Appendix A, Goldwater probability tutorial ⁴ project assignment out (due 9/9) Meryem

8/31
probability, ethics, naive bayes Eisenstein 2, Nathan Schneider's unix notes⁵, Unix for poets⁶, sculpting text⁷ The Social Impact of Natural Language Processing⁸ Presenter: Jon -

9/2 Perceptron, Logistic Regression Eisenstein 3, Charniak 1. Thumbs up? Sentiment Classification using Machine Learning Techniques⁹ Presenter: Zekun HW1 out (due 9/30) Meryem

9/7 LABOR DAY NO CLASS

9/9 Nonlinear classifiers, backpropagation, gradient descent Eisenstein 7 Fast Semantic Extraction Using a Novel Neural Network Architecture¹⁰ Presenter: Tooraj project proposal due Mozhdeh

9/14 POS tags, HMMs Eisenstein 9,2, 10. Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments¹¹ Presenter: Negar Mozhdeh

9/16 cky, constituencies, treebank Eisenstein 11 Building a Large Annotated Corpus of English¹² Presenters: Ani, Sabyasachee Meryem

9/21 viterbi cky, restructuring, dependencies, shift-reduce Eisenstein 4.5. A Fast and Accurate Dependency Parser using Neural Networks¹³ Presenter: Shweta. Mozhdeh

9/23 arc-eager, evaluation, human annotation Eisenstein 13, 14. An Empirical Investigation of Statistical Significance in NLP ¹⁴ Presenter: Nikolaos HW2 out (due 10/21) Meryem

9/28 NO CLASS

9/30 mechanical turk, semantics: word sense, propbank, amr, distributional Eisenstein 7 Linguistic Regularities in Continuous Space Word Representations¹⁵. Presenter: Hongkuan The word analogy testing caveat¹⁶ Presenter: Jihoon HW1 due Meryem

10/5 language models: ngram, feed-forward, recurrent Machine Translation history, evaluation Eisenstein 18.1, 18.2 Bleu: a Method for Automatic Evaluation of Machine Translation¹⁷ Presenter: Paras Towards a Literary Machine Translation: The Role of Referential Cohesion¹⁸ Presenter: Katy Mozhdeh

10/7 Statistical, Neural Machine Translation, summarization, generation Eisenstein 18.3, 19.1, 19.2 Effective Approaches to Attention-based Neural Machine Translation¹⁹ Presenter: Xiou Neural Machine Translation by Jointly Learning to Align and Translate²⁰ Presenter: Soumya Revised proposals due (no late days) Meryem

10/12 Transformers Attention is all you need ²¹, Illustrated Transformer ²²
Six Challenges for Neural Machine Translation²³ Presenter: Yuchen. Get To The Point: Summarization with Pointer-Generator Networks²⁴ Presenter: Qi.
Mozhdeh

10/14 Large Contextualized Language Models (ElMo, BERT, GPT-N, etc.) Illustrated BERT, ElMo, and co.²⁵ Universal Neural Machine Translation for Extremely Low Resource Languages ²⁶ Presenter: Amirhesam. Defending Against Neural Fake News ²⁷ Presenter: Yizhou. HW3 out (due 11/11) Meryem

10/19 Guest Lecture (Xuezhe Ma) Language Models are Unsupervised Multitask Learners ²⁸ Presenter: I-Hung. Language Models are Few-Shot Learners²⁹ Presenters: Yufei, Wenxuan. Mozhdeh

10/21 Information Extraction: Entity/Relation, CRF Eisenstein 17.1, 17.2 25 years of IE³⁰ Presenter: Justin. HW2 Due Meryem

10/26 Information Extraction: Events, Zero-shot Eisenstein 17.3 Events are Not Simple: Identity, Non-Identity, and Quasi-Identity³¹ Presenter: Basel Mozhdeh

10/28 Blade Runner NLP/Bertology GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding³² Presenter: Prateek Meryem

11/2 Text Games and Reinforcement Learning The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives³³ Presenter: Shuai Mozhdeh

11/4 Dialogue Eisenstein 19.3. A Diversity-Promoting Objective Function for Neural Conversation Models³⁴ Presenter: Peifeng Personalizing Dialogue Agents: I have a dog, do you have pets too?³⁵ Presenter: Akshat Meryem

11/9
Power and Ethics Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data³⁶ Presenter: Ani Energy and Policy Considerations for Deep Learning in NLP ³⁷ Presenter: Ali A. Mozhdeh

11/11
How to write a paper Neubig slides on Piazza On Measuring Social Biases in Sentence Encoders ³⁸ Presenter: Katy HW3 Due Meryem

11/16
Presentations Meryem & Mozhdeh

11/18 Presentations Meryem & Mozhdeh

11/23
Presentations Meryem & Mozhdeh

Footnotes

... Eisenstein ¹: https://mitpress.mit.edu/books/introduction-natural-language-processing or free version https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstein-nlp-notes.pdf
... Charniak ²: https://mitpress.mit.edu/books/introduction-deep-learning (first three chapters at https://cs.brown.edu/courses/csci1460/assets/files/deep-learning.pdf)
... Martin ³: https://web.stanford.edu/~jurafsky/slp3/
... tutorial ⁴: http://homepages.inf.ed.ac.uk/sgwater/teaching/general/probability.pdf
... notes ⁵: https://github.com/nschneid/unix-text-commands
... poets ⁶: https://www.cs.upc.edu/~padro/Unixforpoets.pdf
... text ⁷: http://matt.might.net/articles/sculpting-text/
... Processing ⁸: https://www.aclweb.org/anthology/P16-2096/
... Techniques ⁹: https://www.aclweb.org/anthology/W02-1011/
... Architecture ¹⁰: https://www.aclweb.org/anthology/P07-1071/
... Experiments ¹¹: https://www.aclweb.org/anthology/P11-2008/
... English ¹²: https://www.aclweb.org/anthology/J93-2004.pdf
... Networks ¹³: https://www.aclweb.org/anthology/D14-1082/
... NLP ¹⁴: https://www.aclweb.org/anthology/D12-1091/
... Representations ¹⁵: https://www.aclweb.org/anthology/N13-1090.pdf
... caveat ¹⁶: https://www.aclweb.org/anthology/N18-2039.pdf
... Translation ¹⁷: https://www.aclweb.org/anthology/P02-1040
... Cohesion ¹⁸: https://www.aclweb.org/anthology/W12-2503/
... Translation ¹⁹: https://www.aclweb.org/anthology/D15-1166/
... Translate ²⁰: https://arxiv.org/abs/1409.0473
... need ²¹: https://arxiv.org/abs/1706.03762
... Transformer ²²: http://jalammar.github.io/illustrated-transformer/
... Translation ²³: https://www.aclweb.org/anthology/W17-3204/
... Networks ²⁴: https://www.aclweb.org/anthology/P17-1099/
... co.²⁵: http://jalammar.github.io/illustrated-bert/
... Languages ²⁶: https://www.aclweb.org/anthology/N18-1032/
... News ²⁷: https://papers.nips.cc/paper/9106-defending-against-neural-fake-news.pdf
... Learners ²⁸: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
... Learners ²⁹: https://arxiv.org/abs/2005.14165
... IE ³⁰: in piazza or https://www.cambridge.org/core/journals/natural-language-engineering/article/twentyfive-years-of-information-extraction/0E5BB0D6AE906BB3C25037E2D74CA8F3/share/5ce1ad8430e190e282cc234c79c320c49906a7e2
... Quasi-Identity ³¹: https://www.aclweb.org/anthology/W13-1203/
... Understanding ³²: https://www.aclweb.org/anthology/W18-5446/
... Objectives ³³: https://www.aclweb.org/anthology/D19-1448/
... Models ³⁴: https://www.aclweb.org/anthology/N16-1014/
... too?³⁵: https://www.aclweb.org/anthology/P18-1205/
... Data ³⁶: https://www.aclweb.org/anthology/2020.acl-main.463/
... NLP ³⁷: https://aclweb.org/anthology/papers/P/P19/P19-1355/
... Encoders ³⁸: https://www.aclweb.org/anthology/N19-1063/

jonmay@isi.edu

date	material	reading	presentation	Other	TA attending
8/24	intro, applications	Eisenstein 1 (not mandatory)			Meryem & Mozhdeh
8/26	end of intro, probability basics	Eisenstein Appendix A, Goldwater probability tutorial ⁴		project assignment out (due 9/9)	Meryem
8/31	probability, ethics, naive bayes	Eisenstein 2, Nathan Schneider's unix notes⁵, Unix for poets⁶, sculpting text⁷	The Social Impact of Natural Language Processing⁸ Presenter: Jon		-
9/2	Perceptron, Logistic Regression	Eisenstein 3, Charniak 1.	Thumbs up? Sentiment Classification using Machine Learning Techniques⁹ Presenter: Zekun	HW1 out (due 9/30)	Meryem
9/7	LABOR DAY NO CLASS
9/9	Nonlinear classifiers, backpropagation, gradient descent	Eisenstein 7	Fast Semantic Extraction Using a Novel Neural Network Architecture¹⁰ Presenter: Tooraj	project proposal due	Mozhdeh
9/14	POS tags, HMMs	Eisenstein 9,2, 10.	Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments¹¹ Presenter: Negar		Mozhdeh
9/16	cky, constituencies, treebank	Eisenstein 11	Building a Large Annotated Corpus of English¹² Presenters: Ani, Sabyasachee		Meryem
9/21	viterbi cky, restructuring, dependencies, shift-reduce	Eisenstein 4.5.	A Fast and Accurate Dependency Parser using Neural Networks¹³ Presenter: Shweta.		Mozhdeh
9/23	arc-eager, evaluation, human annotation	Eisenstein 13, 14.	An Empirical Investigation of Statistical Significance in NLP ¹⁴ Presenter: Nikolaos	HW2 out (due 10/21)	Meryem
9/28	NO CLASS
9/30	mechanical turk, semantics: word sense, propbank, amr, distributional	Eisenstein 7	Linguistic Regularities in Continuous Space Word Representations¹⁵. Presenter: Hongkuan The word analogy testing caveat¹⁶ Presenter: Jihoon	HW1 due	Meryem
10/5	language models: ngram, feed-forward, recurrent Machine Translation history, evaluation	Eisenstein 18.1, 18.2	Bleu: a Method for Automatic Evaluation of Machine Translation¹⁷ Presenter: Paras Towards a Literary Machine Translation: The Role of Referential Cohesion¹⁸ Presenter: Katy		Mozhdeh
10/7	Statistical, Neural Machine Translation, summarization, generation	Eisenstein 18.3, 19.1, 19.2	Effective Approaches to Attention-based Neural Machine Translation¹⁹ Presenter: Xiou Neural Machine Translation by Jointly Learning to Align and Translate²⁰ Presenter: Soumya	Revised proposals due (no late days)	Meryem
10/12	Transformers	Attention is all you need ²¹, Illustrated Transformer ²²	Six Challenges for Neural Machine Translation²³ Presenter: Yuchen. Get To The Point: Summarization with Pointer-Generator Networks²⁴ Presenter: Qi.		Mozhdeh
10/14	Large Contextualized Language Models (ElMo, BERT, GPT-N, etc.)	Illustrated BERT, ElMo, and co.²⁵	Universal Neural Machine Translation for Extremely Low Resource Languages ²⁶ Presenter: Amirhesam. Defending Against Neural Fake News ²⁷ Presenter: Yizhou.	HW3 out (due 11/11)	Meryem
10/19	Guest Lecture (Xuezhe Ma)		Language Models are Unsupervised Multitask Learners ²⁸ Presenter: I-Hung. Language Models are Few-Shot Learners²⁹ Presenters: Yufei, Wenxuan.		Mozhdeh
10/21	Information Extraction: Entity/Relation, CRF	Eisenstein 17.1, 17.2	25 years of IE³⁰ Presenter: Justin.	HW2 Due	Meryem
10/26	Information Extraction: Events, Zero-shot	Eisenstein 17.3	Events are Not Simple: Identity, Non-Identity, and Quasi-Identity³¹ Presenter: Basel		Mozhdeh
10/28	Blade Runner NLP/Bertology		GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding³² Presenter: Prateek		Meryem
11/2	Text Games and Reinforcement Learning		The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives³³ Presenter: Shuai		Mozhdeh
11/4	Dialogue	Eisenstein 19.3.	A Diversity-Promoting Objective Function for Neural Conversation Models³⁴ Presenter: Peifeng Personalizing Dialogue Agents: I have a dog, do you have pets too?³⁵ Presenter: Akshat		Meryem
11/9	Power and Ethics		Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data³⁶ Presenter: Ani Energy and Policy Considerations for Deep Learning in NLP ³⁷ Presenter: Ali A.		Mozhdeh
11/11	How to write a paper	Neubig slides on Piazza	On Measuring Social Biases in Sentence Encoders ³⁸ Presenter: Katy	HW3 Due	Meryem
11/16	Presentations				Meryem & Mozhdeh
11/18	Presentations				Meryem & Mozhdeh
11/23	Presentations				Meryem & Mozhdeh