AIgrads

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 25 Open GitHub ↗

This is a niche collection of research papers which are proven to be gradients pushing the field of Natural Language Processing, Deep Learning and Artificial Intelligence

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AIgrads
This is a niche collection of research papers which are proven to be gradients pushing the field of Natural Language Processing, Deep Learning and Artificial Intelligence

## RAG

* Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., ... & Yih, S. (2023). ["Ra-dit: Retrieval-augmented dual instruction tuning."](summary/RADIT.md) arXiv preprint arXiv:2310.01352 [[PDF](https://arxiv.org/pdf/2310.01352)].

## Large Language Models

* Wang, Dongsheng, et al. ["DocLLM: A layout-aware generative language model for multimodal document understanding."](summary/DocLLM.md) arXiv preprint arXiv:2401.00908 (2023) [[PDF](https://arxiv.org/pdf/2401.00908.pdf)].

* McIntosh, Timothy R., et al. ["From google gemini to openai q*(q-star): A survey of reshaping the generative artificial intelligence (ai) research landscape."](summary/GenAI_survey_2023.md) arXiv preprint arXiv:2312.10868 (2023) [PDF](https://arxiv.org/pdf/2312.10868.pdf).

* Pelrine, Kellin, et al. ["Exploiting Novel GPT-4 APIs."](summary/GPT4_exploits.md) arXiv preprint arXiv:2312.14302 (2023) [[PDF](https://arxiv.org/pdf/2312.14302.pdf)].

* Jiang, Albert Q., et al. ["Mistral 7B."](summary/Mistral7B.md) arXiv preprint arXiv:2310.06825 (2023) [[PDF](https://arxiv.org/pdf/2310.06825.pdf)].

* Liu, Zhengzhong, et al. ["LLM360: Towards Fully Transparent Open-Source LLMs."](summary/LLM360.md) arXiv preprint arXiv:2312.06550 (2023) [[PDF](https://arxiv.org/pdf/2312.06550.pdf)].

* Dao, Tri, et al. ["Flashattention: Fast and memory-efficient exact attention with io-awareness."](summary/flash.md) Advances in Neural Information Processing Systems 35 (2022): 16344-16359. [[PDF](https://proceedings.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf)]

* Hu, Edward J., et al. ["Lora: Low-rank adaptation of large language models."](summary/LORA.md) arXiv preprint arXiv:2106.09685 (2021). [[PDF](https://arxiv.org/pdf/2106.09685.pdf)]

* Touvron, Hugo, et al. ["Llama: Open and efficient foundation language models."](summary/llama.md) arXiv preprint arXiv:2302.13971 (2023). [[arXiv](https://arxiv.org/pdf/2302.13971.pdf)]

* Wu, Shijie, et al. ["Bloomberggpt: A large language model for finance."](summary/bloombergGPT.md) arXiv preprint arXiv:2303.17564 (2023) [[PDF](https://arxiv.org/pdf/2303.17564.pdf)].

* Yang, H., Liu, X. Y., & Wang, C. D. (2023). [FinGPT: Open-Source Financial Large Language Models.](summary/FinGPT.md) arXiv preprint arXiv:2306.06031. [PDF](https://arxiv.org/pdf/2306.06031.pdf)

* Wei, Jason, et al. [Chain-of-thought prompting elicits reasoning in large language models.](summary/ChainOfThought.md) Advances in Neural Information Processing Systems 35 (2022): 24824-24837. [PDF](https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf)

## NLP Pretraining and Architectures

* [UNILMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](summary/unilmv2.md) Hangbo Bao et al. 2020 [[arXiv](https://arxiv.org/pdf/2002.12804.pdf)]

* [MPNet: Masked and Permuted Pre-training for Language Understanding](summary/mpnet.md) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu et al. NIPS 2020 [[arXiv](https://arxiv.org/pdf/2004.09297.pdf)]

* [ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding](summary/ERNIE2.md) Sun et al. AAAI 2020 [[arXiv](https://arxiv.org/pdf/1907.12412.pdf)]

* [STRUCTBERT: INCORPORATING LANGUAGE STRUCTURES INTO PRETRAINING FOR DEEP LANGUAGE UNDERSTANDING](summary/structbert.md) Wang et al, ICLR 2020 [[arXiv](https://openreview.net/pdf?id=BJgQ4lSFPH)]

* [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](summary/pegasus.md), Jingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J Liu. Google Research, ICML 2020 [[arXiv](https://arxiv.org/pdf/1912.08777.pdf)]

* [GPT3: Language Models are Few Shot Learners](summary/GPT3.md), Brown et al, Open AI, 2020 [[arXiv](https://arxiv.org/pdf/2005.14165.pdf)]

* [ELECTRA: Pre-Training Text Encoders As Discriminators Rather Than Genererators](summary/electra.md), Clark et al., Stanford and Google, 2020 [[arXiv](https://arxiv.org/pdf/2003.10555.pdf)]

* [XLNet: Generalized Autoregressive Pretraining for Language Understanding](summary/xlnet.md), Yang et al. NIPS 2019, Google AI Brain, 2019 [[arXiv](https://arxiv.org/pdf/1906.08237.pdf)]

* [ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations](summary/albert.md), Zhenzhong Lan1, Mingda Chen, 2020 [[arXiv](https://arxiv.org/pdf/1909.11942.pdf)]

* [T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](summary/T5.md), Raffel et al, Google, 2019 [[arXiv](https://arxiv.org/pdf/1910.10683.pdf)]

* [RoBERTa: A Robustly Optimized BERT Pretraining Approach](summary/roberta.md), Liu et al. Facebook AI, 2019 [[arXiv](https://arxiv.org/pdf/1907.11692.pdf)]

* [SpanBERT: Improving Pre-training by Representing and Predicting Spans](summary/spanBERT.md), Joshi, Chen, AllenAI, Facebook Research, 2020 [[arXiv](https://www.mitpressjournals.org/doi/pdf/10.1162/tacl_a_00300)]

* [UniLM: Unified Language Model Pre-training for Natural Language Understanding and Generation](summary/unilm.md), Dong et al., Microsoft Research, 2019 [[NIPS](http://papers.nips.cc/paper/9464-unified-language-model-pre-training-for-natural-language-understanding-and-generation.pdf)]

* [DistilBERT, a distilled version of BERT: smaller,faster, cheaper and lighter](summary/distilbert.md), Victor SANH, Lysandre DEBUT, Julien CHAUMOND, Thomas WOLF, 2020 [[arXiv](https://arxiv.org/abs/1910.01108)]

* [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](summary/bart.md), Lewis et al. [[arXiv](https://arxiv.org/pdf/1910.13461.pdf)]

* [MASS: Masked Sequence to Sequence Pre-training for Language Generation](summary/mass.md), Song et al. [[arXiv](https://arxiv.org/pdf/1905.02450.pdf)]

* [GPT-2: Language Models are Unsupervised Multitask Learners](summary/gpt2.md) Radford et al. 2018, OpenAI [[OpenAI](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)]

* [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](summary/bert.md) Devlin et al. 2018, Google AI Language [[arXiv](https://arxiv.org/pdf/1810.04805.pdf)]

* [GPT: Improving Language Understanding by Generative Pre-Training](summary/gpt.md) Alec Radford, Karthik Narasimhan, Tim Saliman, Ilya Sutskever @ OpenAI, 2018 [[OpenAI](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)]

* [Attention Is All You Need](summary/transformers.md), Vaswani et al, 2017 [[arXiv](https://arxiv.org/pdf/1706.03762.pdf)]

* [GRAPH ATTENTION NETWORKS](summary/GAT.md) Petar Velickovi, Yoshua Bengio et. al., ICLR 2018 [[arXiv](https://arxiv.org/pdf/1710.10903.pdf)]]

* He, Pengcheng, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. ["DeBERTa: Decoding-enhanced BERT with Disentangled Attention."](summary/DeBERTa.md) arXiv preprint arXiv:2006.03654 (2020). [[arXiv](https://arxiv.org/pdf/2006.03654.pdf)]

* Lewis, Mike, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. ["MARGE: Pre-training via paraphrasing."](summary/MARGE.md) arXiv preprint arXiv:2006.15020 (2020).[[arXiv](https://arxiv.org/pdf/2006.15020.pdf)]

* Mao, Yuning, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. ["Generation-augmented retrieval for open-domain question answering."](summary/GAR.md) arXiv preprint arXiv:2009.08553 (2020) [[arXiv](https://arxiv.org/pdf/2009.08553.pdf)].

* Fedus, William, Barret Zoph, and Noam Shazeer. ["Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity."](summary/switch_transformers.md) arXiv preprint arXiv:2101.03961 (2021) [[arXiv](https://arxiv.org/pdf/2101.03961.pdf)].

## Computer Vision

* Bell, Sean, Yiqun Liu, Sami Alsheikh, Yina Tang, Edward Pizzi, M. Henning, Karun Singh, Omkar Parkhi, and Fedor Borisyuk. ["Groknet: Unified computer vision model trunk and embeddings for commerce."](summary/Groknet.md) In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2608-2616. 2020. [[PDF](https://ai.facebook.com/research/publications/groknet-unified-computer-vision-model-trunk-and-embeddings-for-commerce/)]

## Fine Tuning & Down-stream Tasks

* [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](summary/GlossBERT.md) Huang et al. 2020 [[arXiv](summary/https://arxiv.org/pdf/1908.07245.pdf)]

* [Syntax-guided Controlled Generation of Paraphrases](summary/SGCP.md), Ashutosh Kumar, Kabir Ahuja, Raghuram Vadapalli, Partha Talukdar, ACL 2020 [[arXiv](https://arxiv.org/pdf/2005.08417.pdf)]

* [Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension](summary/bert_calculator.md) Daniel Andor, Luheng He, Kenton Lee, Emily Pitler, ACL 2019 [[arXiv](https://arxiv.org/abs/1909.00109)]

* [CTRL: A CONDITIONAL TRANSFORMER LANGUAGE MODEL FOR CONTROLLABLE GENERATION](summary/ctrl.md) Nitish Shirish Keskar, Richard Socher, 2020 [[arXiv](https://einstein.ai/presentations/ctrl.pdf)]

* [Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks](summary/domain_task_adaptive_training.md), Gururangan et al, ACL 2020 [[arXiv](https://www.aclweb.org/anthology/2020.acl-main.740.pdf)]

* [Unifying Question Answering, Text Classification, and Regression via Span Extraction](SpEx-BERT.md) Nitish Keskar, Richard Socher et al [[arXiv](https://arxiv.org/pdf/1904.09286.pdf)]

* [How to Fine-Tune BERT for Text Classification?](summary/finetune_bert_text_classification.md) Sun et al, 2019[[arXiv](https://arxiv.org/abs/1905.05583)]

* [To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks](summary/adapting_pretrained_repr.md) Peters, Ruder, Smith, AI2, ACL 2019 [[arXiv](https://arxiv.org/abs/1903.05987)]

* [To Tune or Not To Tune? How About the Best of Both Worlds?](summary/stack_and_finetune.md), Wang et al, 2019, [[arXiv](https://arxiv.org/abs/1907.05338)]

* [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](summary/seq_generation_from_pretrained.md) Sascha Rothe, Shashi Narayan, Aliaksei Severyn, ACL 2020 [[arXiv](https://arxiv.org/pdf/1907.12461.pdf)]

* [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](summary/finetuning_wi_do.md), Dodge et al. 2020  [[arXiv](https://arxiv.org/pdf/2002.06305.pdf)]

* [If Beam Search is the Answer, What was the Question?](summary/beam_search_objective.md), Clara Meister, Tim Vieira, Ryan Cotterell, Oct 2020, EMNLP [[arXiv](https://arxiv.org/pdf/2010.02650.pdf)]

* Xu, Benfeng, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. ["Curriculum learning for natural language understanding."](summary/NLU_CL.md) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6095-6104. 2020. [[ACLWeb](https://www.aclweb.org/anthology/2020.acl-main.542.pdf)]

* Lee-Thorp, James, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. ["FNet: Mixing Tokens with Fourier Transforms."](summary/FNet.md) arXiv preprint arXiv:2105.03824 (2021).

## Multi task learning

* [MT-DNNKD: Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding](summary/MTDNNKD.md), Xiaodong Liu et al. [[arXiv](https://arxiv.org/pdf/1904.09482.pdf)]

* [Multi-Task Deep Neural Networks for Natural Language Understanding](summary/MTDNN_GLUE.md), Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, 2019 [[arXiv](https://arxiv.org/abs/1901.11504)]

* [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](summary/bam_multi_task.md), Kevin Clark, Christopher D. Manning et al. [[arXiv](https://arxiv.org/pdf/1907.04829.pdf)]

* [Multitask Prompted Training Enables Zero-Shot Task Generalization](summary/Mutlitask_prompted_training_T0.md),Victor Sanh, Albert Webson, Colin Raffel et al. 2021, [[arXiv](https://arxiv.org/abs/2110.08207)]

## Datasets, Benchmarks & Metrics

* [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](summary/conceptNet.md), Robyn Speer et al. AAAI 2017 [[arXiv](https://arxiv.org/pdf/1612.03975.pdf)]

* [Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison](summary/WSD.md),Alessandro Raganato, Jose Camacho-Collados and Roberto Navigli, ACL 2017 [[arXiv](https://www.aclweb.org/anthology/E17-1010.pdf)]

* [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](summary/superGLUE.md), Wang et al, NIPS 2019 [[arXiv](https://arxiv.org/pdf/1905.00537.pdf)]

* [CHECKLIST: Beyond Accuracy: Behavioral Testing of NLP Models with CHECKLIST](summary/checklist.md), Ribeiro, Wu, Guestrin, Sameer Singh, 2020 [[ACLWeb](https://www.aclweb.org/anthology/2020.acl-main.442.pdf)]

* [HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](summary/hotpotqa.md) Zhilin Yang et al. ACL 2018 [[arXiv](https://arxiv.org/pdf/1809.09600.pdf)]

* [LearningQ: A Large-scale Dataset for Educational Question Generation](summary/learningQ.md) Guanliang Chen et al. AAAI 2018, [[PDF](https://doc.rero.ch/record/309023/files/yan_lls.pdf)]

* Petroni, Fabio, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne et al. ["Kilt: a benchmark for knowledge intensive language tasks."](summary/KILT.md) arXiv preprint [[arXiv:2009.02252](https://aclanthology.org/2021.naacl-main.200.pdf) (2020).

* Khot, Tushar, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. ["QASC: A Dataset for Question Answering via Sentence Composition."](summary/QASC.md) In AAAI, pp. 8082-8090. 2020.

## Knowledge Probes

* [Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge](summary/reason_implicit_knowledge.md) Alon Talmor, Peter Clark et al, NIPS 2020 [[arXiv](https://arxiv.org/pdf/2006.06609.pdf)]

* [oLMpics - On what Language Model Pre-training Captures](summary/oLMpics.md) Alon Talmor, Yoav Goldberg et al. The Allen Institute for AI, 2020 [[arXiv](https://arxiv.org/pdf/1912.13283.pdf)]

## Explainable AI

* [A Framework for Understanding Unintended Consequences of Machine Learning](summary/formalising_bias.md) Harini Suresh, John V. Guttag, 2020 [[arXiv](https://arxiv.org/pdf/1901.10002.pdf)]

* [How Can I Explain This to You? An Empirical Study of Deep Neural Network Explanation Methods](summary/explainabilty_emperical_study.md) Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, Mani Srivastava, NIPS 2020 [[arXiv](https://proceedings.neurips.cc//paper/2020/file/2c29d89cc56cdb191c60db2f0bae796b-Paper.pdf)]

* [Explaining Explanations: Axiomatic Feature Interactions for Deep Networks](summary/integrated_hessians.md) Janizek, Sturmfels, Lee, 2020 [[arXiv](https://arxiv.org/pdf/2002.04138.pdf)]

* [Towards Interpretable Natural Language Understanding with Explanations as Latent Variables](summary/latent_explanations.md) Zhou, Hu, Zhang, Liang, Sun, Xiong, Tang et al. [[arXiv](https://arxiv.org/pdf/2011.05268.pdf)]

* [Principles and Practice of Explainable Machine Learning](summary/explainable_ml.md) Vaishak Belle, Ioannis Papantonis [[arXiv](https://arxiv.org/pdf/2009.11698.pdf)]

* [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](summary/xai_concepts.md), Arrieta et al., 2019 [[arXiv](https://arxiv.org/abs/1910.10045)]

* Jhamtani, Harsh, and Peter Clark. ["Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering."](summary/GRC.md) arXiv preprint arXiv:2010.03274 (2020)[[arXiv](https://arxiv.org/pdf/2010.03274.pdf)]

* Narang, Sharan, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. ["WT5?! Training Text-to-Text Models to Explain their Predictions."](summary/WT5.md) arXiv preprint arXiv:2004.14546 (2020).[[arXiv](https://arxiv.org/pdf/2004.14546.pdf)]

## Probing and Interpretability

* [A Survey of the State of Explainable AI for Natural Language Processing](summary/XAI_NLP_survey.md) Danilevsky et al. [[arXiv](https://arxiv.org/pdf/2010.00711v1.pdf)]

* [Definitions, methods, and applications in interpretable machine learning](summary/interpretable_ml.md) W. James Murdocha, Chandan Singhb, Karl Kumbiera, Reza Abbasi-Asl, and Bin Yua, PNAS 2019 [[PNAS](https://www.pnas.org/content/pnas/116/44/22071.full.pdf)]

* [Towards Transparent and Explainable Attention Models](summary/diversity_attention.md) Mohankumar, Mitesh Khapra et al. ACL 2020 [[arXiv](https://arxiv.org/abs/2004.14243)]

* [Revealing the Dark Secrets of BERT](summary/dark_secret_BERT.md) Olga Kovaleva, ACL 2019 [[arXiv](https://arxiv.org/abs/1908.08593)]

* [DeepLift: Learning Important Features Through Propagating Activation Differences](summary/deeplift.md) Avanti Shrikumar et al, Stanford University, ICML 2019 [[arXiv](https://arxiv.org/pdf/1704.02685.pdf)]

* [Analysis Methods in Neural Language Processing: A Survey](summary/survey_nlp_analysis.md), Belinkov, Glass 2019 [[arXiv](https://www.mitpressjournals.org/doi/pdfplus/10.1162/tacl_a_00254)]

* [LIME: "Why Should I Trust You?": Explaining the Predictions of Any Classifier](summary/LIME.md) Ribeiro, Sameer Singh, Guestrin, University of Washington, KDD 2016 [[arXiv](https://arxiv.org/pdf/1602.04938.pdf)]

* [Axiomatic Attribution for Deep Networks](summary/integrated_gradients.md), Sundararajan, Taly, Yan, Google, ICML 2017 [[arXiv](https://arxiv.org/pdf/1703.01365.pdf)]

* [How Important Is a Neuron?](summary/conductance.md), Kedar Dhamdhere, Mukund Sundararajan, Qiqi Yan, Google Research [[arXiv](https://arxiv.org/pdf/1805.12233.pdf)]

* [SHAP: A Unified Approach to Interpreting Model Predictions](summary/shap.md), Lundberg, Lee, University of Washington, NIPS 2017 [[arXiv](http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf)]

* [Attention is not not explanation](summary/attention_not_not.md) Sarah Wiegreffe, Yuval Pinter,  EMNLP 2019 [[arXiv](https://arxiv.org/abs/1908.04626)]

* [Attention is not Explanation](summary/attention_not_explanation.md), Sarthak Jain, Byron C Wallace, NAACL-2019, [[arXiv](https://arxiv.org/pdf/1902.10186.pdf)]

* [What do you Learn from Context? Probing for Sentence Structure in Contextualized Word Representations](summary/edge_probing.md), ICLR 2019, Tenney at el [[openreview](https://openreview.net/pdf?id=SJzSgnRcKX)]

* [Are Sixteen Heads Really Better than One?](summary/sixteen_heads.md), Paul Michel, Omer Levy, Graham Neubig, 2019 [[arXiv](https://arxiv.org/pdf/1905.10650.pdf)]

* [Fine-Grained Analysis of Sentence Embedding Using Auxiliary Prediction Tasks](summary/aux_pred.md), Yossi Adi, Yoav Goldberg et al, ICLR 2017 [[arXiv](https://arxiv.org/pdf/1608.04207.pdf)]

* [Assessing BERT’s Syntactic Abilities](summary/bert_syntactic.md), Yoav Goldberg, 2019 [[arXiv](https://arxiv.org/pdf/1901.05287.pdf)]

* [Generating Derivational Morphology with BERT](summary/bert_derivational_morphology.md) Valentin Hofmann, Janet B. Pierrehumbert, Hinrich Schutze, 2020 [[arXiv](https://arxiv.org/pdf/2005.00672.pdf)]

* [Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs](summary/bert_npi.md) Warstadt et al. [[arXiv](https://arxiv.org/pdf/1909.02597.pdf)]

* [What Does BERT Look At? An Analysis of BERT’s Attention](summary/bert_analysis.md), Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D. Manning, 2019 [[arXiv](https://arxiv.org/abs/1906.04341)]

* [BERT Rediscovers the Classical NLP Pipeline](summary/bert_analysis_nlp_pipeline.md), Ian Tenney, Dipanjan Das, Ellie Pavlick, 2019 [[arXiv](https://arxiv.org/pdf/1905.05950.pdf)]

* [Visualizing and Measuring the Geometry of BERT](summary/bert_geometry.md), Andy Coenen, Martin Wattenberg et al, NIPS 2019 [[arXiv](https://arxiv.org/pdf/1906.02715.pdf)]

* [Designing and Interpreting Probes with Control Tasks](summary/designing_probes.md), John Hewitt, Percy Liang, EMNLP-2019 [[arXiv](https://www.aclweb.org/anthology/D19-1275.pdf)]

* [Open Sesame: Getting Inside BERT’s Linguistic Knowledge](summary/sesame.md), Yongjie Lin, Yi Chern Tan, Robert Frank, 2019 [[arXiv](https://arxiv.org/pdf/1906.01698.pdf)]

* [A Structural Probe for Finding Syntax in Word Representations](summary/structural_probe.md), John Hewitt, Christopher D. Manning, 2019, NAACL 2019, Standford [[arXiv](https://nlp.stanford.edu/pubs/hewitt2019structural.pdf)]

* [On Identifiability in Transformers](summary/identifiability.md), Brunner, Liu, Pascual, Richter, Ciaramita, Wattenhofer, Google Research, ICLR 2020 [[arXiv](https://openreview.net/pdf?id=BJg1f6EFDB)]

* [NILE : Natural Language Inference with Faithful Natural Language Explanations](summary/nile.md), Sawan Kumar, Partha Talukdar, 2020 [[arXiv](https://www.aclweb.org/anthology/2020.acl-main.771.pdf)]

* [Quantifying Attention Flow in Transformers](summary/attention_flow.md), Samira Abnar, Willem Zuidema, ACL 2020 [[arXiv](https://arxiv.org/pdf/2005.00928.pdf)]

* [Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words?](summary/human_attention_comparision.md) Sen et al, ACL 2020 [[arXiv](https://www.aclweb.org/anthology/2020.acl-main.419.pdf)]
]

* [Understanding Attention for Text Classification](summary/understanding_attention.md), Xiaobing Sun and Wei Lu, Singapore University, ACL 2020 [[arXiv](https://www.aclweb.org/anthology/2020.acl-main.312.pdf)]

## Few Shots Learning

* Schick, Timo, and Hinrich Schütze. ["It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners."](summary/PET.md) arXiv preprint [[arXiv:2009.07118](https://arxiv.org/pdf/2009.07118.pdf)] (2020).

## Extreme Classification

* [Taming Pretrained Transformers for Extreme Multi-label Text Classification](summary/XMC_XTransformer.md)], Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, Inderjit S. Dhillon, 2020 [[arXiv](https://assets.amazon.science/32/d7/bb602e97419ead030dde419b9191/taming-pretrained-transformers-for-extreme-multi-label-text-classification.pdf)]

* Dahiya, Kunal, Deepak Saini, Anshul Mittal, Ankush Shaw, Kushal Dave, Akshay Soni, Himanshu Jain, Sumeet Agarwal, and Manik Varma. ["DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents."](summary/DeepXML.md) In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 31-39. 2021.[[arXiv](http://manikvarma.org/pubs/dahiya21-main.pdf)]

## Conversational AI

* [Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Tracking](summary/cai_synthetic_data.md), Giovanni Campagna Agata Foryciarz Mehrad Moradshahi Monica S. Lam, ACL 2020 [[arXiv](https://www.aclweb.org/anthology/2020.acl-main.12.pdf)]

## Automated Essay Scoring
* [Automated Essay Scoring with Discourse-Aware Neural Models](summary/AES_discore_aware.md), Farah Nadeem et al. ACL 2019 [[arXiv](https://www.aclweb.org/anthology/W19-4450.pdf)]

## Summarisation

* [Get To The Point: Summarization with Pointer-Generator Networks](summary/pointer_coverage.md) Abigail See, Peter J. Liu, Christopher D. Manning, 2017 [[arXiv](https://arxiv.org/pdf/1704.04368.pdf)]

## Question Generation

* [A Recurrent BERT-based Model for Question Generation](summary/recurrent_BERT_QG.md) Ying-Hong Chan, Yao-Chung Fan. ACL 2019 workshop on Question Answering, [[arXiv](https://www.aclweb.org/anthology/D19-5821.pdf)]

* [Improving Neural Question Generation using Answer Separation](summary/NQG_answer_sep.md), Yanghoon Kim, Hwanhee Lee, Joongbo Shin and Kyomin Jung, AAAI 2018 [[arXiv](https://arxiv.org/pdf/1809.02393.pdf)]

* [Question Generation for Question Answering](summary/QGforQA.md), Nan Duan, Duyu Tang, Peng Chen, Ming Zhou, EMNLP 2017 [[arXiv](aclweb.org/anthology/D17-1090.pdf)]

* [Toward Subgraph Guided Knowledge Graph Question Generation with Graph Neural Networks](summary/Graph2Seq_QG.md) Yu Chen, Lingfei Wu, Mohammed J. Zaki, 2020 [[arXiv](https://arxiv.org/pdf/2004.06015.pdf)]

* [Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks](summary/QG_maxout.md), Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, Qifa Ke, ACL 2018 [[arXiv](https://www.aclweb.org/anthology/D18-1424.pdf)]

* [CopyBERT: A Unified Approach to Question Generation with Self-Attention](summary/CopyBERT.md), Stalin Varanasi, Saadullah Amin, Gunter Neumann, ACL 2020 [[arXiv](https://www.aclweb.org/anthology/2020.nlp4convai-1.3.pdf)]

* [Generating Natural Language Question-Answer Pairs from a Knowledge Graph Using a RNN Based Question Generation Model](summary/K2Q_RNN.md), Sathish Indurthi, Dinesh Raghu, Mitesh M. Khapra and Sachindra Joshi. ACL 2017 [[arXiv](https://www.aclweb.org/anthology/E17-1036.pdf)]

* [Recent Advances in Neural Question Generation](summary/NQG_survey), Liangming Pan, Min-Yen Kan et al. 2019 [[arXiv](https://arxiv.org/pdf/1905.08949.pdf)]

* [Semantic Graphs for Generating Deep Questions](summary/NQG_semantic_graphs.md), Liangming Pan, Min-Yen Kan et al. 2019 [[arXiv](https://arxiv.org/pdf/2004.12704.pdf)]

* [Learning to Ask: Neural Question Generation for Reading Comprehension](summary/learning_to_ask.md) Xinya Du et al. ACL 2017 [[arXiv](https://arxiv.org/pdf/1705.00106.pdf)]

* [Neural Models for Key Phrase Extraction and Question Generation](phrase_extraction_que_generation.md) Sandeep Subramanian et al. Machine Reading for Question Answering workshop at ACL 2018 [[arXiv](https://arxiv.org/pdf/1706.04560.pdf)]

* Ko, Wei-Jen, Te-Yuan Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. ["Inquisitive Question Generation for High Level Text Comprehension."](summary/inquirer.md) arXiv preprint arXiv:2010.01657 (2020).[[arXiv](https://arxiv.org/pdf/2010.01657.pdf)]

* Sultan, Md Arafat, Shubham Chandel, Ramón Fernandez Astudillo, and Vittorio Castelli. ["On the importance of diversity in question generation for QA."](summary/QG_diversity.md) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5651-5656. 2020.[[arXiv](https://www.aclweb.org/anthology/2020.acl-main.500.pdf)]

* Lopez, Luis Enrico, Diane Kathryn Cruz, Jan Christian Blaise Cruz, and Charibeth Cheng. ["Simplifying Paragraph-level Question Generation via Transformer Language Models."](summary/QG_TLM.md) arXiv preprint arXiv:2005.01107 (2020)[[arXiv](https://arxiv.org/pdf/2005.01107.pdf)].

## Question Answering
* [Retrieve, Rerank, Read, then Iterate: Answering Open-Domain Questions of Arbitrary Complexity from Text](summary/IRRR.md) Peng Qi, Christopher Manning et al. 2020 [[arXiv](https://arxiv.org/pdf/2010.12527.pdf)]

* [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](summary/CAGE.md) Rajani, McCann, Xiong, Richard Socher, 2019 [[arXiv](https://arxiv.org/pdf/1906.02361.pdf)]

* Min, Sewon, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu et al. ["NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned."](summary/EfficientQA_analysis.md) arXiv preprint arXiv:2101.00133 (2021). [[arXiv](https://arxiv.org/abs/2101.00133)]

* Karpukhin, Vladimir, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. ["Dense Passage Retrieval for Open-Domain Question Answering."](summary/DPR.md), arXiv preprint arXiv:2004.04906 (2020).[[arXiv](https://arxiv.org/pdf/2004.04906.pdf)]

* Cheng, Hao, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. ["Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering."](summary/DocumentQA.md) arXiv preprint arXiv:2005.01898 (2020).[[arXiv](https://arxiv.org/pdf/2005.01898.pdf)]

* Ju, Ying, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. ["Technical report on conversational question answering."](summary/QA_AT_KD.md) arXiv preprint arXiv:1909.10772 (2019).[[arXiv](https://arxiv.org/pdf/1909.10772.pdf)]

* Khashabi, Daniel, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. ["Unifiedqa: Crossing format boundaries with a single qa system."](summary/unifiedQA.md) arXiv preprint [[arXiv:2005.00700](https://arxiv.org/pdf/2005.00700.pdf)] (2020).

* Gomez-Perez, Jose Manuel, and Raul Ortega. ["ISAAQ--Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention."](summary/ISAAQ.md) arXiv preprint [[arXiv:2010.00562](https://arxiv.org/pdf/2010.00562.pdf)] (2020).

* Wang, Peifeng, Nanyun Peng, Filip Ilievski, Pedro Szekely, and Xiang Ren. ["Connecting the dots: A knowledgeable path generator for commonsense question answering."](summary/ConnectingDots.md) arXiv preprint arXiv:2005.00691 (2020).

* Saxena, Apoorv, Soumen Chakrabarti, and Partha Talukdar. ["Question Answering Over Temporal Knowledge Graphs."](summary/CRONKGQA.md) arXiv preprint arXiv:2106.01515 (2021)[[arXiv](https://arxiv.org/pdf/2106.01515.pdf)].

## Meta Learning
* [DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference](summary/DReCa.md) Shikhar Murty,  Tatsunori B. Hashimoto, Christopher D. Manning, 2020 [[arXiv](https://openreview.net/pdf?id=PqsalKqGudW)]

## Knowledge Graph Tasks

* [Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Prediction](summary/extrapolate_knowledge.md) Jinheon Baek, Dong Bok Lee, Sung Ju Hwang, NIPS 2020 [[arXiv](https://arxiv.org/pdf/2006.06648.pdf)]

* [Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings](summary/Query2Box.md), Hongyu Ren, Weihua Hu, Jure Leskovec, ICLR 2020 [[arXiv](https://arxiv.org/pdf/2002.05969.pdf)]

## Knowledge Infusion

* [GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction](summary/GATE.md) Oct 2020 [[arXiv](https://arxiv.org/pdf/2010.03009.pdf)]

* [SemBERT: Semantics-aware BERT for Language Understanding](summary/SemBERT.md) Zhuosheng Zhang et al. AAAI 2020 [[arXiv](https://arxiv.org/pdf/1909.02209.pdf)]

* [SenseBERT: Driving Some Sense into BERT](summary/SenseBERT.md) Yoav Levine et al. AI21 Labs, May 2020 [[arXiv](https://arxiv.org/pdf/1908.05646.pdf)]

* [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](summary/RAG.md) Piktus et al, Facebook AI Research, NIPS 2020 [[arXiv](https://arxiv.org/pdf/2005.11401.pdf)]

* [Augmenting Neural Networks with First-order Logic](summary/first-order-logic.md) Tao Li, Vivek Srikumar, Archive Preprint, 2019 [[arXiv](https://arxiv.org/pdf/1906.06298.pdf)]

* [Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering](summary/OCNKI.md) Kaixin Ma et al. COIN workshop, ACL 2019 [[arXiv](https://www.aclweb.org/anthology/D19-6003.pdf)]

* [Neural Natural Language Inference Models Enhanced with External Knowledge](summary/NIL_knowledge_KIM.md) Qian Chen et al. ACL 2018 [[arXiv](https://www.aclweb.org/anthology/P18-1224.pdf)]

* [KEPLER: A Unified Model for Knowledge Embedding and  Pre-trained Language Representation](summary/KEPLER.md) Wang et al, Preprint, Feb 2020 [[arXiv](https://arxiv.org/pdf/1911.06136.pdf)]

* [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](summary/KagNet.md) Bill Yuchen Lin et al. EMNLP-IJCNLP 19 [[arXiv](https://arxiv.org/pdf/1909.02151.pdf)]

* [Improving question answering with external knowledge](summary/qa_external_knowledge.md) Xiaoman Pan et al. MRQA 2019 [[arXiv](https://arxiv.org/pdf/1902.00993.pdf)]

* [Improving Natural Language Inference Using External Knowledge in the Science Questions Domain](summary/ConSeqNet.md) Wang et al, AAAI 2019 [[arXiv](https://www.aaai.org/ojs/index.php/AAAI/article/view/4705)]

* [KG-BERT: BERT for Knowledge Graph Completion](summary/KG_BERT.md), Liang Yao, Chengsheng Mao, Yuan Luo, AAAI 2020 [[arXiv](https://arxiv.org/pdf/1909.03193.pdf)]

* [K-BERT: Enabling Language Representation with Knowledge Graph](summary/k-bert.md), Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, Ping Wang, AAAI 2020 [[arXiv](https://arxiv.org/pdf/1909.07606.pdf)]

* [Structural Information Preserving for Graph-to-Text Generation](summary/graph_to_text_RATrans.md), Linfeng Song et al., ACL 2020 [[arXiv](https://www.aclweb.org/anthology/2020.acl-main.712.pdf)]

* [Low-Dimensional Hyperbolic Knowledge Graph Embeddings](summary/Hyperbolic_KG_embedding.md) Chami et al, ACL 2020 [[arXiv](aclweb.org/anthology/2020.acl-main.617.pdf)]

* [K-Adapters: Infusing Knowledge into Pre-Trained Models with Adapters](summary/k-adapter.md), Ruize Wang, Ming Zhou et al. ACL 2020 [[arXiv](https://arxiv.org/pdf/2002.01808.pdf)]

* [ERWISE: Zero-shot Word Sense Disambiguation using Sense Definition Embeddings](summary/ERWISE.md) Sawan Kumar, Partha Talukdar, ACL 2019 [[arXiv](https://malllabiisc.github.io/publications/papers/EWISE_ACL19.pdf)]

* [KnowBERT: Knowledge Enhanced Contextual Word Representations](summary/knowbert.md) Peters et al, ACL 2019 [[arXiv](https://arxiv.org/abs/1909.04164)]

* [Sequential Latent Knowledge Selection For Knowledge-Grounded Dialogue](summary/KGD_SKT.md), Byeongchang Kim, Jaewoo Ahn, Gunhee Kim, ICLR 2020 [[arXiv](https://arxiv.org/abs/2002.07510)]

* [Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition](summary/KALM.md), Angli Liu, Jingfei Du, Veselin Stoyanov [[arXiv](https://www.aclweb.org/anthology/N19-1117/)]
]

* [Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling](summary/KGLM.md), Logan IV, ACL 2019, [[arXiv](https://www.aclweb.org/anthology/P19-1598/)]

* [Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning](summary/KIL.md), Kursuncu et al. AAAI 2020 [[arXiv](https://arxiv.org/pdf/1912.00512.pdf)]

* [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](summary/comet.md), Bosselut et al. ACL 2019 [[arXiv](https://www.aclweb.org/anthology/P19-1470/)]

* [ERNIE: Enhanced Language Representation with Informative Entities](summary/ernie.md), Zhang et al, ACL 2019 [[arXiv](https://arxiv.org/pdf/1905.07129.pdf)]

* [EmbedKGQA: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings](summary/embed_kgqa.md), Apoorv Saxena, Aditay Tripathi, Partha Talukdar, 2020 [[ACLWeb](https://www.aclweb.org/anthology/2020.acl-main.412.pdf)]

* Feng, Yanlin, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. ["Scalable multi-hop relational reasoning for knowledge-aware question answering."](summary/MHGRN.md) arXiv preprint [[arXiv:2005.00646](https://arxiv.org/pdf/2005.00646.pdf)] (2020).

* Sun, Yu, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu et al. ["ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation."](summary/ERNIE3.md) arXiv preprint [arXiv:2107.02137](https://arxiv.org/pdf/2107.02137.pdf) (2021).

## KG Embeddings

* [Convolutional 2D Knowledge Graph Embeddings](summary/ConvE.md), Dettmers et al, AAAI 2018 [[arXiv](https://arxiv.org/abs/1707.01476)]

* [Translating Embeddings for Modeling Multi-relational Data](summary/TransE.md), Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, NIPS 2013 [[arXiv](https://papers.nips.cc/paper/5071-translating-embeddings-for-modeling-multi-relational-data.pdf)]

* [InteractE: Improving Convolution-based Knowledge Graph Embeddings by Increasing Feature Interactions](summary/InteractE.md), Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, Partha Talukdar, AAAI 2020 [[arXiv](https://arxiv.org/pdf/1911.00219.pdf)]

* Abboud, Ralph, Ismail Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. ["BoxE: A Box Embedding Model for Knowledge Base Completion."](summary/BoxE.md) Advances in Neural Information Processing Systems 33 (2020).

## Knowledge Graphs in Artificial Intelligence

* Ji, Shaoxiong, Shirui Pan, Erik Cambria, Pekka Marttinen, and S. Yu Philip. "A survey on knowledge graphs: Representation, acquisition, and applications." IEEE Transactions on Neural Networks and Learning Systems (2021).

* Lecue, Freddy. ["On the role of knowledge graphs in explainable AI."](summary/RoleKG_XAI.md) Semantic Web 11, no. 1 (2020): 41-51. [[PDF](http://www.semantic-web-journal.net/system/files/swj2198.pdf)]

## Math Word Problems
* Tan, Minghuan, Lei Wang, Lingxiao Jiang, and Jing Jiang. ["Investigating Math Word Problems using Pretrained Multilingual Language Models."](summary/MWP_MLM.md) arXiv preprint arXiv:2105.08928 (2021)[[arXiv](https://arxiv.org/pdf/2105.08928.pdf)].

* Amini, Aida, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. ["MathQA: Towards interpretable math word problem solving with operation-based formalisms."](summary/MathQA.md) arXiv preprint [[arXiv:1905.13319](https://arxiv.org/abs/1905.13319)] (2019).

* Zhang, Jipeng, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. ["Graph-to-tree learning for solving math word problems."](summary/GraphTreeMWF.md) Association for Computational Linguistics, [ACL](https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=6276&context=sis_research), 2020.

* Saxton, David, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. ["Analysing mathematical reasoning abilities of neural models."](summary/MathematicsDataset.md) arXiv preprint [[arXiv:1904.01557](https://arxiv.org/pdf/1904.01557.pdf)] (2019).

* Ran, Qiu, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. ["NumNet: Machine reading comprehension with numerical reasoning."](summary/NumNet.md) arXiv preprint arXiv:1910.06701 (2019).

* Xie, Zhipeng, and Shichao Sun. ["A Goal-Driven Tree-Structured Neural Model for Math Word Problems."](summary/GoalDriveTree_MWP.md) In IJCAI, pp. 5299-5305. 2019.

* Wang, Yan, Xiaojiang Liu, and Shuming Shi. ["Deep neural solver for math word problems."](summary/Math23K.md) In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845-854. 2017 [[PDF](https://aclanthology.org/D17-1088.pdf)].

* Li, Jierui, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang. ["Modeling intra-relation in math word problems with different functional multi-head attentions."](summary/MWP_Intra_Relation.md) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6162-6167. 2019.

* Liang, Zhenwen, Jipeng Zhang, Jie Shao, and Xiangliang Zhang. ["MWP-BERT: A Strong Baseline for Math Word Problems."](summary/MWP-BERT.md) arXiv preprint [[arXiv:2107.13435](https://arxiv.org/abs/2107.13435)] (2021).
* Liu, Qianying, Wenyu Guan, Sujian Li, Fei Cheng, Daisuke Kawahara, and Sadao Kurohashi. ["Reverse Operation based Data Augmentation for Solving Math Word Problems."](summary/MWP_RODA.md) arXiv preprint [[arXiv:2010.01556](https://arxiv.org/pdf/2010.01556.pdf)] (2020).

* Hong, Yining, Qing Li, Daniel Ciao, Siyuan Huang, and Song-Chun Zhu. ["Learning by Fixing: Solving Math Word Problems with Weak Supervision."](summary/MWP_LBF_weaksupervision.md) In AAAI Conference on Artificial Intelligence. [[PDF](https://www.aaai.org/AAAI21Papers/AAAI-5790.HongY.pdf)] 2021.

* Hong, Yining, Qing Li, Ran Gong, Daniel Ciao, Siyuan Huang, and Song-Chun Zhu. ["SMART: A Situation Model for Algebra Story Problems via Attributed Grammar."](summary/MWP_smart.md) In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 14, pp. 13009-13017. 2021. [[PDF](https://arxiv.org/pdf/2012.14011.pdf)]

* Qin, Jinghui, Xiaodan Liang, Yining Hong, Jianheng Tang, and Liang Lin. ["Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks."](summary/NSSolver.md) arXiv preprint [arXiv:2107.01431](https://arxiv.org/pdf/2107.01431.pdf) (2021).

* Lample, Guillaume, and François Charton. ["Deep learning for symbolic mathematics."](summary/DLMath.md) arXiv preprint [arXiv:1912.01412](https://arxiv.org/abs/1912.01412) (2019).

* Miao, Shen-Yun, Chao-Chun Liang, and Keh-Yih Su. ["A diverse corpus for evaluating and developing english math word problem solvers."](summary/MWP_evaluation.md) arXiv preprint [arXiv:2106.15772](https://arxiv.org/pdf/2106.15772.pdf) (2021).

* Jinghui Qin, Lihui Lin, Xiaodan Liang, Rumin Zhang, and Liang Lin. 2020. [Semantically-aligned universal tree-structured solver for math word problems.](summary/SAU_solver.md) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ([EMNLP](https://aclanthology.org/2020.emnlp-main.309.pdf)), pages 3780–3789.

* Patel, Arkil, Satwik Bhattamishra, and Navin Goyal. ["Are NLP Models really able to Solve Simple Math Word Problems?."](summary/SVAMP.md) arXiv preprint [[arXiv:2103.07191](https://arxiv.org/pdf/2103.07191.pdf)] (2021).

* Griffith, Kaden, and Jugal Kalita. ["Solving Arithmetic Word Problems with Transformers and Preprocessing of Problem Text."](summary/MWP_Preprocessing.md) arXiv preprint [[arXiv:2106.00893](https://arxiv.org/pdf/2106.00893.pdf)] (2021).

* Liu, Qianying, Wenyv Guan, Sujian Li, and Daisuke Kawahara. ["Tree-structured decoding for solving math word problems."](summary/TreeStructuredDecoding.md) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2370-2379. 2019. [PDF](https://aclanthology.org/D19-1241.pdf)

* Thawani, Avijit, Jay Pujara, Pedro A. Szekely, and Filip Ilievski. "Representing Numbers in NLP: a Survey and a Vision." arXiv preprint arXiv:2103.13136 (2021).

## Emperical Systems
* [Why Reinvent the Wheel – Let’s Build Question Answering Systems Together](summary/frankenstein.md) Kuldeep Singh et al., IW3C2 2018 [[arXiv](http://jens-lehmann.org/files/2018/www_qa_pipelines.pdf)]

## Software packages on NLP
* [jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models](summary/jiant.md), Pruksachatkun et al, 2020 [[arXiv](https://arxiv.org/pdf/2003.02249.pdf)]

* [Huggingface's Transformers: State-of-the-art Natural Language Processing](summary/huggingface.md), Wolf et al, 2020 [[arXiv](https://arxiv.org/pdf/1910.03771.pdf)]

* [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](summary/allen_nlp.md), Wallace et al, 2019 EMNLP [[arXiv](https://www.aclweb.org/anthology/D19-3002.pdf)]

## AI Ethics & Future

* [Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data](summary/meaning.md) Bender, Koller, ACL 2020 [[arXiv](https://www.aclweb.org/anthology/2020.acl-main.463.pdf)]

* [THIEVES ON SESAME STREET! MODEL EXTRACTION OF BERT-BASED APIS](summary/bert_extraction.md), Krishna, Mohit Iyyer et al. ICLR 2020 [[arXiv](https://arxiv.org/pdf/1910.12366.pdf)]

* [What Can We Do to Improve Peer Review in NLP?](summary/peer_review.md), Anna Rogers, Anna Rogers, 2020 [[arXiv](https://arxiv.org/pdf/2010.03863.pdf)]

## Model Compression

* [Optimal Subarchitecture Extraction For BERT](summary/BORT.md) Adrian de Wynter and Daniel J. Perry, 2020 [[arXiv](https://arxiv.org/pdf/2010.10499.pdf)]

## Deep Learning Building Blocks
* [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](summary/one_cycle_learning.md), Leslie N Smith, 2018 [[arXiv](https://arxiv.org/abs/1803.09820)]

* [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](summary/batch_normalization), Sergey Ioffe, Christian Szegedy, 2015 [[arXiv](https://arxiv.org/pdf/1502.03167.pdf)]

* [Kaiming Initialization: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](summary/kaiming_initialization.md), Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun [[arXiv]](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf)

* [LAMB: Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](summary/lamb.md), Yang You, Jing Li [[arXiv]](https://arxiv.org/abs/1904.00962)

* [Sentencepience: Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](summary/sentencepiece.md) Taku Kudo, Google, 2018 [[arXiv](https://arxiv.org/abs/1804.10959)]

* [Self-Attention with Relative Position Representations](summary/relative_position.md) Peter Shaw, Jakob Uszkoreit, Ashish Vaswani [[arXiv](https://arxiv.org/abs/1803.02155)]

* [Group Normalization](summary/group_norm.md) Wu, Yuxin, and Kaiming He. "Group normalization." Proceedings of the European conference on computer vision (ECCV). 2018. [[arXiv](https://arxiv.org/pdf/1803.08494.pdf)]

* Cheng, Hao, Xiaodong Liu, Lis Pereira, Yaoliang Yu, and Jianfeng Gao. ["Posterior Differential Regularization with f-divergence for Improving Model Robustness."](summary/PDR.md) arXiv preprint arXiv:2010.12638 (2020) [[arXiv](https://arxiv.org/pdf/2010.12638.pdf)].

* Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. ["The curious case of neural text degeneration."](summary/Nucleus.md) arXiv preprint [[arXiv:1904.09751](https://arxiv.org/pdf/1904.09751.pdf)] (2019).

* Sablayrolles, Alexandre, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. ["Spreading vectors for similarity search."](summary/Catelyzer.md) arXiv preprint arXiv:1806.03198 (2018) [[PDF](https://arxiv.org/pdf/1806.03198.pdf)].

## Learning Outcomes 
* [Prerequisite-Driven Deep Knowledge Tracing](summary/PDKTC.md) Penghe Chen, Yu Lu, Vincent W. Zheng, Yang Pian, ICDM 2018 [[arXiv](https://aic-fe.bnu.edu.cn/docs/20190108101850881476.pdf)]

* [Deep Knowledge Tracing](summary/DKT.md) Piech et al, [[NIPS 2015]](https://proceedings.neurips.cc/paper/2015/file/bac9162b47c56fc8a4d2a519803d51b3-Paper.pdf)

* [Individualized Bayesian Knowledge Tracing
Models](summary/Individualised_BKT.md) Michael V. Yudelson, Kenneth R. Koedinger, and Geoffrey J. Gordon, CMU, [[Springer 2013](https://www.cs.cmu.edu/~ggordon/yudelson-koedinger-gordon-individualized-bayesian-knowledge-tracing.pdf)]

## Speech Synthesis
* Li, Naihan, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. ["Neural speech synthesis with transformer network."](summary/TransformerTTS.md) In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 6706-6713. 2019.[ (AAAI)](https://ojs.aaai.org/index.php/AAAI/article/download/4642/4520)

* Ren, Yi, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. ["Fastspeech: Fast, robust and controllable text to speech."](summary/fastspeech.md) arXiv preprint arXiv:1905.09263 (2019).

* Łańcucki, Adrian. ["Fastpitch: Parallel text-to-speech with pitch prediction."](summary/fastpitch.md) In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6588-6592. IEEE, 2021 [[Arxiv](https://arxiv.org/pdf/2006.06873.pdf)].

* Boersma, Paul. ["Accurate short-term analysis of the fundamental frequency and the harmonics-to-noise ratio of a sampled sound."](summary/fundamental_freq.md) In Proceedings of the institute of phonetic sciences, vol. 17, no. 1193, pp. 97-110. 1993.

* Ren, Yi, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. [Fastspeech 2: Fast and high-quality end-to-end text to speech.](summary/fastspeech2.md) arXiv preprint arXiv:2006.04558 (2020).

* McAuliffe, Michael, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. ["Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi."](summary/MFA.md) In Interspeech, vol. 2017, pp. 498-502. 2017. [[PDF](https://www.isca-speech.org/archive/Interspeech_2017/pdfs/1386.PDF)]

* Weiss, Ron J., R. J. Skerry-Ryan, Eric Battenberg, Soroosh Mariooryad, and Diederik P. Kingma. ["Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis."](summary/wave_tacotron.md) In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5679-5683. [[IEEE, 2021](https://arxiv.org/pdf/2011.03568.pdf)].

## Asset Price Prediction

* Gezici, B., & Sefer, E. (2024). [Deep transformer-based asset price and direction prediction.](summary/vit_asset_price_prediction.md) IEEE Access.

* JuHyok, U., Lu, P., Kim, C., Ryu, U., & Pak, K. (2020). ["A new LSTM based reversal point prediction method using upward/downward reversal point feature sets"](summary/lstm_trend_reversal.md). Chaos, Solitons & Fractals, 132, 109559.

* Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). [Why do tree-based models still outperform deep learning on typical tabular data?.](summary/tree_models_on_tabulardata.md) Advances in neural information processing systems, 35, 507-520.

* Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). [Revisiting deep learning models for tabular data.](summary/FT_transformer.md) Advances in Neural Information Processing Systems, 34, 18932-18943.

## Machine Learning Fundametals
* [Linear Regression](summary/Linear_Regression.pdf)
* [Logistic Regression](summary/Logistic_Regression.pdf)
* [Normal Equation and Newton's Method](summary/Normal_Equation_and_Newton_s_Method.pdf)
* [Generalised Linear Models](summary/Generalised_Linear_Models.pdf.pdf)
* [Generative Learning Algorithms](summary/Generative_Learning_Algorithms.pdf.pdf)
* [Support Vector Machines](summary/Support_Vector_Machines.pdf)
* [Learning Theory](summary/Learning_Theory.pdf)
* [Regularisation and Model Selection](summary/Regularization_and_model_selection.pdf)
* [The perceptron and large margin classifer](summary/The_perceptron_and_large_margin_classifiers.pdf.pdf)
* [k-means and EM Algorithm](summary/10__K_Means.pdf)
* [Trees, Bagging and Boosting](summary/trees_bagging_boosting.pdf)
* [Hypothesis Testing](summary/Hypothesis_Testing.pdf)

AIgrads

Today's picks

Was this content helpful?

Recent Feedback

AIgrads

Today's picks

Was this content helpful?

Recent Feedback

Download README