nlp-roadmap

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 113 Open GitHub ↗

🗺️ 一个自然语言处理的学习路线图

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Natural Language Processing Roadmap

🗺️ 一个「自然语言处理」的**学习路线图**。

> ⚠️ 注意:
>
> 1. 这个项目包含一个名为 `PCB` 的小实验，这个的 PCB 不是印刷电路板 `Printed Circuit Board`，也不是进程控制块 `Process Control Block`，而是 `Paper Code Blog` 的缩写。我认为 `论文`、`代码` 和 `博客` 这三个东西，可以让我们兼顾理论和实践同时，快速地掌握知识点！
>
> 2. 每篇论文后面的星星个数代表论文的重要性（*主观意见，仅供参考*）。
>     1. 🌟: 一般；
>     2. 🌟🌟: 重要；
>     3. 🌟🌟🌟: 非常重要。

## 1 分词 `Word Segmentation`

**词是能够独立活动的最小语言单位。** 在自然语言处理中，通常都是以词作为基本单位进行处理的。由于英文本身具有天生的优势，以空格划分所有词。而中文的词与词之间没有明显的分割标记，所以在做中文语言处理前的首要任务，就是把连续中文句子分割成「词序列」。这个分割的过程就叫**分词**。[了解更多](https://www.v2ai.cn/2018/04/26/nature-language-processing/2-word-segmentation/)

### 综述

- 汉语分词技术综述 [{Paper}](http://www.lis.ac.cn/CN/article/downloadArticleFile.do?attachType=PDF&id=9402) 🌟
- 国内中文自动分词技术研究综述 [{Paper}](http://www.lis.ac.cn/CN/article/downloadArticleFile.do?attachType=PDF&id=11361) 🌟
- 汉语自动分词的研究现状与困难 [{Paper}](http://sourcedb.ict.cas.cn/cn/ictthesis/200907/P020090722605434114544.pdf) 🌟🌟
- 汉语自动分词研究评述 [{Paper}](http://59.108.48.5/course/mining/12-13spring/%E5%8F%82%E8%80%83%E6%96%87%E7%8C%AE/02-01%E6%B1%89%E8%AF%AD%E8%87%AA%E5%8A%A8%E5%88%86%E8%AF%8D%E7%A0%94%E7%A9%B6%E8%AF%84%E8%BF%B0.pdf) 🌟🌟
- 中文分词十年又回顾: 2007-2017 [{Paper}](https://arxiv.org/pdf/1901.06079.pdf) 🌟🌟🌟
- chinese-word-segmentation [{Code}](https://github.com/Ailln/chinese-word-segmentation)
- 深度学习中文分词调研 [{Blog}](http://www.hankcs.com/nlp/segment/depth-learning-chinese-word-segmentation-survey.html)

## 2 词嵌入 `Word Embedding`

**词嵌入**就是找到一个映射或者函数，生成在一个新的空间上的表示，该表示被称为「单词表示」。[了解更多](https://www.v2ai.cn/2018/08/27/nature-language-processing/6-word-embedding/)

### 综述

- Word Embeddings: A Survey [{Paper}](https://arxiv.org/pdf/1901.09069.pdf) 🌟🌟🌟
- Visualizing Attention in Transformer-Based Language Representation Models [{Paper}](https://arxiv.org/pdf/1904.02679.pdf) 🌟🌟
- **PTMs**: Pre-trained Models for Natural Language Processing: A Survey [{Paper}](https://arxiv.org/pdf/2003.08271.pdf) [{Blog}](https://zhuanlan.zhihu.com/p/115014536) 🌟🌟🌟
- Efficient Transformers: A Survey [{Paper}](https://arxiv.org/pdf/2009.06732.pdf) 🌟🌟
- A Survey of Transformers [{Paper}](https://arxiv.org/pdf/2106.04554.pdf) 🌟🌟
- Pre-Trained Models: Past, Present and Future [{Paper}](https://arxiv.org/pdf/2106.07139.pdf) 🌟🌟
- Pretrained Language Models for Text Generation: A Survey [{Paper}](https://arxiv.org/pdf/2105.10311.pdf) 🌟
- A Practical Survey on Faster and Lighter Transformers [{Paper}](https://arxiv.org/pdf/2103.14636.pdf) 🌟
- The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures [{Paper}](https://arxiv.org/pdf/2104.10640.pdf) 🌟🌟

### 核心

- **NNLM**: A Neural Probabilistic Language Model [{Paper}](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf) [{Code}](https://github.com/FuYanzhe2/NNLM) [{Blog}](https://zhuanlan.zhihu.com/p/21240807) 🌟
- **W2V**: Efficient Estimation of Word Representations in Vector Space [{Paper}](https://arxiv.org/abs/1301.3781) 🌟🌟
- **Glove**: Global Vectors for Word Representation [{Paper}](https://nlp.stanford.edu/pubs/glove.pdf) 🌟🌟
- **CharCNN**: Character-level Convolutional Networks for Text Classification [{Paper}](https://arxiv.org/pdf/1509.01626.pdf) [{Blog}](https://zhuanlan.zhihu.com/p/51698513) 🌟
- **ULMFiT**: Universal Language Model Fine-tuning for Text Classification [{Paper}](https://arxiv.org/pdf/1801.06146.pdf) 🌟
- **SiATL**: An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models [{Paper}](https://www.aclweb.org/anthology/N19-1213.pdf) 🌟
- **FastText**: Bag of Tricks for Efficient Text Classification [{Paper}](https://arxiv.org/pdf/1607.01759.pdf) 🌟🌟
- **CoVe**: Learned in Translation: Contextualized Word Vectors [{Paper}](https://arxiv.org/pdf/1708.00107.pdf) 🌟
- **ELMo**: Deep contextualized word representations [{Paper}](https://arxiv.org/pdf/1802.05365.pdf) 🌟🌟
- **Transformer**: Attention is All you Need [{Paper}](https://arxiv.org/pdf/1706.03762.pdf) [{Code}](https://github.com/tensorflow/tensor2tensor) [{Blog}](http://jalammar.github.io/illustrated-transformer/) 🌟🌟🌟
- **GPT**: Improving Language Understanding by Generative Pre-Training [{Paper}](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) 🌟
- **GPT2**: Language Models are Unsupervised Multitask Learners [{Paper}](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf) [{Code}](https://github.com/openai/gpt-2) [{Blog}](https://openai.com/blog/better-language-models/) 🌟🌟
- **GPT3**: Language Models are Few-Shot Learners [{Paper}](https://arxiv.org/pdf/2005.14165.pdf) [{Code}](https://github.com/openai/gpt-3) 🌟🌟🌟
- **GPT4**: GPT-4 Technical Report [{Paper}](https://arxiv.org/pdf/2303.08774.pdf) 🌟🌟🌟
- **BERT**: Pre-training of Deep Bidirectional Transformers for Language Understanding [{Paper}](https://arxiv.org/pdf/1810.04805.pdf) [{Code}](https://github.com/google-research/bert) [{Blog}](https://zhuanlan.zhihu.com/p/49271699) 🌟🌟🌟
- **UniLM**: Unified Language Model Pre-training for Natural Language Understanding and Generation [{Paper}](https://arxiv.org/pdf/1905.03197.pdf) [{Code}](https://github.com/microsoft/unilm) [{Blog}](https://zhuanlan.zhihu.com/p/68327602) 🌟🌟
- **T5**: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer [{Paper}](https://arxiv.org/pdf/1910.10683.pdf) [{Code}](https://github.com/google-research/text-to-text-transfer-transformer) [{Blog}](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) 🌟
- **ERNIE**(Baidu): Enhanced Representation through Knowledge Integration [{Paper}](https://arxiv.org/pdf/1904.09223.pdf) [{Code}](https://github.com/PaddlePaddle/ERNIE) 🌟
- **ERNIE**(Tsinghua): Enhanced Language Representation with Informative Entities [{Paper}](https://arxiv.org/pdf/1905.07129.pdf) [{Code}](https://github.com/thunlp/ERNIE) 🌟
- **RoBERTa**: A Robustly Optimized BERT Pretraining Approach [{Paper}](https://arxiv.org/pdf/1907.11692.pdf) 🌟
- **ALBERT**: A Lite BERT for Self-supervised Learning of Language Representations [{Paper}](https://arxiv.org/pdf/1909.11942.pdf) [{Code}](https://github.com/google-research/ALBERT) 🌟🌟
- **TinyBERT**: Distilling BERT for Natural Language Understanding [{Paper}](https://arxiv.org/pdf/1909.10351.pdf) 🌟🌟
- **FastFormers**: Highly Efficient Transformer Models for Natural Language Understanding [{Paper}](https://arxiv.org/pdf/2010.13382.pdf) [{Code}](https://github.com/microsoft/fastformers) 🌟🌟

### 其他

- word2vec Parameter Learning Explained [{Paper}](https://arxiv.org/pdf/1411.2738.pdf) 🌟🌟
- Semi-supervised Sequence Learning [{Paper}](https://arxiv.org/pdf/1511.01432.pdf) 🌟🌟
- BERT Rediscovers the Classical NLP Pipeline [{Paper}](https://arxiv.org/pdf/1905.05950.pdf) 🌟
- Pre-trained Languge Model Papers [{Blog}](https://github.com/thunlp/PLMpapers)
- HuggingFace Transformers [{Code}](https://github.com/huggingface/transformers)
- Fudan FastNLP [{Code}](https://github.com/fastnlp/fastNLP)

## 3 文本分类 `Text Classification`

### 综述

- A Survey on Text Classification: From Shallow to Deep Learning [{Paper}](https://arxiv.org/pdf/2008.00364.pdf) 🌟🌟🌟
- Deep Learning Based Text Classification: A Comprehensive Review [{Paper}](https://arxiv.org/pdf/2004.03705.pdf) 🌟🌟

### CNN

- **TextCNN**:Convolutional Neural Networks for Sentence Classification [{Paper}](https://arxiv.org/pdf/1408.5882.pdf) [{Code}](https://github.com/dennybritz/cnn-text-classification-tf) 🌟🌟🌟
- Convolutional Neural Networks for Text Categorization: Shallow Word-level vs. Deep Character-level [{Paper}](https://arxiv.org/pdf/1609.00718.pdf) 🌟
- **DPCNN**: Deep Pyramid Convolutional Neural Networks for Text Categorization [{Paper}](https://www.aclweb.org/anthology/P17-1052.pdf) [{Code}](https://github.com/Cheneng/DPCNN) 🌟🌟

## 4 序列标注 `Sequence Labeling`

### 综述

- Sequence Labeling 的发展史（DNNs+CRF）[{Blog}](https://zhuanlan.zhihu.com/p/34828874)

### Bi-LSTM + CRF

- End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF [{Paper}](https://www.aclweb.org/anthology/P16-1101) 🌟🌟

- pytorch_NER_BiLSTM_CNN_CRF [{Code}](https://github.com/bamtercelboo/pytorch_NER_BiLSTM_CNN_CRF)
- NN_NER_tensorFlow [{Code}](https://github.com/LopezGG/NN_NER_tensorFlow)
- End-to-end-Sequence-Labeling-via-Bi-directional-LSTM-CNNs-CRF-Tutorial [{Code}](https://github.com/jayavardhanr/End-to-end-Sequence-Labeling-via-Bi-directional-LSTM-CNNs-CRF-Tutorial)
- Bi-directional LSTM-CNNs-CRF [{Code}](https://zhuanlan.zhihu.com/p/30791481)

### 其他

- Sequence to Sequence Learning with Neural Networks [{Paper}](https://proceedings.neurips.cc/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf) 🌟
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks [{Paper}](https://arxiv.org/pdf/1506.03099.pdf) 🌟

## 5 对话系统 `Dialogue Systems`

### 综述

- A Survey on Dialogue Systems: Recent Advances and New Frontiers [{Paper}](https://arxiv.org/pdf/1711.01731v1.pdf) [{Blog}](https://zhuanlan.zhihu.com/p/45210996) 🌟🌟
- 小哥哥，检索式chatbot了解一下？ [{Blog}](https://mp.weixin.qq.com/s/yC8uYwti9Meyt83xkmbmcg) 🌟🌟🌟
- Recent Neural Methods on Slot Filling and Intent Classification for Task-Oriented Dialogue Systems: A Survey [{Paper}](https://arxiv.org/pdf/2011.00564.pdf) 🌟🌟

### Open Domain Dialogue Systems

- **HERD**: Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models [{Paper}](https://arxiv.org/pdf/1507.04808v3.pdf) [{Code}](https://github.com/hsgodhia/hred) 🌟🌟
- Adversarial Learning for Neural Dialogue Generation [{Paper}](https://arxiv.org/pdf/1701.06547.pdf) [{Code}](https://github.com/liuyuemaicha/Adversarial-Learning-for-Neural-Dialogue-Generation-in-Tensorflow) [{Blog}](https://blog.csdn.net/liuyuemaicha/article/details/60581187) 🌟🌟

### Task Oriented Dialogue Systems

- **Joint NLU**: Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling [{Paper}](https://arxiv.org/pdf/1609.01454.pdf) [{Code}](https://github.com/Ailln/chatbot) 🌟🌟
- BERT for Joint Intent Classification and Slot Filling [{Paper}](https://arxiv.org/pdf/1902.10909.pdf) 🌟
- Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures [{Paper}](https://www.aclweb.org/anthology/P18-1133.pdf) [{Code}](https://github.com/WING-NUS/sequicity) 🌟🌟
- Attention with Intention for a Neural Network Conversation Model [{Paper}](https://arxiv.org/pdf/1510.08565.pdf) 🌟
- **REDP**: Few-Shot Generalization Across Dialogue Tasks [{Paper}](https://arxiv.org/pdf/1811.11707.pdf) [{Blog}](http://www.xuwei.io/2019/03/18/%E3%80%8Afew-shot-generalization-across-dialogue-tasks%E3%80%8B%E8%AE%BA%E6%96%87%E7%AC%94%E8%AE%B0/) 🌟🌟
- **TEDP**: Dialogue Transformers [{Paper}](https://arxiv.org/pdf/1910.00486.pdf) [{Code}](https://github.com/RasaHQ/TED-paper) [{Blog}](https://zhuanlan.zhihu.com/p/336977835) 🌟🌟🌟

### Conversational Response Selection

- Multi-view Response Selection for Human-Computer Conversation [{Paper}](https://aclweb.org/anthology/D16-1036.pdf) 🌟🌟
- **SMN**: Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots [{Paper}](https://www.aclweb.org/anthology/P17-1046.pdf) [{Code}](https://github.com/MarkWuNLP/MultiTurnResponseSelection) [{Blog}](https://zhuanlan.zhihu.com/p/65062025) 🌟🌟🌟:
- **DUA**: Modeling Multi-turn Conversation with Deep Utterance Aggregation [{Paper}](https://www.aclweb.org/anthology/C18-1317.pdf) [{Code}](https://github.com/cooelf/DeepUtteranceAggregation) [{Blog}](https://zhuanlan.zhihu.com/p/60618158) 🌟🌟
- **DAM**: Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network [{Paper}](https://www.aclweb.org/anthology/P18-1103.pdf) [{Code}](https://github.com/baidu/Dialogue/tree/master/DAM) [{Blog}](https://zhuanlan.zhihu.com/p/65143297) 🌟🌟🌟
- **IMN**: Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots [{Paper}](https://arxiv.org/pdf/1901.01824.pdf) [{Code}](https://github.com/JasonForJoy/IMN) [{Blog}](https://zhuanlan.zhihu.com/p/68590678) 🌟🌟
- Dialogue Transformers [{Paper}](https://arxiv.org/pdf/1910.00486.pdf) 🌟🌟

## 6 主题模型 `Topic Model`

### LDA

- Latent Dirichlet Allocation [{Paper}](https://jmlr.org/papers/volume3/blei03a/blei03a.pdf) [{Blog}](https://arxiv.org/pdf/1908.03142.pdf) 🌟🌟🌟

## 7 知识图谱 `Knowledge Graph`

### 综述

- Towards a Definition of Knowledge Graphs [{Paper}](http://ceur-ws.org/Vol-1695/paper4.pdf) 🌟🌟🌟

## 8 提示学习 `Prompt Learning`

### 综述

- **PPP**: Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing [{Paper}](https://arxiv.org/pdf/2107.13586.pdf) [{Blog}](https://zhuanlan.zhihu.com/p/395115779) 🌟🌟🌟

## 9 图神经网络 `Graph Neural Network`

### 综述

- Graph Neural Networks for Natural Language Processing: A Survey [{Paper}](https://arxiv.org/pdf/2106.06090.pdf) 🌟🌟

## 10 句嵌入 `Sentence Embedding`

### 核心

- **InferSent**: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data [{Paper}](https://arxiv.org/pdf/1705.02364.pdf) [{Code}](https://github.com/facebookresearch/InferSent) 🌟🌟
- **Sentence-BERT**: Sentence Embeddings using Siamese BERT-Networks [{Paper}](https://arxiv.org/pdf/1908.10084.pdf) [{Code}](https://github.com/UKPLab/sentence-transformers) 🌟🌟🌟
- **BERT-flow**: On the Sentence Embeddings from Pre-trained Language Models [{Paper}](https://arxiv.org/pdf/2011.05864.pdf) [{Code}](https://github.com/bohanli/BERT-flow) [{Blog}](https://zhuanlan.zhihu.com/p/337134133) 🌟🌟
- **SimCSE**: Simple Contrastive Learning of Sentence Embeddings [{Paper}](https://arxiv.org/pdf/2104.08821.pdf) [{Code}](https://github.com/princeton-nlp/SimCSE) 🌟🌟🌟

## 参考

- [thunlp/NLP-THU](https://github.com/thunlp/NLP-THU)
- [iwangjian/Paper-Reading](https://github.com/iwangjian/Paper-Reading)
- [thunlp/PromptPapers](https://github.com/thunlp/PromptPapers)

nlp-roadmap

Today's picks

Was this content helpful?

Recent Feedback

nlp-roadmap

Today's picks

Was this content helpful?

Recent Feedback

Download README