Awesome-Multi-Token-Prediction

KO | EN

gitlite — search

#python #javascript #tensorflow #machine-learning #golang #android #c #vuejs #python3 #deep-learning #mongodb #vue

Awesome-Multi-Token-Prediction

★ 194 Open GitHub ↗

A curated list of papers, tools, and resources on Multi-Token Prediction (MTP) and related techniques in Large Language Models (LLMs), Speech-Language Models (SLMs), and more.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# <img height=45 src="https://raw.githubusercontent.com/Tarikul-Islam-Anik/Animated-Fluent-Emojis/master/Emojis/Travel%20and%20places/Rocket.png"/>  Awesome Multi-Token Prediction (MTP!)

> *A curated list of papers, tools, and resources on Multi-Token Prediction (MTP) and related techniques in Large Language Models (LLMs), Speech-Language Models (SLMs), and more.*

Multi-Token Prediction (MTP) is an emerging paradigm that enhances the efficiency and capability of language and multimodal models by allowing them to predict multiple tokens simultaneously. This repository collects recent research and implementations in this exciting direction.

---

![](https://raw.githubusercontent.com/Xiaohao-Liu/Awesome-Multi-Token-Prediction/main/img.png)

## 🔬 Recent Papers (2026)

| Title | Institution | Year | Paper | Code |
|------|-------------|------|-------|------|
| **AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction** | CUHK | 2026 | [PDF](https://arxiv.org/pdf/2608.00434) | - |
| **Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing** | Qualcomm AI Research | ICML 2026 | [PDF](https://arxiv.org/pdf/2603.17942) | - |
| **Fast and Expressive Multi-Token Prediction with Probabilistic Circuits** | University of Edinburgh | ICML 2026 | [PDF](https://arxiv.org/pdf/2511.11346) | - |
| **Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries** | Mila / CMU / FAIR at Meta | ICLR 2026 | [PDF](https://arxiv.org/pdf/2510.14751) | - |
| **Parallel Token Prediction for Language Models** | University of California | ICLR 2026 | [PDF](https://arxiv.org/pdf/2512.21323) | - |
| **Multi-Token Prediction via Self-Distillation** | University of Maryland | 2026 | [PDF](https://arxiv.org/pdf/2602.06019) | [Code](https://github.com/jwkirchenbauer/mtp-lm) |
| **Self-Distillation for Multi-Token Prediction** | Tencent | 2026 | [PDF](https://arxiv.org/pdf/2603.23911) | - |
| **Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs** | Renmin University of China | 2026 | [PDF](https://arxiv.org/pdf/2605.27255) | [Code](https://github.com/AlbertTan404/PIPO) |
| **Temporal Guidance for Large Language Models** | NUAA | 2026 | [PDF](https://arxiv.org/pdf/2601.21744) | - |
| **MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-Token Prediction** | NEU | ICASSP 2026 | [PDF](https://arxiv.org/pdf/2510.10003) | - |
| **How Transformers Learn to Plan via Multi-Token Prediction** | UCLA / SJTU / UPenn / RIKEN AIP | COLM 2026 | [PDF](https://arxiv.org/pdf/2604.11912) | - |
| **Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement** | Shenzhen University / Microsoft Research Asia | 2026 | [PDF](https://arxiv.org/pdf/2604.06155) | [Code](https://github.com/QiminZhong/LSE-MTP) |
| **Beyond Token-Level Policy Gradients for Complex Reasoning with LLMs** | HIT / Baidu | 2026 | [PDF](https://arxiv.org/pdf/2602.14386) | - |
| **BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion** | SJTU / MMLab CUHK / CAS | 2026 | [PDF](https://arxiv.org/pdf/2605.11577) | - |
| **Efficient Document Parsing via Parallel Token Prediction** | Tencent / Renmin University of China | 2026 | [PDF](https://arxiv.org/pdf/2603.15206) | - |
| **DFlash: Block Diffusion for Flash Speculative Decoding** | z-lab | ICML 2026 | [PDF](https://arxiv.org/pdf/2602.06036) | [Code](https://github.com/z-lab/dflash) |
## 🔬 Recent Papers (2025)

| Title | Institution | Year | Paper | Code |
|------|-------------|------|-------|------|
| **L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models** | NUS | NeurIPS 2025 | [PDF](https://arxiv.org/pdf/2505.17505) | [Code](https://github.com/Xiaohao-Liu/L-MTP) |
| **Peeking Into The Future For Contextual Biasing** | Samsung Research America | 2025 | [PDF](https://arxiv.org/pdf/2512.17657) | - |
| **MiMo-V2-Flash Technical Report** | Xiaomi | 2025 | [PDF](https://github.com/XiaomiMiMo/MiMo-V2-Flash/blob/main/paper.pdf) | [Code](https://github.com/XiaomiMiMo/MiMo-V2-Flash) |
| **FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction** | Tencent | 2025 | [PDF](https://arxiv.org/pdf/2509.18362) | [Code]([Tencent-BAC/FastMTP](https://github.com/Tencent-BAC/FastMTP)) |
| **Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariesn** | Meta | 2025 | [PDF](https://arxiv.org/pdf/2510.14751) | - |
| **Enhancing Visual Planning with Auxiliary Tasks and Multi-token Prediction** | Meta | 2025 | [PDF](https://arxiv.org/pdf/2507.15130) | - |
| **Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential** | Apple | 2025 | [PDF](https://arxiv.org/pdf/2507.11851v1) | - |
| **Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction** (Outstanding paper) | Google | ICML 2025 | [PDF](https://arxiv.org/pdf/2504.15266) | - |
| **Improving Large Language Models with Concept-Aware Fine-Tuning** | NTU | 2025 | [PDF](https://arxiv.org/pdf/2506.07833) | [Code](https://github.com/michaelchen-lab/caft-llm) |
| **Speech-Language Models with Decoupled Tokenizers and Multi-Token Prediction** | Fudan University | 2025 | [PDF](https://arxiv.org/pdf/2506.12537) | [Code](https://github.com/cnxupupup/SLM-Decoupled-MTP) |
| **Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation** | ByteDance Seed | 2025 | [PDF](https://arxiv.org/pdf/2506.09990) | [Project Page](https://chain-of-action.github.io/) |
| **DONUT: A Decoder-Only Model for Trajectory Prediction** | RWTH Aachen University | 2025 | [PDF](https://arxiv.org/pdf/2506.06854) | [Code](https://vision.rwth-aachen.de/DONUT) |
| **Generating Long Semantic IDs in Parallel for Recommendation** | UC Berkeley | KDD 2025 | [PDF](https://arxiv.org/pdf/2506.05781) | [Code](https://github.com/facebookresearch/RPG_KDD2025) |
| **Pre-Training Curriculum for Multi-Token Prediction in Language Models** | Humboldt-Universität zu Berlin | 2025 | [PDF](https://arxiv.org/pdf/2505.22757) | [Code](https://github.com/aynetdia/mtp_curriculum) |
| **Multi-Token Prediction Needs Registers** | Athena Research Center | NeurIPS 2025 | [PDF](https://arxiv.org/pdf/2505.10518) | - |
| **MiMo: Unlocking the Reasoning Potential of Language Model – From Pretraining to Posttraining** | Xiaomi | 2025 | [PDF](https://arxiv.org/pdf/2505.07608) | [Code](https://github.com/xiaomimimo/MiMo) |
| **VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation** | SJTU | 2025 | [PDF](https://arxiv.org/pdf/2504.04060) | - |
| **GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLM** | TeleAI | 2025 | [PDF](https://arxiv.org/pdf/2504.12339) | - |
| **On multi-token prediction for efficient LLM inference** | Sony | 2025 | [PDF](https://arxiv.org/pdf/2502.09419) | - |

---

## 📚 Earlier Works & Foundations

| Title | Institution | Year | Paper | Code |
|------|-------------|------|-------|------|
| **Deepseek-v3 Technical Report** | DeepSeek AI | 2024 | [PDF](https://arxiv.org/pdf/2412.19437) | - |
| **Better & Faster Large Language Models via Multi-token Prediction** | Meta | ICML 2024 | [PDF](https://arxiv.org/pdf/2404.19737) | - |
| **ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training** | USTC | 2020 | [PDF](https://arxiv.org/pdf/2001.04063) | - |

---

## 🧠 Speculative Decoding + MTP

| Title | Institution | Year | Paper | Code |
|------|-------------|------|-------|------|
| **DFlash: Block Diffusion for Flash Speculative Decoding** | z-lab | 2026 | [PDF](https://arxiv.org/pdf/2602.06036) | [Code](https://github.com/z-lab/dflash) |
| **EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test** | Peking University | 2025 | [PDF](https://arxiv.org/pdf/2503.01840) | [Code](https://github.com/SafeAILab/EAGLE) |
| **Accelerating Codec-Based Speech Synthesis with Multi-Token Prediction and Speculative Decoding** | KAIST | ICASSP 2025 | [PDF](https://arxiv.org/pdf/2410.13839) | [Project Page](https://mm.kaist.ac.kr/projects/mtp/) |
| **EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty** | Peking University | ICML 2024 | [PDF](https://arxiv.org/pdf/2401.15077) | [Code](https://github.com/SafeAILab/EAGLE?tab=readme-ov-file) |
| **Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding** | MIT | COLM 2024 | [PDF](https://arxiv.org/pdf/2402.05109) | [Code](https://github.com/zankner/Hydra) |
| **MEDUSA: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads** | Princeton University | ICML 2024 | [PDF](https://arxiv.org/pdf/2401.10774) | [Code](https://github.com/FasterDecoding/Medusa) |
| **Blockwise Parallel Decoding for Deep Autoregressive Models** | UC Berkeley | NeurIPS 2018 | [PDF](https://proceedings.neurips.cc/paper/2018/file/c4127b9194fe8562c64dc0f5bf2c93bc-Paper.pdf) | - |

---

## 🧩 Want to Contribute?

We welcome contributions! Please feel free to submit a PR or open an issue if you'd like to add new papers, tools, or correct any mistakes.

### ✅ Guidelines:
- Add relevant papers or projects related to MTP.
- Use consistent formatting.
- Include links where available.

Awesome-Multi-Token-Prediction

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README