Precision-RL

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 197 Open GitHub ↗

Defeating the Training-Inference Mismatch via FP16

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Defeating the Training-Inference Mismatch via FP16
[Penghui Qi](https://scholar.google.com/citations?user=CLRsGEMAAAAJ&hl=en), [Zichen Liu](https://lkevinzc.github.io/), [Xiangxin Zhou](https://zhouxiangxin1998.github.io/)
[Tianyu Pang](https://p2333.github.io/), [Chao Du](https://duchao0726.github.io/), [Wee Sun Lee](https://scholar.google.com/citations?user=8PCrLgwAAAAJ&hl=en), [Min Lin](https://scholar.google.com.sg/citations?user=BGONmkIAAAAJ&hl=en)

[![Paper](https://img.shields.io/badge/paper-A42C25?style=for-the-badge&logo=arxiv&logoColor=white )](https://arxiv.org/pdf/2510.26788)
[![Github](https://img.shields.io/badge/Precision_RL-000000?style=for-the-badge&logo=github&logoColor=000&logoColor=white)](https://github.com/sail-sg/Precision-RL)

</div>

## Updates

* 14/11/2025: 🎉 VeRL has supported FP16 training for both [FSDP](https://github.com/volcengine/verl/pull/4036) and [Megatron](https://github.com/volcengine/verl/pull/4086) (dense model only). The native FP16 support of MoE in Megatron is [on the way](https://github.com/nvidia/megatron-lm/issues/2102).
* 31/10/2025: FP16 training has been supported by [Oat](https://github.com/sail-sg/oat), a highly modular, research-friendly and efficient LLM RL framework.

## Overview

![](https://raw.githubusercontent.com/sail-sg/Precision-RL/main/figures/bf16_vs_fp16_training.png)
*Figure 1: Training reward comparison between BF16 and FP16. We evaluate across diverse settings: our Sanity test with various algorithms (GRPO, GSPO, TIS, MIS, PG); different model families (R1D, Qwen and OctoThinker); alternative fine-tuning methods of Lora; and larger scale models (Dense-14B, MoE). Results are validated on two independent frameworks (VeRL and Oat).*

![](https://raw.githubusercontent.com/sail-sg/Precision-RL/main/figures/bf16_vs_fp16_evaluation.png)
*Figure 2: Evaluation comparisons between BF16 and FP16 across various frameworks, algorithms, datasets and training regimes.*

![](https://raw.githubusercontent.com/sail-sg/Precision-RL/main/figures/verl_oat_fix.png)
*Figure 3: Simply switching from BF16 to FP16 stabilizes and prolongs RL training. The basic importance-weighted policy gradient algorithm in FP16 outperforms all baselines in BF16.*

![](https://raw.githubusercontent.com/sail-sg/Precision-RL/main/figures/fp16_comparison.png)
*Figure 4: Comparisons between various algorithms based on FP16.*

![](https://raw.githubusercontent.com/sail-sg/Precision-RL/main/figures/offline_mismatch.png)
*Figure 5: FP16 significantly reduces the training-inference mismatch. The left two plots show the token-level probability distribution, and the right two plots present the distribution of sequence-level log probability ratio between the inference policy ($\mu$) and the training policy ($\pi$).*

## Reproduce the Sanity Test 🎯

### OAT

Find related code in `oat` folder.

### VeRL

**We provide a [minimal patch](verl_fp16.patch) for VeRL to enable FP16 training.**

Find related code in `Precision-RL-verl` folder to reproduce our experiments.

### Sanity-Test Data
The sanity test dataset for [`DeepSeek-R1-Distill-Qwen-1.5B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) can be viewed and downloaded from huggingface: [sail/Sanity-Test-R1D-1.5B](https://huggingface.co/datasets/sail/Sanity-Test-R1D-1.5B).

They are also included in [`this folder`](https://github.com/sail-sg/Precision-RL-verl/tree/main/sanity_test).

## Citation
If you find our works useful for your research, please consider citing:

```bibtex
@article{qi2025precisionrl,
  title={Defeating the Training-Inference Mismatch via FP16},
  author={Qi, Penghui and Liu, Zichen and Zhou, Xiangxin and Pang, Tianyu and Du, Chao and Lee, Wee Sun and Lin, Min},
  journal={arXiv preprint arXiv:2510.26788},
  year={2025}
}
```

Precision-RL

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README