Diffusion-Reward-Modeling-for-Text-Rendering

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

Diffusion-Reward-Modeling-for-Text-Rendering

★ 72

Comparison of Diffusion RLHF algorithms for improved Text Rendering

Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Diffusion Reward Modeling for text rendering on images ✨

## 🌟 Main Contributions

### 1) Text Rendering Dataset
🔗 [Diffusion-Reward-Modeling-for-Text-Rendering-Dataset](https://huggingface.co/datasets/leffff/Diffusion-Reward-Modeling-for-Text-Rendering-Dataset)  
📊 14,000 curated prompts for text-on-image generation

### 2) 6 Alignment Pipelines
- Supervised Fine-Tuning (SFT)
- Reward Weighted Regression (RWR)
- Direct Preference Optimization (DPO)
- DRaFT
- ReFL
- GRPO

### 3) 2 Text Rendering Quality Metrics
- OCR Accuracy Metric
- Reward Model Score

## 📊 Visualizations

## SD3.5 Medium base
<p align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/original/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/original/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/original/2.png" width="30%">
</p>

<div>A superrealistic panda holding a sign that says "I Love SMILES 2025"</div>  
<div>An asian dragon holding a sign with "Summer of Machine Learning by Skoltech 2025 !"</div>  
<div>"I love Harbin Institute of Technology" written on a chinese office building</div>

### SFT Results
#### OCR Metric
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft/sft_ocr/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft/sft_ocr/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft/sft_ocr/2.png" width="30%">
</div>

#### Reward Metric
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft/sft_reward/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft/sft_reward/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft/sft_reward/2.png" width="30%">
</div>

---

### RWR Results
#### OCR Metric
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/rwr/rwr_ocr/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/rwr/rwr_ocr/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/rwr/rwr_ocr/2.png" width="30%">
</div>

#### Reward Metric
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/rwr/rwr_reward/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/rwr/rwr_reward/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/rwr/rwr_reward/2.png" width="30%">
</div>

---

### DPO Results
#### OCR Metric
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/dpo/dpo_ocr/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/dpo/dpo_ocr/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/dpo/dpo_ocr/2.png" width="30%">
</div>

#### Reward Metric
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/dpo/dpo_reward/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/dpo/dpo_reward/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/dpo/dpo_reward/2.png" width="30%">
</div>

---

### SFT + DPO Results
#### OCR Metric
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft_dpo/sft_dpo_ocr/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft_dpo/sft_dpo_ocr/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft_dpo/sft_dpo_ocr/2.png" width="30%">
</div>

#### Reward Metric
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft_dpo/sft_dpo_reward/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft_dpo/sft_dpo_reward/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/sft_dpo/sft_dpo_reward/2.png" width="30%">
</div>

---

### DRaFT Results (Reward only)
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/draft/reward/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/draft/reward/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/draft/reward/2.png" width="30%">
</div>

---

### ReFL Results (Reward only)
<div align="center">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/refl/reward/0.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/refl/reward/1.png" width="30%">
  <img src="https://github.com/leffff/SMILES-2025/blob/main/assets/refl/reward/2.png" width="30%">
</div>

---
### GRPO
#### Reward Metric

## 🔤 Text Rendering Quality Metrics

## 📊 Text Rendering Quality Metric Distributions

## 📈 Method Comparison

## 🚀 Usage
SFT:
```bash
sh run_train_sd3_sft.sh
```

RWR:
```bash
sh run_train_sd3_rwr.sh
```

DPO:
```bash
sh run_train_sd3_dpo.sh
```

ReFL:
```bash
sh run_train_sd3_refl.sh
```

DRaFT:
```bash
sh run_train_sd3_draft.sh
```

GRPO:
```bash
sh run_train_sd3_grpo.sh
```
---
Generate Latents:
```bash
generate_visuals_sd3_480p.sh
```
Encode text prompts into embeddings
```bash
generate_text_embeds_sd3.sh
```
---
Calculate OCR + Levenstein metric
```bash
calculate_levenstein_metric.sh
```
Calculate Reward metric
```bash
calculate_reward_metric.sh
```

## ⚠️ Warning
Quality of DRaFT, ReFL and GRPO is worse in the examples we provide than the ones of DPO, this is due to a small batch size in comparison to DPO. DRaFT, ReFL and GRPO quality could be improved by introducing EMA.

## 📜 Citation

```bibtex
@article{liu2025improving,
  title={Improving video generation with human feedback},
  author={Liu, Jie and Liu, Gongye and Liang, Jiajun and Yuan, Ziyang and Liu, Xiaokun and Zheng, Mingwu and Wu, Xiele and Wang, Qiulin and Qin, Wenyu and Xia, Menghan and others},
  journal={arXiv preprint arXiv:2501.13918},
  year={2025}
}
```

```bibtex
@article{clark2023directly,
  title={Directly fine-tuning diffusion models on differentiable rewards},
  author={Clark, Kevin and Vicol, Paul and Swersky, Kevin and Fleet, David J},
  journal={arXiv preprint arXiv:2309.17400},
  year={2023}
}
```

```bibtex
@article{xu2023imagereward,
  title={Imagereward: Learning and evaluating human preferences for text-to-image generation},
  author={Xu, Jiazheng and Liu, Xiao and Wu, Yuchen and Tong, Yuxuan and Li, Qinkai and Ding, Ming and Tang, Jie and Dong, Yuxiao},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  pages={15903--15935},
  year={2023}
}
```

```bibtex
@article{liu2025flow,
  title={Flow-grpo: Training flow matching models via online rl},
  author={Liu, Jie and Liu, Gongye and Liang, Jiajun and Li, Yangguang and Liu, Jiaheng and Wang, Xintao and Wan, Pengfei and Zhang, Di and Ouyang, Wanli},
  journal={arXiv preprint arXiv:2505.05470},
  year={2025}
}
```

```bibtex
@article{gao2025seedream,
  title={Seedream 3.0 technical report},
  author={Gao, Yu and Gong, Lixue and Guo, Qiushan and Hou, Xiaoxia and Lai, Zhichao and Li, Fanshi and Li, Liang and Lian, Xiaochen and Liao, Chao and Liu, Liyang and others},
  journal={arXiv preprint arXiv:2504.11346},
  year={2025}
}
```

## 📧 Contact
* [Lev Novitskiy](https://t.me/leffffffffffff): Data, SD3 pipeline, DPO, GRPO 
* [Maria Kovaleva](https://t.me/makovka2000): Rewards, RWR, DraFT, ReFL
* [Daniel Kniazev](https://t.me/Oorgien): DraFT, ReFL
* [Aleksandr Kutsakov](https://t.me/askutsakov): SFT, DPO
* [Ilia Statsenko](https://t.me/iljastazenko): Metrics pipeline, SFT
* [S. Panova](https://t.me/mittemim): Paper, Presentation
* [Viacheslav Vasilev](https://t.me/ViacheslavVasilev): Project Coordinator

### Supported by:

[<img src="https://static.tildacdn.com/tild3166-3634-4164-a563-643364373239/SKOLTECH_MACHINE-LEA.png" width="20%">](https://smiles.skoltech.ru/)

Diffusion-Reward-Modeling-for-Text-Rendering

Download

Today's picks

Was this content helpful?

Recent Feedback

Diffusion-Reward-Modeling-for-Text-Rendering

Download

Today's picks

Was this content helpful?

Recent Feedback

Download README