TexOCR

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 15)

ACL 2026 Main

TexOCR Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstruction

<p align="center">
  <a href="https://github.com/QDRhhhh/TexOCR">🌐 Github</a> •
  <a href="https://arxiv.org/abs/2604.22880">📖 Paper</a> •
  <a href="https://huggingface.co/collections/chengyewang/texocr">🤗 Data</a>
</p>

---

**TexOCR** studies page-level reconstruction of document images into **compilable, structurally faithful LaTeX**. Unlike conventional OCR systems that primarily produce plain text or Markdown, TexOCR evaluates whether a model can recover executable LaTeX while preserving equations, tables, section hierarchies, citations, labels, cross-references, and float structure.

We introduce **TexOCR-Bench**, containing **2,135 expert-annotated documents** from diverse technical and public-domain sources, together with a nine-metric evaluation suite covering transcription fidelity, structural faithfulness, and end-to-end usability. We also release **TexOCR-Train**, constructed from approximately **57K papers** and containing **404K aligned page-image–LaTeX/BibTeX training pairs**.

Using this corpus, we train a 2B-parameter model based on Qwen3-VL through supervised fine-tuning followed by reinforcement learning with verifiable rewards. The rewards are derived from deterministic LaTeX unit tests that directly target transcription quality, structural consistency, compilability, and referential integrity.

> **Can document OCR systems reconstruct scientific pages as usable LaTeX—not merely transcribe their visible text?**

---

## News

- **[2026-07]** The TexOCR paper is published in the proceedings of **ACL 2026 Main Conference**.
- **[2026-04]** TexOCR is accepted to **ACL 2026 Main Conference**.
- **[2026-04]** TexOCR-Bench, TexOCR-Train, the RLVR corpus, and trained checkpoints are released on Hugging Face.
- **[2026-04]** Evaluation, SFT, and GRPO/RLVR training code are released in this repository.

---

## Overview

TexOCR covers three end-to-end workflows:

1. **Evaluation** — evaluate open- or closed-source vision-language models on TexOCR-Bench using nine deterministic metrics and a zero-touch LaTeX compilation test.
2. **Supervised Fine-Tuning** — train page-to-LaTeX models on the 404K-pair TexOCR-Train corpus using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory).
3. **RLVR with GRPO** — further optimize the SFT model using [TRL](https://github.com/huggingface/trl) and verifiable rewards derived from LaTeX unit tests.

### Highlights

- **2,135 expert-annotated benchmark documents** covering scientific papers, mathematical documents, and scanned public-domain materials.
- **57K source papers and 404K aligned page-image–LaTeX/BibTeX pairs** for large-scale training.
- **Nine evaluation metrics** spanning transcription fidelity, structural faithfulness, and end-to-end usability.
- **Zero-touch compilation testing** that measures whether reconstructed projects compile without manual correction.
- **SFT + RLVR training** for a 2B-parameter Qwen3-VL-based model.
- **Evaluation of 21 frontier MLLMs and OCR systems** under a unified page-level inference protocol.
- **State-of-the-art performance among evaluated open-source models**, with RLVR consistently improving structural and compilation-related metrics over SFT alone.

---

## TexOCR-Bench

TexOCR-Bench evaluates generated LaTeX along three complementary dimensions.

| Dimension               | Metric                    | Abbreviation | Description                                                  |
| ----------------------- | ------------------------- | ------------ | ------------------------------------------------------------ |
| Transcription Fidelity  | Complex Text Preservation | CTP          | Checks whether representative body-text sentences are faithfully recovered. |
| Transcription Fidelity  | Formula Accuracy          | FA           | Measures normalized token-level correctness of mathematical expressions. |
| Transcription Fidelity  | Table Accuracy            | TA           | Measures recovery of numerical content and structure in tables. |
| Structural Faithfulness | Section Accuracy          | SA           | Checks section-title recovery and hierarchy consistency.     |
| Structural Faithfulness | Citation Coverage         | CC           | Checks whether in-text citations and citation keys are correctly reconstructed. |
| Structural Faithfulness | Reference Validity        | RV           | Verifies that figure and table references resolve to valid labels. |
| End-to-End Usability    | Document-Level Similarity | DS           | Computes normalized character-level similarity to the reference LaTeX. |
| End-to-End Usability    | Baseline Validity Check   | Baseline     | Detects empty, truncated, malformed, or degenerate generations. |
| End-to-End Usability    | Compilation Success Rate  | CSR          | Compiles the reconstructed project without manual intervention. |

The evaluation pipeline processes each document page independently, concatenates the generated LaTeX in document order, and then computes document-level metrics. Compilation is performed with `pdflatex`, with optional fallback to `xelatex` or `lualatex` when supported by the evaluation script.

---

## Released Resources

All models and datasets are grouped in the [TexOCR Hugging Face collection](https://huggingface.co/collections/chengyewang/texocr).

| Stage        | Hugging Face repository                                      | Contents                                                     |
| ------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| Evaluation   | [`chengyewang/TexOCR-eval`](https://huggingface.co/datasets/chengyewang/TexOCR-eval) | TexOCR-Bench annotations, rendered pages, figure archives, queries, and evaluation metadata. |
| SFT data     | [`chengyewang/TexOCR-SFT-json`](https://huggingface.co/datasets/chengyewang/TexOCR-SFT-json) | Page-level LaTeX/BibTeX supervision in JSON format.          |
| SFT images   | [`chengyewang/TexOCR-SFT-figures`](https://huggingface.co/datasets/chengyewang/TexOCR-SFT-figures) | Page-image archives for supervised fine-tuning.              |
| RLVR data    | [`chengyewang/TexOCR-RL-json`](https://huggingface.co/datasets/chengyewang/TexOCR-RL-json) | Harder training examples used for GRPO/RLVR.                 |
| RLVR images  | [`chengyewang/TexOCR-RL-figures`](https://huggingface.co/datasets/chengyewang/TexOCR-RL-figures) | Page images used during GRPO/RLVR training.                  |
| SFT model    | [`chengyewang/TexOCR-SFT`](https://huggingface.co/chengyewang/TexOCR-SFT) | Qwen3-VL-2B checkpoint after supervised fine-tuning.         |
| TexOCR model | [`chengyewang/TexOCR-RL`](https://huggingface.co/chengyewang/TexOCR-RL) | Final checkpoint after SFT and RLVR.                         |

---

## Repository Structure

```text
TexOCR/
├── README.md
├── requirements/
│   ├── OCR_evaluation.txt          # Evaluation environment
│   ├── OCR_Train.txt               # SFT environment (LLaMA-Factory)
│   └── TexRL.txt                   # RLVR environment (TRL / GRPO)
├── evaluation/
│   ├── main.py                     # Batched VLM inference entry point
│   ├── get_query.py                # Build query files from rendered page images
│   ├── evaluate_arxiv.py           # Nine-metric scorer for the arXiv split
│   ├── evaluate_others.py          # Nine-metric scorer for the other-source split
│   ├── combine_results.py          # Merge metrics from the two splits
│   ├── download_eval.sh            # Download TexOCR-Bench and extract archives
│   ├── data/                       # Populated by download_eval.sh
│   ├── model_inference/
│   │   ├── azure_gpt.py            # Hosted GPT / Claude backend
│   │   ├── openai_compatible.py    # OpenAI-compatible API backend
│   │   ├── vllm_inference.py       # Local vLLM backend
│   │   └── vllm_model_list.json    # Supported vLLM checkpoints
│   ├── scripts/
│   │   ├── api.sh                  # Hosted-API launcher
│   │   ├── batch_vllm_small.sh     # Batched small-model launcher
│   │   ├── vllm_small.sh           # vLLM launcher for models up to 8B
│   │   └── vllm_large.sh           # vLLM launcher for 32B+ models
│   └── utils/
│       ├── api_utils.py
│       ├── constant.py             # Prompt and generation defaults
│       ├── input_processing.py     # Image preprocessing and base64 packaging
│       └── vllm_input_preparation.py
└── training/
    ├── sft/
    │   ├── prepare_trainset.py     # Convert SFT JSON to LLaMA-Factory JSONL
    │   └── download_sft.sh         # Download and extract the SFT corpus
    └── grpo/
        ├── grpo.py                 # TRL GRPO trainer with nine reward functions
        ├── grpo_utils.py           # Reward helpers and LaTeX compilation checks
        └── download_rl.sh          # Download and extract the RLVR corpus
```

---

## Quickstart

### 0. Clone the Repository

```bash
git clone https://github.com/QDRhhhh/TexOCR.git
cd TexOCR
```

TexOCR uses three independent Conda environments so that the evaluation, SFT, and RLVR dependencies do not conflict over versions of `torch`, `transformers`, `vllm`, and related libraries.

```bash
# 1. Evaluation
conda create -n OCR_evaluation python=3.11 -y
conda activate OCR_evaluation
pip install -r requirements/OCR_evaluation.txt

# 2. Supervised fine-tuning
conda create -n OCR_Train python=3.11 -y
conda activate OCR_Train
pip install -r requirements/OCR_Train.txt

# 3. RLVR / GRPO
conda create -n TexRL python=3.11 -y
conda activate TexRL
pip install -r requirements/TexRL.txt
```

Authenticate with Hugging Face before running the download scripts:

```bash
huggingface-cli login
```

Alternatively, export an access token through `HF_TOKEN`.

---

## 1. Evaluation

```bash
conda activate OCR_evaluation
cd evaluation

# Download TexOCR-Bench and extract the figure/page archives.
bash download_eval.sh

# The download script provides the released query files.
# Re-run this only after regenerating or replacing page images.
python get_query.py

# Hosted API models.
bash scripts/api.sh

# Open-source models served through vLLM.
bash scripts/vllm_small.sh
bash scripts/vllm_large.sh

# Score predictions on the two source groups.
python evaluate_arxiv.py
python evaluate_others.py

# Merge the per-split metrics.
python combine_results.py
```

The scorer performs an actual LaTeX compilation test. Install a TeX distribution such as TeX Live and ensure that `pdflatex` is available on `$PATH`. Depending on the document and local configuration, the evaluation pipeline may fall back to `xelatex` or `lualatex`.

Before launching inference, review the model/API configuration in:

- `evaluation/utils/constant.py`
- `evaluation/model_inference/`
- `evaluation/scripts/`

---

## 2. Supervised Fine-Tuning

TexOCR-Train is released as separate JSON and image repositories to simplify large-file distribution.

```bash
conda activate OCR_Train
cd training/sft

# Download the SFT JSON and page-image archives, then extract them.
bash download_sft.sh

# Convert the released data to the LLaMA-Factory multimodal JSONL format.
# Edit input_path, output_path, and image_root in the script first.
python prepare_trainset.py
```

Register the converted dataset in LLaMA-Factory and launch multimodal SFT from the LLaMA-Factory repository. See the [LLaMA-Factory documentation](https://github.com/hiyouga/LLaMA-Factory) for dataset registration, templates, and distributed-training options.

---

## 3. RLVR with GRPO

The second training stage applies group-based policy optimization with deterministic LaTeX unit-test rewards.

```bash
conda activate TexRL
cd training/grpo

# Download the GRPO/RLVR JSON and page-image archives.
bash download_rl.sh

# Example: launch training on two GPUs.
torchrun --nproc_per_node=2 grpo.py
```

The reward suite checks the same broad properties as TexOCR-Bench, including text preservation, formula and table accuracy, section structure, citation and reference validity, output sanity, document similarity, and compilability.

`reward_fn5` invokes a LaTeX compiler for generated outputs. Ensure that:

1. A TeX distribution is installed and visible on `$PATH`.
2. `OCR_grpo/latex_test/`, or the `out_dir` passed to `get_tex_file`, exists and is writable.
3. The training process has sufficient temporary disk space for generated `.tex`, auxiliary, and log files.

---

## Citation

If you use TexOCR, TexOCR-Bench, TexOCR-Train, the released checkpoints, or the evaluation code, please cite the following paper:

```bibtex
@inproceedings{wang-etal-2026-texocr,
    title = "{T}ex{OCR}: Advancing Document {OCR} Models for Compilable Page-to-{L}a{T}e{X} Reconstruction",
    author = "Wang, Chengye  and
      Fu, Lin  and
      Kuang, Zexi  and
      Zhao, Yilun",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1658/",
    doi = "10.18653/v1/2026.acl-long.1658",
    pages = "35821--35845",
    ISBN = "979-8-89176-390-6"
}
```

---

## Contributing

Contributions are welcome, including:

- adapters for additional hosted or open-source models;
- improved inference and batching support;
- extensions to the evaluation suite;
- fixes for LaTeX compilation edge cases;
- documentation and reproducibility improvements.

Please open an issue before submitting a substantial change so that its scope can be discussed.

---

## License

The source code in this repository is released under the **MIT License**.

The released TexOCR datasets on Hugging Face are distributed under **CC BY 4.0**. Individual source documents may remain subject to their original licenses or terms of use; users are responsible for complying with the applicable source licenses.

---

## Acknowledgements

TexOCR uses and builds upon open-source tools and ecosystems including [Hugging Face](https://huggingface.co/), [Transformers](https://github.com/huggingface/transformers), [TRL](https://github.com/huggingface/trl), [vLLM](https://github.com/vllm-project/vllm), and [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory).

TexOCR

TexOCR Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

TexOCR

TexOCR Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README