KO
|
EN
gitlite — search
Search
#python
#java
#python3
#arduino
#golang
#machine-learning
#rust
#html
#flask
#javascript
#seismology
#nodejs
TexOCR
★ 15
Open GitHub ↗
ACL 2026 Main
Download README (.md)
Explore Similar Repositories
verimg
:
Tool for creating customized FreeBSD VM image(s) and also for unattended installation of PKGBASE based FreeBSD system.
buku-pa-pens-latex
:
Template LaTeX Proyek Akhir PENS
vibemove
:
A vibe coding companion for macOS. Camera-driven gestures and body motions — thumbs up to dictate, pinch to send, squat to turn on the mic. No hardware.
vlm-posttraining-ecommerce
:
基于Qwen3-VL-8B模型的数据构建/微调/强化学习/Agentic RAG项目
mujoco-workbench
:
a CLI and set of agent skills to prototype custom scenes with coding agents like Codex and Claude Code.
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
TexOCR
?
Download (.md)
# TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstruction <p align="center"> <a href="https://github.com/QDRhhhh/TexOCR">🌐 Github</a> • <a href="https://arxiv.org/abs/2604.22880">📖 Paper</a> • <a href="https://huggingface.co/collections/chengyewang/texocr">🤗 Data</a> </p> --- **TexOCR** studies page-level reconstruction of document images into **compilable, structurally faithful LaTeX**. Unlike conventional OCR systems that primarily produce plain text or Markdown, TexOCR evaluates whether a model can recover executable LaTeX while preserving equations, tables, section hierarchies, citations, labels, cross-references, and float structure. We introduce **TexOCR-Bench**, containing **2,135 expert-annotated documents** from diverse technical and public-domain sources, together with a nine-metric evaluation suite covering transcription fidelity, structural faithfulness, and end-to-end usability. We also release **TexOCR-Train**, constructed from approximately **57K papers** and containing **404K aligned page-image–LaTeX/BibTeX training pairs**. Using this corpus, we train a 2B-parameter model based on Qwen3-VL through supervised fine-tuning followed by reinforcement learning with verifiable rewards. The rewards are derived from deterministic LaTeX unit tests that directly target transcription quality, structural consistency, compilability, and referential integrity. > **Can document OCR systems reconstruct scientific pages as usable LaTeX—not merely transcribe their visible text?** --- ## News - **[2026-07]** The TexOCR paper is published in the proceedings of **ACL 2026 Main Conference**. - **[2026-04]** TexOCR is accepted to **ACL 2026 Main Conference**. - **[2026-04]** TexOCR-Bench, TexOCR-Train, the RLVR corpus, and trained checkpoints are released on Hugging Face. - **[2026-04]** Evaluation, SFT, and GRPO/RLVR training code are released in this repository. --- ## Overview TexOCR covers three end-to-end workflows: 1. **Evaluation** — evaluate open- or closed-source vision-language models on TexOCR-Bench using nine deterministic metrics and a zero-touch LaTeX compilation test. 2. **Supervised Fine-Tuning** — train page-to-LaTeX models on the 404K-pair TexOCR-Train corpus using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). 3. **RLVR with GRPO** — further optimize the SFT model using [TRL](https://github.com/huggingface/trl) and verifiable rewards derived from LaTeX unit tests. ### Highlights - **2,135 expert-annotated benchmark documents** covering scientific papers, mathematical documents, and scanned public-domain materials. - **57K source papers and 404K aligned page-image–LaTeX/BibTeX pairs** for large-scale training. - **Nine evaluation metrics** spanning transcription fidelity, structural faithfulness, and end-to-end usability. - **Zero-touch compilation testing** that measures whether reconstructed projects compile without manual correction. - **SFT + RLVR training** for a 2B-parameter Qwen3-VL-based model. - **Evaluation of 21 frontier MLLMs and OCR systems** under a unified page-level inference protocol. - **State-of-the-art performance among evaluated open-source models**, with RLVR consistently improving structural and compilation-related metrics over SFT alone. --- ## TexOCR-Bench TexOCR-Bench evaluates generated LaTeX along three complementary dimensions. | Dimension | Metric | Abbreviation | Description | | ----------------------- | ------------------------- | ------------ | ------------------------------------------------------------ | | Transcription Fidelity | Complex Text Preservation | CTP | Checks whether representative body-text sentences are faithfully recovered. | | Transcription Fidelity | Formula Accuracy | FA | Measures normalized token-level correctness of mathematical expressions. | | Transcription Fidelity | Table Accuracy | TA | Measures recovery of numerical content and structure in tables. | | Structural Faithfulness | Section Accuracy | SA | Checks section-title recovery and hierarchy consistency. | | Structural Faithfulness | Citation Coverage | CC | Checks whether in-text citations and citation keys are correctly reconstructed. | | Structural Faithfulness | Reference Validity | RV | Verifies that figure and table references resolve to valid labels. | | End-to-End Usability | Document-Level Similarity | DS | Computes normalized character-level similarity to the reference LaTeX. | | End-to-End Usability | Baseline Validity Check | Baseline | Detects empty, truncated, malformed, or degenerate generations. | | End-to-End Usability | Compilation Success Rate | CSR | Compiles the reconstructed project without manual intervention. | The evaluation pipeline processes each document page independently, concatenates the generated LaTeX in document order, and then computes document-level metrics. Compilation is performed with `pdflatex`, with optional fallback to `xelatex` or `lualatex` when supported by the evaluation script. --- ## Released Resources All models and datasets are grouped in the [TexOCR Hugging Face collection](https://huggingface.co/collections/chengyewang/texocr). | Stage | Hugging Face repository | Contents | | ------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | Evaluation | [`chengyewang/TexOCR-eval`](https://huggingface.co/datasets/chengyewang/TexOCR-eval) | TexOCR-Bench annotations, rendered pages, figure archives, queries, and evaluation metadata. | | SFT data | [`chengyewang/TexOCR-SFT-json`](https://huggingface.co/datasets/chengyewang/TexOCR-SFT-json) | Page-level LaTeX/BibTeX supervision in JSON format. | | SFT images | [`chengyewang/TexOCR-SFT-figures`](https://huggingface.co/datasets/chengyewang/TexOCR-SFT-figures) | Page-image archives for supervised fine-tuning. | | RLVR data | [`chengyewang/TexOCR-RL-json`](https://huggingface.co/datasets/chengyewang/TexOCR-RL-json) | Harder training examples used for GRPO/RLVR. | | RLVR images | [`chengyewang/TexOCR-RL-figures`](https://huggingface.co/datasets/chengyewang/TexOCR-RL-figures) | Page images used during GRPO/RLVR training. | | SFT model | [`chengyewang/TexOCR-SFT`](https://huggingface.co/chengyewang/TexOCR-SFT) | Qwen3-VL-2B checkpoint after supervised fine-tuning. | | TexOCR model | [`chengyewang/TexOCR-RL`](https://huggingface.co/chengyewang/TexOCR-RL) | Final checkpoint after SFT and RLVR. | --- ## Repository Structure ```text TexOCR/ ├── README.md ├── requirements/ │ ├── OCR_evaluation.txt # Evaluation environment │ ├── OCR_Train.txt # SFT environment (LLaMA-Factory) │ └── TexRL.txt # RLVR environment (TRL / GRPO) ├── evaluation/ │ ├── main.py # Batched VLM inference entry point │ ├── get_query.py # Build query files from rendered page images │ ├── evaluate_arxiv.py # Nine-metric scorer for the arXiv split │ ├── evaluate_others.py # Nine-metric scorer for the other-source split │ ├── combine_results.py # Merge metrics from the two splits │ ├── download_eval.sh # Download TexOCR-Bench and extract archives │ ├── data/ # Populated by download_eval.sh │ ├── model_inference/ │ │ ├── azure_gpt.py # Hosted GPT / Claude backend │ │ ├── openai_compatible.py # OpenAI-compatible API backend │ │ ├── vllm_inference.py # Local vLLM backend │ │ └── vllm_model_list.json # Supported vLLM checkpoints │ ├── scripts/ │ │ ├── api.sh # Hosted-API launcher │ │ ├── batch_vllm_small.sh # Batched small-model launcher │ │ ├── vllm_small.sh # vLLM launcher for models up to 8B │ │ └── vllm_large.sh # vLLM launcher for 32B+ models │ └── utils/ │ ├── api_utils.py │ ├── constant.py # Prompt and generation defaults │ ├── input_processing.py # Image preprocessing and base64 packaging │ └── vllm_input_preparation.py └── training/ ├── sft/ │ ├── prepare_trainset.py # Convert SFT JSON to LLaMA-Factory JSONL │ └── download_sft.sh # Download and extract the SFT corpus └── grpo/ ├── grpo.py # TRL GRPO trainer with nine reward functions ├── grpo_utils.py # Reward helpers and LaTeX compilation checks └── download_rl.sh # Download and extract the RLVR corpus ``` --- ## Quickstart ### 0. Clone the Repository ```bash git clone https://github.com/QDRhhhh/TexOCR.git cd TexOCR ``` TexOCR uses three independent Conda environments so that the evaluation, SFT, and RLVR dependencies do not conflict over versions of `torch`, `transformers`, `vllm`, and related libraries. ```bash # 1. Evaluation conda create -n OCR_evaluation python=3.11 -y conda activate OCR_evaluation pip install -r requirements/OCR_evaluation.txt # 2. Supervised fine-tuning conda create -n OCR_Train python=3.11 -y conda activate OCR_Train pip install -r requirements/OCR_Train.txt # 3. RLVR / GRPO conda create -n TexRL python=3.11 -y conda activate TexRL pip install -r requirements/TexRL.txt ``` Authenticate with Hugging Face before running the download scripts: ```bash huggingface-cli login ``` Alternatively, export an access token through `HF_TOKEN`. --- ## 1. Evaluation ```bash conda activate OCR_evaluation cd evaluation # Download TexOCR-Bench and extract the figure/page archives. bash download_eval.sh # The download script provides the released query files. # Re-run this only after regenerating or replacing page images. python get_query.py # Hosted API models. bash scripts/api.sh # Open-source models served through vLLM. bash scripts/vllm_small.sh bash scripts/vllm_large.sh # Score predictions on the two source groups. python evaluate_arxiv.py python evaluate_others.py # Merge the per-split metrics. python combine_results.py ``` The scorer performs an actual LaTeX compilation test. Install a TeX distribution such as TeX Live and ensure that `pdflatex` is available on `$PATH`. Depending on the document and local configuration, the evaluation pipeline may fall back to `xelatex` or `lualatex`. Before launching inference, review the model/API configuration in: - `evaluation/utils/constant.py` - `evaluation/model_inference/` - `evaluation/scripts/` --- ## 2. Supervised Fine-Tuning TexOCR-Train is released as separate JSON and image repositories to simplify large-file distribution. ```bash conda activate OCR_Train cd training/sft # Download the SFT JSON and page-image archives, then extract them. bash download_sft.sh # Convert the released data to the LLaMA-Factory multimodal JSONL format. # Edit input_path, output_path, and image_root in the script first. python prepare_trainset.py ``` Register the converted dataset in LLaMA-Factory and launch multimodal SFT from the LLaMA-Factory repository. See the [LLaMA-Factory documentation](https://github.com/hiyouga/LLaMA-Factory) for dataset registration, templates, and distributed-training options. --- ## 3. RLVR with GRPO The second training stage applies group-based policy optimization with deterministic LaTeX unit-test rewards. ```bash conda activate TexRL cd training/grpo # Download the GRPO/RLVR JSON and page-image archives. bash download_rl.sh # Example: launch training on two GPUs. torchrun --nproc_per_node=2 grpo.py ``` The reward suite checks the same broad properties as TexOCR-Bench, including text preservation, formula and table accuracy, section structure, citation and reference validity, output sanity, document similarity, and compilability. `reward_fn5` invokes a LaTeX compiler for generated outputs. Ensure that: 1. A TeX distribution is installed and visible on `$PATH`. 2. `OCR_grpo/latex_test/`, or the `out_dir` passed to `get_tex_file`, exists and is writable. 3. The training process has sufficient temporary disk space for generated `.tex`, auxiliary, and log files. --- ## Citation If you use TexOCR, TexOCR-Bench, TexOCR-Train, the released checkpoints, or the evaluation code, please cite the following paper: ```bibtex @inproceedings{wang-etal-2026-texocr, title = "{T}ex{OCR}: Advancing Document {OCR} Models for Compilable Page-to-{L}a{T}e{X} Reconstruction", author = "Wang, Chengye and Fu, Lin and Kuang, Zexi and Zhao, Yilun", editor = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David", booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)", month = jul, year = "2026", address = "San Diego, California, United States", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2026.acl-long.1658/", doi = "10.18653/v1/2026.acl-long.1658", pages = "35821--35845", ISBN = "979-8-89176-390-6" } ``` --- ## Contributing Contributions are welcome, including: - adapters for additional hosted or open-source models; - improved inference and batching support; - extensions to the evaluation suite; - fixes for LaTeX compilation edge cases; - documentation and reproducibility improvements. Please open an issue before submitting a substantial change so that its scope can be discussed. --- ## License The source code in this repository is released under the **MIT License**. The released TexOCR datasets on Hugging Face are distributed under **CC BY 4.0**. Individual source documents may remain subject to their original licenses or terms of use; users are responsible for complying with the applicable source licenses. --- ## Acknowledgements TexOCR uses and builds upon open-source tools and ecosystems including [Hugging Face](https://huggingface.co/), [Transformers](https://github.com/huggingface/transformers), [TRL](https://github.com/huggingface/trl), [vLLM](https://github.com/vllm-project/vllm), and [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory).
