benchmarking-ocr-gepa

KO | EN

gitlite — search

#ai #typescript #llm #golang #android #docker #macos #java #awesome #awesome-list #vue #security #ios #openai

benchmarking-ocr-gepa

(★ 17)

No description available.

benchmarking-ocr-gepa 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# OCR Benchmarking with GEPA

Automatic prompt optimization for document OCR using [DSPy](https://github.com/stanfordnlp/dspy) and the [GEPA optimizer](https://github.com/gepa-ai/gepa).

This project benchmarks and improves a two-stage OCR pipeline on the [Omni OCR Benchmark](https://github.com/getomni-ai/benchmark):
1. **Image → Markdown**: Vision model extracts text from document images
2. **Markdown → JSON**: Language model extracts structured data based on a schema

GEPA (Generalized Error-Prompt Alignment) automatically analyzes failures and optimizes the prompts to improve accuracy.

![GEPA Optimization Results](https://raw.githubusercontent.com/Studio-Intrinsic/benchmarking-ocr-gepa/main/results.png)

*GEPA optimization improved JSON extraction accuracy from baseline to optimized performance*

## Prerequisites

- Python 3.11+
- Node.js (for JSON evaluation scoring)
- OpenRouter API key
- Weights & Biases API key (optional, for experiment tracking)

## Setup

### 1. Clone and create virtual environment

```bash
git clone <repository-url>
cd benchmarking-ocr-gepa
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

### 2. Install dependencies

```bash
pip install -r requirements.txt
npm install json-diff  # For scoring
```

### 3. Set up environment variables

Create a `.env` file:

```bash
OPENROUTER_API_KEY=your_key_here
WANDB_API_KEY=your_key_here  # Optional
```

Or export them:

```bash
export OPENROUTER_API_KEY=your_key_here
export WANDB_API_KEY=your_key_here
```

## Usage

### Step 1: Download the benchmark dataset

Download 1000 OCR examples from HuggingFace:

```bash
cd ocr-benchmarking
python download_data.py --max-rows 1000 --image-workers 20
```

This creates:
- `data/` - 1000 JSON files with schemas and ground truth
- `image-cache/` - Document images (converted to PNG)

**Options:**
- `--no-images` - Skip downloading images (JSON only)
- `--max-rows N` - Limit number of examples
- `--image-workers N` - Concurrent download workers (default: 20)

### Step 2: Run GEPA optimization

Optimize the OCR pipeline prompts:

```bash
python gepa_optimizer.py
```

This will:
1. Load the dataset (30 validation, 970 training examples)
2. Run the baseline pipeline
3. Use GEPA to analyze failures and improve prompts
4. Save the optimized program to `optimized_programv1.json`
5. Track experiments in Weights & Biases (if configured)

**Results saved to:**
- `v1/` - Optimization logs and candidate programs
- `optimized_programv1.json` - Final optimized prompts

## How It Works

### Pipeline Architecture

```
Document Image → [Vision Model] → Markdown → [LLM + Schema] → Structured JSON
                  Gemini 2.0 Flash         GPT-4.1
```

### GEPA Optimization Loop

1. **Evaluate**: Run pipeline on training examples
2. **Analyze**: GPT-5 reflection model identifies failure patterns
3. **Improve**: Generate better prompts based on analysis
4. **Validate**: Test on validation set
5. **Iterate**: Repeat until convergence

### Scoring

Uses the official TypeScript evaluator from the Omni benchmark:
- Structural diff between predicted and ground truth JSON
- Score = `1 - (additions + deletions + modifications) / total_fields`

## Configuration

Edit model settings in `gepa_optimizer.py`:

```python
REFLECTION_MODEL = "openai/gpt-5"           # GEPA reflection
OCR_MODEL = "google/gemini-2.0-flash-001"  # Image → Markdown
JSON_MODEL = "openai/gpt-4.1"              # Markdown → JSON
```

## Project Structure

```
ocr-benchmarking/
├── gepa_optimizer.py      # Main optimization script
├── download_data.py       # Dataset download utilities
├── open_router_client.py  # OpenRouter API client
├── parsing_utils.py       # Chain-of-thought parsing
├── scorer.py              # Evaluation metrics
├── evaluate_json.js       # TypeScript scoring logic
├── data/                  # Benchmark JSON files (gitignored)
└── image-cache/           # Document images (gitignored)
```

## License

Apache 2.0 - See [LICENSE](LICENSE) for details

## References

- [Omni OCR Benchmark](https://github.com/getomni-ai/benchmark)
- [DSPy Framework](https://github.com/stanfordnlp/dspy)
- [GEPA Optimizer](https://github.com/gepa-ai/gepa)

benchmarking-ocr-gepa

benchmarking-ocr-gepa 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

benchmarking-ocr-gepa

benchmarking-ocr-gepa 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README