DataEvolver

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 10 Open GitHub ↗

DataEvolver: Self-Evolving Multi-Agent Data Construction for Text-Rich Image Generation

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">DataEvolver</h1>

Self-Evolving Multi-Agent Data Construction for Text-Rich Image Generation

<a href="https://sgysy.github.io/">Siyu Yan</a>1,2,*&nbsp;&nbsp;
 <a href="#">Yizhen Gao</a>1,*&nbsp;&nbsp;
 <a href="#">Yilin Wang</a>1&nbsp;&nbsp;
 <a href="#">Dongxing Mao</a>1&nbsp;&nbsp;
 <a href="https://fingerrec.github.io/">Alex Jinpeng Wang</a>1,†

1Central South University&nbsp;&nbsp;&nbsp;
 2The Hong Kong University of Science and Technology 
 *Equal contribution&nbsp;&nbsp;&nbsp;
 †Corresponding author

<a href="#the-idea">The Idea</a> ·
 <a href="#closed-loop-architecture">Architecture</a> ·
 <a href="#four-cooperative-agents">Agents</a> ·
 <a href="#results">Results</a> ·
 <a href="#quick-start">Quick Start</a> ·
 <a href="#citation">Citation</a>

---

## The Idea

> Rejected samples are not waste. They are the feedback signal that should shape the next construction round.

**DataEvolver** is a self-evolving multi-agent data construction framework for text-rich image generation. Instead of following a static *crawl → filter → freeze* pipeline, DataEvolver treats rejected samples as feedback signals and uses them to improve subsequent construction rounds.

The key idea is simple: **rejected samples should not be discarded silently; they should guide the next round of retrieval, filtering, and generation.** Construction-time failures, including OCR errors, semantic mismatches, duplicates, and topic coverage gaps, are converted into actionable feedback for policy revision.

<table>
 <tr>
 <td width="25%">01 · Policy evolution Turn rejection patterns into reusable guidance.</td>
 <td width="25%">02 · Four agents Retriever, Verifier, Critic, and Generator close the loop.</td>
 <td width="25%">03 · Training-free Evolve the construction policy in natural-language space.</td>
 <td width="25%">04 · Strong transfer Improve multiple text-rich image generators.</td>
 </tr>
</table>

## Closed-Loop Architecture

DataEvolver replaces a static data pipeline with an adaptive loop: retrieve or generate candidates, verify them, diagnose rejection causes, update the policy, and begin the next round with better queries and prompts.

DataEvolver pipeline. Feedback drives query updates, experience updates, and prompt refinement across construction rounds.

## Four Cooperative Agents

<table>
 <thead>
 <tr>
 <th width="24%">Agent</th>
 <th width="48%">Role</th>
 <th width="28%">Backend</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td width="24%">🔍&nbsp;Retriever</td>
 <td width="48%">Discovers candidate images with optimized search queries and updates query strategies using feedback from previous rounds.</td>
 <td width="28%"><code>llm.model</code> via Ollama, default <code>mistral:latest</code></td>
 </tr>
 <tr>
 <td width="24%">✅&nbsp;Verifier</td>
 <td width="48%">Filters candidates through OCR, image quality, watermark, deduplication, CLIP semantic relevance, and text-consistency checks.</td>
 <td width="28%"><code>PaddleOCR</code>/<code>Tesseract</code> + <code>CLIP</code> + <code>Sentence-Transformers</code> + pHash</td>
 </tr>
 <tr>
 <td width="24%">📊&nbsp;Critic</td>
 <td width="48%">Summarizes rejection patterns into semantic feedback, maintains an experience library, and revises construction policies.</td>
 <td width="28%"><code>llm.cir_model</code> via Ollama, default <code>qwen3.5:4b</code></td>
 </tr>
 <tr>
 <td width="24%">🎨&nbsp;Generator</td>
 <td width="48%">Uses CountAnalyse and PromptPlanner to identify coverage gaps, write prompts, synthesize text-aware images, and re-filter generated samples.</td>
 <td width="28%"><code>mistral:latest</code> + <code>qwen3.5:4b</code> agents, <code>Qwen/Qwen-Image</code> T2I, optional <code>qwen3-vl:latest</code> annotation</td>
 </tr>
 </tbody>
</table>

The entire loop is **traceable**: accepted samples retain query, caption, OCR, quality, semantic, and filtering metadata, while rejected samples preserve the evidence needed for policy revision.

## Results

On **PixArt-α** at the **0.75M** data scale, DataEvolver improves OCR-F1 over the strongest matched-budget baseline.

Overview and scaling trend. DataEvolver converts discarded samples into iterative policy updates and improves OCR-F1 under matched data budgets.

| Evaluation Set | Relative OCR-F1 Improvement |
| --- | ---: |
| **TextScenesHQ** | **+85.3%** |
| **LongTextBench** | **+35.3%** |

Ablation studies show that both the **Critic** and the **Generator** contribute to the final performance, indicating that feedback-based policy revision and targeted completion are both necessary for effective text-rich data construction.

## Quick Start

### 1. Install

```bash
git clone https://github.com/CSU-JPG/DataEvolver.git
cd DataEvolver

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

### 2. Run the pipeline

For an existing environment, run the main entrypoint directly:

```bash
python main.py --config config.yaml --shard-id host-a --keep-rejects
```

`--shard-id` is required and should be unique for each concurrent machine or shard. Use `--keep-rejects` if you want rejected samples retained for diagnosis.

<details>
<summary>Full CUDA and Ollama bootstrap</summary>

For a full CUDA/Ollama bootstrap, use:

```bash
bash run.sh
```

The script creates a Python 3.9 Conda environment, installs CUDA-oriented dependencies, starts eight local Ollama servers on ports `11437`-`11444`, pulls the default Ollama models, and launches `main.py`.

</details>

## Configuration

All major settings are controlled by `config.yaml`.

<details>
<summary>Paths — data, logs, and output directories</summary>

```yaml
paths:
  images_crawled: "/path/to/crawled/images"
  images_generated: "/path/to/generated/images"
  ann_dir: "/path/to/annotations"
  log_dir: "/path/to/logs"
  accepted_dir: "/path/to/accepted"
  gen_dir: "/path/to/generated/raw"
  gen_accepted_dir: "/path/to/gen/accepted"
```

</details>

<details>
<summary>LLM & Agent Models — Ollama-compatible local models</summary>

```yaml
llm:
  model: "mistral:latest"          # Retriever / QueryGenerator / QueryPlannerAgent / CountAnalyse
  cir_model: "qwen3.5:4b"          # Critic agents / PromptPlanner / prompt rewriting
```

All LLM agents are served through local Ollama instances. `src/agents/local_agents.py` round-robins requests over ports `11437`-`11444`.

</details>

<details>
<summary>Pipeline Control — crawl/generate phases and feedback loops</summary>

```yaml
dataset_num: 1000000

generation:
  enabled: true
  max_regeneration_rounds: 3
  prompt_critic_enabled: true

critic:
  enabled: true
  strategy:
    enabled: true
    queries_enabled: true
    warmup_rounds: 3
  experience:
    use_agent: true

dynamic_query:
  enabled: true
  low_watermark: 30

feedback_query:
  enabled: true
  min_samples: 20
```

The main run first performs asynchronous retrieval and filtering, then enters the optional generation phase when `generation.enabled` is true.

</details>

<details>
<summary>Retrieval, OCR, and Filtering — Bing, OCR, quality, semantic, and dedup checks</summary>

```yaml
bing:
  per_subtopic_images: 1500
  mkt: "auto"
  safesearch: "moderate"

ocr:
  engine: "paddle"
  lang: "ch"
  use_multi_gpu: true
  use_multiprocess: true

quality:
  min_ocr_coverage: 0.25
  min_legibility: 0.5
  min_side: 384
  max_side: 4096

semantic:
  enabled: true
  model_name: "openai/clip-vit-base-patch32"

text_consistency:
  enabled: true
  model_name: "sentence-transformers/all-MiniLM-L6-v2"

hash_deduplication:
  enabled: true
  max_distance: 3
  scope: "global"
```

The Verifier combines OCR-derived quality signals, watermark detection, pHash deduplication, CLIP similarity, and sentence-transformer text consistency before writing accepted samples.

</details>

<details>
<summary>Generation and Annotation Models — Qwen-Image and Qwen-VL</summary>

```yaml
qwen:
  true_cfg_scale: 4.5
  num_inference_steps: 25
  seed: 42

annotation:
  qwen_vl:
    enabled: true
    model: "qwen3-vl:latest"
    verbose: false
```

Image synthesis uses the Diffusers pipeline `Qwen/Qwen-Image` by default. Override it with the `QWEN_IMAGE_MODEL` environment variable if needed. The Qwen-VL setting is used for optional vision-language caption/annotation, not for text-to-image generation.

</details>

<details>
<summary>Topics & Seed Queries — target domains for data construction</summary>

```yaml
topics:
  - name: Store_Signs_and_Shopfronts
    seed_queries:
      - "store signboard shop front"
      - "restaurant sign food street sign"
      - "Chinese shop plaque traditional signage"

- name: Book_Covers
    seed_queries:
      - "book cover title author text"
      - "novel cover book jacket design"
```

Seed queries define the initial retrieval space. During construction, the Retriever and Critic refine these queries based on acceptance and rejection patterns.

</details>

## Quick Customization

1. Edit `paths` in `config.yaml`.
2. Define target `topics` and seed queries.
3. Choose Ollama-compatible LLM backends.
4. Start Ollama instances and make sure `mistral:latest`, `qwen3.5:4b`, and `qwen3-vl:latest` are available, or replace them in `config.yaml`.
5. Adjust OCR, semantic, text-consistency, deduplication, and quality thresholds.
6. Enable or disable generation with `generation.enabled`.
7. Run `python main.py --config config.yaml --shard-id <unique-shard-id>`.

## Citation

If you find DataEvolver useful, please cite:

```bibtex
@article{yan2026dataevolver,
  title   = {DataEvolver: Self-Evolving Multi-Agent Data Construction for Text-Rich Image Generation},
  author  = {Yan, Siyu and Gao, Yizhen and Wang, Yilin and Mao, Dongxing and Wang, Alex Jinpeng},
  journal = {arXiv preprint},
  year    = {2026}
}
```

## Acknowledgments

This project builds on several open-source tools, including PaddleOCR, Tesseract OCR, Ollama, Hugging Face Diffusers, Qwen-Image, Qwen-VL, CLIP, and Sentence-Transformers.

---

Reject. Reflect. Revise. Evolve.

DataEvolver

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README