OverSearchGuard

KO | EN

gitlite — search

#python #php #nodejs #machine-learning #golang #android #deep-learning #dashboard #game #react #data-analysis #csharp

OverSearchGuard

★ 71 Open GitHub ↗

OverSearchGuard is a drop-in step between retrieval and generation: it selects a small, high-confidence, conflict-aware subset of evidence lines before calling the LLM. No fine-tuning required.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# OverSearchGuard

Conflict-aware evidence thinning for search-augmented LLMs (RAG / agents).

Language: English | Chinese: `README.zh-CN.md`

## Problem

In agent-style RAG workflows, the system often runs multiple searches/retrievals and keeps appending evidence into a growing prompt. This frequently leads to:

- Over-searching: more context increases noise and cost, not accuracy.
- Conflicting sources: "stuff everything" makes the model vulnerable to wrong-but-repeated claims.
- Context window pressure: relevant facts get buried or truncated as the prompt grows.

OverSearchGuard is a drop-in step between retrieval and generation: it selects a small, high-confidence, conflict-aware subset of evidence lines before calling the LLM. No fine-tuning required.

## What makes it different

OverSearchGuard is an evidence-governance layer: it decides which evidence lines are worth paying token budget for under conflict. It is designed around three product-facing requirements that are not always addressed together by relevance-only reranking, summarization-style compression, or training-heavy conflict detection.

### A) Robust to "wrong-but-repeated" evidence: anti-spam / anti-dup

If a wrong claim is repeated by many low-quality sources, relevance-only selection can still surface it repeatedly. OverSearchGuard makes this failure mode explicit (`duplicate_attack`) and adds a hard control knob: `cap-per-source`, which limits how much repeated low-quality evidence can dominate the decision.

### B) Jointly optimizes cost and correctness: not "shorter at any price"

OverSearchGuard reports a practical efficiency metric, TPC = total_tokens / #correct, and provides BEA (Budgeted Evidence Accumulation): add evidence in rounds (1, 2, 4, ...) until the answer stabilizes. This mirrors a real product constraint: pay for extra context only when it helps.

### C) Explainable, tunable, and pluggable: a baseline you can ship

Training-based conflict detection requires data, evaluation loops, and ongoing maintenance. Summarization-style compression adds extra model calls and can introduce a new failure mode (summary hallucination). OverSearchGuard is a lightweight, interpretable module with a single integration point (retrieval -> OverSearchGuard -> LLM) and a small set of knobs you can reason about.

### Comparison (typical tradeoffs)

The table below compares common solution directions. Individual implementations vary.

| Dimension | OverSearchGuard | Relevance rerank (BM25/dense) | LLM rerank | Summarization / compression | Training-based conflict models |
|---|---|---|---|---|---|
| Robust to wrong-but-repeated duplicates | Yes (`cap-per-source`) | Usually no | Varies | Varies | Varies (depends on training) |
| Explicit source reliability handling | Yes (source weights) | Usually no | Varies | Varies | Varies (depends on training) |
| Recency handling | Yes (half-life) | Usually no | Varies | Varies | Varies |
| Cost-aware decision policy | Yes (TPC + optional BEA) | No (fixed top-k) | Usually no (extra calls) | Partial (extra calls) | Varies |
| Requires training data | No | No | No | No | Yes |
| Extra model calls besides the final answer | No (thin); optional (BEA) | No | Often yes | Often yes | No at inference (but training cost) |
| Explainable knobs | High | Medium | Low | Medium | Low |
| Integration complexity | Low (retrieval -> thin -> LLM) | Low | Medium | Medium | High |

OverSearchGuard is orthogonal to many pipelines: you can place it before a reranker, before summarization, or before the final generator.

## Results (reproducible)

Benchmark: `paper` suite (synthetic stress tests for conflicting/noisy evidence), `n=300`, model `google/flan-t5-small`, device `cuda`. Numbers come from `runs/latest.json` generated by `.\run.ps1`.

| method | accuracy (higher) | avg_total_tokens (lower) | TPC (lower) |
|---|---:|---:|---:|
| full (stuff all evidence) | 0.143 | 491.5 | 3428.8 |
| bm25_k | 0.177 | 387.4 | 2192.8 |
| random_k | 0.160 | 387.9 | 2424.1 |
| thin (OverSearchGuard / CACT) | 0.943 | 107.5 | 113.9 |
| bea | 0.943 | 117.3 | 124.4 |
| bea_fallback (best) | 0.970 | 117.3 | 121.0 |

Where TPC = total_tokens / #correct (lower is better).

![Overall metrics](https://raw.githubusercontent.com/LongWeihan/OverSearchGuard/main/assets/readme/overall.svg)

![Efficiency frontier](https://raw.githubusercontent.com/LongWeihan/OverSearchGuard/main/assets/readme/tradeoff.svg)

![Scenario breakdown](https://raw.githubusercontent.com/LongWeihan/OverSearchGuard/main/assets/readme/scenarios.svg)

## Quickstart (Windows)

```powershell
cd OverSearchGuard
.\run.ps1 --suite paper --n 300 --device cuda
Start-Process .\runs\latest.html
```

Notes:
- `run.ps1` creates a local `.venv`, installs deps, and prefers CUDA PyTorch wheels when CUDA/NVIDIA is detected.
- CPU: `.\run.ps1 --device cpu`
- Change model: `.\run.ps1 --model google/flan-t5-base`

## Quickstart (Linux/macOS)

```bash
cd OverSearchGuard
bash ./run.sh --suite paper --n 300 --device cuda
# open ./runs/latest.html in your browser
```

## How it works (CACT + optional BEA)

OverSearchGuard consumes evidence as a list of lines. Lines can include optional tags:

```
[source=official][doc=example.com/page][date=2025-09] The capital of Asteria is Aurum.
```

### 1) CACT: Conflict-Aware Candidate Thinning

1. Extract candidate facts from evidence lines (this repo's benchmark uses lightweight patterns).
2. Score candidate values using:
   - source reliability (official > report > news > blog > forum > social)
   - recency decay (configurable half-life)
   - contradiction penalty via negation
   - anti-dup defense via `cap-per-source` (limits how much repeated low-quality sources can dominate)
3. Output the top candidate and up to `max_support_lines` strongest supporting lines.

### 2) BEA: Budgeted Evidence Accumulation (optional)

Instead of sending all selected support at once, BEA queries the model in rounds with 1, 2, 4, ... lines until the output stabilizes (or a max round limit is hit). `bea_fallback` returns the CACT candidate if the model still disagrees at the end.

## Use as a library (between retrieval and your LLM)

```python
from oversearchguard.api import build_thin_prompt

question = "What is the capital of Asteria?"
lines = [
    "[source=blog][doc=x][date=2021-01] The capital of Asteria is Fjord.",
    "[source=official][doc=gov.example][date=2025-09] The capital of Asteria is Aurum.",
]

payload = build_thin_prompt(question, lines)
prompt = payload["prompt"]          # ready to send to any LLM
candidate = payload["candidate"]    # CACT's best value (useful for guardrails)
```

## Benchmark design

The included `paper` suite targets common agent/RAG failure modes:

- `noisy_long`: long contexts with many distractors
- `duplicate_attack`: many low-quality sources repeating the same wrong claim
- `negations`: contradictions expressed via negation
- `dated_recency_shift`: older official info becomes outdated; newer report is correct
- `paraphrases`: varied phrasing
- `mixed_hard`: all of the above combined

Run:

```powershell
.\run.ps1 --suite paper --n 300 --max-support-lines 12 --cap-per-source 1
```

## License

MIT (see `LICENSE`).

OverSearchGuard

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README