KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #dsh #dsh-plugin #ai-agent #codex #ai-agents #deepseek-harness #claude-code #cli #react #mcp

ikp

★ 107 Open GitHub ↗

IKP: Incompressive Knowledge Probes

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Incompressible Knowledge Probes (IKP)

Evaluation toolkit and reproduction bundle for the paper:

> **Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter
> Counts via Factual Capacity.** Bojie Li, Pine AI.

IKP is a 1,400-question factual benchmark — 200 items × 7 obscurity tiers
(T1: universal knowledge … T7: extreme long-tail). Accuracy on IKP
scales log-linearly with parameter count across 93 open-weight models
from 135M to 1.6T (R² = 0.910, no-penalty λ=0 scoring), so a single black-box API call budget is
enough to estimate the effective knowledge capacity of any deployed
model — including closed-source frontier models whose sizes are
undisclosed.

- **Paper PDF:** `paper/main.pdf`
- **Companion website (interactive):** https://01.me/research/ikp
- **Source:** https://github.com/19PINE-AI/ikp

## Quickstart — estimate a model

```bash
# 1. Install deps (Python ≥ 3.10)
pip install -r requirements.txt

# 2. Point at any OpenAI-compatible endpoint and run
export OPENROUTER_API_KEY=sk-or-...
python scripts/ikp_estimate.py --model openai/gpt-4.1
```

Output:

```
  ╔══════════════════════════════════════════════════════════╗
  ║  IKP Estimation Results                                 ║
  ║  Model:     openai/gpt-4.1                              ║
  ║  Probes:    1400                                         ║
  ║  Accuracy:  63.9% (λ=0, no penalty; raw 65.2%)         ║
  ║  N_eff:    ~  315B effective parameters                  ║
  ╚══════════════════════════════════════════════════════════╝
  T1   99%  …  T7    4%
  Effective tier: T6
  Effective size: ~315B IKP-effective parameters (calibrated on 93 open models, R²=0.910)
  Refusal interval: 315B – 355B (1.2% refusals; Reliable)
  90% band (new-vendor LOFO): 90B – 1.1T
  N_eff is the open-cohort-equivalent size, not the true weight count — see PARAM_POLICY.md.
```

N_eff ("IKP-effective parameters") is the size of an average open model
with the same incompressible-knowledge capacity — the quantity the
calibration measures exactly. Converting it to a true weight count
requires a knowledge-density prior; [`PARAM_POLICY.md`](PARAM_POLICY.md)
defines the estimand, the parameter-counting conventions (MoE, embeddings,
quantization), the λ=0 scoring rationale, the refusal policy, and a
bias-detection checklist.

Faster stratified sample (200 probes, ~1 min):

```bash
python scripts/ikp_estimate.py --model openai/gpt-4.1 --sample 200
```

Non-OpenRouter endpoint (vLLM, OpenAI, Together, local):

```bash
python scripts/ikp_estimate.py \
    --api-base http://localhost:8000/v1 \
    --api-key  <your-key> \
    --model    my-local-model
# Judge always runs on OpenRouter (google/gemini-3-flash-preview);
# OPENROUTER_API_KEY must still be set for the judge.
```

Full CLI reference, including how to plug in a different judge or export
per-probe verdicts: see [`TOOLKIT.md`](TOOLKIT.md).

## Interactive CLI — explore the benchmark

A second, lighter CLI (`python -m cli`) lets readers poke at the
benchmark without running the full estimator. It has two modes.

**Research mode** — query the six tier landmarks plus three frontier
models (GPT-5.5, DeepSeek V4 Pro, Claude Opus 4.7) with a researcher
name or any free-form factual question:

```bash
export OPENROUTER_API_KEY=sk-or-...

# Look up a researcher (substring match against the probe set)
python -m cli research --researcher "Stjepan Picek"

# Ask any factual question
python -m cli research --question "Who founded the field of cache-oblivious algorithms?"
```

**Evaluation mode** — re-run any probe against the preset models plus
any models you specify, scored with the paper's exact judge prompt
(`google/gemini-3-flash-preview`, CORRECT / WRONG / REFUSAL):

```bash
# Score a single tier-7 probe against the preset 9 models
python -m cli eval IKP_T7_1234

# Add your own models; --model is repeatable
python -m cli eval IKP_T5_0123 \
    --model openai/gpt-4o \
    --model id=qwen/qwen3-32b,name=q3-32b,thinking=true
```

T1 uses a local Ollama landmark (`qwen2.5:0.5b`); install Ollama or
ignore that row. The other eight models all run via OpenRouter.

## Reproducing the paper

Every figure and table in the paper is generated from the scored results in
`data/results/` — the paper itself names no scripts, so this is the
authoritative map. [`REPRODUCTION.md`](REPRODUCTION.md) has the full version
with inputs and expected outputs.

**Figures** (`paper/figures/*.pdf`, regenerated by `make figs`):

| Figure | Generator |
|---|---|
| Fig 1 calibration · 2 tier heatmap · 3 thinking · 4 MoE · 5 researcher · 6 fingerprint · 8 densing | `paper/figures/generate_figures.py` |
| Fig 7 LOO-CV validation | `scripts/loo_cv_analysis.py` |
| Appendix A1–A4 | `paper/figures/generate_appendix_figures.py` |
| λ-sensitivity (2-panel) | `paper/figures/generate_lambda_figure.py` |

**Tables** (`paper/tables/*.tex`, `\input` by the paper):

| Table | Generator |
|---|---|
| Frontier parameter estimates | `scripts/frontier_table.py` |
| λ sensitivity sweep | `scripts/lambda_sensitivity.py` |
| λ × flooring ablation | `scripts/lambda_floor_ablation.py` |
| Dense-vs-MoE fits + frontier sensitivity | `scripts/moe_dense_analysis.py` |
| Full per-model results (accuracy + hallucination) | `scripts/full_results_tables.py` |

**Scoring and probe cleaning.** Scoring is **no-penalty (λ = 0)**: accuracy is
simply correct/total, so there is no penalty or per-tier flooring choice. The
released probe set is audited for name collisions and label ambiguity by
`scripts/clean_flag_researchers.py` (OpenAlex) and
`scripts/clean_flag_wikidata.py`, combined into `data/probes/clean_mask.json`
by `scripts/clean_build_mask.py`; all paper results are scored on the cleaned
1,311-probe subset. That subset is materialized as a single drop-in file,
`data/probes/final_probe_set_clean.json` (regenerate with
`scripts/build_clean_probe_set.py`), for anyone who wants the released
benchmark directly without applying the mask.

Short path:

```bash
make figs                                # every figure from already-scored results
for s in frontier_table lambda_sensitivity lambda_floor_ablation \
         moe_dense_analysis full_results_tables; do python scripts/$s.py; done
cd paper && latexmk -pdf main.tex        # rebuild the PDF (TeX Live)
```

To score additional models and extend the dataset:

```bash
python scripts/run_all_models.py --skip-existing
python scripts/run_evaluation.py --rebuild-summary  # refreshes evaluation_summary.json
```

## Build the paper / website

The `Makefile` is the single entry point.

```bash
make help              # list every target

# Paper
make figs              # regenerate every figure under paper/figures/
make pdf               # one pdflatex pass (fast, no bibtex)
make full              # full rebuild with bibtex (4 passes)

# Calibration / data refresh after a new model lands in data/results/
make calibration       # rerun loo_cv_analysis.py + analyze_results.py
make website           # rebuild website/public/data/*.json (must precede website-build)
make data              # = calibration + website

# Website
make website-dev       # vite dev server  → http://localhost:5173
make website-build     # static build     → website/dist/
make website-preview   # preview the production build
make website-deploy    # rsync website/dist/ to DEPLOY_HOST:DEPLOY_PATH
                       # override per invocation:
                       #   make website-deploy DEPLOY_HOST=user@host \
                       #                       DEPLOY_PATH=/var/www/research/ikp/

make all               # data → figs → pdf
```

For subpath deploys (e.g. `https://example.com/research/ikp/`), set
`BASE_URL=/research/ikp/ make website-build`. See `website/README.md` for full
website documentation, nginx config, and GitHub Pages instructions.

## Repo layout

```
ikp-paper/
├── README.md               ← this file
├── TOOLKIT.md              ← ikp_estimate.py reference
├── REPRODUCTION.md         ← figure/table ⇄ script map
├── ADVERSARIAL_IKP.md      ← how cheaply an operator can game its estimate
├── IKP_V2.md               ← gaming-resistant estimator (refusal interval + held-out split)
├── requirements.txt
│
├── paper/                  ← LaTeX sources
│   ├── main.tex  main.pdf  appendix.tex  references.bib
│   ├── research-plan.md    ← original planning document
│   └── figures/            ← PDF/PNG figures + generators (all main & appendix figs)
│       ├── generate_figures.py            (main-text figs 1–6, 8)
│       └── generate_appendix_figures.py   (appendix figs A1–A4)
│
├── configs/
│   ├── experiment.json     ← tier definitions, API settings, seeds
│   ├── models.json         ← calibration-set models (open, known size)
│   └── all_models.json     ← full roster (188 models evaluated)
│
├── data/                   ← see data/README.md for schemas
│   ├── probes/
│   │   ├── final_probe_set_v8.json  ← THE 1,400 probes (the benchmark)
│   │   ├── final_probe_set_clean.json ← the cleaned 1,311-probe subset (paper results)
│   │   ├── clean_mask.json          ← flags the 89 dropped probes; clean_ids
│   │   ├── researcher_probes.json   ← researcher sub-probe source
│   │   └── archive/                 ← earlier probe versions (v1..v7, batches, candidates)
│   ├── results/<model>.json         ← per-model raw evaluations (188 files)
│   ├── results/evaluation_summary.json  ← aggregated, consumed by every figure
│   ├── calibration/calibration_fit.json ← fitted log-linear calibration
│   ├── researcher_citations.json        ← T4–T7 researcher metadata
│   ├── researcher_recognition_rates.json
│   ├── densing_analysis_data.csv        ← Densing-Law table (for Fig 8)
│   ├── notes/                           ← exploratory analysis markdown
│   └── archive/                         ← superseded runs (results_v7, …)
│
├── results/
│   ├── figures/archive/    ← early-draft plots (superseded by paper/figures/)
│   └── tables/             ← .tex tables \input'ed by the paper
│
├── scripts/                ← see scripts/README.md for a full index
│   ├── ikp_estimate.py     ← one-model estimator (public entrypoint)
│   ├── run_all_models.py   ← bulk evaluation across the full roster
│   ├── run_evaluation.py   ← single-model evaluator
│   ├── 01_..15_*.py        ← numbered dataset pipeline
│   ├── analyze_results.py, loo_cv_analysis.py, show_progress.py
│   └── legacy/             ← one-off / superseded dev scripts (kept for audit)
│
├── pipeline/               ← probe generation + calibration library
├── src/                    ← evaluation runtime (api_client, probe_runner, scorer, …)
├── cli/                    ← interactive reader CLI (research + eval modes)
└── website/                ← React companion site
```

All active scripts resolve paths via `Path(__file__).parent.parent`, so
they expect to live in `scripts/`. Scripts under `scripts/legacy/`
have been patched to three-`..` (`.parent.parent.parent`) and still
work when invoked directly.

## How it works (one paragraph)

Each probe is a short factual question with a gold answer, scored by a
Gemini 3 Flash Preview judge. Researcher subfield probes use a 4-way
evidence-aware judge (CORRECT_STRONG = subfield + verifiable evidence
item; CORRECT_WEAK = subfield only; REFUSAL; WRONG); other probes use
a 3-way judge (CORRECT / REFUSAL / WRONG). Scoring is no-penalty
(`λ = 0`): probes score `{+1.0, +0.5, 0, 0}` for the four classes, so
wrong answers and refusals both score zero. Capacity and honesty are
different latent axes — penalizing hallucinations (`λ = -1`, kept only
as an ablation) mixes a vendor-dependent honesty policy into the size
estimate; see `PARAM_POLICY.md` §4. The
calibration curve, loaded at runtime from
`data/results/effective_params.json` (λ=0), is
`log10(N_eff_B) = 6.298 · accuracy − 1.526` (equivalently the fitted
`accuracy = 0.159 · log10(N_B) + 0.242`; R² ≈ 0.91, no-penalty λ=0,
LOO median fold error ≈1.5×). For MoE models, *total* parameters
predict accuracy (R² = 0.667) better than active parameters
(R² = 0.412) — so the curve is fit against total parameter count. This is
a fitted convention, not an assumption: the continuous MoE exponent γ
(1 = total, 0 = active) lands at γ̂ = 1.00 with bootstrap 90% CI
[0.86, 1.00] (`scripts/19_effective_params.py`).

## Requirements

- Python ≥ 3.10
- An API key for the model(s) you want to evaluate (OpenRouter covers
  all 188 evaluated models; OpenAI-compatible endpoints also work)
- An `OPENROUTER_API_KEY` for the judge (always Gemini 3 Flash Preview)
- ~$0.10–$3 per model to score the full 1,400 probes, depending on the
  model priced at OpenRouter rates. Get the exact figure up-front with
  `python scripts/ikp_budget.py --model <id>` (no API key needed) — see
  the "Budgeting a run" section of `TOOLKIT.md`.

## Citing

```bibtex
@misc{li2026incompressibleknowledgeprobesestimating,
  title         = {Incompressible Knowledge Probes: Estimating Black-Box LLM
                   Parameter Counts via Factual Capacity},
  author        = {Bojie Li},
  year          = {2026},
  eprint        = {2604.24827},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2604.24827}
}
```

## License

Code: MIT. Probe set and per-model results: CC BY 4.0.

ikp

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README