llm-fingerprinting

KO | EN

gitlite — search

#hacktoberfest #go #android #machine-learning #rust #docker #macos #nodejs #awesome #awesome-list #vue #security #mcp #open-source

llm-fingerprinting

(★ 25)

single-token behavioural fingerprints (arXiv:2607.10252) One Token Is Enough

llm-fingerprinting Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# One Token Is Enough

This is a CLI tool that probes whether the models advertised by an OpenAI-compatible
API endpoint are genuinely **distinct backends** — using the single-token
behavioural fingerprinting method from **"One Token Is Enough: Fingerprinting
and Verifying Large Language Models from Single-Token Output Distributions"**
([arXiv:2607.10252](https://arxiv.org/abs/2607.10252)).

Default target: **[vivgrid.com](https://vivgrid.com)**
(`https://api.vivgrid.com/v1`, OpenAI-compatible, `Authorization: Bearer <key>`).

---

## Why this works

Ask a model a trivial one-word question — *"name a random number between 1 and
100"* — and it does **not** answer uniformly. Each model has a sharp, idiosyncratic
bias (famously ~27–42 for the 1–100 task). The paper shows that:

- The empirical distribution over ~30 single-token answers is a stable
  **behavioural fingerprint**.
- **Split-halves of one model's samples land an order of magnitude closer**
  (Jensen-Shannon divergence) **than samples from two different models.**
- So you can verify whether an endpoint really serves the advertised model —
  without the provider's cooperation, without long generations, and without
  logprob access.

This tool applies that to a practical question: *are all the models vivgrid
lists actually different models, or are some the same backend wearing different
name tags?*

### The battery (40 cells)

10 tasks × 4 languages (English, Russian, Chinese, Arabic) = **40 probe cells**,
matching the paper. Tasks: random number 1–100, random number 1–10, favorite
number, random letter, random word, random color, favorite color, random
animal, random city, coin flip. Prompts live in
[`config/prompts.json`](config/prompts.json) and are fully editable.

Each cell is sampled **30× at temperature 1.0** by default (the paper's standard;
lower `--samples` to trade accuracy for tokens). That's **1200 single-token calls
per model** for the full 40-cell battery.
Answers are normalized (digit-system mapping incl. Arabic-Indic & Chinese
numerals, case folding, punctuation stripping, coin-term canonicalization) into
a categorical empirical distribution `p̂[task,lang]`.

### The metric

Distance between two models:

```
D(A, B) = mean over shared cells of  JSD(p̂_A[cell] || p̂_B[cell])      (JSD in base 2)
```

only counting cells with ≥10 valid samples in both. `D = 0` → identical
behaviour; larger → more different. Same model split-half ≈ small; different
models ≈ an order of magnitude larger.

---

## Install

Two zero-dependency implementations ship in this repo — use whichever you like.
They share `config/prompts.json` and produce identical fingerprint JSON.

**TypeScript (Bun)** — no `npm install` needed; Bun's built-in `fetch` + `crypto`
cover everything:

```bash
cd compare-llm-fingerprints
bun run otie.ts --help          # or:  bun run otie <command>   (via package.json)
```

**Python** — pure 3.8+ standard library:

```bash
python3 otie.py --help          # or: python3 -m otie --help
```

Configure the endpoint with two environment variables:

```bash
export API_KEY="sk-..."                      # required (get it from the vivgrid console)
export BASE_URL="https://api.vivgrid.com/v1" # optional; this is the default if unset
```

- **`API_KEY`** — Bearer key for the endpoint. Required unless you pass `--mock`;
  if it's missing the tool stops and tells you.
- **`BASE_URL`** — OpenAI-compatible base URL. Defaults to
  `https://api.vivgrid.com/v1`; if unset the tool uses that default and prints a
  note. Override per-run with `--base-url` (use `--base-url mock` for offline).

Get a key from the [vivgrid console](https://console.vivgrid.com). Both `--help`
outputs document these variables.

> The rest of this README shows `python3 otie.py …`; for the Bun version just
> swap in `bun run otie.ts …` — every command, flag, and default is identical.

---

## Quick start — audit vivgrid

One command fingerprints every advertised model, cross-compares them, and flags
any pair that is suspiciously identical:

```bash
python3 otie.py audit                      # audits the models in config/models.txt
python3 otie.py audit --models claude-opus-4-8 gpt-4o gemini-3.5-flash glm-5
```

Output (abridged):

```
[1] Split-half self-consistency (lower = stabler fingerprint)
    claude-opus-4-8       self-JSD = 0.09
    ...
[2] Nearest other-model neighbour
    claude-opus-4-8       nearest=gpt-4o   JSD=0.18  <== COLLISION
    gemini-3.5-flash      nearest=glm-5    JSD=0.82
[3] Verdict
    SUSPECTED DUPLICATE / MISLABELLED BACKENDS:
      - 'claude-opus-4-8' and 'gpt-4o' fingerprint almost identically ...
```

Reads: a model whose nearest neighbour sits at **mean JSD ≤ 0.35** is flagged —
two names, one backend. Genuinely distinct models sit far higher (≈0.6–0.9).

Artifacts written to `fingerprints/`: one `<model>.json` per model,
`audit_report.txt`, and `audit_summary.json`.

### Try it with no key (offline demo)

A deterministic mock backend lets you see the whole pipeline — and the collision
detector firing — without any network or key. `OTIE_MOCK_ALIAS` forces two names
onto the same backend to simulate a mislabelled model:

```bash
OTIE_MOCK_ALIAS="gpt-4o=claude-opus-4-8" \
  python3 otie.py audit --mock \
  --models claude-opus-4-8 claude-sonnet-5 gpt-4o gemini-3.5-flash glm-5
```

You'll see `claude-opus-4-8` and `gpt-4o` flagged as a collision while the rest
stay far apart.

---

## Commands

| Command | Purpose |
|---|---|
| `audit` | Fingerprint many models, cross-compare, flag duplicate/mislabelled backends. **Main command.** |
| `fingerprint --model M` | Build and save one model's fingerprint. |
| `compare A.json B.json …` | Print the pairwise JSD distance matrix. |
| `verify --probe P.json --reference R.json` | Decide MATCH / MISMATCH vs a threshold τ. |
| `selftest F.json` | Split-half self-consistency of one fingerprint (sanity check). |
| `list-models` | Print the model IDs from `config/models.txt`. |

Common flags: `--samples` (default **30**, the paper standard), `--temperature` (1.0),
`--tasks`, `--langs`, `--concurrency` (8), `--base-url` (overrides `$BASE_URL`;
`mock` for offline), `--api-key-env` (read the key from a different env var;
default `API_KEY`).

---

## Detecting a *specific* fake (not just duplicates)

`audit` proves whether the advertised models are **distinct from each other**.
To prove a vivgrid model **is the specific model it claims** (e.g. that
vivgrid's `claude-opus-4-8` really is Anthropic's), fingerprint the genuine
model from its official endpoint and `verify`:

```bash
# 1. reference fingerprint from the official provider
#    (point API_KEY + --base-url at the genuine endpoint for this call)
API_KEY=... python3 otie.py fingerprint \
  --model claude-opus-4-8 \
  --base-url https://api.official-provider.example/v1 \
  --out ref/opus.json

# 2. probe fingerprint from vivgrid (uses your exported API_KEY / BASE_URL)
python3 otie.py fingerprint --model claude-opus-4-8 --out probe/opus.json

# 3. verify
python3 otie.py verify --probe probe/opus.json --reference ref/opus.json
#   mean JSD = 0.05  -> VERDICT: MATCH
#   mean JSD = 0.61  -> VERDICT: MISMATCH (not the claimed model)
```

Exit codes: `verify` returns 0 on MATCH, 1 on MISMATCH, 3 on insufficient data —
handy in CI.

---

## Thresholds & tuning

- `--samples` (default **30**, the paper standard): lower it to save tokens at
  the cost of noisier per-cell distributions, so borderline distances get
  shakier; costs scale linearly. Keep it at 30 to confirm any borderline flag.
- `audit --collision-threshold` (default **0.35** mean JSD): a model is flagged
  when its nearest neighbour sits at or below this absolute distance. This is
  the primary, sample-count-robust signal. The `ratio_to_self` column is shown
  for context only — at low `--samples` the split-half self-noise is too
  jittery to threshold on, which is why the flag is absolute, not a ratio.
- `verify --threshold` (default **0.10** mean JSD): below → MATCH. The paper
  reports a 7.3% equal-error-rate operating point on the full 40-cell battery;
  0.10 is conservative and assumes well-sampled fingerprints — raise it if you
  verify with few samples. Calibrate on your own genuine-vs-impostor pairs.

---

## Caveats

- **Same-family models are genuinely similar.** Two real snapshots of the same
  family (e.g. two Claude tiers) can fingerprint closer than two unrelated
  families. Read a collision as "these two behave like the same backend," then
  judge whether that's plausible for the names involved — it's evidence, not a
  courtroom verdict.
- **Providers can change behaviour** (system prompts, samplers, quantization,
  routing) without swapping the model, which shifts fingerprints. Re-baseline
  reference fingerprints periodically.
- vivgrid's docs note that for some product surfaces the model is selected in
  the console rather than the request body. This tool sends the standard
  OpenAI `model` field; if your project pins a single model server-side, every
  "model" you audit may resolve to that one — which the audit will correctly
  surface as a wall of collisions.
- This is a **statistical** test. Treat flags as leads to investigate.

## Layout

```
config/prompts.json     40-cell probe battery (10 tasks x 4 languages) — shared
config/models.txt       default model IDs to audit, one per line — shared, editable

src/                    TypeScript (Bun) implementation
  types.ts              shared types
  prompts.ts            load/expand the battery
  client.ts            OpenAI-compatible fetch client + offline mock backend
  normalize.ts          raw answer -> categorical label (numeral/coin/text)
  fingerprint.ts        concurrent sampling, distribution build, split-half
  divergence.ts         Jensen-Shannon divergence + mean per-cell distance
  cli.ts                CLI (audit/fingerprint/compare/verify/selftest)
otie.ts                 Bun launcher  ->  bun run otie.ts <cmd>
package.json            bun scripts (audit/fingerprint/verify/…) + typecheck

otie/                   Python implementation (mirror of the above)
  prompts.py  client.py  normalize.py  fingerprint.py  divergence.py  cli.py
otie.py                 Python launcher  ->  python3 otie.py <cmd>

fingerprints/           output fingerprints + audit reports (from either impl)
```

## Examples

```sh
$ bun run otie.ts audit --samples 30
Note: $BASE_URL is not set; using default https://api.vivgrid.com/v1

== Fingerprinting deepseek-v4-pro ==
  [deepseek-v4-pro] 800/800 samples

== Fingerprinting deepseek-v4-flash ==
  [deepseek-v4-flash] 800/800 samples

== Fingerprinting glm-5.2 ==
  [glm-5.2] 800/800 samples
======================================================================
ONE-TOKEN-IS-ENOUGH  —  MODEL AUTHENTICITY AUDIT
======================================================================

Method: single-token behavioural fingerprints (arXiv:2607.10252).
Two genuinely distinct models have a large mean cross-model JSD;
two names for the SAME backend collapse to a near-zero distance.
Collision flag: nearest-neighbour mean JSD <= 0.35.

[1] Split-half self-consistency (lower = stabler fingerprint)
    deepseek-v4-pro       self-JSD = 0.2079
    deepseek-v4-flash     self-JSD = 0.1561
    glm-5.2               self-JSD = 0.2071
    (with few samples per cell this is only a rough sanity signal)

[2] Nearest other-model neighbour
    deepseek-v4-pro       nearest=deepseek-v4-flash  JSD=0.2904  ratio_to_self=1.4x  <== COLLISION: too close to be a different model
    deepseek-v4-flash     nearest=deepseek-v4-pro    JSD=0.2904  ratio_to_self=1.9x  <== COLLISION: too close to be a different model
    glm-5.2               nearest=deepseek-v4-pro    JSD=0.3284  ratio_to_self=1.6x  <== COLLISION: too close to be a different model

[3] Verdict
    SUSPECTED DUPLICATE / MISLABELLED BACKENDS:
      - 'deepseek-v4-pro' and 'deepseek-v4-flash' fingerprint almost identically (mean JSD=0.2904). At least one is likely
        not the distinct model it claims to be.
        deepseek-v4-pro  vs  deepseek-v4-flash   —  per-cell breakdown (JSD 0=identical .. 1=disjoint)
          most divergent cells (evidence these are DIFFERENT models):
            1.000  word|ru             deepseek-v4-pro: яблоко×10, апельсин×2, фонарь×2, …   |   deepseek-v4-flash: облако×11, карандаш×3, библиотека×1, …
            0.778  animal|ru           deepseek-v4-pro: капибара×8, панда×7, квокка×1, …   |   deepseek-v4-flash: кенгуру×8, слон×5, коала×3, …
            0.775  color|zh            deepseek-v4-pro: 靛蓝×14, 钴蓝色×1, 蓝色×1, …   |   deepseek-v4-flash: 蓝色×15, 紫色×4, 绿色×1
            0.703  color|ru            deepseek-v4-pro: бирюзовый×10, фиолетовый×3, лавандовый×3, …   |   deepseek-v4-flash: синий×11, фиолетовый×4, зеленый×3, …
            0.695  city|zh             deepseek-v4-pro: 雷克雅未克×10, 东京×3, 苏黎世×2, …   |   deepseek-v4-flash: 东京×17, 上海×2, 巴黎×1
            0.643  city|ru             deepseek-v4-pro: кито×3, токио×2, киото×2, …   |   deepseek-v4-flash: токио×12, киото×2, копенгаген×1, …
          most identical cells (trivial agreement — pulls the mean down):
            0.000  favorite_color|zh   deepseek-v4-pro: 蓝色×20   |   deepseek-v4-flash: 蓝色×20
            0.000  favorite_color|en   deepseek-v4-pro: blue×20   |   deepseek-v4-flash: blue×20
            0.004  coin|ar             deepseek-v4-pro: heads×17, tails×3   |   deepseek-v4-flash: heads×18, tails×2
            0.010  favorite_number|zh  deepseek-v4-pro: 7×16, 42×4   |   deepseek-v4-flash: 7×14, 42×6
            0.025  number_1_10|ru      deepseek-v4-pro: 7×20   |   deepseek-v4-flash: 7×19, 3×1
            0.025  number_1_10|ar      deepseek-v4-pro: 7×19, 4×1   |   deepseek-v4-flash: 7×20
      - 'glm-5.2' and 'deepseek-v4-pro' fingerprint almost identically (mean JSD=0.3284). At least one is likely
        not the distinct model it claims to be.
        glm-5.2  vs  deepseek-v4-pro   —  per-cell breakdown (JSD 0=identical .. 1=disjoint)
          most divergent cells (evidence these are DIFFERENT models):
            0.931  city|ru             glm-5.2: калуга×3, лиссабон×2, воронеж×2, …   |   deepseek-v4-pro: кито×3, токио×2, киото×2, …
            0.919  city|zh             glm-5.2: 京都×5, 成都×4, 巴塞罗那×2, …   |   deepseek-v4-pro: 雷克雅未克×10, 东京×3, 苏黎世×2, …
            0.879  word|ru             glm-5.2: облако×10, ветер×4, лампа×1, …   |   deepseek-v4-pro: яблоко×10, апельсин×2, фонарь×2, …
            0.769  city|ar             glm-5.2: لشبونة×6, روما×2, طوكيو×2, …   |   deepseek-v4-pro: طوكيو×17, باريس×2, الدار×1
            0.760  animal|zh           glm-5.2: 水豚×7, 狐狸×4, 企鹅×3, …   |   deepseek-v4-pro: 树懒×4, 海豚×4, 猫×4, …
            0.749  word|zh             glm-5.2: 斑马×4, 琥珀×3, 微风×3, …   |   deepseek-v4-pro: 苹果×8, 斑马×2, 星空×1, …
          most identical cells (trivial agreement — pulls the mean down):
            0.000  number_1_10|ru      glm-5.2: 7×20   |   deepseek-v4-pro: 7×20
            0.000  favorite_color|zh   glm-5.2: 蓝色×20   |   deepseek-v4-pro: 蓝色×20
            0.000  favorite_color|ru   glm-5.2: синий×20   |   deepseek-v4-pro: синий×20
            0.000  favorite_color|en   glm-5.2: blue×20   |   deepseek-v4-pro: blue×20
            0.004  coin|ar             glm-5.2: heads×18, tails×2   |   deepseek-v4-pro: heads×17, tails×3
            0.007  coin|zh             glm-5.2: heads×19, tails×1   |   deepseek-v4-pro: heads×18, tails×2

Interpretation guide (mean cross-model JSD, 0=identical .. 1=disjoint):
  <= 0.35  -> suspiciously similar / same backend (flagged)
  >  0.35  -> plausibly distinct models
  Note: with only a few samples per cell, distances are noisier;
  re-run with --samples 15 to confirm any borderline flag.
======================================================================

Wrote /Users/cc/tmp/one-token-is-enough-llm-fingerprint-detector/fingerprints/audit_report.txt and audit_summary.json
```

llm-fingerprinting

llm-fingerprinting Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

llm-fingerprinting

llm-fingerprinting Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README