signoff

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #typescript #llm #golang #machine-learning #docker #macos #java #kubernetes #mcp #open-source

(★ 77)

Statistical sign-off for the model swap: N-run, nondeterminism-aware A/B equivalence with an audit-ready report.

signoff 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# signoff

**Statistical sign-off for the model swap.** N-run, nondeterminism-aware A/B equivalence testing for LLM migrations, with an audit-ready report.

Model deprecations force migrations on someone else's schedule. "We eyeballed ten outputs" is not evidence a regulated enterprise can take to a risk committee. `signoff` runs your real prompt suite on the old and new model N times each, measures per-case flip rates (nondeterminism), scores answers through layered matching (exact, normalized, numeric, then LLM-judge), and applies a paired sign test before declaring the swap safe.

Vendor-neutral by design: it judges an OpenAI to Anthropic migration with exactly the same rigor as the reverse.

## Install

```bash
pip install -e .
export OPENAI_API_KEY=...      # for openai: models and/or the judge
export ANTHROPIC_API_KEY=...   # for anthropic: models
```

## Use

```bash
signoff run \
  --suite examples/extraction.yaml \
  --model-a openai:gpt-4o \
  --model-b anthropic:claude-sonnet-4-5 \
  --runs 5 \
  --judge openai:gpt-4o-mini \
  --out report.html
```

Output: a one-line verdict in the terminal, plus `report.html` (human) and `report.json` (machine).

```
cases=48 agreement=95.8% flip_a=2.1% flip_b=1.7%
accuracy: A=93.8% B=95.8% p=0.512
verdict: PASS   report: report.html / report.json
```

**Verdict logic:** with ground truth (`expected` per case), FAIL only when model B is less accurate *and* the regression is statistically significant (exact sign test on discordant pairs, p <= 0.05). Without ground truth, 90%+ majority-answer agreement passes; below that, REVIEW.

## Suite format

```yaml
cases:
  - id: facility-amount
    prompt: "Extract the total facility amount. ..."
    expected: "$250,000,000"   # optional; omit for agreement-only mode
```

## Why N runs?

Single-run comparisons confuse nondeterminism with regression. A prompt that flips answers 30% of the time will "regress" or "improve" by luck. `signoff` reports the flip rate per case per model, so instability is visible before it's misattributed to the migration.

## Roadmap

- `signoff stability`: flip-rate matrix and prompt-line blame for a single model
- Semantic match layer (embeddings) between numeric and judge
- Governance evidence annex: mapping results to ISO/IEC 42001 change control, NIST AI RMF MEASURE, EU AI Act Annex IV
- Adapters: Bedrock, Azure OpenAI, Gemini
- Equivalence bands (TOST) as an alternative to the sign test

## License

Apache-2.0

signoff

signoff 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

signoff

signoff 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README