bayes_evals

KO | EN

gitlite — search

#hacktoberfest #react #ai #android #deep-learning #linux #java #windows #awesome #pytorch #security #mcp #openai #open-source

(★ 25)

A lightweight library for Bayesian analysis of LLM evals (ICML 2025 Spotlight Position Paper)

bayes_evals 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# `bayes_evals`: A lightweight library for Bayesian analysis of LLM evals

Implementing the methods from the ICML 2025 spotlight paper "[Position: Don't use the CLT in LLM evals with fewer than a few hundred datapoints](https://arxiv.org/abs/2503.01747)".
(Code to reproduce the experiments in that paper can be found at a [different repo](https://github.com/sambowyer/no_clt_paper).)

## Installation
This is a simple package, so you can just download the `bayes_evals.py` file (located at [src/bayes_evals/bayes_evals.py](src/bayes_evals/bayes_evals.py)) and put it in your project directory, e.g. with `wget`:
```bash
wget https://raw.githubusercontent.com/sambowyer/bayes_evals/main/src/bayes_evals/bayes_evals.py
```
or `curl`:
```bash
curl -L -O https://raw.githubusercontent.com/sambowyer/bayes_evals/main/src/bayes_evals/bayes_evals.py
```

Alternatively, you can install the package by cloning this repository and using pip:
```bash
pip install -e .
```

## Usage
```python
import bayes_evals as be
import pandas as pd

# Load the data (should NOT contain an index column)
eval_data = pd.from_csv('data/evals.csv')

# Get the results either for individual LLMs (each column in the data)
# with a specified confidence level alpha (default=0.05)...
indep_intervals = be.independent_intervals(eval_data, alpha=0.05)

# ... in which case you can also do independent LLM comparisons...
indep_comparisons = be.independent_comparisons(eval_data)

# ... or get comparisons between LLMs assuming a paired evals model...
# (i.e. with the same questions asked to each LLM)
paired_comparisons = be.paired_comparisons(eval_data)
```
Each of the `indep_results`, `indep_comparisons`, and `paired_comparisons` objects are just `pd.DataFrame` objects with model names as the column names.
The `indep_results` object has two rows: `lower` and `upper`.
The `indep_comparisons` and `paired_comparisons` objects have a row for each model, with entry at row $i$ and column $j$ being the probability that model $i$ is better than model $j$.

## Data format
The data should be in a `pandas` DataFrame, with $Q$ = no. questions rows and $M$ = no. LLMs columns.
The data should be binary, with 1 indicating a correct answer and 0 indicating an incorrect answer.
The columns should be named with the LLMs' names.

## Displaying results
You can also make matplotlib plots of the results using the following functions:
```python
be.plot_intervals(eval_data, indep_intervals, filename='plots/indep_intervals.png')
```
![Independent LLM intervals](https://raw.githubusercontent.com/sambowyer/bayes_evals/main/examples/plots/indep_intervals.png)

```python
be.plot_comparisons(indep_comparisons, filename='plots/indep_comparisons.png', title="Independent LLM comparisons")
```
![Independent LLM comparisons](https://raw.githubusercontent.com/sambowyer/bayes_evals/main/examples/plots/indep_comparisons.png)

```python
be.plot_comparisons(paired_comparisons, filename='plots/paired_comparisons.png', title="Paired LLM comparisons")
```
![Paired LLM comparisons](https://raw.githubusercontent.com/sambowyer/bayes_evals/main/examples/plots/paired_comparisons.png)

## Examples
See the `examples` directory for a Jupyter notebook and basic script that generate the above plots:
- [Jupyter notebook](examples/basic_example.ipynb)
- [Script](examples/basic_example.py)

# Citing
If you find this work useful, please consider citing the accompanying paper:
```bibtex
@inproceedings{bowyer2025positiondontuseclt,
      title={Position: Don't use the CLT in LLM evals with fewer than a few hundred datapoints}, 
      author={Sam Bowyer and Laurence Aitchison and Desi R. Ivanova},
      year={2025},
      booktitle={Forty-second International Conference on Machine Learning Position Paper Track},
      url={https://arxiv.org/abs/2503.01747}, 
}
```

bayes_evals

bayes_evals 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

bayes_evals

bayes_evals 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README