humanities_data_benchmark

KO | EN

gitlite — search

#javascript #python #ai #go #android #machine-learning #rust #linux #docker #nodejs #awesome #cli #openai #open-source

humanities_data_benchmark

(★ 19)

LLM Benchmark Suite for Humanities Data

humanities_data_benchmark 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# RISE Humanities Data Benchmark

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21789055.svg)](https://doi.org/10.5281/zenodo.21789055)

This repository contains benchmark datasets (images and text files), prompts, ground truths, and evaluation scripts for
assessing the performance of large language models (LLMs) on humanities-related tasks. The suite is
designed as a resource for researchers and practitioners interested in systematically evaluating
how well various LLMs perform on digital humanities (DH) tasks involving visual and text-like materials.

> **ℹ Looking for benchmark results?**
> This README provides an overview of the benchmark suite and explains how to use it.
> For detailed test results and model comparisons, visit our [results dashboard](https://rise-services.rise.unibas.ch/benchmarks/).

## What is Benchmarking and Why Should You Care?

Benchmarking is the process of systematically evaluating and ranking various models for specific tasks using well-defined ground truths and metrics. For humanities research, benchmarking provides:

- **Evidence-based decision-making** about which model(s) to use for which humanities-specific task(s)
- **Quantifiable comparisons** between different AI models on humanities data, including cost efficiency analysis
- **Standardized evaluation** of model performance on tasks like historical document analysis, transcription, and metadata extraction

This benchmark suite focuses on tasks essential to digital humanities work with visual materials, helping researchers make informed choices about which AI systems best suit their specific research needs.

> **ℹ Looking for more background?**
> 
> Hindermann, M., Kasper, L. K., Marti, S., &amp; Bosse, A. (2026). From Experiments to Epistemic Practice: The RISE Humanities Data Benchmark. *Journal of Open Humanities Data*, *12*(1), 38. https://doi.org/10.5334/johd.470
> 
> Hindermann, M., Marti, S., Kasper, L. K., & Bosse, A. (2026). The RISE Humanities Data Benchmark: A Framework for Evaluating Large Language Models for Humanities Tasks. *Journal of Open Humanities Data*, *12*(1), 24. https://doi.org/10.5334/johd.481
>
> Hindermann, M., & Marti, S. (2026). The RISE Humanities Data Benchmark: From Anecdote to Evidence. DH Benelux 2026, Maastricht, Netherlands. Zenodo. https://doi.org/10.5281/zenodo.20595263
> 
> Hindermann, M., & Marti, S. (2025, March 19). *RISE Crash Course: "AI Benchmarking"*. Zenodo. https://doi.org/10.5281/zenodo.15062831

## Table of Contents

- [1. Overview](#1-overview)
  - [1.1. Terminology](#11-terminology)
  - [1.2. Available Benchmarks](#12-available-benchmarks)
  - [1.3. How it Works](#13-how-it-works)
  - [1.4. Practical Considerations](#14-practical-considerations)
- [2. Use it!](#2-use-it)
  - [2.1. Fork and prepare](#21-fork-and-prepare)
  - [2.2. Run a configured test](#22-run-a-configured-test)
  - [2.3. Create a new Benchmark](#23-create-a-new-benchmark)
  - [2.4. Run an adhoc test](#24-run-an-adhoc-test)
  - [2.5. Generate a result render](#25-generate-a-result-render)
  - [2.6. Run a local model](#26-run-a-local-model)
- [3. Share it!](#3-share-it)
  - [3.1. Before Submitting](#31-before-submitting)
  - [3.2. Create a pull request](#32-create-a-pull-request)
  - [3.3. Review & Publication](#33-review--publication)
- [4. Providers & Models](#4-providers-and-models)
- [5. Benchmarking Methodology](#5-benchmarking-methodology)
  - [5.1. Ground Truth](#51-ground-truth)
  - [5.2. Metrics](#52-metrics)
- [6. Project Status](#6-project-status)
  - [6.1. Current Limitations](#61-current-limitations)
  - [6.2. Outlook](#62-outlook)
- [7. Contributors](#7-contributors)

## 1. Overview

### 1.1. Terminology
- **Adhoc-Test**: A specific instance of a benchmark run which is only run once with the run-tests-tool CLI for testing reasons.
- **Benchmark**: A task for models to perform, consisting of images, ground truths, prompts, dataclasses, and scoring functions. Each benchmark is stored in a separate directory.
- **Configured Test**: A specific instance of a benchmark run with a particular configuration (ID, provider, model, temperature, role description, prompt file, dataclass).
- **Dataclass**: Pydantic models for structured output, supported across all providers.
- **Ground Truth**: The correct answer used to evaluate the model's response.
- **Image**: Visual input for the task. Images are paired with ground truth files.
- **Model**: Specific model used to perform the task.
- **Prompt**: Text given to the model to guide its response. 
- **Local Provider**: Provider handled by a local backend instead of an API (`contour_local`, `sam3_local`, `grounding_dino_local`, `doclayout_yolo_local`). Registered in `local/__init__.py`; no API key required.
- **Provider**: Company or service providing model access (`openai`, `genai`, `anthropic`, `cohere`, `mistral`, `openrouter`, `scicore`, `deepseek`, `x-ai`, or `alibaba`).
- **Request**: API call(s) made during a test, consisting of images and prompts.
- **Response**: Model's answer containing metadata and output.
- **Score**: Evaluation result indicating model performance.
- **Scoring Function**: Function that evaluates the model's response, implemented via the `score_request_answer` and `score_benchmark` methods.
- **Test Configuration**: Parameters for running a test, stored in `benchmarks_tests.csv`.
- **Text file**: Textual input for the task. Text files are paired with ground truth files.

### 1.2. Available Benchmarks
This benchmark suite currently includes the following benchmarks for evaluating LLM performance on humanities tasks:

| Benchmark | Description |
|-----------|-------------|
| **[Bibliographic Data](benchmarks/bibliographic_data/)** | Extract bibliographic information (publication details, authors, dates, metadata) from historical documents |
| **[Blacklist Cards](benchmarks/blacklist_cards/)** | Extract and structure information from historical blacklist cards |
| **[Book Advert XML](benchmarks/book_advert_xml/)** | Correct malformed XML from 18th century book advertisements |
| **[Business Letters](benchmarks/business_letters/)** | Extract structured metadata (names, organizations, dates, locations) from 20th century Swiss historical correspondence |
| **[Company Lists](benchmarks/company_lists/)** | Extract structured company information from historical business listings and directories |
| **[Duty Rosters](benchmarks/duty_rosters/)** | Extract structured schedule data (shift types, half-day splits, unit assignments, staff metadata) from Swiss nursing staff duty rosters |
| **[Fraktur Adverts](benchmarks/fraktur_adverts/)** | Recognize and transcribe historical German Fraktur script (16th-20th centuries) |
| **[General Meeting Minutes](benchmarks/general_meeting_minutes/)** | Extract structured data from general meeting minutes of historical companies |
| **[Library Cards](benchmarks/library_cards/)** | Catalog card analysis and information extraction from historical library catalog systems |
| **[Magazine Pages](benchmarks/magazine_pages/)** | Detect and locate advertisements on historical magazine pages using bounding boxes |
| **[Medieval Manuscripts](benchmarks/medieval_manuscripts/)** | Page segmentation and handwritten text extraction from 15th century medieval German manuscripts |
| **[Personnel Cards](benchmarks/personnel_cards/)** | Extract structured employment data (position, location, salary, dates) from 20th century Swiss personnel card tables |
| **Test Benchmarks** | System validation and basic functionality testing ([test_benchmark](benchmarks/test_benchmark/), [test_benchmark2](benchmarks/test_benchmark2/)) |

### 1.3. How it Works
The RISE Humanities Data Benchmark is designed to be modular and extensible. There are a number of datasets which are submitted to tests and their results are saved.
The whole framework, the datasets and the results are part of this repository.

Refer to the [next chapter](#2-use-it) in order to learn about usage in your own research.

### 1.4. Practical Considerations
When using this benchmark suite for your own research, consider the following:

| Category | Consideration | Description |
|----------|---------------|-------------|
| **Resource Requirements** | Skills | Operationalizing tasks requires both domain knowledge and technical expertise |
| | Ground Truth Creation | Requires domain expertise and careful curation |
| | Metric Selection | Requires understanding of both the humanities domain and evaluation methods |
| **Technical** | Local vs. API Models | Determine if you need to run models locally or can use API services |
| | Data Privacy | Ensure you're allowed to share your data via APIs if needed |
| | Infrastructure | Consider if you have access to appropriate computing resources |
| **Compliance** | Legal Requirements | Check for any legal restrictions on data sharing or model usage |
| | Ethical Guidelines | Consider any ethical implications of your benchmarking approach |
| | Funder Requirements | Verify if there are any funding agency requirements |
| | FAIR Data Principles | Consider how to make your benchmark data Findable, Accessible, Interoperable, and Reusable |

## 2. Use it!

> **ℹ We welcome your contributions**     
> The benchmark suite is designed to be extensible and welcomes contributions from the digital humanities community. Whether you're adding new benchmarks, improving existing ones, or enhancing the evaluation framework, your contributions help advance AI evaluation for humanities research.
> For detailed contribution guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md). To report bugs, suggest features, or discuss improvements, please open an issue on our [GitHub Issues page](https://github.com/rise-unibas/humanities_data_benchmark/issues).

### 2.1. Fork and prepare
In order to start, the following steps are in order:
- Fork this repository and clone your fork
- Install the dependencies (Python 3.12 or newer is recommended)
- Obtain API keys to the providers you want to test
- Create a `.env` file in the root directory of the repository.

```bash
pip install -r requirements.txt
```

Add the following lines to the `.env` file as needed with the obtained API keys.
```bash
OPENAI_API_KEY=<your_openai_api_key>
GENAI_API_KEY=<your_genai_api_key>
ANTHROPIC_API_KEY=<your_anthropic_api_key>
COHERE_API_KEY=<your_cohere_api_key>
MISTRAL_API_KEY=<your_mistral_api_key>
OPENROUTER_API_KEY=<your_openrouter_api_key>
SCICORE_API_KEY=<your_scicore_api_key>
DEEPSEEK_API_KEY=<your_deepseek_api_key>
ALIBABA_API_KEY=<your_alibaba_api_key>
X-AI_API_KEY=<your_xai_api_key>
```

The key name is derived from the provider name in `benchmarks_tests.csv` as `<PROVIDER>_API_KEY` (uppercased). Note the hyphen in `X-AI_API_KEY`, which follows from the provider name `x-ai`. Local providers (see [2.6.](#26-run-a-local-model)) need no key.

### 2.2. Run a configured test
To test if your installation works, it's easiest to run one of the configured tests. Define either `OPENAI_API_KEY` (= `T0001`) or `GENAI_API_KEY` (= `T0193`) to get started.
Start the script from the root of your project, like so:

```
python scripts/run_single_test.py --test_id T0001
```

This executes the `test_benchmark` (one image, one request) and saves the results to `results/YYYY-MM-DD/T0001`. Once these results are present, the test will **not** send requests for existing results on the same day. If you want to overwrite the existing results, you can:

```
python scripts/run_single_test.py --test_id T0001 --regenerate
```

You also can run the script without any parameters for the interactive interface. It lets you search for and select the test you might be looking for.

Tests marked `legacy_test=true` in `benchmarks_tests.csv` are deprecated and are skipped by both `run_single_test.py` and `run_benchmarks.py`; they are kept only for historical results and cannot be run by ID.

### 2.3. Create a new Benchmark
Start with the CLI tool to create the basic structure:

```
python scripts/create_benchmark.py
```
This creates a new dataset environment, like so:

1. **Directory Structure:**
   - `benchmarks/[your_benchmark_name]/`
   - `images/` or `texts/` directories (based on your choice)
   - `ground_truths/` directory
   - `prompts/` directory

2. **Required Files:**
   - `benchmark.py` - Main benchmark class with scoring logic templates
   - `meta.json` - Benchmark metadata
   - `README.md` - Documentation
   - `prompts/prompt.txt` - Default prompt
   - `dataclass.py` - Pydantic schema (optional)

#### Step-by-Step Process
When starting the `create_benchmark.py` script, you will be guided through the creation of the following data:

**1. Benchmark Name**
- Must be lowercase with underscores (e.g., `personal_letters`)
- **Important:** Should describe the SOURCE, not the task
  - Good: `personal_letters`, `company_registers`, `manuscript_pages`
  - Bad: `date_recognition`, `entity_extraction`
- **Cannot be changed later** - choose carefully!
- Will be converted to CamelCase for the class name (e.g., `PersonalLetters`)

**2. Basic Information**
- **Title:** Full descriptive title
- **Short Title:** Abbreviated version for displays
- **Description:** Multi-line description (press Enter twice to finish)
  - Should describe both SOURCES and TASK

**3. Tags**
- Structured tags from specific categories:
  - **Source Type:** index-cards, letter-pages, manuscript-pages, book-pages, article-pages, essay-pages, registers, lists
  - **Structure:** text-like, list-like, table-like, mixed
  - **Text Type:** handwritten-source, typed-source, printed-source
  - **Century:** century-15th, century-16th, century-17th, century-18th, century-19th, century-20th
  - **Languages:** language-german, language-english
  - **Entry Type:** company-entries, bibliographic-entries, ner-entries
  - **Task:** ner-extraction, metadata-extraction, transcription, classification
  - **Misc:** test-benchmark
- Enter as comma-separated values

**4. Contributors**
- Add contributors by role:
  - Domain Expert
  - Data Curator
  - Annotator
  - Analyst
  - Engineer
- **Format:** firstname_lastname (e.g., john_doe, jane_smith)
- Details of non-existing users can be added later

**5. Scoring Configuration**
- Defaults to `fuzzy` metric with `descending` order
- Can be customized later in `meta.json`

**6. Data Structure**
- **Dataclass:** Strongly recommended - defines expected output structure using Pydantic
  - Enables automatic validation
  - Ensures consistent output format
- **Dataclass Name:** Should reflect result content (e.g., Page, Letter, Document)
  - Use CamelCase
- **Images:** Whether the benchmark uses images
- **Text Files:** Whether the benchmark uses text files

**7. Prompt**
- Enter the default prompt text (multi-line)
- **This prompt is editable** and can be changed later in `prompts/prompt.txt`
- Role description for system prompt

**8. Review and Edit**
- Review all collected settings
- **Edit any field** by entering its number (1-12)
- Enter 'c' to continue and create benchmark
- Enter 'q' to quit without creating

**After Creation:**

1. **Add Context Data:**
   - Place images in `benchmarks/[name]/images/`
   - Place ground truth JSON files in `benchmarks/[name]/ground_truths/`
   - Each ground truth filename must match its corresponding image (e.g., `image.jpg` → `image.json`)

2. **Implement Scoring:**
   - Edit `benchmarks/[name]/benchmark.py`
   - Implement `score_request_answer()` method
   - Implement `score_benchmark()` method

3. **Define Schema (if using dataclass):**
   - Edit `benchmarks/[name]/dataclass.py`
   - Add fields to your Pydantic model

#### Add Context Data
You need to add at least one image or text file. This is the context data. 
Context data are the inputs that will be sent to the LLM. Depending on the benchmark, this may include:

- .txt, .json and other text-only files (historical texts, metadata records, descriptions, OCR fragments)
- .jpg, .png or other image types (manuscript pages, document snippets, photos)

_Naming convention_:
- The whole filename without its ending is treated as context object name
- This means that all files with the same basename in the context directories (images, texts) are sent at the same time
- For each basename you must provide a ground truth file

#### Implement Scoring
Each benchmark has a corresponding benchmark class in benchmark.py. Two methods have to be implemented in order for the scoring to work:

_Implement the scoring of a single object/request:_
Implement the scoring for a single request. Most of the times it is not as easy as to ask if the llm-response and the ground truth are equal.

```python
def score_request_answer(self, object_name, response, ground_truth):
   # object_name: basename of the processed files
   # response: large language model response
   # ground_truth: corresponding_ground_truth
   
   calculated_score = 0
   # implement scoring for one object
   
   return {"fuzzy": calculated_score}
```

_Implement the scoring of the whole test run:_
Take the average or the mean or use any other functionality to score across all requests for the test run.

```python
def score_benchmark(self, all_scores):
       total_score = 0
       for score in all_scores:
           total_score += score['fuzzy']
       return {"fuzzy": total_score / len(all_scores)}
```

Return at least one metric. Commonly used metrics are fuzzy, f1_score, cer

#### Define Schema
If you want the model to return structured output, define a Pydantic model in `benchmarks/[name]/dataclass.py` and
reference its class name in the `dataclass` column of `benchmarks_tests.csv`. Nested models are supported, and field
descriptions are passed on to the provider, so use them to disambiguate fields.

```python
from typing import List, Optional
from pydantic import BaseModel, Field

class Entry(BaseModel):
    """A single entry on the page."""

name: str = Field(description="Name as written in the source")
    year: Optional[str] = Field(default=None, description="Year, if given")

class Page(BaseModel):
    """Main output structure; its name goes into the `dataclass` column."""

entries: List[Entry] = Field(default_factory=list, description="All entries on the page")
```

Beyond the two mandatory scoring methods, the benchmark class inherits a number of methods from `Benchmark`
(`scripts/benchmark_base.py`) that you can override to configure its behaviour:

| Method | Purpose |
|--------|---------|
| `remove_none_values()` | Whether `None` values are stripped from the response before scoring (default `True`) |
| `convert_truth_to_json()` | Whether a ground truth stored as a JSON string is parsed into an object (default `True`) |
| `resize_images()` | Whether oversized images are resized before being sent to the model (default `False`) |
| `get_title()` | Title of the benchmark as shown in the result table |
| `get_prompt_kwargs(basename, filenames)` | Values interpolated into the prompt file, e.g. file information |
| `skip_object(object_basename)` | Whether a given object is excluded from the run |

### 2.4. Run an adhoc test
When you have created a benchmark you should test it first and ensure that everything works. That's what adhoc-tests are for.
Run the following command and select from the options to create an on-the-fly configuration to test.

```
python scripts/run_single_test.py --adhoc
```

The results are saved to `test_runs/` directory instead of `results/` which you can easily delete and is ignored by the repository.
Perfect for experimentation and quick testing, ID format: `ADHOC_YYYYMMDD_HHMMSS`.

### 2.5. Generate a result render
To render a single test run as a standalone HTML report:

```
python scripts/generate_test_report.py results/YYYY-MM-DD/T0001
```

The report is written to `reports/<test_id>_report.html` by default; use `-o` to choose another path, `--include-raw`
to embed the raw API responses, `--no-images` to omit the input images, and `--open` to open the result in a browser.

To render all test runs of a given day into one overview report:

```
python scripts/generate_date_report.py results/YYYY-MM-DD
```

### 2.6. Run a local model
Not every model needs an API. Local backends are registered in `local/__init__.py` and are used by putting their
provider name into the `provider` column of `benchmarks_tests.csv`; no API key is required.

| Provider | Backend | Requirements |
|----------|---------|--------------|
| `contour_local` | OpenCV contour detection | any OS, no GPU needed |
| `doclayout_yolo_local` | DocLayout-YOLO | Windows / Linux, NVIDIA GPU |
| `grounding_dino_local` | Grounding DINO | Windows / Linux, NVIDIA GPU |
| `sam3_local` | SAM 3 | macOS / Apple Silicon |

To add a new backend, create `local/backends/<name>.py` with a class extending `LocalBackend`, register it in
`LOCAL_PROVIDERS`, and add a test row with the new provider name.

## 3. Share it!
We welcome contributions to the RISE Humanities Data Benchmark.

### 3.1. Before submitting
Before submitting a pull request, please make sure your benchmark meets all of the following criteria:

**Data Requirements**
- Dataset is not too large
  Recommended: < 50 MB total
- Ground truths are manually checked and reliable
- Data is legally usable
- Must be openly available
- No copyrighted or sensitive data
  - A clear license is indicated (preferably open license)
- Context files are properly named and paired with ground_truths

**Technical Requirements**
- Benchmark runs locally without errors
- Scoring metric is clearly defined and documented
- Directory structure follows the template
- README.md inside the benchmark folder is fully completed
  (template is created automatically by create_benchmark.py)

**Quality Requirements**
- Outputs are deterministic enough for fair evaluation
- The benchmark fills a clear research gap (new task, domain, or corpus)
- Instructions do not bias the LLM toward “right answers” via over-specification

### 3.2. Create a pull request
- Go to your fork on GitHub.
- Click “Compare & pull request”.
- Target: `RISE-UNIBAS/humanities_data_benchmark → main`

Add a short description:
- What your benchmark tests
- Example ground truth formats
- Scoring logic summary
- How you validated the dataset
- Any remaining issues or questions

### 3.3. Review & Publication
The maintainers will:

- run the benchmark locally
- check the metric
- confirm licensing
- validate folder structure
- potentially request revisions

Once everything is green, your benchmark will be merged into the main repository.

## 4. Providers and Models

This benchmark suite currently tests models from the following providers:

| Provider | Model                                 | Notes                                                     |
|----------|---------------------------------------|-----------------------------------------------------------|
| **Alibaba** | qwen3.5-plus-2026-02-15                   | Qwen 3.5 Plus (Feb 2026); multimodal                       |
| | qwen3.5-35b-a3b                           | Qwen 3.5, 35B MoE (3B active)                              |
| | qwen3.5-27b                               | Qwen 3.5, 27B dense                                        |
| | qwen3.5-122b-a10b                         | Qwen 3.5, 122B MoE (10B active)                            |
| | qwen3.5-397b-a17b                         | Qwen 3.5, 397B MoE (17B active)                            |
| | qwen3.5-flash-2026-02-23                  | Qwen 3.5 Flash (Feb 2026)                                  |
| **Anthropic** | ~~claude-3-5-sonnet-20241022~~            | ~~Claude 3.5 Sonnet~~ (legacy)                             |
| | ~~claude-3-7-sonnet-20250219~~            | ~~Claude 3.7 Sonnet~~ (legacy)                             |
| | ~~claude-3-opus-20240229~~                | ~~Claude 3 Opus~~ (legacy)                                 |
| | ~~claude-3-5-haiku-20241022~~             | ~~Claude 3.5 Haiku~~ (legacy)                              |
| | claude-haiku-4-5-20251001                 | Claude Haiku 4.5                                           |
| | claude-opus-4-1-20250805                  | Claude Opus 4.1                                            |
| | ~~claude-opus-4-20250514~~                | ~~Claude Opus 4~~ (legacy)                                 |
| | claude-opus-4-5-20251101                  | Claude Opus 4.5                                            |
| | ~~claude-sonnet-4-20250514~~              | ~~Claude Sonnet 4~~ (legacy)                               |
| | claude-sonnet-4-5-20250929                | Claude Sonnet 4.5                                          |
| | claude-opus-4-6                           | Claude Opus 4.6                                            |
| | claude-opus-4-7                           | Claude Opus 4.7                                            |
| | claude-opus-4-8                           | Claude Opus 4.8                                            |
| | claude-sonnet-4-6                         | Claude Sonnet 4.6                                          |
| | claude-sonnet-5                           | Claude Sonnet 5                                            |
| | claude-fable-5                            | Claude Fable 5                                             |
| | claude-opus-5                             | Claude Opus 5                                              |
| **Cohere** | command-a-03-2025                         | Command A (Mar 2025)                                       |
| | command-a-vision-07-2025                  | Command A Vision (Jul 2025); multimodal                    |
| | command-r-08-2024                         | Command R (Aug 2024)                                       |
| | command-r-plus-08-2024                    | Command R+ (Aug 2024)                                      |
| | command-r7b-12-2024                       | Command R 7B (Dec 2024)                                    |
| **DeepSeek** | deepseek-chat                             | DeepSeek V3 (chat)                                         |
| | deepseek-reasoner                         | DeepSeek R1; reasoning                                     |
| | deepseek-v4-flash                         | DeepSeek V4 Flash                                          |
| | deepseek-v4-pro                           | DeepSeek V4 Pro                                            |
| **Google/Gemini** | ~~gemini-1.5-flash~~                      | ~~Gemini 1.5 Flash~~ (legacy)                              |
| | ~~gemini-1.5-pro~~                        | ~~Gemini 1.5 Pro~~ (legacy)                                |
| | ~~gemini-2.0-flash~~                      | ~~Gemini 2.0 Flash~~ (legacy)                              |
| | ~~gemini-2.0-flash-lite~~                 | ~~Gemini 2.0 Flash-Lite~~ (legacy)                         |
| | ~~gemini-2.0-pro-exp-02-05~~              | ~~Gemini 2.0 Pro (experimental)~~ (legacy)                 |
| | gemini-2.5-flash                          | Gemini 2.5 Flash                                           |
| | gemini-2.5-flash-lite                     | Gemini 2.5 Flash-Lite                                      |
| | ~~gemini-2.5-flash-lite-preview-09-2025~~ | ~~Gemini 2.5 Flash-Lite (preview, Sep 2025)~~ (legacy)     |
| | ~~gemini-2.5-flash-preview-04-17~~        | ~~Gemini 2.5 Flash (preview, Apr 2025)~~ (legacy)          |
| | ~~gemini-2.5-flash-preview-09-2025~~      | ~~Gemini 2.5 Flash (preview, Sep 2025)~~ (legacy)          |
| | gemini-2.5-pro                            | Gemini 2.5 Pro                                             |
| | ~~gemini-2.5-pro-exp-03-25~~              | ~~Gemini 2.5 Pro (experimental)~~ (legacy)                 |
| | ~~gemini-2.5-pro-preview-05-06~~          | ~~Gemini 2.5 Pro (preview, May 2025)~~ (legacy)            |
| | ~~gemini-exp-1206~~                       | ~~Gemini experimental (Dec 2024)~~ (legacy)                |
| | gemini-3-flash-preview                    | Gemini 3 Flash (preview)                                   |
| | ~~gemini-3.1-flash-lite-preview~~         | ~~Gemini 3.1 Flash-Lite (preview)~~ (legacy)               |
| | gemini-3.1-flash-lite                     | Gemini 3.1 Flash-Lite                                      |
| | gemini-3.1-pro-preview                    | Gemini 3.1 Pro (preview)                                   |
| | ~~gemini-3-pro-preview~~                  | ~~Gemini 3 Pro (preview)~~ (legacy)                        |
| | gemini-3.5-flash                          | Gemini 3.5 Flash                                           |
| | gemini-3.6-flash                          | Gemini 3.6 Flash                                           |
| | gemini-3.5-flash-lite                     | Gemini 3.5 Flash-Lite                                      |
| **Mistral AI** | ~~magistral-medium-2509~~                 | ~~Magistral Medium (Sep 2025); reasoning~~ (legacy)        |
| | ~~magistral-small-2509~~                  | ~~Magistral Small (Sep 2025); reasoning~~ (legacy)         |
| | ministral-14b-2512                        | Ministral 3 14B (Dec 2025)                                 |
| | ministral-8b-2512                         | Ministral 3 8B (Dec 2025)                                  |
| | ~~mistral-large-2411~~                    | ~~Mistral Large (Nov 2024)~~ (legacy)                      |
| | mistral-large-2512                        | Mistral Large (Dec 2025)                                   |
| | ~~mistral-medium-2505~~                   | ~~Mistral Medium (May 2025)~~ (legacy)                     |
| | ~~mistral-medium-2508~~                   | ~~Mistral Medium (Aug 2025)~~ (legacy)                     |
| | mistral-medium-3.5                        | Mistral Medium 3.5 (Apr 2026)                              |
| | ~~mistral-small-2506~~                    | ~~Mistral Small (Jun 2025)~~ (legacy)                      |
| | ~~pixtral-12b~~                           | ~~Pixtral 12B; multimodal~~ (legacy)                       |
| | ~~pixtral-large-2411~~                    | ~~Pixtral Large (Nov 2024); multimodal~~ (legacy)          |
| **OpenAI** | gpt-4.1                                   | GPT-4.1                                                    |
| | gpt-4.1-mini                              | GPT-4.1 Mini                                               |
| | gpt-4.1-nano                              | GPT-4.1 Nano                                               |
| | ~~gpt-4.5-preview~~                       | ~~GPT-4.5 (preview)~~ (legacy)                             |
| | gpt-4o                                    | GPT-4o; multimodal                                         |
| | gpt-4o-mini                               | GPT-4o Mini; multimodal                                    |
| | gpt-5                                     | GPT-5                                                      |
| | gpt-5.1-2025-11-13                        | GPT-5.1 (Nov 2025)                                         |
| | gpt-5.2-2025-12-11                        | GPT-5.2 (Dec 2025)                                         |
| | gpt-5.3-codex                             | GPT-5.3 Codex; coding                                      |
| | gpt-5.4-2026-03-05                        | GPT-5.4 (Mar 2026)                                         |
| | gpt-5.5-2026-04-23                        | GPT-5.5 (Apr 2026)                                         |
| | gpt-5.6-sol                               | GPT-5.6 Sol                                                |
| | gpt-5.6-terra                             | GPT-5.6 Terra                                              |
| | gpt-5.6-luna                              | GPT-5.6 Luna                                               |
| | gpt-5-mini                                | GPT-5 Mini                                                 |
| | gpt-5-nano                                | GPT-5 Nano                                                 |
| | o3                                        | OpenAI o3; reasoning                                       |
| **OpenRouter** | google/gemma-4-26b-a4b-it                 | Gemma 4, 26B MoE (4B active), instruction-tuned            |
| | google/gemma-4-31b-it                     | Gemma 4, 31B, instruction-tuned                            |
| | meta-llama/llama-4-maverick               | Llama 4 Maverick                                           |
| | qwen/qwen3-vl-30b-a3b-instruct            | Qwen3-VL, 30B MoE (3B active), instruction-tuned; multimodal |
| | qwen/qwen3-vl-8b-instruct                 | Qwen3-VL 8B, instruction-tuned; multimodal                 |
| | qwen/qwen3-vl-8b-thinking                 | Qwen3-VL 8B, reasoning; multimodal                         |
| | qwen/qwen3.8-max                          | Qwen 3.8 Max; multimodal                                   |
| | qwen/qwen3.7-plus                         | Qwen 3.7 Plus                                              |
| | qwen/qwen3.6-plus                         | Qwen 3.6 Plus                                              |
| | qwen/qwen3.5-122b-a10b                    | Qwen 3.5, 122B MoE (10B active)                            |
| | qwen/qwen3.5-27b                          | Qwen 3.5, 27B dense                                        |
| | qwen/qwen3.5-35b-a3b                      | Qwen 3.5, 35B MoE (3B active)                              |
| | qwen/qwen3.5-397b-a17b                    | Qwen 3.5, 397B MoE (17B active)                            |
| | qwen/qwen3.5-plus-02-15                   | Qwen 3.5 Plus (Feb 2026)                                   |
| | qwen/qwen3.5-flash-02-23                  | Qwen 3.5 Flash (Feb 2026)                                  |
| | qwen/qwen3.5-9b                           | Qwen 3.5 9B                                                |
| | ~~x-ai/grok-4~~                           | ~~Grok 4; multimodal~~ (legacy)                            |
| | meta-llama/llama-4-scout                  | Llama 4 Scout                                              |
| | stepfun/step-3.7-flash                    | StepFun Step 3.7 Flash                                     |
| | moonshotai/kimi-k3                        | Kimi K3                                                    |
| **sciCORE** | ~~GLM-4.5V-FP8~~                              | ~~GLM-4.5V, FP8 quantization; multimodal (Univ. of Basel HPC)~~ (legacy) |
| | ~~qwen3-235b-fp8~~                            | ~~Qwen3 235B, FP8 quantization (Univ. of Basel HPC)~~ (legacy)          |
| | qwen35-397b-a17b-fp8                      | Qwen3.5 397B-A17B, FP8 quantization (Univ. of Basel HPC)   |
| **xAI** | grok-4.20-0309-reasoning                  | Grok 4.20; reasoning                                       |
| | grok-4.3                                  | Grok 4.3                                                   |
| | grok-4.5                                  | Grok 4.5; multimodal                                       |

**Note:** OpenRouter provides access to models from multiple providers through a unified API. sciCORE provides access to models hosted on the University of Basel's high-performance computing infrastructure.

## 5. Benchmarking Methodology

### 5.1. Ground Truth
In this benchmark suite, a model's output for a task is compared to the ground truth (gold standard) for that task given the same input. Ground truth is the correct or verified output created by domain experts.

When selecting ground truth samples, we ensure:
- They are representative of the overall dataset
- They cover various edge cases and scenarios relevant to humanities tasks
- The sample size is large enough to achieve statistical significance

### 5.2. Metrics
We use two categories of metrics to evaluate model performance:

#### 5.2.1. Internal Metrics (Task Performance)
These metrics evaluate how well the model performs the specific task. Examples include:

- **F1 Score**: The harmonic mean of precision and recall, balancing both metrics
- **Precision**: The ratio of correctly predicted positive observations to all predicted positives
- **Recall**: The ratio of correctly predicted positive observations to all actual positives
- **Character/Word Error Rate**: Used for evaluating text generation and transcription accuracy

#### 5.2.2. External Metrics (Practical Considerations)
These metrics evaluate factors beyond task performance that impact usability:

- **Compute Cost**: Automatically tracked based on token usage and date-based pricing data (`scripts/data/pricing.json`). Each run includes cost breakdown and historical pricing via Wayback Machine snapshots.
  - **Cost per Performance Point**: Efficiency metric ($/performance point) calculated per test, averaged per benchmark, then globally. Uses multi-level normalization for fair comparison across different test configurations and benchmark scales.
- **Test Time**: Automatically tracked for each API call.
  - **Time per Performance Point**: Efficiency metric (seconds/point per item) using the same multi-level normalization as cost calculation for fair comparison across different item counts and benchmark complexities.
- **Deployment Options**: Whether the model can be run locally or requires API calls
- **Legal and Ethical Considerations**: Including data privacy, IP compliance, and model bias

## 6. Project Status

### 6.1. Current Limitations

The benchmark suite currently has several limitations that could be addressed in future iterations:

| Category | Limitation | Description |
|----------|------------|-------------|
| **Models** | Local/self-hosted models | Local backends exist for vision tasks (see [2.6.](#26-run-a-local-model)), but no locally run generative LLMs are covered yet |
| **Capabilities** | Domain-specific fine-tuned models | Models specifically optimized for historical research not included |
| | OCR-specialized models | Models with particular strength in document processing/OCR not included |
| | Multilingual capabilities | Systematic testing across different languages not covered |
| **Benchmark Coverage** | Limited benchmark diversity | Currently focused on document analysis; missing art history, archaeology, musicology domains |
| | Language coverage | Primarily German and English; limited coverage of other European languages and non-Western scripts |
| | Historical period coverage | Concentrated on 19th-20th century; limited medieval, early modern, or contemporary sources |
| **Evaluation** | Context window testing | Evaluation across different context window sizes and document lengths not implemented |
| | Standardized error analysis | More granular error categorization and failure mode analysis needed |

### 6.2. Outlook
TODO

## 7. Contributors

This project is developed by a multidisciplinary team at the University of Basel's RISE (Research and Infrastructure Support).

| Name                        | GitHub                                                 | ORCID                                                        |
|-----------------------------|--------------------------------------------------------|--------------------------------------------------------------|
| Anthea Alberto              | [@antheajeanne](https://github.com/antheajeanne)       | [0009-0007-0430-0050](https://orcid.org/0009-0007-0430-0050) |
| Alexandra Binnenkade        | —                                                      | [0000-0002-9518-7212](https://orcid.org/0000-0002-9518-7212) |
| Arno Bosse                  | [@kintopp](https://github.com/kintopp)                 | [0000-0003-3681-1289](https://orcid.org/0000-0003-3681-1289) |
| Sven Burkhardt              | [@Sveburk](https://github.com/Sveburk)                 | [0009-0001-4954-4426](https://orcid.org/0009-0001-4954-4426) |
| Eric Decker                 | [@edecker](https://github.com/edecker)                 | [0000-0003-3035-2413](https://orcid.org/0000-0003-3035-2413) |
| Pema Frick                  | [@pwmff](https://github.com/pwmff)                     | [0000-0002-8733-7161](https://orcid.org/0000-0002-8733-7161) |
| Maximilian Hindermann       | [@MHindermann](https://github.com/MHindermann)         | [0000-0002-9337-4655](https://orcid.org/0000-0002-9337-4655) |
| Lea Kasper                  | [@lekasp](https://github.com/lekasp)                   | [0000-0002-4671-1700](https://orcid.org/0000-0002-4671-1700) |
| Ana-Maria Leonte            | —                                                      | [0009-0005-4678-2807](https://orcid.org/0009-0005-4678-2807) |
| José Luis Losada Palenzuela | [@editio](https://github.com/editio)                   | [0000-0002-6530-1328](https://orcid.org/0000-0002-6530-1328) |
| Sven Lienhard               | —                                                      | [0009-0005-9981-4286](https://orcid.org/0009-0005-9981-4286) |
| Sorin Marti                 | [@sorinmarti](https://github.com/sorinmarti)           | [0000-0002-9541-1202](https://orcid.org/0000-0002-9541-1202) |
| Gabriel Müller              | [@gbmllr1](https://github.com/gbmllr1)                 | [0000-0001-8320-5148](https://orcid.org/0000-0001-8320-5148) |
| Ina Serif                   | [@wissen-ist-acht](https://github.com/wissen-ist-acht) | [0000-0003-2419-4252](https://orcid.org/0000-0003-2419-4252) |
| Elena Spadini               | [@elespdn](https://github.com/elespdn)                 | [0000-0002-4522-2833](https://orcid.org/0000-0002-4522-2833) |
| Tabea Wullschleger          | [@tabea-w](https://github.com/tabea-w) | [0000-0001-9841-0005](https://orcid.org/0000-0001-9841-0005) |
| Franziska Zúñiga            | —                                                      | [0000-0002-8844-4903](https://orcid.org/0000-0002-8844-4903) |

For detailed attribution by benchmark and contribution type, see our [CONTRIBUTORS.md](CONTRIBUTORS.md) file.

humanities_data_benchmark

humanities_data_benchmark 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

humanities_data_benchmark

humanities_data_benchmark 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README