KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #codex #typescript #developer-tools #ai-agents #deepseek-harness #claude-code #ai-agent #cli #react #mcp

Counting-Stars

★ 83 Open GitHub ↗

Counting-Stars (★)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
  <h2>Counting-Stars (★): A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Models</h1>
</div>

In this work, we propose **a multi-evidence, position-aware, and scalable benchmark** for evaluating long-context LLMs, named **Counting-Stars**, which evaluates long-context LLMs by using two tasks: multi-evidence acquisition and multi-evidence reasoning. 
- **Multi-evidence**: *Counting-Stars is the most evidence-intensive evaluation in known long-context benchmarks*.
- **Position-aware**: *The position of the evidence in the context can be adjusted as desired and tested in a targeted manner*.
- **Scalable**: *Both the context length and the amount of evidence can be expanded arbitrarily*.

Based on the Counting-Stars test, we conduct experiments to evaluate long-context LLMs (i.e., GPT-4 Turbo, Gemini 1.5 Pro, Claude3 Opus, GLM-4, and Moonshot-v1). Experimental results demonstrate that Gemini 1.5 Pro achieves the best overall results, while the performance of GPT-4 Turbo is the most stable across various tasks. Furthermore, our analysis of these LLMs, which are extended to handle long-context scenarios, indicates that there is potential for improvement as the length of the input context and the intricacy of the tasks are increasing.

> Please find more details of this work in the [paper](https://arxiv.org/pdf/2403.11802).
## Note

We'd like to encourage you to test the Counting-Stars using
- Me-Acq. (EN) means the English version of Multi-evidence Acquisition in the Counting-Stars.
  - ```Counting_Stars_EN_acquisition_128000_32_32.jsonl```
- Me-Acq. (ZH) means the Chinese version of Multi-evidence Acquisition in the Counting-Stars.
  - ```Counting_Stars_ZH_acquisition_128000_32_32.jsonl```
- Me-Rea. (EN) means the English version of Multi-evidence Reasoning in the Counting-Stars.
  - ```Counting_Stars_EN_reasoning_128000_32_32.jsonl```
- Me-Rea. (ZH) means the Chinese version of Multi-evidence Reasoning in the Counting-Stars.
  - ```Counting_Stars_ZH_reasoning_128000_32_32.jsonl```

, the 128K English and Chinese versions of the Counting-Stars.

|Rank|Models|Claimed Length|Me-Acq.(ZH)|Me-Acq.(EN)|Me-Rea.(ZH)|Me-Rea.(EN)|Avg.|
|----|----|----|----|----|----|----|----|
|1| Gemini 1.5 Pro|1M|0.775|0.833|0.575|0.371|0.639|
|2| GPT-4 Turbo (1106)|128K|0.697|0.718|0.473|0.651|0.635|
|3| Claude3 Opus|200K|0.807|0.705|0.488|0.374|0.594|
|4| GPT-4 Turbo (0125)|128K|0.663|0.662|0.386|0.610|0.580|
|5| Moonshot-v1|200K|0.606|0.559|0.344|0.460|0.492|
|6| GLM-4|128K|0.682|0.389|0.475|0.179|0.431|
|-| Claude3 Sonnet|200K|0.788|-|-|-|-|
|-| Claude3 Haiku|200K|0.698|-|-|-|-|
|-| Baichuan3-Turbo|128K|0.759|0.490|-|-|-|

## Task Description

## Evaluation Results

> Visualization of the results on the Chinese version of the Counting-Stars-32-(Multi-evidence Acquisition).

> Visualization of the results on the Chinese version of the Counting-Stars-32-(Multi-evidence Reasoning).

## Cite
If you find our work helpful, feel free to give us a cite.

```
@inproceedings{song-etal-2025-counting,
    title = "Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Models",
    author = "Song, Mingyang and Zheng, Mao and Luo, Xuan",
    booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
    year = "2025",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.coling-main.253",
    pages = "3753--3763"
}
```

## CONTACT
For any questions, feel free to create an issue, and we will try our best to solve it. \
**If the problem is more urgent**, you can email me simultaneously (I check email almost daily).
```
NAME: Mingyang Song
EMAIL: nickmysong@tencent.com
```
Our visualization code is built on the source code from [NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack). Thanks for their work.

Counting-Stars

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README