KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #cli #claude-code #codex #developer-tools #dsh-plugin #deepseek-harness #windows #open-source #agent-skills #ai

Video-Holmes

★ 97 Open GitHub ↗

[ECCV 2026] Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?

**[Junhao Cheng1,2](https://donahowe.github.io/), 
[Yuying Ge1,&#9993;](https://geyuying.github.io/), 
[Teng Wang1,&#9993;](http://ttengwang.com/), 
[Yixiao Ge1](https://geyixiao.com/), 
[Jing Liao2](https://scholar.google.com/citations?user=3s9f9VIAAAAJ&hl=en), 
[Ying Shan1](https://scholar.google.com/citations?user=4oXBp9UAAAAJ&hl=en)**
 
1ARC Lab, Tencent PCG, 
2City University of Hong Kong

## 🔎 Introduction

Video-Holmes is a benchmark designed to evaluate the complex video reasoning capabilities of MLLMs.

Video-Holmes consists of 1,837 questions derived from 270 manually annotated suspense short films (ranging from 1 to 5 minutes), which spans seven carefully designed tasks. Each task is constructed by first identifying key events and causal relationships within films, and then designing questions that require models to actively locate and connect multiple relevant visual clues scattered across different video segments.

⭐ Key Aspects of Video-Holmes:

<ul style="list-style-type: disc; padding-left: 20px;">
<li>One-Click Evaluation: Videos, questions, and evaluation codes are packaged on GitHub and <a href="https://huggingface.co/datasets/TencentARC/Video-Holmes" target="_blank">Huggingface</a>.</li>
<li>High Reasoning Demand: Significant performance gap between reasoning models and non-reasoning models.</li>
<li>Reasoning Process Analysis: Clearly visualizes the reasons behind correct and incorrect model responses.</li>
</ul>

We aim that Video-Holmes can serve as a "Holmes-test" for multimodal reasoning, motivating models to reason more like humans and emphasizing the ongoing challenges in this field. Please visit our [homepage](https://video-holmes.github.io/Page.github.io/) for more details!

## 📅 News

* [2025-05-29] 🔥We released the training set of Video-Holmes, which consists of 233 videos and 1,551 questions.
* [2025-05-28] 🔥We released Video-Holmes and corresponding evaluation codes.

## 🚩 Plan
- [x] Release suspense short film annotations
- [x] Release benchmark construction codes
- [x] Release training data
- [x] Support evaluation from [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)
## 🏆 Leaderboard
🏅 Best performance model: [Gemini-2.5-Pro](https://gemini.google.com/)

🏅 Best thinking model based on Qwen2.5-VL-7B: [Video-R1](https://github.com/tulerfeng/Video-R1)

➡️ [Full leaderboard](https://video-holmes.github.io/Page.github.io#leaderboard)

> Welcome to contact us at Howe4884@outlook.com to upload your model to the leaderboard.

## 🚀 Quick Start

To download Video-Holmes, you can run the following commands:
```shell
git clone https://github.com/TencentARC/Video-Holmes.git
cd Video-Holmes
pip install huggingface_hub
python download.py --hf_token YOUR_HUGGINGFACE_ACCESS_TOKEN
unzip Benchmark/videos.zip -d Benchmark/
unzip Benchmark/annotations.zip -d Benchmark/
```

We provide all-in-one evaluation codes for baseline models:
```shell
python evaluate.py --model_name YOUR_MODEL_NAME --model_path YOUR_MODEL_PATH (optional)
```

Supported Model List:

| QwenVL | QwenVL-RL | InternVL | Gemini |
|----------------|----------------|----------------|----------------|
| Qwen2.5-VL-7B  | VideoChat-R1  | InternVL2.5-8B | gemini-2.0-flash |
| Qwen2.5-VL-32B | Video-R1  | InternVL3-8B | gemini-2.0-pro-exp |

You can also customize your model by specifying the `--model_path` argument, or by implementing the following functions: `prepare_your_model` (line 388) and `generate_your_model` (line 439).

<details>
<summary>🧐 Reasoning Process Analysis</summary>
 
You first need to apply a [DeepSeek API key](https://platform.deepseek.com/api_keys) and then you can run the following commands to analyze the reasoning process of your models:

```shell
python evaluate_reasoning.py --model_name YOUR_MODEL_NAME --api_key YOUR_API_KEY
```

</details>

<details>
<summary>🪄 Generate Your Holmes-Test</summary>
 
To generate questions for your videos with annotations, you can run the following commands:

```shell
cd Pipeline
python generate_questions.py --api_key YOUR_API_KEY
```

> Note: You can down load the video on YouTube according to the `VIDEO_ID` by `https://www.youtube.com/watch?v=VIDEO_ID`
</details>

## 🧠 Training (𝓃𝑒𝓌🔥)

We release the [training set](https://huggingface.co/datasets/TencentARC/Video-Holmes/blob/main/train_Video-Holmes.json) of Video-Holmes, which consists of 233 videos and 1,551 questions. Experimental results (as shown in the table below) demonstrate that performing RL post-training on this training set can further enhance the model's complex reasoning ability.

| Model | SR |        IMC        | TCI        | TA        |        MHR        |        PAR        |        CTI        |        Avg |
|---|----|----|----|----|----|----|----|----|
| Qwen2.5-VL-7B  | 38.4|        34.8|        17.6|        30.0|        27.1|        18.6|        25.2|        27.8|
| [Qwen2.5-VL-7B-GRPO-CARE](https://github.com/TencentARC/GRPO-CARE) |42.8|        35.1|        25.6|        40.5|        29.2|        29.9|        32.6|  33.5 |
| [Qwen2.5-VL-7B-GRPO-CARE*](https://github.com/TencentARC/GRPO-CARE) | **46.2** | **44.9** |**31.5** |**49.5** |**39.2** |**37.1** |**37.4**|**40.7** |

> \* denotes models training on Video-Holmes.

## 🛠️ Construction Pipeline

We select 270 high-quality suspense short films for human annotation. Next, we design 7 challenging tasks and employ DeepSeek to generate questions. Finally, we evaluate SOTA MLLMs and use DeepSeek to analyze their responses (optional).
<img src="assets/pipeline.png" alt="Teaser Image" style="width: 100%; height: auto;">

## 🗝️ Question Types

Existing benchmarks primarily involve clue-given questions, where models depend on explicitly provided clues to derive answers. In contrast, Video-Holmes adopts an active seeking paradigm, requiring models to actively locate and connect multiple relevant visual clues scattered across different video segments.
<img src="assets/Teaser2.png" alt="Teaser Image" style="width: 100%; height: auto;">

## :closed_book: License
- Video-Holmes is released under the Apache-2.0 license for academic purpose only.
- All videos of the Video-Holmes are obtained from the Internet which are not property of our institutions. Our institution are not responsible for the content nor the meaning of these videos. The copyright remains with the original owners of the video.
- If any video in our dataset infringes upon your rights, please contact us for removal.
  
## 📜 Citation

If you find our work helpful, please consider giving a star ⭐ and citation 📝

```BibTeXw
@article{cheng2025video,
  title={Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?},
  author={Cheng, Junhao and Ge, Yuying and Wang, Teng and Ge, Yixiao and Liao, Jing and Shan, Ying},
  journal={arXiv preprint arXiv:2505.21374},
  year={2025}
}
```

## 🤗 Acknowledgements

We refer to [MovieDreamer](https://github.com/aim-uofa/MovieDreamer) and [VCR-Bench](https://github.com/zhishuifeiqian/VCR-Bench) to build our codebase and homepage. Thanks for their wonderful project.

Video-Holmes

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README