KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #typescript #ai-agent #developer-tools #claude-code #codex #deepseek-harness #ai-agents #react #mcp #rust

HoPE

★ 29 Open GitHub ↗

[NeurIPS 2025] HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h2 align="center" style="font-size: 30px;">HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models</h2>

<h5 align="center">
  
[![Paper](https://img.shields.io/badge/paper-A42C25?style=for-the-badge&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2505.20444) 
[![Hugging Face Collection](https://img.shields.io/badge/HuggingFace-fcd022?style=for-the-badge&logo=huggingface&logoColor=000)](https://huggingface.co/papers/2505.20444)
[![X](https://img.shields.io/badge/X-000000?style=for-the-badge&logo=X&logoColor=white)](https://x.com/HaoranLi_/status/1994940409257365733?s=20)

</h5>

## 📢 News
- **\[09/18/2025\]** 🎉🔥 Our paper is accepted by **NeurIPS 2025**. See you in SD!
- **\[07/13/2025\]** All training and evaluation scripts are released. Check it out!
- **\[06/29/2025\]** Our work is covered by [JIQIZHIXIN (机器之心)](https://mp.weixin.qq.com/s/KQHGw8_v0rEY8pS7jufRbQ)!
- **\[05/26/2025\]** Release our paper on [arXiv](https://arxiv.org/abs/2505.20444).

## 🔭 Overview

Extending RoPE to multimodal scenarios typically involves allocating different frequencies to encode different positional components (*t*, *x*, *y*). In this paper:

1️⃣ We first investigate **how different frequency allocation strategies impact the semantic modeling capabilities of VLMs**. Our analysis reveals that current multimodal RoPEs, which keep all frequencies, are unreliable in long-term semantic modeling. HoPE tackles this issue by **Hybrid Frequency Allocation (HFA)**, which integrates *zero frequencies* for reliable semantic modeling over extended contexts.

2️⃣ Moreover, we point out that existing temporal index scaling of visual tokens lacks flexibility and robustness during inference, where videos exhibit varying speeds and information densities. To address this, HoPE introduces **Dynamic Temporal Scaling (DTS)**, which enables VLMs to learn multi-scale temporal relationships during training and adaptively select temporal scaling during inference.

## 🛠️ Requirements
1. Clone this repository and install `transformers==4.45.2` from source
```
git clone https://github.com/hrlics/HoPE.git
cd HoPE
wget https://github.com/huggingface/transformers/archive/refs/tags/v4.45.2.tar.gz
tar -xzf v4.45.2.tar.gz
```

2. Install required packages
```
bash setup_env.sh
```

3. Replace the code in
```
HoPE/transformers-4.45.2/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py
```
with `HoPE/modeling_hope.py`. The differences are marked with `# MODIFIED`.

## 🚀 Train

We utilize a subset of [LLaVA-Video-178K](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K) as training data, which comprises 30k videos with durations under 2 minutes and 3k videos with durations between 2 to 3 minuts (~300k pairs).

Under `LLaMA-Factory/`, run the following script to start training:
```
train_hope.sh
```
Adjustments are made to `LLaMA-Factory/src/llamafactory/data/mm_plugin.py` to accomodate Qwen2-VL's training recipe.

## 🔍 Evaluation

#### Long Video Understanding

Evaluations on long video understanding are based on [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval). The first step is to install relevant dependencies: 
```
cd lmms-eval
pip install -e .
```

Then, run the following script to start evaluations on MLVU, LongVideoBench, and Video-MME:
```
bash eval_LVU.sh
```

Adjustments are made to `lmms-eval/lmms_eval/models/qwen2_vl.py` to accomodate our evaluation configs.

#### Long Video Retrieval

Under `vision_niah/`, run the following script to produce haystack and needle embeddings for long video retrieval:
```
bash produce_haystack_and_needle_embedding.sh
```

Now, we can run evaluations:
```
bash eval.sh
```

## 📖 Citation
If you find our work helpful, please consider citing 📝 and giving us a star ⭐
```
@article{li2025hope,
  title={HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models},
  author={Li, Haoran and Qin, Yingjie and Ou, Baoyuan and Xu, Lai and Xu, Ruiwen},
  journal={arXiv preprint arXiv:2505.20444},
  year={2025}
}
```

HoPE

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README