CrossLMM

KO | EN

gitlite — search

#javascript #hacktoberfest #typescript #llm #go #golang #linux #docker #nodejs #awesome #kubernetes #pytorch #security #openai

(★ 25)

CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanisms

CrossLMM 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
<br>
<h3>CrossLMM: Decoupling Long Video Sequences from LMMs via <br> Dual Cross-Attention Mechanisms</h3>

[Shilin Yan](https://scholar.google.com/citations?user=2VhjOykAAAAJ&hl=zh-CN&oi=ao)<sup>1†</sup>, [Jiaming Han](https://scholar.google.com/citations?user=vgcxKEcAAAAJ&hl=en&oi=ao)<sup>2</sup>, [Joey Tsai]()<sup>3</sup>, [Hongwei Xue](https://scholar.google.com/citations?user=k5CJa5YAAAAJ&hl=en&oi=ao)<sup>1</sup>, [Rongyao Fang](https://scholar.google.com/citations?user=FtH3CW4AAAAJ&hl=en&oi=ao)<sup>2</sup>, <br> [Lingyi Hong](https://scholar.google.com/citations?user=wHh_m_IAAAAJ&hl=en&oi=ao)<sup></sup>, [Ziyu Guo](https://scholar.google.com/citations?user=a6ZGNTEAAAAJ&hl=en&oi=ao)<sup>2</sup>, [Ray Zhang](https://scholar.google.com/citations?user=YlL3xN4AAAAJ&hl=en&oi=ao)<sup>2‡</sup>

<sup>1</sup>Accio Team, Alibaba Group <sup>2</sup>CUHK MMLab <sup>3</sup>Tsinghua University

<div class="is-size-6 publication-authors">
  <p class="footnote">
    <span class="footnote-symbol"><sup>†</sup></span>Project Leader
    <span class="footnote-symbol"><sup>‡</sup></span>Corresponding author
  </p>
</div>

<div align="center">
  <img src="figures/teaser.jpg" width="100%" alt="CrossLMM Framework" />
</div>
<hr>
<div align="center" style="line-height: 1.2;">
  <a href="https://arxiv.org/pdf/2505.17020" target="_blank"><b>Paper</b></a> •
  <a href="#-introduction">Introduction</a> •
  <a href="#-model">Model</a> •


</div>

## 🔥 News
* [2025-05-23]🔥🔥🔥 We release the [paper](https://arxiv.org/pdf/2505.17020)

## 🧠 Introduction

We present  **CrossLMM**, decoupling long video sequences from LMMs via a dual cross-attention mechanism, which substantially reduces visual token quantity with minimal performance degradation. 
Specifically, we first implement a significant token reduction from pretrained visual encoders through a pooling methodology.
Then, within LLM layers, we employ a visual-to-visual cross-attention mechanism, wherein the pooled visual tokens function as queries against the original visual token set. This module enables more efficient token utilization while retaining fine-grained informational fidelity. 
In addition, we introduce a text-to-visual cross-attention mechanism, for which the text tokens are enhanced through interaction with the original visual tokens, enriching the visual comprehension of the text tokens.

## 👀 Model

---

### 🚩 Main Innovations

#### 1. 🌟 **Token Reduction via Pooling**
- Significantly compress the number of tokens from pretrained visual encoders for efficient representation.
- Apply a simple pooling strategy to **retain critical visual information** while reducing token count.

#### 2. 🚀 **Visual-to-Visual Cross-Attention**
- Novel architecture design: **Pooled visual tokens act as queries** attending over the original visual token set.
- Enables the model to **capture fine-grained visual details**, maintaining fidelity even under strong token compression.

#### 3. 🔮 **Text-to-Visual Cross-Attention**
- Enhances text token representations through interaction with the original visual tokens.
- Deepens text-visual alignment, offering **richer contextual understanding** for multimodal downstream tasks.

---

### 🔗 **Framework Benefits**

- The **dual attention mechanism** maximizes model efficiency while preserving the ability to handle long-form video content.
- Achieves a strong balance between computational efficiency and fine-grained multimodal understanding, empowering advanced video-language applications.

---

> _This architecture enables efficient and scalable video-text modeling while maintaining state-of-the-art accuracy._

## 🥳 Acknowledgements
We would like to thank [LLAVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT/tree/main), upon which our repo is built.

## 📄 Cite
```
@article{yan2025crosslmm,
  title={CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanisms},
  author={Yan, Shilin and Han, Jiaming and Tsai, Joey and Xue, Hongwei and Fang, Rongyao and Hong, Lingyi and Guo, Ziyu and Zhang, Ray},
  journal={arXiv preprint arXiv:2505.17020},
  year={2025}
}
```

## 📧 Concat
If you have any question about this project, please feel free to contact tattoo.ysl@gmail.com.

CrossLMM

CrossLMM 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

CrossLMM

CrossLMM 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README