ReVisiT

KO | EN

gitlite — search

#python #docker #nodejs #javascript #java #mysql #golang #cli #vue #vuejs #html #express

★ 26 Open GitHub ↗

[ACL 2026 Main] Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center"> Revisit What You See: Revealing Visual Semantics in <br>Vision Tokens to Guide LVLM Decoding
</h1>

[![arXiv](https://img.shields.io/badge/arXiv%20paper-2506.09522-b31b1b.svg)](https://arxiv.org/abs/2506.09522)&nbsp;
[![huggingface](https://img.shields.io/badge/huggingface-2506.09522-yellow.svg)](https://huggingface.co/papers/2506.09522)&nbsp;
[![LICENSE](https://img.shields.io/badge/License-Apache%202.0-g.svg)](https://raw.githubusercontent.com/bscho333/ReVisiT/main/LICENSE)

<div align="center">
  <a href="https://bscho333.notion.site/" target="_blank">Beomsik&nbsp;Cho</a><sup>1</sup> &ensp; <b>&middot;</b> &ensp;
  <a href="https://sites.google.com/view/jaehyungkim" target="_blank">Jaehyung&nbsp;Kim</a><sup>1</sup> &ensp; <br>
  <sup>1</sup> Yonsei University &emsp; <br>
</div>
<br>

## Overview
  <p align="center">
    <img src="figs/overview.png"/>
  </p>

**ReVisiT** is a decoding-time algorithm for LVLMs that guides text generation by **Re**ferencing **Visi**on **T**okens. It projects vision tokens into the text token space, selects the most relevant one through constrained divergence minimization, and guides generation to better align with visual semantics without modifying the underlying model.

## Implementation

Due to differences in the supported Transformers versions for each LVLM family, we provide separate implementations for LLaVA-1.5.
LLaVA-1.5 is based on Transformers v4.31.0, while Qwen2.5-VL and InternVL3 are based on v4.50.0, reflecting compatibility requirements with their respective tokenizer and model wrappers.  
Although the core ReVisiT decoding logic remains the same, these version-specific dependencies necessitate isolated environments and tailored integration scripts per model.

If you want to integrate ReVisiT into your own environment, simply add the corresponding decoding function to the Huggingface Transformers source code.
Specifically, copy the code from:
- For v4.31.0: [`src/transformers-v4.31.0/src/transformers/generation/utils.py` (lines 2473–2852)](https://github.com/bscho333/ReVisiT/blob/main/src/transformers-v4.31.0/src/transformers/generation/utils.py#L2473-L2852)
- For v4.50.0: [`src/transformers-v4.50.0/src/transformers/generation/utils.py` (lines 2579–2830)](https://github.com/bscho333/ReVisiT/blob/main/src/transformers-v4.50.0/src/transformers/generation/utils.py#L2579-L2830)

and paste it into your local `transformers/generation/utils.py`.

The following section provides CHAIR evaluation scripts and instructions for each model.

### Environment setup

**LLaVA1.5 (transformers==4.31.0)**
  ```bash
  conda env create -f prerequisites/ReVisiT_LLaVA.yaml
  conda activate revisit_llava
  pip install numpy==1.26.4
  cd src/transformers-v4.31.0
  pip install -e .
  cd ../..
  ```

**Qwen2.5-VL & InternVL3 (transformers==4.50.0)**
  ```bash
  conda create -n revisit python=3.9.21
  conda activate revisit
  pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu118
  conda env update -f prerequisites/ReVisiT.yaml
  cd src/transformers-v4.50.0
  pip install -e .
  cd ../..
  ```

### Download model weights & dataset
  **Download MSCOCO**
  ```bash
  bash prerequisites/download_coco.sh
  ```
  **LLaVA-1.5**
  ```bash
  conda activate revisit_llava
  python prerequisites/download_from_huggingface.py --llava
  ```
  **Qwen2.5-VL**
  ```bash
  conda activate revisit
  python prerequisites/download_from_huggingface.py --qwenvl
  ```
  **InternVL3**
  ```bash
  conda activate revisit
  python prerequisites/download_from_huggingface.py --internvl
  ```

### CHAIR Evaluation
  **LLaVA-1.5**
  ```bash
  conda activate revisit_llava
  bash eval_chair_llava.sh
  ```
  **Qwen2.5-VL**
  ```bash
  conda activate revisit
  bash eval_chair_qwenvl.sh
  ```
  **InternVL3**
  ```bash
  conda activate revisit
  bash eval_chair_internvl.sh
  ```
## Acknowledgements
This repository builds upon the open-source implementations of [LLaVA](https://github.com/haotian-liu/LLaVA), [QwenVL](https://huggingface.co/Qwen), [InternVL](https://github.com/OpenGVLab/InternVL), [VCD](https://github.com/DAMO-NLP-SG/VCD), and [RITUAL](https://github.com/sangminwoo/RITUAL).  
We sincerely thank the authors for making their code publicly available.

## Citation
If you find our work helpful, please consider citing:

```bibtex
@article{cho2025revisit,
  title     = {Revisit What You See: Disclose Language Prior in Vision Tokens for Efficient Guided Decoding of LVLMs},
  author    = {Beomsik Cho and Jaehyung Kim},
  journal   = {arXiv preprint arXiv:2506.09522},
  year      = {2025},
  url       = {https://arxiv.org/abs/2506.09522}
}
```

ReVisiT

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README