KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

DenseVLM

★ 53 Open GitHub ↗

[ICCV 2025] Unbiased Region-Language Alignment for Open-Vocabulary Dense Prediction

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">Unbiased Region-Language Alignment for Open-Vocabulary Dense Prediction</h1>
 
 <a href="https://lyhisme.github.io/">Yunheng Li</a>
 ·
 <a href="https://zcablii.github.io/">Yuxuan Li</a>
 ·
 <a href="https://github.com/ashun989">Quansheng Zeng</a>
 ·
 <a href="https://whai362.github.io/">Wenhai Wang</a>
 ·
 <a href="https://houqb.github.io/">Qibin Hou†</a>
 ·
 <a href="https://mmcheng.net/cmm/">Ming-Ming Cheng</a>
 
 <h2 align="center">Accepted By ICCV 2025!</h2>

### [[Paper](https://arxiv.org/pdf/2412.06244)] [[Github](https://github.com/HVision-NKU/DenseVLM)] [[Pretrained models](https://github.com/HVision-NKU/DenseVLM/tree/main#)]

## Contributions
- 🔥 We identify the foreground bias issue in existing VLMs and propose region-text alignment by incorporating explicit semantic structuring through category guidance. 
- 🔥 We propose DenseVLM, a region-language alignment framework that uses a strong VLM to retrieve categories for unlabeled regions and decouples foreground and background features to reduce bias. 
- 🔥 Extensive experiments on dense prediction benchmarks show that our DenseVLM outperforms previous methods and exhibits promising scalability.

## Overview

DenseVLM is an annotation-free fine-tuning framework for open-vocabulary dense prediction tasks, which retrieves region-level semantics from a powerful vision-language model and decouples foreground and background features to achieve unbiased region-language alignment and improved open-vocabulary dense prediction.

## TODO
- [x] Release the training and inference code of DenseVLM.
- [x] Supports training and inference code for RegionCLIP and CLIPSelf.
- [ ] Release the code to integrate DenseVLM into CAT-Seg.
- [ ] Release the code to integrate DenseVLM into F-ViT.

## Quick Start
- 🚀 Linux system with CUDA 11.8
- 🚀 At least one RTX 3090 GPU (4 GPUs are default for training ～23min/epoch)

#### 1. Create Conda Environment
- The provided environment is suggested for reproducing our results, similar configurations may also work.

```
git clone git@github.com:HVision-NKU/DenseVLM.git
cd DenseVLM

conda create -n DenseVLM python=3.8.20
conda activate DenseVLM
pip install -r requirements.txt
pip install -e . -v
```

#### 2. Data Preparation
The main experiments are conducted using images from [COCO](https://cocodataset.org/#home) and [ADE20k](http://sceneparsing.csail.mit.edu) datasets. Please prepare datasets and organize them like the 
following:

```text
DenseVLM/
├── data
    ├── coco
        ├── annotations
            ├── instances_train2017.json  
            ├── panoptic_val2017.json
            ├── panoptic_val2017/    
        ├── train2017/
        ├── val2017/
        ├── coco_pseudo_4764.json
        ├── coco_proposals.json     
    ├── ADEChallengeData2016
        ├── ade20k_panoptic_val/
        ├── images/validation/
        ├── ade20k_panoptic_val.json 
```

#### 3. Checkpoints
Please download the pretrained weights from [huggingface](https://huggingface.co/lyhisme/DenseVLM) and organize them like the 
```text
DenseVLM/
├── checkpoints
    ├── EVA02_CLIP_B_psz16_s8B.pt    
    ├── clipself_coco_6_save6_512_eva_vitl14_24layers.pt  
    ├── densevlm_coco_6_save6_512_eva_vib16_12layers.pt
```

If using a fine-tuned CLIP, you can directly use it. For example:

```python
model = open_clip.create_model(
    'EVA02-CLIP-B-16', pretrained='eva', cache_dir='checkpoints/densevlm_coco_6_save6_512_eva_vib16_12layers.pt'
)
```

#### 4. Training and Testing

To fine-tune the CLIP model using densevlm, run:
```bash
bash scripts/train_densevlm_coco_image_patches_eva_vitb16.sh 
```

To evaluate the CLIP model fine-tuned with densevlm, run:
```bash
bash scripts/test_coco_eva_vitb16_macc_boxes_masks.sh path/to/checkpoint.pt 2 densevlm_coco_test

bash scripts/test_ade_eva_vitb16_macc_boxes_masks.sh path/to/checkpoint.pt 2 densevlm_ade_test
```

## 🙏 Citation:
If you find this project useful, please consider citing:

```bibtex
@article{li2024densevlm,
  title={Unbiased Region-Language Alignment for Open-Vocabulary Dense Prediction},
  author={Li, Yunheng and Li, Yuxuan and Zeng, Quansheng and Wang, Wenhai and Hou, Qibin and Cheng, Ming-Ming},
  journal={arXiv preprint arXiv:2412.06244},
  year={2024}
}

@InProceedings{li2024cascadeclip,
  title={Cascade-{CLIP}: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation},
  author={Li, Yunheng and Li, Zhong-Yu and Zeng, Quan-Sheng and Hou, Qibin and Cheng, Ming-Ming},
  booktitle={Proceedings of the 41st International Conference on Machine Learning},
  pages={28243--28258},
  year={2024},
  volume={235},
  month={21--27 Jul},
  publisher={PMLR}
}
```

## License
Licensed under a [Creative Commons Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/) for Non-commercial use only.
Any commercial use should get formal permission first.

DenseVLM

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README