KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #ai-agent #claude-code #mcp #typescript #codex #cli #rust #deepseek-harness

JiuTian-LION

★ 154 Open GitHub ↗

[CVPR 2024] LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h2 class="papername"> <img src="./assets/LION_logo.png" style="vertical-align: middle; height: 1em; padding: 0 0.2em;"> LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge </h2>
<div>
<div>
    <a href="https://scholar.google.com/citations?user=Mpg0w3cAAAAJ" target="_blank">Gongwei Chen</a>,
    <a href="https://www.slywiki.cn/" target="_blank">Leyang Shen</a>,
    <a href="https://rshaojimmy.github.io/" target="_blank">Rui Shao*</a>,
    <a href="https://xiang-deng-dl.github.io/" target="_blank">Xiang Deng</a>,
    <a href="https://liqiangnie.github.io/" target="_blank">Liqiang Nie*</a>
</div>

School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen<br>
*Corresponding author

IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2024

[[Paper]](https://arxiv.org/abs/2311.11860) [[Project Page]](https://rshaojimmy.github.io/Projects/JiuTian-LION) [[Video(YouTube)]](https://www.youtube.com/watch?v=YzJ5MZFS5RA) [[Video(bilibili)]](https://www.bilibili.com/video/BV1kH4y1y7UR/)

:fire: Details will be released. Stay tuned :beers: :+1:

[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fwww.slywiki.cn&count_bg=%2379C83D&title_bg=%23555555&icon=github.svg&icon_color=%23E7E7E7&title=Visitors&edge_flat=false)](https://hits.seeyoufarm.com)

</div>
<br>
  
<img src='assets/LION-Introduction.jpg' width='90%'>

</div>

## If you find this work useful for your research, please kindly cite our paper and star our repo.

## Updates
- [07/2024] Code and checkpoints are released.
- [02/2024] LION has been accepted by CVPR 2024.
- [11/2023] [Arxiv paper](https://arxiv.org/abs/2311.11860) released.
- [11/2023] [Project page](https://rshaojimmy.github.io/Projects/JiuTian-LION) released.

## Introduction

This is the github repository of *LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge*. In this work, we enhance MLLMs by integrating fine-grained spatial-aware visual knowledge and high-level semantic visual evidence, boosting capabilities and alleviating hallucinations.

The framework of the proposed LION model:

## Installation

### Download
```bash
git clone https://github.com/JiuTian-VL/JiuTian-LION.git
cd JiuTian-LION
```

### Environment

```bash
conda create -n LION python=3.12
conda activate LION
conda install pip
pip install -r requirements.txt
```

## Checkpoints

| Version | Checkpoint |
| --- | --- |
| LION-FlanT5-XL| [daybreaksly/LION-FlanT5-XL](https://huggingface.co/daybreaksly/LION-FlanT5-XL) |
| LION-FlanT5-XXL| [daybreaksly/LION-FlanT5-XXL](https://huggingface.co/daybreaksly/LION-FlanT5-XXL) |

## Usage

### Prepare models

1. Download the pre-trained vit model [eva_vit_g](https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/eva_vit_g.pth).
2. Download the pre-trained RAM model [ram_swin_large_14m](https://huggingface.co/spaces/xinyu1205/Recognize_Anything-Tag2Text/blob/main/ram_swin_large_14m.pth).
3. Download the pre-trained FlanT5 model [FlanT5-XL](https://huggingface.co/google/flan-t5-xl).
4. Download the pre-trained BERT model [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased)
5. Fill in the paths to these models into the corresponding locations in the config file `configs\models\lion_flant5xl.yaml`

### Inference

We provide inference examples for **Image-Level** and **Region-Level** tasks in `playground.ipynb`.

### Training

We provide a training script and instruction to do stage4 training as an example.

- Download dataset from [huggingface](https://huggingface.co/datasets/daybreaksly/LION-data-train)
- Download images and organized them in one folder:

Please download the following datasets:

- **Training images**
  - `OCR-VQA`
  - `coco-2014`
  - `coco-2017`
  - `okvqa-2014`
  - `textcaps`
  - `vqav2-2014`
  - `visual_genome`

After downloading, place all these folders under a **single directory**.  
For example:

```bash
/path/to/data/images/
├── OCR-VQA/images
├── coco/images/train2014
├── coco_2017/train2017
├── okvqa/images/train/train2014
├── textcaps/images/train_images
├── vqav2/images/train2014
├── visual_genome/VG_100K
└── visual_genome/VG_100K_2

---
```
In your config file, add the unified image folder path:

```yaml
train_datasets:
  - ann_path: "/path/to/image_level_data.json"
    vis_root: "/path/to/image_folder"
    is_train: true
    sample_ratio: 1
  - ann_path: "/path/to/region_level_data.json"
    vis_root: "/path/to/image_folder"
    is_train: true
    sample_ratio: 1
```

- Configure training with `configs/lion_train_stage4.yaml` (update model paths and dataset paths)
- Run multi‑GPU training:

```
cd JiuTian-LION
bash scripts/start_train.sh
```

Or manually:

```
CUDA_VISIBLE_DEVICES=0,1,2,3 TOKENIZERS_PARALLELISM=true \
  torchrun --master_port 12345 --nproc_per_node=4 \
  train.py --cfg-path configs/lion_train_stage4.yaml
```

Outputs and checkpoints are written to `outputs/lion_stage4/<timestamp>/` by default.

## Evaluation results

For <b>image-level</b> tasks, we focus on image captioning and Visual Question Answering (VQA). For <b>region-level</b> tasks, we evaluate LION on three REC datasets including RefCOCO, RefCOCO+ and RefCOCOg. The results, detailed in Table 1~2, highlight LION's superior performance compared to baseline models.

![Score](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-Score.jpg)

![Image-level](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-Image-level.jpg)
![Region-level](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-Region-level.jpg)

We further evaluate LION on a object hallucination benchmark([POPE](https://github.com/AoiDragon/POPE)) and the most popular MLLM benchmark ([MMBench](https://mmbench.opencompass.org.cn/home)). The results in Table 1~2 show that LION has strong performances across various skills and also demonstrates a strong resistance to hallucinations, particularly in popular and adversarial settings in POPE.

![MMBench](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-MMBench.jpg)
![POPE](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-POPE.jpg)

## Qualitative Comparison

![Qualitative Comparison](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-Examples.jpg)
![Qualitative Comparison](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-CapVQA.jpg)
![Qualitative Comparison](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-REC.jpg)

## More Examples
![Qualitative Comparison](https://raw.githubusercontent.com/JiuTian-VL/JiuTian-LION/main/assets/LION-6Examples.jpg)

## Citation

If you find this work useful for your research, please kindly cite our paper:
```
@inproceedings{chen2024lion,
    title={LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge}, 
    author={Chen, Gongwei and Shen, Leyang and Shao, Rui and Deng, Xiang and Nie, Liqiang},
    booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2024}
}
```

JiuTian-LION

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README