SC-Tune

KO | EN

gitlite — search

#javascript #hacktoberfest #react #llm #machine-learning #rust #linux #docker #macos #java #kubernetes #vue #pytorch #mcp

(★ 16)

Official code for CVPR 2024 paper, "SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models"

SC-Tune 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align='center' style="text-align:center; font-weight:bold; font-size:2.0em;letter-spacing:2.0px;"> SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models </h1>

<a href="https://scholar.google.com/citations?user=OrICiVQAAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Tongtian Yue1,3*</a>&nbsp;,&nbsp;
 <a href="https://scholar.google.com/citations?user=IOiro9MAAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Jie Cheng2,3*</a>&nbsp;,&nbsp;
 <a href="https://scholar.google.com/citations?user=OaGRHWYAAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Longteng Guo1,3*</a>&nbsp;,&nbsp;
 <a href="https://scholar.google.com/citations?user=R4Rn7dMAAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Xingyuan Dai2,3</a>&nbsp;,&nbsp;
 <a href="https://scholar.google.com/citations?user=g_9KoD8AAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Zijia Zhao1,3</a>&nbsp;,&nbsp; 
 <a href="https://scholar.google.com/citations?user=XWunp9YAAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Xingjian He1,3</a>&nbsp;&nbsp;
 <a href="https://scholar.google.com/citations?user=F4ypDHIAAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Gang Xiong2,3</a>&nbsp;&nbsp;
 <a href="https://scholar.google.com/citations?user=RRKqjKAAAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Yisheng Lv2,3</a>&nbsp;&nbsp;
 <a href="https://scholar.google.com/citations?user=sOI-S7oAAAAJ&hl=zh-CN" target="_blank" style="text-decoration: none;">Jing Liu1,3†</a>&nbsp;&nbsp;
	 
1Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA&nbsp;&nbsp;&nbsp; 
2State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA&nbsp;&nbsp;&nbsp; 
3School of Artificial Intelligence, University of Chinese Academy of Sciences&nbsp;&nbsp;&nbsp;

CVPR, 2024

<!-- ## Abstract

Recent trends in Large Vision Language Models (LVLMs) research have been increasingly focusing on advancing beyond general image understanding towards more nuanced, object-level referential comprehension. In this paper, we present and delve into the self-consistency capability of LVLMs, a crucial aspect that reflects the models' ability to both generate informative captions for specific objects and subsequently utilize these captions to accurately re-identify the objects in a closed-loop process. This capability significantly mirrors the precision and reliability of fine-grained visual-language understanding. Our findings reveal that the self-consistency level of existing LVLMs falls short of expectations, posing limitations on their practical applicability and potential. To address this gap, we introduce a novel fine-tuning paradigm named **Self-Consistency Tuning (SC-Tune)**. It features the synergistic learning of a cyclic describer-locator system. This paradigm is not only data-efficient but also exhibits generalizability across multiple LVLMs. Through extensive experiments, we demonstrate that SC-Tune significantly elevates performance across a spectrum of object-level vision-language benchmarks and maintains competitive or improved performance on image-level vision-language benchmarks. -->

## Requirements

### Installation

Create a conda environment and install dependencies:

```bash
conda create -n sc_tune python=3.10
conda activate sc_tune
pip install -r requirements.txt
```

### Data

Download the [Qwen-VL-Chat checkpoint](https://huggingface.co/Qwen/Qwen-VL-Chat/tree/main) (10 *.bin files in total) to the path `Qwen-VL-Chat/` and [Object365 images](https://www.objects365.org/download.html).

> [!NOTE]
> We have modified the codes in `Qwen-VL-Chat/visual.py`. Please replace the original file with the one in this repo if necessary.

## Get Started

### Configs

Set the path of Object365 images in `scripts/finetune_ds.sh`. Other hyperparameters can also be found in this file.

### Running

```bash
sh scripts/finetune_ds.sh
```

### Main codes

The main codes to implement sc-tune method are in `transformers/trainer.py` and `transformers/trainer_utils.py`.

## Acknowledgement

This repo benefits from [Qwen-VL](https://github.com/QwenLM/Qwen-VL), [TRL](https://github.com/huggingface/trl), and [MOSS](https://github.com/OpenLMLab/MOSS-RLHF). Thanks for their wonderful work.

## Citation

```latex
@article{yue2024sc,
  title={SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models},
  author={Yue, Tongtian and Cheng, Jie and Guo, Longteng and Dai, Xingyuan and Zhao, Zijia and He, Xingjian and Xiong, Gang and Lv, Yisheng and Liu, Jing},
  journal={arXiv preprint arXiv:2403.13263},
  year={2024}
}
```

SC-Tune

SC-Tune 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

SC-Tune

SC-Tune 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README