less-is-more

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

★ 58 Open GitHub ↗

Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective (ACL 2024)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Checklist

What you can find in this repo:

- Selective EOS Supervision
  - training code
  - trained model checkpoints
- Scoring EOS Supervision
  - data filtering code
  - filtered data
- CHAIR evaluation
  - evaluation scripts
  - our test set data
- others
  - [generated captions](./CHAIR-eval/data/Table1-results) in [Table 1](https://arxiv.org/html/2402.14545v2#S3.T1)

## Selective EOS Supervision

### Training

Following the instruction of [LLaVA](https://github.com/haotian-liu/LLaVA) to prepare the environment, data (`LLaVA-Instruction-150K`) and pretraining models (e.g., `LLaVA-1.5-7b`).

Train the model with Selective EOS Supervision. The default configuration is set to train the `llava-1.5-7b` model with `Detail23k` for one epoch.

```bash
cd LLaVA
bash scripts/v1_5/selective_eos_finetune.sh
```
The main modifications to the original LLaVA code for Selective EOS Supervision are detailed in [./assets/selective-eos-supervision.md](./assets/selective-eos-supervision.md).

### Checkpoint

Our models (LoRA weights) finetuned with Selective EOS Supervision:

## Scoring EOS Supervision

### Data Scoring

For the LLaVA codebase, due to some constraints related to deepspeed, currently I have no idea about how to efficiently score a dataset with a standalone script. Our scoring relies on the training process, i.e., for each training step:
- Score the data in the minibatch and save the scores;
- Cancel loss backward (can be achieved by modifying the trainer code).

The core code for data scoring are provided in `./LLaVA/llava/model/language_model/llava_llama_filter.py`.

### Filtered Data

Our data filtered with Scoring EOS Supervision:

Basic Data | Filtered Data
:- | :-
`LLaVA-Instruction-150K` | LLaVA-Instruction-150K-filtered [[OneDrive]](https://1drv.ms/u/c/97dec68abb271787/EdVLzADx-SVFtHL2sHyg4LEBQfY6SlyuNFY6f0_qcLPMjA?e=UWGWbF)

### Training

Instruction tune the LLaVA-7b model on our filtered data with:

```bash
cd LLaVA
bash scripts/finetune_qlora_filtered.sh
```

## CHAIR Evaluation

### Data

The test set used in our paper for CHAIR evaluation is provided in [./CHAIR-eval/data/chair-500.jsonl](./CHAIR-eval/data/chair-500.jsonl). The data is randomly sampled from the MSCOCO validation set with a random seed of 0.

#### CHAIR Images

We provide two ways to collect test set images:

- a [python script](./CHAIR-eval/prepare_data.py) to collect images from the original MSCOCO images with softlinks. Please specify the path of your own MSCOCO image path. The script will create a folder `./CHAIR-eval/data/chair-500` for the CHAIR images.
  ```bash
  python ./CHAIR-eval/prepare_data.py
  ```
- a [OneDrive link](https://1drv.ms/u/c/97dec68abb271787/ERCcXm0ZeiFGl96K6tG1xz4Bs0YYDFu9pcSA6OacLgZhTw?e=8e6eA8) to download the 500 images. Unzip the images to `./CHAIR-eval/data/chair-500`.

#### MSCOCO Annotation

Use the following command to download the annotation files of MSCOCO detection, which will be used for CHAIR evaluation:
```bash
cd ./CHAIR-eval/data
wget http://images.cocodataset.org/annotations/annotations_trainval2014.zip
mkdir MSCOCO
unzip -d MSCOCO/annotation annotations_trainval2014.zip
  ```

### Evaluation

We provide a [script](./CHAIR-eval/eval.sh) for CHAIR inference and evaluation.  
Set your model in the following script and then run it:

```bash
bash ./CHAIR-eval/eval.sh
```
`MODEL_NAME`: lora weights, e.g., `yuezih/llava-v1.5-7b-selective-23k-lora`  
`MODEL_BASE`: base model checkpoint, e.g., `liuhaotian/llava-v1.5-7b`

The first-time evaluation can be slow because of the ground-truth object set construction. Subsequent evaluations will be faster with the cache.

## Citation

If you find this repo helpful, please consider citing our paper:

```bibtex
@misc{yue2024less,
      title={Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective}, 
      author={Zihao Yue and Liang Zhang and Qin Jin},
      year={2024},
      eprint={2402.14545},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

## Acknowledgement

This repo is built on [LLaVA](https://github.com/haotian-liu/LLaVA) (models) and [OPERA](https://github.com/shikiw/OPERA) (CHAIR evaluation). Many thanks for their efforts. The use of our code should also follow the original licenses.

less-is-more

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README