OMNEX-VL

KO | EN

gitlite — search

#hacktoberfest #android #machine-learning #linux #docker #java #nodejs #awesome #cli #kubernetes #vue #security #mcp #openai

(★ 13)

Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization

OMNEX-VL Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# OMNEX-VL
Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization

[[📖 Paper](https://arxiv.org/abs/2601.06224)] [[🤗 OMNEX-VL-7B](https://huggingface.co/OmniAI-ZJU/OMNEX-VL)] [[🤗 OMNEX-VL-train-data](https://huggingface.co/datasets/OmniAI-ZJU/OMNEX-VL-DATA)]

</div>

## 🔥 News
- [2025/11/08] Our work is accepted by AAAI 2026.🎇🎇🎇

## 👀 Introduction
**Three types of illusions in the field of multimodal large models:**
<img src="./intro.png" style="zoom:100%;" />

## 🧠 Method
<img src="./method.png" style="zoom:100%;" />

## 🏆 Performance
<img src="./performance.png" style="zoom:100%;" />

## 📐 Set up
### 0. Environment
```bash
git clone https://github.com/omniAI-Lab/Ground-What-You-See.git
cd Ground-What-You-See

# build environment
conda create -n video-ground python=3.11 
conda activate video-ground
pip install -r requirements.txt
```
### 1. Dataset
1. Cold-Start Dataset:
* [Llava-COT](https://huggingface.co/datasets/Xkev/LLaVA-CoT-100k)
* [Llava-COT-modify-json](https://huggingface.co/datasets/teardrop-gwj/omni-rl-train-data)
2. RL Dataset:
* [Video-R1-Data](https://huggingface.co/datasets/Video-R1/Video-R1-data)
* [Train_RL_Medium_24K.json](https://huggingface.co/datasets/teardrop-gwj/omni-rl-train-data)

### 2. Train
#### 2.1 cold start  
`train/experiment/main/cold-start/run_sft.sh`
```bash
export DEBUG_MODE="true"
export LOG_PATH="/train/outputs/Main/ColdStart/debug_log.txt" # define your log path

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --nproc_per_node="8" \
    --nnodes="1" \
    --node_rank="0" \
    --master_addr="127.0.0.1" \
    --master_port="12349" \
    train/experiment/main/cold-start/train/sft.py \
    --output_dir "train/outputs/Main/ColdStart/Qwen2.5-VL-7B-coldstart-sft" \
    --model_name_or_path "model/Qwen2.5-VL-7B-Instruct" \
    --dataset_name "Llava-COT-modify.json" \ # Replace with the address of the Llava COT modified json you downloaded
    --deepspeed train/local_scripts/zero2.json \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --learning_rate 1e-6 \
    --logging_steps 1 \
    --bf16 \
    --report_to wandb \
    --gradient_checkpointing true \
    --attn_implementation flash_attention_2 \
    --num_train_epochs 2 \
    --run_name Qwen2.5-VL-7B-coldstart-sft \
    --save_steps 1000 \
    --max_grad_norm 5 \
    --save_only_model false \
```
#### 2.2 RL
`train/experiment/main/RL-OURS/run_grpo.sh`
```bash
export DEBUG_MODE="true"
export LOG_PATH="/train/outputs/EXPERIENTS/Main/MediumSample/Qwen2.5-VL-7B-GRPO/debug_log.txt"

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --nproc_per_node="8" \
    --nnodes="1" \
    --node_rank="0" \
    --master_addr="127.0.0.1" \
    --master_port="12365" \
    /train/experiment/EXPERIMENTS/Main/RL/grpo/grpo.py \
    --output_dir "/train/outputs/EXPERIENTS/Main/MediumSample/Qwen2.5-VL-7B-GRPO" \
    --model_name_or_path '/train/outputs/Main/ColdStart/Qwen2.5-VL-7B-coldstart-sft' \ # The address of the model trained during the cold start phase
    --dataset_name "/data/train_data/OmniVisionData/Train_RL_Medium_24K.json" \ # Download JSON data address
    --deepspeed /train/local_scripts/zero3.json \
    --max_prompt_length 16384 \
    --max_completion_length 768 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --learning_rate 1e-6 \
    --lr_scheduler_type "cosine" \
    --weight_decay 0.01 \
    --bf16 \
    --logging_steps 1 \
    --gradient_checkpointing true \
    --attn_implementation flash_attention_2 \
    --max_pixels 401408 \
    --num_train_epochs 1 \
    --run_name OmniVisionR1-MAIN-Finally-Medium-GRPO-Training \
    --save_steps 100 \
    --beta 0.04 \
    --max_grad_norm 5 \
    --save_only_model false \
    --num_generations 8
```

## 📖 Citation
If you want to cite our paper:

```bibtex
@inproceedings{pan2026ground,
  title={Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization},
  author={Pan, Miao and Gan, Wangjie and Chen, Jintao and Zhang, Wenqi and Bing, Sun and Yin, Jianwei and Zhang, Xuhong},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={40},
  number={10},
  pages={8242--8250},
  year={2026}
}
```

OMNEX-VL

OMNEX-VL Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

OMNEX-VL

OMNEX-VL Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README