IBISAgent

KO | EN

gitlite — search

#flutter #dart #css #html #python #android #javascript #tutorial #cryptography #discord #game-development #raspberry-pi

★ 52 Open GitHub ↗

Discover the repository for "IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation," a pioneering study that has been accepted for presentation at CVPR 2026.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
    <h1 align="center">[CVPR 2026] IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation
    </h1>
</div>

- **Authors**: Yankai Jiang, Qiaoru Li, Binlu Xu, Haoran Sun, Chao Ding, Junting Dong, [Yuxiang Cai](mailto:caiyuxiang@zju.edu.cn)📧, Xuhong Zhang, Jianwei Yin
- **Institutes**: Zhejiang University; Shanghai AI Laboratory
- **Resources**: [📖[Paper](https://arxiv.org/abs/2601.03054)]  [🤗[Huggingface](https://huggingface.co/manglu3935/IBIS/tree/main)]

## 📖 Introduction

**IBISAgent** is a novel agentic Multimodal Large Language Model (MLLM) framework designed to address the limitations of existing medical MLLMs in fine-grained pixel-level understanding. unlike previous approaches that rely on implicit segmentation tokens and single-pass reasoning, IBISAgent reformulates segmentation as a **vision-centric, multi-step decision-making process**.

By treating segmentation tools (e.g., MedSAM2) as plug-and-play modules controllable through natural language, IBISAgent iteratively generates interleaved **reasoning (Thinking)** and **text-based click actions (Action)** to progressively refine segmentation masks. This approach mimics the interactive behavior of human experts, allowing for self-correction and high-quality mask generation without requiring architectural modifications to the MLLM.

## 💡 Highlights

- 🔥 **Agentic Reasoning Framework.** We reformulate medical image segmentation as a multi-step Markov Decision Process (MDP), enabling the model to "think" and "act" iteratively to solve complex visual grounding tasks.
- 🔥 **No Implicit Tokens.** IBISAgent eliminates the need for special `<SEG>` tokens and external pixel decoders, preserving the LLM's inherent text generation capabilities and ensuring better generalization.
- 🔥 **Two-Stage Training Strategy.**
  - **Cold-Start SFT**: Initialized with high-quality trajectory data synthesized from automatic click simulation and self-reflection error correction.
  - **Agentic Reinforcement Learning.** Further optimized using **GRPO** with novel fine-grained rewards (Region-based Click Placement, Progressive Improvement), enabling the model to discover advanced segmentation strategies beyond imitation.
- 🔥 **SOTA Performance.** IBISAgent significantly outperforms existing medical MLLMs on both in-domain and out-of-domain benchmarks, demonstrating superior robustness and pixel-level reasoning ability.

![IBISAgent Performance](https://raw.githubusercontent.com/Yankai96/IBISAgent/main/assets/table.jpg)

## Model Weights

Please refer to our [Huggingface repository](https://huggingface.co/manglu3935/IBIS/tree/main) for the pre-trained model weights.

## 🤖 Inference

Taking the example of [SAM2](https://github.com/facebookresearch/sam2) as the segmentation tool, we provide the inference code for IBISAgent.

1. Create a new conda environment and install the required packages.

```bash
conda create -n ibisagent python=3.12
conda activate ibisagent
pip install -r infer/requirements.txt
```

2. Install `sam2` python library from the [official repo](https://github.com/facebookresearch/sam2).

```batch
git clone https://github.com/facebookresearch/sam2.git
cd sam2
pip install -e .
```

3. Download our RL-trained model weights to `infer/models/mllm` from [here](https://huggingface.co/manglu3935/IBIS/tree/main/qwen2_5vl-7b-RL).

```bash
huggingface-cli download manglu3935/IBIS \
    --include "qwen2_5vl-7b-RL/*" \
    --local-dir infer/models/mllm \
    --local-dir-use-symlinks False
```

4. Run the multi-turn inference script.

```bash
python infer/multi_turn.py \
    --image "infer/test_img.png" \
    --prompt "Can you find a liver in this image?" \
    --mllm_path "infer/models/mllm"
```

Parameters:

| Parameter | Description | Default | Required |
| --------- | ----------- | ------- | -------- |
| `--image` | Path to the input medical image | `None` | Yes |
| `--prompt` | User text prompt (e.g., 'Is there a colon tumor?') | `None` | Yes |
| `--mllm_path` | Path to the MLLM model | `infer/models/mllm` | No |
| `--max_turns` | Maximum number of iterations | `20` | No |
| `--use_history` | Whether to enable chat history (1 for True, 0 for False) | `0` | No |
| `--output_dir` | Directory to save results | `./outputs` | No |

## 🏋️ Training

IBISAgent uses a two-stage training strategy: **Cold-Start SFT** followed by **Agentic Reinforcement Learning (GRPO)**.

1. Create a new conda environment and install the required packages.

```bash
conda create -n ibisagent-train python=3.12
conda activate ibisagent-train
pip install -r requirements.txt
```

### Stage 1: Cold-Start SFT

We use [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) for supervised fine-tuning with LoRA. The configuration file is provided at `training_scripts/qwen2_5vl_lora_sft.yaml`.

1. Set up the dataset following the `dataset_info.json` format required by LLaMA-Factory, and update the `dataset` and `output_dir` fields in the config file.

2. Specify the base model path in `model_name_or_path`.

3. Run SFT training:

```bash
llamafactory-cli train training_scripts/qwen2_5vl_lora_sft.yaml
```

### Stage 2: Agentic Reinforcement Learning (GRPO)

We use [VERL](https://github.com/volcengine/verl) for GRPO-based reinforcement learning. The training script is at `training_scripts/grpo.sh`.

1. Prepare your training data in `.parquet` format and update the `data.train_files` and `data.val_files` paths in `training_scripts/grpo.sh`.

3. Set the `model` variable to the path of your SFT-merged checkpoint, and run:

```bash
bash training_scripts/grpo.sh
```

## 🔧 Build Your SFT Data

Building the Supervised Fine-Tuning (SFT) data consists of two main parts:
1. **Building Trajectory Data.** Generating simulated interactive point prompts using SAM2.
2. **CoT Distillation.** Generating textual reasoning steps using a Teacher MLLM (Please refer to **Appendix B.3 Reasoning Generation for Our SFT Dataset** in our paper for details on this step).

Here we provide the script and instructions for the **first part**: generating your own trajectory data for custom 2D images or 3D NIfTI volumes.

### Generate Trajectories

We provide a versatile script `trajectory_gen/run_tr_gen.py` to automatically generate high-quality interactive click trajectories. It simulates human-like iterative clicking behavior on your segmentation data.

**For 2D Images (e.g., PNG, JPG)**
```bash
python trajectory_gen/run_tr_gen.py \
    --type img \
    --image /path/to/image.jpg \
    --mask /path/to/mask.png \
    --output /path/to/output.json \
    --iou_threshold 0.8 \
    --max_steps 20
```
*(Note: The script assumes white pixels (>128) in the mask represent the target. If your mask uses black for the target, add the `--reverse_mask` flag.)*

**For 3D NIfTI Volumes**
You can specify the slicing axis, slice index, mask label value, and optional windowing parameters for NIfTI data.
```bash
python trajectory_gen/run_tr_gen.py \
    --type nii \
    --image /path/to/volume.nii.gz \
    --mask /path/to/segmentation.nii.gz \
    --output /path/to/output.json \
    --slice_axis z \
    --slice_idx 55 \
    --mask_value 2 \
    --window_min -125 \
    --window_max 1000 \
    --iou_threshold 0.85
```

The script will output a JSON file containing the file metadata and the generated sequence of steps, including the normalized coordinates of the simulated clicks and the RLE-compressed predicted masks at each step.

**Note:** The RLE mask string can be converted into a binary mask image using the following code snippet:

```python
import numpy as np
import base64
import zlib

def decode_rle_zlib_b64_to_mask(rle_b64_str: str, shape: tuple) -> np.ndarray:
    """Decode a zlib-compressed base64 RLE string back to a binary mask."""
    if not rle_b64_str:
        return np.zeros(shape, dtype=np.uint8)
        
    compressed = base64.b64decode(rle_b64_str)
    runs_str = zlib.decompress(compressed).decode('utf-8')
    runs = np.array([int(x) for x in runs_str.split()])
    
    pixels = np.zeros(np.prod(shape), dtype=np.uint8)
    
    current_idx = 0
    current_val = 0
    for run in runs:
        pixels[current_idx:current_idx + run] = current_val
        current_idx += run
        current_val = 1 - current_val  # Alternate between 0 and 1
        
    return pixels.reshape(shape)

# Example usage:
# shape = (512, 512) # Use the size from 'nii_data' or 'img_data' in the JSON
# mask = decode_rle_zlib_b64_to_mask("eJxVUtuS2jAM/...", shape)
```

## 🔧 Build Your RL Data

Once you have the trajectory JSON files (from the SFT data generation step above), use `trajectory_gen/convert_to_rl_data.py` to convert them into the `.parquet` format required by VERL for GRPO training.

```bash
python trajectory_gen/convert_to_rl_data.py \
    --dataset your_dataset \
    --data_root /path/to/med-seg-rl \
    --output_root /path/to/output \
    --split train
```

The output Parquet file will be saved to `{output_root}/{dataset}/{split}_rl.parquet`, which can be directly used as `data.train_files` in `training_scripts/grpo.sh`.

## 📜 News

- **[2026/02/28]** 🚀 Code and dataset release preparation.
- **[2026/02/21]** 🎉 **IBISAgent** is accepted to **CVPR 2026**!

## 👨‍💻 Todo

- [x] Release training scripts (SFT & RL)
- [x] Release inference code
- [x] Release pre-trained model weights
- [ ] Release **Cold-Start** and **RL** datasets

## ✒️ Citation

If you find our work helpful for your research, please consider giving one star ⭐️ and citing:

```bibtex
@inproceedings{jiang2026ibisagent,
  title={IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation},
  author={Jiang, Yankai and Li, Qiaoru and Xu, Binlu and Sun, Haoran and Ding, Chao and Dong, Junting and Cai, Yuxiang and Zhang, Xuhong and Yin, Jianwei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}
```

## ❤️ Acknowledgments

- [VERL](https://github.com/volcengine/verl): The reinforcement learning framework we built upon.
- [Qwen2.5-VL-7B](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct): The MLLM used in our agent.
- [SAM2](https://github.com/facebookresearch/sam2): The segmentation tool used in our agent.
- [MedSAM2](https://github.com/bowang-lab/MedSAM2): The segmentation tool used in our agent.

IBISAgent

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README