IBISAgent

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 52)

Discover the repository for "IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation," a pioneering study that has been accepted for presentation at CVPR 2026.

IBISAgent 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
    <h1 align="center">[CVPR 2026] IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation
    </h1>
</div>

- **Authors**: Yankai Jiang, Qiaoru Li, Binlu Xu, Haoran Sun, Chao Ding, Junting Dong, [Yuxiang Cai](mailto:caiyuxiang@zju.edu.cn)📧, Xuhong Zhang, Jianwei Yin
- **Institutes**: Zhejiang University; Shanghai AI Laboratory
- **Resources**: [📖[Paper](https://arxiv.org/abs/2601.03054)]  [🤗[Huggingface](https://huggingface.co/manglu3935/IBIS/tree/main)]

## 📖 Introduction

**IBISAgent** is a novel agentic Multimodal Large Language Model (MLLM) framework designed to address the limitations of existing medical MLLMs in fine-grained pixel-level understanding. unlike previous approaches that rely on implicit segmentation tokens and single-pass reasoning, IBISAgent reformulates segmentation as a **vision-centric, multi-step decision-making process**.

By treating segmentation tools (e.g., MedSAM2) as plug-and-play modules controllable through natural language, IBISAgent iteratively generates interleaved **reasoning (Thinking)** and **text-based click actions (Action)** to progressively refine segmentation masks. This approach mimics the interactive behavior of human experts, allowing for self-correction and high-quality mask generation without requiring architectural modifications to the MLLM.

## 💡 Highlights

- 🔥 **Agentic Reasoning Framework.** We reformulate medical image segmentation as a multi-step Markov Decision Process (MDP), enabling the model to "think" and "act" iteratively to solve complex visual grounding tasks.
- 🔥 **No Implicit Tokens.** IBISAgent eliminates the need for special `<SEG>` tokens and external pixel decoders, preserving the LLM's inherent text generation capabilities and ensuring better generalization.
- 🔥 **Two-Stage Training Strategy.**
  - **Cold-Start SFT**: Initialized with high-quality trajectory data synthesized from automatic click simulation and self-reflection error correction.
  - **Agentic Reinforcement Learning.** Further optimized using **GRPO** with novel fine-grained rewards (Region-based Click Placement, Progressive Improvement), enabling the model to discover advanced segmentation strategies beyond imitation.
- 🔥 **SOTA Performance.** IBISAgent significantly outperforms existing medical MLLMs on both in-domain and out-of-domain benchmarks, demonstrating superior robustness and pixel-level reasoning ability.

![IBISAgent Performance](https://raw.githubusercontent.com/Yankai96/IBISAgent/main/assets/table.jpg)

## Model Weights

Please refer to our [Huggingface repository](https://huggingface.co/manglu3935/IBIS/tree/main) for the pre-trained model weights.

## 🤖 Inference

Taking the example of [SAM2](https://github.com/facebookresearch/sam2) as the segmentation tool, we provide the inference code for IBISAgent.

1. Create a new conda environment and install the required packages.

```bash
conda create -n ibisagent python=3.12
conda activate ibisagent
pip install -r infer/requirements.txt
```

2. Install `sam2` python library from the [official repo](https://github.com/facebookresearch/sam2).

```batch
git clone https://github.com/facebookresearch/sam2.git
cd sam2
pip install -e .
```

3. Download our RL-trained model weights to `infer/models/mllm` from [here](https://huggingface.co/manglu3935/IBIS/tree/main/qwen2_5vl-7b-RL).

```bash
huggingface-cli download manglu3935/IBIS \
    --include "qwen2_5vl-7b-RL/*" \
    --local-dir infer/models/mllm \
    --local-dir-use-symlinks False
```

4. Run the multi-turn inference script.

```bash
python infer/multi_turn.py \
    --image "infer/test_img.png" \
    --prompt "Can you find a liver in this image?" \
    --mllm_path "infer/models/mllm"
```

Parameters:

| Parameter | Description | Default | Required |
| --------- | ----------- | ------- | -------- |
| `--image` | Path to the input medical image | `None` | Yes |
| `--prompt` | User text prompt (e.g., 'Is there a colon tumor?') | `None` | Yes |
| `--mllm_path` | Path to the MLLM model | `infer/models/mllm` | No |
| `--max_turns` | Maximum number of iterations | `20` | No |
| `--use_history` | Whether to enable chat history (1 for True, 0 for False) | `0` | No |
| `--output_dir` | Directory to save results | `./outputs` | No |

## 🏋️ Training

IBISAgent uses a two-stage training strategy: **Cold-Start SFT** followed by **Agentic Reinforcement Learning (GRPO)**.

1. Create a new conda environment and install the required packages.

```bash
conda create -n ibisagent-train python=3.12
conda activate ibisagent-train
pip install -r requirements.txt
```

### Stage 1: Cold-Start SFT

We use [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) for supervised fine-tuning with LoRA. The configuration file is provided at `training_scripts/qwen2_5vl_lora_sft.yaml`.

1. Set up the dataset following the `dataset_info.json` format required by LLaMA-Factory, and update the `dataset` and `output_dir` fields in the config file.

2. Specify the base model path in `model_name_or_path`.

3. Run SFT training:

```bash
llamafactory-cli train training_scripts/qwen2_5vl_lora_sft.yaml
```

### Stage 2: Agentic Reinforcement Learning (GRPO)

We use [VERL](https://github.com/volcengine/verl) for GRPO-based reinforcement learning. The training script is at `training_scripts/grpo.sh`.

1. Prepare your training data in `.parquet` format and update the `data.train_files` and `data.val_files` paths in `training_scripts/grpo.sh`.

3. Set the `model` variable to the path of your SFT-merged checkpoint, and run:

```bash
bash training_scripts/grpo.sh
```

## 🔧 Build Your SFT Data

Building the Supervised Fine-Tuning (SFT) data consists of two main parts:
1. **Building Trajectory Data.** Generating simulated interactive point prompts using SAM2.
2. **CoT Distillation.** Generating textual reasoning steps using a Teacher MLLM (Please refer to **Appendix B.3 Reasoning Generation for Our SFT Dataset** in our paper for details on this step).

Here we provide the script and instructions for the **first part**: generating your own trajectory data for custom 2D images or 3D NIfTI volumes.

### Generate Trajectories

We provide a versatile script `trajectory_gen/run_tr_gen.py` to automatically generate high-quality interactive click trajectories. It simulates human-like iterative clicking behavior on your segmentation data.

**For 2D Images (e.g., PNG, JPG)**
```bash
python trajectory_gen/run_tr_gen.py \
    --type img \
    --image /path/to/image.jpg \
    --mask /path/to/mask.png \
    --output /path/to/output.json \
    --iou_threshold 0.8 \
    --max_steps 20
```
*(Note: The script assumes white pixels (>128) in the mask represent the target. If your mask uses black for the target, add the `--reverse_mask` flag.)*

**For 3D NIfTI Volumes**
You can specify the slicing axis, slice index, mask label value, and optional windowing parameters for NIfTI data.
```bash
python trajectory_gen/run_tr_gen.py \
    --type nii \
    --image /path/to/volume.nii.gz \
    --mask /path/to/segmentation.nii.gz \
    --output /path/to/output.json \
    --slice_axis z \
    --slice_idx 55 \
    --mask_value 2 \
    --window_min -125 \
    --window_max 1000 \
    --iou_threshold 0.85
```

The script will output a JSON file containing the file metadata and the generated sequence of steps, including the normalized coordinates of the simulated clicks and the RLE-compressed predicted masks at each step.

**Note:** The RLE mask string can be converted into a binary mask image using the following code snippet:

```python
import numpy as np
import base64
import zlib

def decode_rle_zlib_b64_to_mask(rle_b64_str: str, shape: tuple) -> np.ndarray:
    """Decode a zlib-compressed base64 RLE string back to a binary mask."""
    if not rle_b64_str:
        return np.zeros(shape, dtype=np.uint8)
        
    compressed = base64.b64decode(rle_b64_str)
    runs_str = zlib.decompress(compressed).decode('utf-8')
    runs = np.array([int(x) for x in runs_str.split()])
    
    pixels = np.zeros(np.prod(shape), dtype=np.uint8)
    
    current_idx = 0
    current_val = 0
    for run in runs:
        pixels[current_idx:current_idx + run] = current_val
        current_idx += run
        current_val = 1 - current_val  # Alternate between 0 and 1
        
    return pixels.reshape(shape)

# Example usage:
# shape = (512, 512) # Use the size from 'nii_data' or 'img_data' in the JSON
# mask = decode_rle_zlib_b64_to_mask("eJxVUtuS2jAM/...", shape)
```

## 🔧 Build Your RL Data

Once you have the trajectory JSON files (from the SFT data generation step above), use `trajectory_gen/convert_to_rl_data.py` to convert them into the `.parquet` format required by VERL for GRPO training.

```bash
python trajectory_gen/convert_to_rl_data.py \
    --dataset your_dataset \
    --data_root /path/to/med-seg-rl \
    --output_root /path/to/output \
    --split train
```

The output Parquet file will be saved to `{output_root}/{dataset}/{split}_rl.parquet`, which can be directly used as `data.train_files` in `training_scripts/grpo.sh`.

## 📜 News

- **[2026/02/28]** 🚀 Code and dataset release preparation.
- **[2026/02/21]** 🎉 **IBISAgent** is accepted to **CVPR 2026**!

## 👨‍💻 Todo

- [x] Release training scripts (SFT & RL)
- [x] Release inference code
- [x] Release pre-trained model weights
- [ ] Release **Cold-Start** and **RL** datasets

## ✒️ Citation

If you find our work helpful for your research, please consider giving one star ⭐️ and citing:

```bibtex
@inproceedings{jiang2026ibisagent,
  title={IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation},
  author={Jiang, Yankai and Li, Qiaoru and Xu, Binlu and Sun, Haoran and Ding, Chao and Dong, Junting and Cai, Yuxiang and Zhang, Xuhong and Yin, Jianwei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}
```

## ❤️ Acknowledgments

- [VERL](https://github.com/volcengine/verl): The reinforcement learning framework we built upon.
- [Qwen2.5-VL-7B](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct): The MLLM used in our agent.
- [SAM2](https://github.com/facebookresearch/sam2): The segmentation tool used in our agent.
- [MedSAM2](https://github.com/bowang-lab/MedSAM2): The segmentation tool used in our agent.

IBISAgent

IBISAgent 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

IBISAgent

IBISAgent 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README