KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #typescript #developer-tools #claude-code #ai-agent #codex #deepseek-harness #ai-agents #cli #react #mcp

Seg-ReSearch

★ 50 Open GitHub ↗

(ICML 2026) Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h3>Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search</h3>

[![arXiv](https://img.shields.io/badge/arXiv-2602.04454-b31b1b.svg)](https://arxiv.org/abs/2602.04454)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://raw.githubusercontent.com/iSEE-Laboratory/Seg-ReSearch/main/LICENSE)

[Tianming Liang](https://tmliang.github.io/)  &emsp;
Qirui Du  &emsp;
[Jian-Fang Hu](https://isee-ai.cn/~hujianfang/)  &emsp;
Haichao Jiang  &emsp;
Zicheng Lin  &emsp;
[Wei-Shi Zheng](https://www.isee-ai.cn/~zhwshi/)

ISEE Lab, Sun Yat-sen University

</div>

## 🌟 Highlights
![visual](https://raw.githubusercontent.com/iSEE-Laboratory/Seg-ReSearch/main/assets/example.png)
Through multi-turn interleaved reasoning and web search, **Seg-ReSearch** is able to localize and segment any text-guided target in images or videos, even those involving new concepts or up-to-date information that lies beyond the internal knowledge of MLLMs.

## 📢 News
* [2026/05/01] 🥳 Seg-ReSearch is accepted to ICML 2026 ! See you in Seoul. 
* [2026/02/06] All training and inference code for Seg-ReSearch has been released. Check it out!
* [2026/02/06] We have released [**OK-VOS**](https://huggingface.co/datasets/iSEE-Laboratory/OK_VOS/tree/main), a challenging VOS Benchmark explicitly requiring external knowledge.
* [2026/02/04] The paper is available on [arXiv](https://arxiv.org/abs/2602.04454).
## 🔎 Framework
![model](https://raw.githubusercontent.com/iSEE-Laboratory/Seg-ReSearch/main/assets/model.png)
**Seg-ReSearch** conducts multi-turn interactions with the search engine throughout the dynamic Multi-modal Chain-of-Thought (MCoT). This capability is incentivized by a hierarchical reward design: **IGR** pilots the initial planning, **TPR** encourages extensive exploration, and **OR** ensures final task accuracy.

## ⚙️ Getting Started
To support Qwen3-VL, we use `verl==0.7.0.dev0` and `vllm==0.11.0`, which require `pytorch>=2.8.0` and `cuda>=12.6`.

### Installation
```bash
git submodule update --init --recursive
conda create --name seg-research python=3.10
conda activate seg-research
pip install -e verl
pip install -e ".[vllm,search_tool]"
pip install "flash-attn==2.8.3" --no-build-isolation
```

### Retrieval Server Setup
We recommend creating a separate environment for the retrieval server to avoid dependency conflicts.
```bash
conda create --name retrieval python=3.10
conda activate retrieval
conda install -c pytorch -c nvidia faiss-gpu=1.8.0
pip install transformers datasets fastapi numpy torch uvicorn
```

## 📦 Data Preparation
### Download OK-VOS Benchmark
```bash
bash data/download_okvos.sh
```

### Preprocess
Run the preprocessing script to prepare the datasets for training and evaluation. This script includes extracting the `bbox` and `point`.
```bash
# Training Set
python examples/data_preprocess/okvos.py --split train --num_frames 6 --max_size 448 --min_size 448 --model_type Qwen3VL

# Test set
python examples/data_preprocess/okvos.py --split test --num_frames 6 --max_size 448 --min_size 448 --model_type Qwen3VL
```

## 🚀 Training
1. Activate the `retrieval` environment and start the retrieval server.
```bash
bash examples/train/okvos/start_retrieval.sh
```

2. Activate the `seg-research` environment and run one of the following scripts.

* **Qwen3-VL-4B-Instruct:**
```bash
# GRPO Training
bash examples/train/okvos/train_4b.sh

# DAPO Training
bash examples/train/okvos/train_4b_dapo.sh
```

* **Qwen3-VL-8B-Instruct:**

Note: We set `tensor_model_parallel_size=2` for 48G GPU memory. You can reduce it to 1 if you have larger memory.

```bash
# GRPO Training
bash examples/train/okvos/train_8b.sh

# DAPO Training
bash examples/train/okvos/train_8b_dapo.sh
```

## 📊 Evaluation

### 1. Reasoning & Search

Remember to replace the checkpoint path in the evaluation script before running.

```bash
# Serper API (Google Search)
bash examples/train/okvos/eval.sh

# DuckDuckGo (Free API)
bash examples/train/okvos/eval_ddg.sh
```
The prediction results will be saved in a `jsonl` file.

### 2. Segmentation

Based on the generated jsonl file, run the segmentation model to generate object masks:

```bash
python post_segmentation/sam2_okvos.py [path_to_jsonl]
```

## 🧩 Web Browsing
You can integrate an auxiliary LLM (e.g., `Qwen3-Next-80B-A3B-Instruct-FP8`) to act as a summarizer, empowering Seg-ReSearch with web browsing capabilities for more precise retrieval results. Give it a try! 😊

To enable this feature,

1. Launch the LLM service:
```bash
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \
     --port 8000 \
     --tensor-parallel-size 2 \
     --max-model-len 262144
```
2. Then, set the `SUMM_MODEL_URL` and `SUMM_MODEL_PATH` in `eval.sh`, like 
```bash
export SUMM_MODEL_URL="http://localhost:8000/v1"
export SUMM_MODEL_PATH="Qwen/Qwen3-Next-80B-A3B-Instruct-FP8"
```

## 🤝 Acknowledgements
Our work is built upon [verl-tool](https://github.com/TIGER-AI-Lab/verl-tool), [Seg-Zero](https://github.com/JIA-Lab-research/Seg-Zero) and [SeC](https://github.com/OpenIXCLab/SeC). We sincerely appreciate these excellent works.

## 📜 Citation
If you find our work helpful for your research, please consider citing our paper.
```bibtex
@inproceedings{liang2026segresearch,
     title={Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search},
     author={Tianming Liang and Qirui Du and Jian-Fang Hu and Haichao Jiang and Zicheng Lin and Wei-Shi Zheng},
     booktitle={Forty-third International Conference on Machine Learning},
     year={2026},
     url={https://openreview.net/forum?id=H0tGv2kBi8}
}
```

Seg-ReSearch

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README