3D-RFT

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 9 Open GitHub ↗

[ICML 2026] 3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h2 align="center">
  <span><img src="assets/logo.png" width="4%" style="transform: translate(0,9px)"></span><b>3D-RFT: Reinforcement Fine-Tuning for Video-based <br/> 3D Scene Understanding</b>
</h2>

<div align="center" margin-bottom="6em">
<a target="_blank" rel="external nofollow noopener" href="https://xiongkunlinghu.github.io/">Xiongkun Linghu*</a>,
<a target="_blank" rel="external nofollow noopener" href="https://huangjy-pku.github.io/">Jiangyong Huang*</a>,
<a target="_blank" rel="external nofollow noopener" href="https://buzz-beater.github.io/">Baoxiong Jia</a>,
<a target="_blank" rel="external nofollow noopener" href="https://siyuanhuang.com/">Siyuan Huang</a>
</div>
&nbsp;

<div align="center">
    <a href="https://arxiv.org/abs/2603.04976" target="_blank" rel="external nofollow noopener">
    <img src="https://img.shields.io/badge/Paper-arXiv-deepgreen" alt="Paper arXiv"></a>
    <a href="https://3d-rft.github.io/" target="_blank" rel="external nofollow noopener">
    <img src="https://img.shields.io/badge/Page-3D RFT-9cf" alt="Project Page"></a>
    <a href="https://github.com/3D-RFT/3D-RFT-Perception" rel="external nofollow noopener" target="_blank">
    <img src="https://img.shields.io/badge/Code-Perception-pink" alt="Perception"></a>
    <a href="https://github.com/3D-RFT/3D-RFT-Reasoning" rel="external nofollow noopener" target="_blank">
    <img src="https://img.shields.io/badge/Code-Reasoning-blue" alt="Reasoning"></a>
</div>
&nbsp;

This repository contains the official implementation of the **3D-RFT Perception** track, including reinforcement fine-tuning for 3D visual grounding on ScanRefer and 3D object detection on ScanNet.

## Codebase

Choose the codebase according to the task family.

| Component | Status | Repository |
| --- | --- | --- |
| 3D-RFT Perception | Released | This repository |
| 3D-RFT Reasoning | Released | [3D-RFT/3D-RFT-Reasoning](https://github.com/3D-RFT/3D-RFT-Reasoning) |

## Environment Setup

The reference environment uses Python 3.11, CUDA 12.4, and GCC 11.3 or newer. Create the environment from the repository root:

```bash
git clone https://github.com/3D-RFT/3D-RFT-Perception.git
cd 3D-RFT-Perception

conda create -n 3d-rft python=3.11 -y
conda activate 3d-rft
bash setup.sh
```

The setup script installs the local Qwen-VL utilities with Decord video support.

The training code was developed with the following key package versions:

- `vllm==0.7.2`
- `trl==0.16.0`
- `flash-attn==2.5.5`

Qwen2.5-VL is sensitive to the Transformers version. Download the compatible source archive from [Google Drive](https://drive.google.com/file/d/1Kc81WZitEhUZYWXpL6y2GXuSXufLSYcF/view?usp=sharing), then install it into the environment:

```bash
unzip transformers-main.zip
pip install ./transformers-main
```

The VGGT-based training scripts additionally require [PyTorch3D](https://github.com/facebookresearch/pytorch3d). Install a PyTorch3D build compatible with the PyTorch and CUDA versions in your environment.

> [!NOTE]
> Building FlashAttention requires a working CUDA toolchain. Our reference setup uses CUDA 12.4, GCC >= 11.3, and `flash-attn==2.5.5`.

## Data and Checkpoints

All processed data, pretrained weights, and released checkpoints will be hosted in the [3D-RFT Hugging Face repository](https://huggingface.co/EricLHK/3D-RFT/tree/main). After the assets are released, download them with:

```bash
hf download EricLHK/3D-RFT --local-dir 3D-RFT-assets
```

Prepare the training annotations and the image/video data referenced by them. The released scripts expect the following annotation files:

```text
3D-RFT-data/
├── ScanRefer_train_64.78K.json
├── ScanNetDet_train_144.16K.json
└── <images and frames referenced by the JSON files>
```

Place the pretrained weights in the following default locations:

```text
pretrained_weights/
├── Qwen2.5-VL-3B-Instruct/
├── Qwen2.5-VL-3B-Instruct-ScanRefer-Scan2Cap-ScanNet-Det-Template-VGGT/
└── VGGT-1B/
```

If the Hugging Face assets are kept under `3D-RFT-assets/`, point the scripts to them without moving any files:

```bash
export DATA_ROOT="$PWD/3D-RFT-assets/3D-RFT-data"
export MODEL_ROOT="$PWD/3D-RFT-assets/pretrained_weights"
```

The scripts use repository-relative paths by default. You can override them without editing a script by exporting any of the following variables:

- `PROJECT_ROOT`
- `DATA_ROOT`
- `MODEL_ROOT`, `MODEL_PATH`, and `VGGT_PATH`
- `OUTPUT_ROOT`
- `DEEPSPEED_CONFIG`
- `RESUME_FROM_CHECKPOINT` (optional; unset by default)

For example, resume a job from a repository-relative checkpoint with:

```bash
RESUME_FROM_CHECKPOINT=logs/ScanNetDet-3D-RFT/checkpoint-6000 \
sbatch src/scripts/Sulrm/RL/VGGT/ScanNetDet/run_grpo_slurm_ScanNetDet_standalone.slurm
```

Submit Slurm jobs from the repository root. The single-GPU debug scripts resolve the repository root from their own locations.

## Training

All runnable training scripts are located in `src/scripts`. The `.slurm` files are multi-GPU Slurm jobs, while the `*_debug.sh` files run directly on one GPU for debugging.

### ScanRefer

Run the standard four-GPU GRPO training job:

```bash
sbatch src/scripts/Sulrm/RL/ScanRefer/run_grpo_slurm_ScanRefer.sh
```

Run 3D-RFT with VGGT and the 3D bounding-box reward on eight GPUs:

```bash
sbatch src/scripts/Sulrm/RL/VGGT/ScanRefer/run_qwen2.5-vl-instruct_grpo_pretrained_bbox3d_iou_reward_8GPU_optimize_vgllm_beta_0.04.slurm
```

For a single-GPU debug run:

```bash
bash src/scripts/Sulrm/RL/VGGT/ScanRefer/run_qwen2.5-vl-instruct_grpo_pretrained_bbox3d_iou_reward_8GPU_optimize_vgllm_beta_0.04_debug.sh
```

### ScanNet 3D Object Detection

Run the eight-GPU 3D-RFT training job:

```bash
sbatch src/scripts/Sulrm/RL/VGGT/ScanNetDet/run_grpo_slurm_ScanNetDet_standalone.slurm
```

For a single-GPU debug run:

```bash
bash src/scripts/Sulrm/RL/VGGT/ScanNetDet/run_grpo_slurm_ScanNetDet_standalone_debug.sh
```

The scripts use a per-device batch size of 1. Adjust `--gradient_accumulation_steps`, `--num_generations`, and the Slurm resource requests according to the available GPUs and memory.

## Citation

If you find this project useful, please cite our paper:

```bibtex
@inproceedings{linghu20263d,
  title={3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding},
  author={Linghu, Xiongkun and Huang, Jiangyong and Jia, Baoxiong and Huang, Siyuan},
  booktitle={International Conference on Machine Learning},
  year={2026}
}
```

## Acknowledgements

This codebase is developed based on [Video-R1](https://github.com/tulerfeng/Video-R1). We thank the authors for releasing their code.

3D-RFT

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README