SAMPro3D

KO | EN

gitlite — search

#javascript #react #typescript #llm #rust #deep-learning #nodejs #cli #kubernetes #security #pytorch #mcp #openai #open-source

(★ 171)

SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation (3DV 2025)

파일 탐색기

최종 버전 다운로드 (.zip)

- preprocess_2d_scannet.py
- preprocess_util.py
- README.md
- scannet_sensordata.py
- - __init__.py
  - setup.py
- main_utils.py
- sam_utils.py
- vis_utils.py
.DS_Store
.gitignore
3d_prompt_proposal.py
installation.md
LICENSE
main.py
README.md
requirements.txt

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<p align="center">
  <img src="https://mutianxu.github.io/sampro3d/static/images/icon_final.jpg" alt="" width="150" height="50"/>
  <h1 align="center">SAMPro3D: Locating SAM Prompts in 3D 
  
  for Zero-Shot Instance Segmentation</h1>
  <p align="center">
    <a href="https://mutianxu.github.io"><strong>Mutian Xu</strong></a>
    ·
    <strong>Xingyilang Yin</strong></a>
    ·
    <a href="https://lingtengqiu.github.io/"><strong>Lingteng Qiu</strong></a>
    ·
    <a href="https://xueyuhanlang.github.io/"><strong>Yang Liu</strong></a>
    ·
    <a href="https://www.microsoft.com/en-us/research/people/xtong/"><strong>Xin Tong</strong></a>
    ·
    <a href="https://gaplab.cuhk.edu.cn/"><strong>Xiaoguang Han</strong></a>
    <br>
    SSE, CUHKSZ
    ·
    FNii, CUHKSZ
    ·
    Microsoft Research Asia

</p>
  <h2 align="center">3DV 2025</h2>
  <h3 align="center"><a href="https://arxiv.org/abs/2311.17707">Paper</a> | <a href="https://mutianxu.github.io/sampro3d/">Project Page</a></h3>
  <div align="center"></div>
</p>
<p align="center">
  <a href="">
    <img src="https://mutianxu.github.io/sampro3d/static/images/teaser.jpg" alt="Logo" width="80%">
  </a>
</p>

*SAMPro3D* can segment **ANY** 😯😯😯 3D indoor scenes <b>WITHOUT</b> training ❗️❗️❗️. It achieves higher quality and more diverse segmentation than previous zero-shot or fully supervised approaches, and in many cases even surpasses human-level annotations.
<br>

If you find our code or work helpful, please cite:
```bibtex
@inproceedings{xu2025sampro3d,
        title={SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation}, 
        author={Mutian Xu and Xingyilang Yin and Lingteng Qiu and Yang Liu and Xin Tong and Xiaoguang Han},
        year={2025},
        booktitle = {International Conference on 3D Vision (3DV)}
  }
```

<details open="open" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'>
  <summary>Table of Contents</summary>
  <ol>
    <li>
      <a href="#-news">News</a>
    </li>
    <li>
      <a href="#requirements-and-installation">Requirements and Installation</a>
    </li>
    <li>
      <a href="#data-preparation">Data Preparation</a>
    </li>
    <li>
      <a href="#run-sampro3d">Run SAMPro3D</a>
    </li>
    <li>
      <a href="#animated-qualitative-comparison">Animated Qualitative Comparison</a>
    </li>
    <li>
      <a href="#segment-your-own-3d-scene">Segment Your Own 3D Scene</a>
    </li>
    <li>
      <a href="#todo">TODO</a>
    </li>
    <li>
      <a href="#contact">Contact</a>
    </li>
    <li>
      <a href="#acknowledgement">Acknowledgement</a>
    </li>
  </ol>
</details>

## 📢 News
- The ScanNet200-Fine50 test data is released. (Mar.11, 2025 UTC)
- The first major revision of code is out. Try the latest code! 💪💪💪 (Jan.2, 2024 UTC)
- The initial code is released. :fire::fire::fire: (Dec.31, 2023 UTC)

## Requirements and Installation

### Hardware requirements
At least 1 GPU to hold around 8000MB. Moreover, it is highly recommended to utilize both a CPU with ample processing power and a disk with fast I/O capabilities. Additionally, the disk needs to be large enough (about 50 MB for a 2D frame of resolution 240*320, totally around 160 GB for 2500 frames of a large-scale scene).

### Software installation
Follow the [installation instruction](installation.md) to install all required packages.

## Data Preparation

Follow the [data pre-processing instruction](dataset_preprocess/README.md) to download and preprocess data.

## Run SAMPro3D

### 3D Prompt Proposal
The initial stage of SAMPro3D involves generating a 3D prompt and executing SAM segmentation, followed by saving the SAM outputs for subsequent stages. To initiate this process, simply run:
```
python 3d_prompt_proposal.py --data_path /PATH_TO/ScanNet_data --scene_name sceneXXXX_XX --prompt_path /PATH_TO/initial_prompt --sam_output_path /PATH_TO/SAM_outputs --device cuda:0
```
This stage will be the only step to perform SAM inference, accounting for the majority of computational time and memory usage within our entire pipeline.

**Note** on time efficiency: This stage will save SAM outputs into `.npy` files for later use. Due to different hardware conditions (CPU and disk), the I/O speed of SAM output files may vary a lot and impact the running time of our pipeline. Please refer to the hardware recommendations mentioned before to prepare your hardware for the best efficiency.

(Optional: *Partial-Area* Segmentation): At this stage, you can also perform 3D segmentation on partial point clouds captured by limited 2D frames, by simply changing the `frame_id_init` and `frame_id_end` at [here](https://github.com/GAP-LAB-CUHK-SZ/SAMPro3D/blob/main/3d_prompt_proposal.py#L169), then running the script. Sometimes this works better than segmenting the whole point clouds (thanks to less complicated scenes and better frame-consistency).

### Finish Segmentation and Result Visualization
Next, we will proceed with filtering and consolidating the initial prompts, leveraging the saved SAM outputs generated during the 3D Prompt Proposal phase, to obtain the final 3D segmentations. This can be realized by executing the following command:
```
python main.py --data_path /PATH_TO/ScanNet_data --scene_name sceneXXXX_XX --prompt_path /PATH_TO/initial_prompt --sam_output_path /PATH_TO/SAM_outputs --pred_path /PATH_TO/sampro3d_predictions --output_vis_path /PATH_TO/result_visualization --device cuda:0
```
After finishing this, the visualization result of the final 3D segmentation will be *automatically* 😊 saved as `sceneXXXX_XX.ply` file in the path specified by `--output_vis_path`.

**Note** on post-processing of the floor: Using our framework, you can usually get a decent segmentation of the floor. However, for a large-scale floor, we use post-processing for perfect segmentation of the floor. For small-scale scenes (e.g., scene0050_00 in ScanNet), you can skip this step by simply adding `--args.post_floor False` to the previous command.

### Time estimation ⚡️
If everything goes well, the entire pipeline will just take 15 min for a large-scale 3D scene captured by 2000 2D frames. (**WE DO NOT NEED TRAIN❗️**)

## Animated Qualitative Comparison

https://github.com/GAP-LAB-CUHK-SZ/SAMPro3D/assets/48080726/3a459c80-ac17-4750-a763-d477d33640bd

https://github.com/GAP-LAB-CUHK-SZ/SAMPro3D/assets/48080726/ac7aaf2c-9223-4d0e-94c4-cf38413eba74

## 🌟 Segment Your Own 3D Scene 
With our advanced framework, you can generate high-quality segmentations on your own 3D scene without the need for training! Here are the steps you can follow:

- Data preparation: Follow the tips mentioned in [data pre-processing instruction](dataset_preprocess/README.md) to prepare your data.
- Familiarize yourself with the instructions: Read and understand the guidelines, and pay attention to any specific recommendations.
- Run SAMPro3D: Execute the segmentation framework on your prepared data. You may also need to adjust some parameters such as *eps* at [here](https://github.com/GAP-LAB-CUHK-SZ/SAMPro3D/blob/main/utils/main_utils.py#L139) according to the code comments. 
- Monitor the segmentation process: Keep an eye on the segmentation process to ensure it is running smoothly. Depending on the size and complexity of your scene, the segmentation may take some time to complete.
- Evaluate the segmentation output: Once the segmentation process is finished, assess the quality of the segmentation results. Check if the segments align with the desired objects or regions in your 3D scene. You may also compare the output to ground truth data or visually inspect the results for accuracy.
- Refine if necessary: If the segmentation output requires improvement or refinement, consider adjusting the parameters or settings of SAMPro3D or applying post-processing techniques to enhance the segmentation quality.
- Analyze and utilize the segmentation: Utilize the segmented output for your intended purposes, such as further analysis, visualization, or integration with other applications or systems.

## ScanNet200-Fine50 Test Data
In our paper, we contribute a ScanNet200-Fine50 test data, providing fine-grained high-quality annotations on 50 scenes from ScanNet200. The instance label files (organized in .npy, point index keeps same with original ScanNet200 data) can be accessed via: [Google Drive Link](https://drive.google.com/file/d/1jJmSmonPjZlXAWMcoYd-viALJmR4ePcF/view?usp=sharing) and [BaiduYun](https://pan.baidu.com/s/1gmOwBYLhODd-1-gbnWJSRw?pwd=ftgd).

## Contact
You are welcome to submit issues, send pull requests, or share some ideas with us. If you have any other questions, please contact Mutian Xu (mutianxu@link.cuhk.edu.cn).

## Acknowledgement
Our code base is partially borrowed or adapted from [SAM](https://github.com/facebookresearch/segment-anything), [OpenScene](https://github.com/pengsongyou/openscene) and [Pointcept](https://github.com/Pointcept).

SAMPro3D

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

SAMPro3D

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README