2D-3D-Lifting

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #llm #deep-learning #linux #macos #java #cli #vue #security #mcp #ios

(★ 77)

No description available.

2D-3D-Lifting 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

#  **Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting**

[![ICCV 2025](https://img.shields.io/badge/ICCV-2025%20Highlight-blue)](https://iccv.com/)
[![Paper](https://img.shields.io/badge/Paper-arXiv-red)](https://www.arxiv.org/abs/2507.18678)
[![Project Page](https://img.shields.io/badge/Project-Page-green)](https://zhanggongjie.github.io/TowardsSSI-page/)

![Pipeline](https://raw.githubusercontent.com/ZhangGongjie/2D-3D-Lifting/main/resources/Pipeline.png)
</div>

--------------------

## 📝 Abstract
<details>
<summary><b>Click to expand full abstract</b></summary>

Spatial intelligence is emerging as a transformative frontier in AI, yet it remains constrained by the scarcity of large-scale 3D datasets. Unlike the abundant 2D imagery, acquiring 3D data typically requires specialized sensors and laborious annotation.

In this work, we present a scalable pipeline that converts single-view images into comprehensive, scale- and appearance-realistic 3D representations - including point clouds, camera poses, depth maps, and pseudo-RGBD - via integrated depth estimation, camera calibration, and scale calibration.

Our method bridges the gap between the vast repository of imagery and the increasing demand for spatial scene understanding. By automatically generating authentic, scale-aware 3D data from images, we significantly reduce data collection costs and open new avenues for advancing spatial intelligence.

We release two generated spatial datasets, i.e., COCO-3D and Objects365-v2-3D, and demonstrate through extensive experiments that our generated data can benefit various 3D tasks, ranging from fundamental perception to MLLM-based reasoning. These results validate our pipeline as an effective solution for developing AI systems capable of perceiving, understanding, and interacting with physical environments.

</details>

## 🔧 Installation

### Prerequisites
- Python 3.8+
- PyTorch > 2.0 with CUDA support

### Step 1: System Dependencies

```bash
# Update system packages
sudo apt-get update

# Install essential libraries
apt-get install -y \
    libgl1-mesa-dev \
    libglib2.0-0 \
    ffmpeg libsm6 libxext6 \
    aria2 \
    git-lfs \
    vim tmux wget unzip htop rsync

# Configure HuggingFace endpoint (only for China mainland users)
echo 'export HF_ENDPOINT=https://hf-mirror.com' >> ~/.bashrc
source ~/.bashrc
```

### Step 2: Python Dependencies

```bash
# Install core dependencies
pip install -r requirements.txt

# Install flash-attention (requires special build configuration)
pip install flash-attn==2.5.8 --no-build-isolation

# Only for H20 platform compatibility (run at the end if needed)
pip install nvidia-cublas-cu12==12.4.5.8
```

### Step 3: PerspectiveFields Setup

```bash
cd ./PerspectiveFields 
pip install -r requirements.txt
python setup.py install
cd ..
```

## 🚀 Quick Start

### Generate Spatial Images from MS-COCO

```bash
# Single GPU version
# Note: First run will download pre-trained checkpoints (~10 minutes)
python generate_spatial_img_coco.py \
    -i /path/to/Datasets/coco/train2017 \
    -a /path/to/Datasets/coco/annotations/instances_train2017.json \
    -o ./path/to/output/
```

### 💡 Example Usage

```bash
# Process COCO validation set
python generate_spatial_img_coco.py \
    -i ./data/coco/val2017 \
    -a ./data/coco/annotations/instances_val2017.json \
    -o ./output/coco_val_3d/
```

## ✅ Validation

### 1️⃣ Validate Spatial Image Generation

Verify that spatial images are correctly generated by reconstructing the 3D scene:

```bash
python validate_spatial_img_coco.py \
    --rgb_image_path ./demo_output/rgb/000000000632.png \
    --depth_image_path ./demo_output/depth/000000000632_remove_edges.png \
    --camera_params_path ./demo_output/camera_parameters/000000000632.json \
    --output_ply_path ./validation/output.ply \
    --visualize  # Optional: directly visualize the point cloud
```

**Expected Result**: 
- ✅ Semantically meaningful point cloud
- ✅ Correct scale representation
- ✅ Z-axis pointing upward

### 2️⃣ Validate Point Cloud Annotations

Ensure point clouds and annotations are correctly aligned:

```bash
python validate_pointcloud_and_anno_coco.py \
    --point_cloud_path ./demo_output/point/000000000632.ply \
    --json_path ./demo_output/json/000000000632.json \
    --output_ply_path ./validation/annotated.ply \
    --visualize  # Optional: directly visualize
```

**Expected Result**: 
- ✅ Instances marked with distinct colors
- ✅ Correct spatial boundaries

## 📖 Citation

If you find our work useful in your research, please consider citing:

```bibtex
@inproceedings{miao2025towards,
  title={Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting},
  author={Miao, Xingyu and Duan, Haoran and Qian, Quanhao and 
          Wang, Jiuniu and Long, Yang and Shao, Ling and 
          Zhao, Deli and Xu, Ran and Zhang, Gongjie},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2025}
}
```

2D-3D-Lifting

2D-3D-Lifting 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

2D-3D-Lifting

2D-3D-Lifting 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README