LangScene-X - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #android #machine-learning #deep-learning #linux #nodejs #windows #cli #vue #ios #openai #open-source

LangScene-X

(★ 302)

[ICCV 2025] LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion

File Explorer

Download Latest Version (.zip)

- pipeline.png
- teaser.png
- - __init__.py
  - automatic_mask_generator.py
  - build_sam.py
  - sam2_image_predictor.py
  - sam2_video_predictor.py
- - __init__.py
  - sam2_hiera_b+.yaml
  - sam2_hiera_l.yaml
  - sam2_hiera_s.yaml
  - sam2_hiera_t.yaml
- auto-mask-align.py
- sam2_hiera_l.yaml
- - colormaps.py
  - colors.py
  - config_utils.py
  - misc.py
- datasets.py
- losses.py
- lpips.py
- pipeline.py
- field_construction.yaml
- test_config.py
- unet_config_c16.py
- unet_config_c32.py
- - .DS_Store
  - __init__.py
  - network_gui.py
- - .DS_Store
  - __init__.py
- - __init__.py
  - utils.py
- - .DS_Store
  - __init__.py
  - app_model.py
  - cameras.py
  - colmap_loader.py
  - dataset_readers.py
  - gaussian_model.py
  - per_point_adam.py
- - .DS_Store
- - .DS_Store
  - camera_utils.py
  - edit_utils.py
  - fusion_util.py
  - general_utils.py
  - graphics_utils.py
  - image_utils.py
  - loss_utils.py
  - pose_utils.py
  - sh_utils.py
  - step_track.py
  - system_utils.py
- - __init__.py
- auto_encoder.py
- extract_with_openseg.py
- gaussian_field.py
- pipeline.py
- preprocessor.py
- - align_traj.py
  - comp_ate.py
  - lie_group_helper.py
  - relative_pose.py
  - vis_cam_traj.py
  - vis_pose_utils.py
- align_traj.py
- camera_utils.py
- general_utils.py
- graphics_utils.py
- image_utils.py
- interp_utils.py
- loss_utils.py
- pose_utils.py
- sfm_utils.py
- sh_utils.py
- stepfun.py
- system_utils.py
- - camera_head.py
  - dpt_head.py
  - head_act.py
  - track_head.py
  - utils.py
- - __init__.py
  - attention.py
  - block.py
  - drop_path.py
  - layer_scale.py
  - mlp.py
  - patch_embed.py
  - rope.py
  - swiglu_ffn.py
  - vision_transformer.py
- - aggregator.py
  - vggt.py
- - geometry.py
  - load_fn.py
  - pose_enc.py
  - rotation.py
  - visual_track.py
.gitignore
entry_point.py
get_normal.py
LICENSE
quick_start.sh
README.md
requirements.txt
train_all.sh
video_inference.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">

# ✨LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion✨

<p align="center">
<a href="https://liuff19.github.io/">Fangfu Liu</a><sup>1</sup>,
<a href="https://lifuguan.github.io/">Hao Li</a><sup>2</sup>,
<a href="https://github.com/chijw">Jiawei Chi</a><sup>1</sup>,
<a href="https://hanyang-21.github.io/">Hanyang Wang</a><sup>1,3</sup>,
<a href="https://github.com/liuff19/LangScene-X">Minghui Yang</a><sup>3</sup>,
<a href="https://github.com/liuff19/LangScene-X">Fudong Wang</a><sup>3</sup>,   
<a href="https://duanyueqi.github.io/">Yueqi Duan</a><sup>1</sup>
<br>
    <sup>1</sup>Tsinghua University, <sup>2</sup>NTU, <sup>3</sup>Ant Group     
</p>
<h3 align="center">ICCV 2025 🔥</h3>
<a href="https://arxiv.org/abs/2507.02813"><img src='https://img.shields.io/badge/arXiv-2507.02813-b31b1b.svg'></a> &nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://liuff19.github.io/LangScene-X"><img src='https://img.shields.io/badge/Project-Page-Green'></a> &nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://huggingface.co/chijw/LangScene-X"><img src='https://img.shields.io/badge/LangSceneX-huggingface-yellow'></a> &nbsp;&nbsp;&nbsp;&nbsp;
<a><img src='https://img.shields.io/badge/License-MIT-blue'></a> &nbsp;&nbsp;&nbsp;&nbsp;

![Teaser Visualization](https://raw.githubusercontent.com/THU-SI/LangScene-X/main/assets/teaser.png)
</div>

**LangScene-X:** We propose LangScene-X, a unified model that generates RGB, segmentation map, and normal map, enabling to reconstruct 3D field from sparse views input.

## 📢 News
- 🔥 [04/07/2025] We release "LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion". Check our [project page](https://liuff19.github.io/LangScene-X) and [arXiv paper](https://arxiv.org/abs/2507.02813).

## 🌟 Pipeline

![Pipeline Visualization](https://raw.githubusercontent.com/THU-SI/LangScene-X/main/assets/pipeline.png)

Pipeline of LangScene-X. Our model is composed of a TriMap Video Diffusion model which generates RGB, segmentation map, and normal map videos, an Auto Encoder that compresses the language feature, and a field constructor that reconstructs 3DGS from the generated videos.

## 🎨 Video Demos from TriMap Video Diffusion

https://github.com/user-attachments/assets/55346d53-eb04-490e-bb70-64555e97e040

https://github.com/user-attachments/assets/d6eb28b9-2af8-49a7-bb8b-0d4cba7843a5

https://github.com/user-attachments/assets/396f11ef-85dc-41de-882e-e249c25b9961

## ⚙️ Setup

### 1. Clone Repository
```bash
git clone https://github.com/liuff19/LangScene-X.git
cd LangScene-X
```
### 2. Environment Setup

1. **Create conda environment**

```bash
conda create -n langscenex python=3.10 -y
conda activate langscenex
```
2. **Install dependencies**
```bash
conda install pytorch torchvision -c pytorch -y
pip install -e field_construction/submodules/simple-knn
pip install -e field_construction/submodules/diff-langsurf-rasterizer
pip install -e auto-seg/submodules/segment-anything-1
pip install -e auto-seg/submodules/segment-anything-2
pip install -r requirements.txt
```

### 3. Model Checkpoints
The checkpoints of SAM, SAM2 and fine-tuned CogVideoX can be downloaded from our [huggingface repository](https://huggingface.co/chijw/LangScene-X).

## 💻Running

### Quick Start
You can start quickly by running the following scripts:
```bash
chmod +x quick_start.sh
./quick_start.sh <first_rgb_image_path> <last_rgb_image_path>
```
### Render
Run the following command to render from the reconstructed 3DGS field:
```bash 
python entry_point.py \
    pipeline.rgb_video_path="does/not/matter" \
    pipeline.normal_video_path="does/not/matter" \
    pipeline.seg_video_path="does/not/matter" \
    pipeline.data_path="does/not/matter" \
    gaussian.dataset.source_path="does/not/matter" \
    gaussian.dataset.model_path="output/path" \
    pipeline.selection=False \
    gaussian.opt.max_geo_iter=1500 \
    gaussian.opt.normal_optim=True \
    gaussian.opt.optim_pose=True \
    pipeline.skip_video_process=True \
    pipeline.skip_lang_feature_extraction=True \
    pipeline.mode="render"
```
You can also configurate by editting `configs/field_construction.yaml`.

## ✒️ TODO List
- [x] Per-scene Auto Encoder released
- [x] Fine-tuned CogVideoX checkpoints released
- [ ] Generalizable Auto Encoder (LQC)
- [ ] Improved TriMap Video Diffusion model

## 🔗Acknowledgement

We are thankful for the following great works when implementing LangScene-X:

- [CogVideoX](https://github.com/THUDM/CogVideo), [CogvideX-Interpolation](https://github.com/feizc/CogvideX-Interpolation), [LangSplat](https://github.com/minghanqin/LangSplat), [LangSurf](https://github.com/lifuguan/LangSurf), [VGGT](https://github.com/facebookresearch/vggt), [3DGS](https://github.com/graphdeco-inria/gaussian-splatting), [SAM2](https://github.com/facebookresearch/sam2)

## 📚Citation

```bibtex
@misc{liu2025langscenexreconstructgeneralizable3d,
      title={LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion}, 
      author={Fangfu Liu and Hao Li and Jiawei Chi and Hanyang Wang and Minghui Yang and Fudong Wang and Yueqi Duan},
      year={2025},
      eprint={2507.02813},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2507.02813}, 
}
```