LiftImage3D

KO | EN

gitlite — search

#python #hacktoberfest #react #golang #android #machine-learning #rust #macos #nodejs #awesome-list #vue #pytorch #mcp #openai

(★ 176)

LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors

File Explorer

Download Latest Version (.zip)

- method.png
- teaser.png
- - __init__.py
- - __init__.py
  - network_gui.py
- - __init__.py
- - __init__.py
  - camera.py
  - cameras.py
  - colmap_loader.py
  - dataset.py
  - dataset_readers.py
  - deformation.py
  - gaussian_model.py
  - grid.py
  - hexplane.py
  - neural_3D_dataset_NDC.py
  - regulation.py
  - utils.py
- - camera_utils.py
  - general_utils.py
  - graphics_utils.py
  - image_utils.py
  - loader_utils.py
  - loss_utils.py
  - params_utils.py
  - point_utils.py
  - pose_utils.py
  - render_utils.py
  - scene_utils.py
  - sh_utils.py
  - system_utils.py
  - timer.py
  - TIMES.TTF
  - TIMESBD.TTF
  - TIMESBI.TTF
  - TIMESI.TTF
- cmd_3dgs_motionctrl_inthewild.py
- requirements.txt
- train.py
- testimg001.png
- testimg002.png
- - dinov2.py
  - dpt.py
- - demo.py
  - README.md
  - requirements.txt
  - requirements_optional.txt
  - train.py
  - visloc.py
- - __init__.py
  - catmlp_dpt_head.py
  - demo.py
  - fast_nn.py
  - losses.py
  - model.py
- - colmap_loader.py
- calc_pose_inthewild_motionctrl.py
- LICENSE
- NOTICE
- README.md
- requirements.txt
- - showcase_svd.md
  - tutorial.md
- - .DS_Store
- - camera_utils.py
  - flow_utils.py
  - motionctrl_cmcm_gradio.py
  - traj_utils.py
  - utils.py
- - __init__.py
- - __init__.py
  - lr_scheduler.py
  - util.py
- app.py
- main.py
- mp4topng_iterative_args.py
- pytest.ini
- README.md
- requirements.txt
- test_xformers.py
- visualize_trajectory.py
README.md
requirements.txt
run_liftimg3d_motionctrl.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors

### [Project Page](https://liftimage3d.github.io/) | [Arxiv Paper](https://arxiv.org/pdf/2412.09597)

[Yabo Chen](https://scholar.google.com/citations?user=6aHx1rgAAAAJ&hl=zh-TW) 1*, [Chen Yang](https://scholar.google.com/citations?hl=zh-CN&user=StdXTR8AAAAJ) 1*,
[Jiemin Fang](https://jaminfong.cn/) 2‡, [Xiaopeng Zhang](https://scholar.google.com/citations?user=Ud6aBAcAAAAJ&hl=zh-CN) 2 ,[Lingxi Xie](http://lingxixie.com/) 2 , [Wei Shen](https://shenwei1231.github.io/) 1 ,[Wenrui Dai](https://scholar.google.com/citations?user=Xg8MhyAAAAAJ&hl=en) 1 , [Hongkai Xiong](https://scholar.google.com/citations?user=bB16iN4AAAAJ&hl=en&oi=ao) 1 , [Qi Tian](https://www.qitian1987.com/) 2

1 Shanghai Jiao Tong University &emsp; 2 Huawei Inc. &emsp;

\* equal contributions in no particular order. $\ddagger$ project lead.

![block](https://raw.githubusercontent.com/AbrahamYabo/LiftImage3D/main/assets/teaser.png)   
Leveraging Latent Video Diffusion Models (LVDMs) priors effectively faces three key challenges: (1) degradation in quality across large camera motions, 
(2) difficulties in achieving precise camera control, and (3) geometric distortions inherent to the diffusion process that damage 3D consistency.

![block](https://raw.githubusercontent.com/AbrahamYabo/LiftImage3D/main/assets/method.png)
The overall pipeline of LiftImage3D. Firstly, we extend LVDM to generate diverse video clips from a single image using an
articulated camera trajectory strategy. Then all generated frames are matching using the robust neural matching module and registered in
to a point cloud. After that we initialize Gaussians from registered point clouds and construct a distortion field to model the independent
distortion of each video frame upon canonical 3DGS.

## 🦾 Updates (Still Under Construction)
- 12/13/2024: Post the arxiv paper and project page.
- 12/23/2024: Post the pipeline of LiftImage3D and requirements.
##  On Coming
- Release the code based on ViewCrafter
- Release the code of test prototype
- Release the local 3DGS viewer

## Requirements
1. Clone LiftImage3D.
```bash
git clone --recursive https://github.com/AbrahamYabo/LiftImage3D
cd LiftImage3D
# if you have already cloned LiftImage3D:
# git submodule update --init --recursive
```

2. Pytorch 2.0 for faster training and inference.
```bash
conda create -n liftimage3d python=3.11
conda activate liftimage3d
conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia  # use the correct version of cuda for your system

pip install -r requirements.txt
```

3. Install [xformer](https://github.com/facebookresearch/xformers#installing-xformers) properly to enable efficient transformers.
```bash
conda install xformers -c xformers
# from source
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
```

5. Install the submodules of 3DGS. We have two diff-gaussian-rasterization for different rendering strategy. [diff-w](https://github.com/rmurai0610/diff-gaussian-rasterization-w-pose/tree/43e21bff91cd24986ee3dd52fe0bb06952e50ec7) and [diff-ori](https://github.com/graphdeco-inria/diff-gaussian-rasterization/tree/9c5c2028f6fbee2be239bc4c9421ff894fe4fbe0) 
```bash
# We have two diff-gaussian-rasterization for different rendering strategy
cd distort-3dgs
pip install -e submodules/diff-w
pip install -e submodules/diff-ori
pip install -e submodules/simple-knn
cd ../
```

6. Optional, compile the cuda kernels for RoPE (as in CroCo v2).
```bash
# DUST3R relies on RoPE positional embeddings for which you can compile some cuda kernels for faster runtime.
cd mast3r/dust3r/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../../
```

7. Download all the checkpoints needed
```bash
mkdir -p checkpoints/
wget https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth -P checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth -P checkpoints/
wget https://huggingface.co/TencentARC/MotionCtrl/resolve/main/motionctrl_svd.ckpt -P checkpoints/
```
Because the computing resources I have cannot directly access the web network, I choose to keep laion/CLIP-ViT-H-14-laion2B-s32B-b79K locally. You can also download it from the website https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/tree/main for access.
All the checkpoints should be organize as follows.
```
├── checkpoints
│     ├── depth_anything_v2_vitl.pth
│     ├── MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth
│     ├── motionctrl_svd.ckpt
├── laion
│   | CLIP-ViT-H-14-laion2B-s32B-b79K
│     ├── config.json
│     ├── open_clip_config.json
│     ├── open_clip_pytorch_model.bin
│     ├── ...
```

8. Try LiftImage3D now
```bash
python run_liftimg3d_motionctrl.py --cache_dir ./output --input_file input_images/testimg001.png --width 1024 --height 768 
#Note that --width and --height need to match the actual resolution of the input image. 
#To maintain the generation performance of motionctrl, it is recommended to choose a width of 1024."
```

##  Acknowledgement
This repository is based on original [MotionCtrl](https://github.com/TencentARC/MotionCtrl), [ViewCrafter](https://github.com/Drexubery/ViewCrafter), [DUSt3R](https://github.com/naver/dust3r), [MASt3R](https://github.com/naver/mast3r), [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2), [3DGS](https://github.com/graphdeco-inria/gaussian-splatting), and [4DGS](https://github.com/hustvl/4DGaussians),. Thanks for their awesome works.

##  Citation
If you find this work repository/work helpful in your research, welcome to cite the paper and give a ⭐:

```
@misc{chen2024liftimage3d,
    title={LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors},
    author={Yabo Chen and Chen Yang and Jiemin Fang and Xiaopeng Zhang and Lingxi Xie and Wei Shen and Wenrui Dai and Hongkai Xiong and Qi Tian},
    year={2024},
    eprint={2412.09597},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
    }
```

LiftImage3D

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

LiftImage3D

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README