Envision3D

KO | EN

gitlite — search

#javascript #hacktoberfest #react #llm #deep-learning #linux #docker #macos #windows #cli #awesome-list #security #mcp #open-source

(★ 116)

Envision3D: One Image to 3D with Anchor Views Interpolation

File Explorer

Download Latest Version (.zip)

- - profiles_settings.xml
- .gitignore
- deployment.xml
- misc.xml
- modules.xml
- repo.iml
- vcs.xml
- .DS_Store
- teaser1.jpg
- teaser2.jpg
- teaser3.jpg
- 0.txt
- 1.txt
- 10.txt
- 11.txt
- 12.txt
- 13.txt
- 14.txt
- 15.txt
- 16.txt
- 17.txt
- 18.txt
- 19.txt
- 2.txt
- 20.txt
- 21.txt
- 22.txt
- 23.txt
- 24.txt
- 25.txt
- 26.txt
- 27.txt
- 28.txt
- 29.txt
- 3.txt
- 30.txt
- 31.txt
- 4.txt
- 5.txt
- 6.txt
- 7.txt
- 8.txt
- 9.txt
- s1.yaml
- s2.yaml
- .DS_Store
- single_image.py
- thriple_image.py
- pumpkin.png
- - .DS_Store
  - neuralangelo-pinhole-wmask-opt.yaml
- - .DS_Store
  - __init__.py
  - ortho.py
  - utils.py
- - __init__.py
  - base.py
  - geometry.py
  - nerf.py
  - network_utils.py
  - neus.py
  - ray_utils.py
  - texture.py
  - utils.py
- - .DS_Store
  - __init__.py
  - base.py
  - criterions.py
  - nerf.py
  - neus.py
  - neus_ortho.py
  - neus_pinhole_opt.py
  - utils.py
- - __init__.py
  - callbacks.py
  - loggers.py
  - misc.py
  - mixins.py
  - obj.py
- .DS_Store
- launch.py
- requirements.txt
- transformer_mv2d.py
- transformer_temporal.py
- unet_3d_blocks.py
- unet_mv2d_blocks.py
- unet_mv2d_condition.py
- unet_spatio_temporal_condition.py
- pipeline_mvdiffusion_image.py
- pipeline_mvdiffusion_ref_image.py
- pipeline_stable_video_diffusion.py
- dummy
- base_utils.py
- dpt.py
- misc.py
- normal_utils.py
- rmbg.py
.DS_Store
gen_s1.py
gen_s2.py
process_img.py
README.md
req.txt

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Envision3D: One Image to 3D with Anchor Views Interpolation, [ArXiv](https://arxiv.org/abs/2403.08902), [Project Page](https://pang-yatian.github.io/Envision3D-webpage/), [Model Weights](https://huggingface.co/pytttttt/Envision3D/tree/main)
[Yatian Pang](https://scholar.google.com/citations?user=AZQyNWkAAAAJ&hl=en), Tanghui Jia, Yujun Shi, Zhenyu Tang, Junwu Zhang, Xinhua Cheng, Xing Zhou, Francis E.H. Tay, [Li Yuan](https://yuanli2333.github.io/)

## TL;DR

We propose a novel cascade diffusion framework to efficiently generate dense(32) multi-view consistent images and extract high-quality 3D content. The cascade diffusion framework inference only takes less than 12GB VRAM.

![img](https://raw.githubusercontent.com/PKU-YuanGroup/Envision3D/main/assets/teaser1.jpg)

## Abstract

We present Envision3D, a novel method for efficiently generating high-quality 3D content from a single image. Recent methods that extract 3D content from multi-view images generated by diffusion models show great potential. However, it is still challenging for diffusion models to generate dense multi-view consistent images, which is crucial for the quality of 3D content extraction. To address this issue, we propose a novel cascade diffusion framework, which decomposes the challenging dense views generation task into two tractable stages, namely anchor views generation and anchor views interpolation. In the first stage, we train the image diffusion model to generate global consistent anchor views conditioning on image-normal pairs. Subsequently, leveraging our video diffusion model fine-tuned on consecutive multi-view images, we conduct interpolation on the previous anchor views to generate extra dense views. This framework yields dense, multi-view consistent images, providing comprehensive 3D information. To further enhance the overall generation quality, we introduce a coarse-to-fine sampling strategy for the reconstruction algorithm to robustly extract textured meshes from the generated dense images. Extensive experiments demonstrate that our method is capable of generating high-quality 3D content in terms of texture and geometry, surpassing previous image-to-3D baseline methods.

# Set up

```
pip install -r req.txt
pip install carvekit --no-deps
```

# Inference
## 1. Download model checkpoints
Download our pre-trained model checkpoints from [here](https://huggingface.co/pytttttt/Envision3D/tree/main).

Download the image normal estimation model omnidata_dpt_normal_v2.ckpt from [here](https://huggingface.co/clay3d/omnidata/tree/main).

Place all of them under ```pretrained_models``` dir.

## 2. Pre-process input image

Run the following command to resize the input image and predict the normal map.
```
CUDA_VISIBLE_DEVICES=0 python process_img.py example_imgs/pumpkin.png processed_imgs/ --size 256 --recenter
```
## 3. Inference stage I and stage II

Modify config files in the cfgs directory and run the following command to inference.

```
CUDA_VISIBLE_DEVICES=0 python gen_s1.py --config cfgs/s1.yaml  validation_dataset.filepaths=['pumpkin.png'] validation_dataset.crop_size=224
CUDA_VISIBLE_DEVICES=0 python gen_s2.py --config cfgs/s2.yaml  validation_dataset.scene=pumpkin
```
## 4. Textured mesh extraction

After getting 32 views of images, first set the correct path to the output images and then run the following command for 3D content extraction.

```
cd instant-nsr-pl/
python launch.py --config configs/neuralangelo-pinhole-wmask-opt.yaml --gpu 0 --train dataset.scene=pumpkin
```
# Results
![img](https://raw.githubusercontent.com/PKU-YuanGroup/Envision3D/main/assets/teaser2.jpg)
![img](https://raw.githubusercontent.com/PKU-YuanGroup/Envision3D/main/assets/teaser3.jpg)

Work in progress ...
- [x] Inference code
- [x] Checkpoints
- [x] Instructions
- [ ] Training code

## Acknowledgements

We thank the authors of [Wonder3D](https://github.com/xxlong0/Wonder3D), [Stable Video Diffusion](https://stability.ai/research/stable-video-diffusion-scaling-latent-video-diffusion-models-to-large-datasets), [omnidata](https://github.com/EPFL-VILAB/omnidata/tree/main/omnidata_tools/torch), [Diffusers](https://github.com/huggingface/diffusers) and [AnimateAnything](https://github.com/alibaba/animate-anything/tree/main) for their great works and open-source codes.

## Citation
```
@misc{pang2024envision3d,
      title={Envision3D: One Image to 3D with Anchor Views Interpolation}, 
      author={Yatian Pang and Tanghui Jia and Yujun Shi and Zhenyu Tang and Junwu Zhang and Xinhua Cheng and Xing Zhou and Francis E. H. Tay and Li Yuan},
      year={2024},
      eprint={2403.08902},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```

Envision3D

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Envision3D

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README