KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #codex #typescript #developer-tools #ai-agents #deepseek-harness #claude-code #ai-agent #cli #react #mcp

WonderWorld

★ 741 Open GitHub ↗

Code release for https://kovenyu.com/WonderWorld/

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Interactive 3D Scene Generation from a Single Image

[![a](https://img.shields.io/badge/Website-WonderWorld-blue)](https://kovenyu.com/wonderworld/)
[![arXiv](https://img.shields.io/badge/arXiv-2406.09394-red)](https://arxiv.org/abs/2406.09394)
[![twitter](https://img.shields.io/twitter/url?label=Koven_Yu&url=https%3A%2F%2Ftwitter.com%2FKoven_Yu)](https://x.com/Koven_Yu/status/1835769026934673595)
</div>

https://github.com/user-attachments/assets/fdf63ae1-362d-4916-9733-71516d61a437

> #### [WonderWorld: *Interactive* 3D Scene Generation from a Single Image](https://arxiv.org/abs/2406.09394)
> ##### [Hong-Xing "Koven" Yu*](https://kovenyu.com/), [Haoyi Duan*](https://haoyi-duan.github.io/), [Charles Herrmann](https://scholar.google.com/citations?user=LQvi5XAAAAAJ&hl=en), [William T. Freeman](https://billf.mit.edu/), [Jiajun Wu](https://jiajunwu.com/) ("*" denotes equal contribution)

## Getting Started

### Installation
For the installation to be done correctly, please proceed only with CUDA-compatible GPU available.
It requires 48GB GPU memory to run.

Clone the repo and create the environment:
```bash
git clone https://github.com/KovenYu/WonderWorld.git && cd WonderWorld
mamba create --name wonderworld python=3.10
mamba activate wonderworld
```
We are using  <a href="https://github.com/facebookresearch/pytorch3d" target="_blank">Pytorch3D</a> to perform rendering.
Run the following commands to install it or follow their <a href="https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md" target="_blank">installation guide</a> (it may take some time). We tested on `cuda=12.4`, other `cuda` versions should also work.

```bash
# switch to cuda 12.4, other versions should also work
mamba install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.4 -c pytorch -c nvidia
mamba install -c fvcore -c iopath -c conda-forge fvcore iopath
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
pip install submodules/depth-diff-gaussian-rasterization-min/
pip install submodules/simple-knn/
```

Install the rest of the requirements:

```bash
pip install -r requirements.txt
cd ./RepViT/sam && pip install -e . && cd ../..
python -m spacy download en_core_web_sm
```

Export your OpenAI api_key (If you want to use GPT-4 to generate scene descriptions):

```bash
export OPENAI_API_KEY='your_api_key_here'
```

Download RepViT model and put it to the root directory.
```bash
wget https://github.com/THU-MIG/RepViT/releases/download/v1.0/repvit_sam.pt
```

### Run examples

- Example config file

To run an example, first you need to write a config. An example config `./config/example.yaml` is shown below (more examples are located at `config/more_examples`, feel free to try):

```yaml
  runs_dir: output/real_campus_2
  example_name: real_campus_2
  
  seed: 1
  # enable guided depth diffusion
  depth_conditioning: True
  
  # use gpt to generate scene description
  use_gpt: False
  debug: True
  
  # depth model and camera/depth parameters
  depth_model: marigold
  camera_speed: 0.001
  fg_depth_range: 0.015
  depth_shift: 0.001
  sky_hard_depth: 0.02
  init_focal_length: 960
  
  # re-generate sky panorama images
  gen_sky_image: False
  # generate sky point cloud
  gen_sky: False
  
  # enable layer-wise generation
  gen_layer: True
  # load previously generated gaussians
  load_gen: False
  ```

- Run

###### Local Visualization Setup:
  
  On your local laptop, `git clone https://github.com/haoyi-duan/splat.git` and open `index_stream.html`.
  
  To enable interactive visualization of your results through this local web browser, follow these steps:
  
  - Ensure you have `'ssh'` installed on your local machine.
  - The main program will run on server user_id@server_name
  
  ```shell
  # On your local machine
  ssh -L 7777:localhost:7777 server_name
  ```
  
  ###### Main Program Running:
  
  On the server, run the main program:
  
  ```bash
  # On user_id@server_name
  python run.py --example_config config/example.yaml --port 7777
  ```
  More examples are located at `config/more_examples`, feel free to try!
  
  ###### Interactive Generation Step:
  
  Open the `index_stream.html` on your local machine, and you should see the scene in it. You can navigate with `WSAD` and arrow keys.
  
  1. If you specify `use_gpt=True` in your example configuration file, the scene description for this new scene will be automatically generated by LLM; if you specify `use_gpu=False`, you can manually input scene description you want in the text box of the local browser. Remember to click 'Next scene is ...' after you are done.  
  2. Next you need to set a proper camera view for the program to generate new scene. You can do this by wondering through the browser to a novel view, then press key `'R'` to let program interactively generate new scene in this view for you. 
  3. If you are not satisfied with the current generation, you can press key `Z` to delete the previous one generation, and follow step 1 and 2 to do a new generation.
  4. Repeat 1-3, you will interactively generate a large-scale connected scene, and you can wonder through the scene freely during the whole process.
  5. After some generation, you can press key `X` to save the current scene. Next time, you can load the generated scene by specifying `load_gen=True` in your configuration file.

### How to add more examples?

We highly encourage you to add new images and try new stuff!
You would need to do the image-caption pairing separately (e.g., using DALL-E to generate image and GPT4V to generate description).

- Add a new image in `./examples/images/`.

- Add content of this new image in `./examples/examples.yaml`.

Here is an example:

```yaml
  - name: new_example
    image_filepath: examples/images/new_example.png
    style_prompt: DSLR 35mm landscape
    content_prompt: scene name, object 1, object 2, object 3
    negative_prompt: ''
    background: ''
  ```

- **content_prompt**: "scene name", "object 1", "object 2", "object 3"

- **negative_prompt** and **background** are optional

- Write a config `config/new_example.yaml` like `./config/example.yaml` for the new example.

- Run the program following the [previous section](#run-examples). (For the first time use, the model will automatically generate the panorama sky images for the example, which takes about 20 minutes on A6000 GPU. After the corresponding sky images for the example are stored, later use of this example will automatically skip this step)

## Citation

```
@article{yu2024wonderworld,
    title={WonderWorld: Interactive 3D Scene Generation from a Single Image},
    author={Hong-Xing Yu and Haoyi Duan and Charles Herrmann and William T. Freeman and Jiajun Wu},
    journal={arXiv:2406.09394},
    year={2024}
}
```

## Related Project

- [CVPR2024] [**WonderJourney**: Going from Anywhere to Everywhere](https://kovenyu.com/wonderjourney/)

## Acknowledgement

We appreciate the authors of [Marigold](https://github.com/prs-eth/Marigold), [SyncDiffusion](https://github.com/KAIST-Visual-AI-Group/SyncDiffusion), [RepViT](https://github.com/THU-MIG/RepViT), [Stable Diffusion](https://huggingface.co/stabilityai/stable-diffusion-2-inpainting), and [OneFormer](https://github.com/SHI-Labs/OneFormer) to share their code.

WonderWorld

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README