KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agent #developer-tools #codex #typescript #cli #deepseek-harness #claude-code #ai-agents #mcp #rust

4K4DGen

★ 28 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# [ICLR 25 Spotlight] 4K4DGen: Panoramic 4D Generation at 4K Resolution

[Renjie Li1,4](https://shadowiterator.github.io/), [Panwang Pan1†](https://paulpanwang.github.io), [Bangbang Yang1](https://ybbbbt.com/), [Dejia Xu2](https://ir1d.github.io/), [Shijie Zhou3](https://shijiezhou-ucla.github.io/), [Xuanyang Zhang1](https://scholar.google.com/citations?user=oPV20eMAAAAJ&hl=zh-CN), [Zeming Li1](https://www.zemingli.com/), [Achuta Kadambi3](https://samueli.ucla.edu/people/achuta-kadambi/), [Zhangyang Wang2](https://www.ece.utexas.edu/people/faculty/atlas-wang), [Zhengzhong Tu4](https://vztu.github.io/), [Zhiwen Fan2](https://zhiwenfan.github.io/)
</div>

<div>
 1ByteDance 
 &nbsp;&nbsp;
 2UT Austin 
 &nbsp;&nbsp;
 3UCLA 
 &nbsp;&nbsp;
 4TAMU

</div>

Feel free to contact me (paulpanwang@gmail.com) or open an issue if you have any questions or suggestions.
</h4>

-----------------------------

![HeadDemo](https://raw.githubusercontent.com/ShadowIterator/4K4DGen/main/output/example/I2.gif)

## Release Checklist
- [x] Complete documents
- [x] Code release for Animation Phase
- [x] Code release for Lifting Phase
- [x] Testing 16 scene data set release

## Setup

### Installation
```
git clone git@github.com:ShadowIterator/4K4DGen.git
cd animating
conda env create -f environment.yml 
cd ../4dlifting
conda env create -f environment.yml

conda activate 4dlifting
pip install submodules/diff-gaussian-rasterization-depth 
pip install submodules/simple-knn
```

### Animating
Please first set your working directory to `./animating`

#### Prepare Checkpoint
Download the checkpoint in [googledrive](https://drive.google.com/file/d/1H6v06DZuHeYhXCz3Rj8OL55y92sp990h/view?usp=sharing). Extract it at `animating/pretrained`.

### Lifting
Please first set your working directory to `./4dlifting`
#### Prepare Data
Put the data under the `./data` folder. Organize the files as following:
```
data
|-- <scene1>
| |-- <scene1>.gif
| |-- <scene1>_mask.png
|-- <scene2>
|-- ...
```
You can also refer to the `I2` folder provided as an example under the `data` folder.

#### Prepare Checkpoints
Please download the checkpoints from [googledrive](https://drive.google.com/file/d/1UKvuTp4wNKotrFOVb3OOZlW9TbOf4rkz/view?usp=sharing). Place them under the `./pre_checkpoints` folder

## Usage

### Animating
To run the animating phase, please run
```
cd ./animating
conda activate animating
# do animating
python gen4k.py --config ./svd_mask_config.yaml --eval validation_data.prompt_image=<path_to_image> validation_data.mask=<path_to_mask> validation_data.config=<path_to_config> pretrained_model_path=pretrained/animate_anything_svd_v1.0 --diff_mode return_latents
# decode latent codes
python gen4k.py --config ./svd_mask_config.yaml --eval validation_data.prompt_image=<path_to_image> validation_data.mask=<path_to_mask> validation_data.config=<path_to_config> pretrained_model_path=pretrained/animate_anything_svd_v1.0 --diff_mode decode_latents
```
You can also run the example bash file:
```
bash ./generate_example.sh
```
### 4D Lifting
to run 4D lifting, please refer to the following.
```
cd 4dlifting
conda activate 4dlifting
# generate initial geometry for each frame
python generate_init_geo_4k.py -s <source_path> -m <target_path>
# lifting the frames
python train.py -s <source_path>/<frame_id> -m <target_path>/<frame_id>
```

### One Command Running
using the script as following:
```
python run.py --input_img <your_image_path_or_folder>
```
If you have setup the data and environments as above, it will generate a running script `run.sh`. You can simply run `bash ./run.sh` to generate the 4D scene from your input images. For more options of the script, please see `python run.py --help`.

## Acknowledgements
We built this from [AnimateAnything](https://github.com/alibaba/animate-anything), [MultiDiffusion](https://github.com/omerbt/MultiDiffusion), and [DreamScene360](https://github.com/ShijieZhou-UCLA/DreamScene360).

## Bibtex
If you find our work useful for your project, please consider citing the following paper.
```
@misc{li20244k4dgenpanoramic4dgeneration,
      title={4K4DGen: Panoramic 4D Generation at 4K Resolution}, 
      author={Renjie Li and Panwang Pan and Bangbang Yang and Dejia Xu and Shijie Zhou and Xuanyang Zhang and Zeming Li and Achuta Kadambi and Zhangyang Wang and Zhengzhong Tu and Zhiwen Fan},
      year={2024},
      eprint={2406.13527},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2406.13527}, 
}
```

4K4DGen

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README