RAIN - GitLite

KO | EN

gitlite — search

#javascript #python #ai #llm #golang #android #java #awesome #cli #vue #awesome-list #mcp #openai #open-source

RAIN

(★ 92)

No description available.

File Explorer

Download Latest Version (.zip)

- rain_morpher.yaml
- unet.json
- test1.png
- test10.png
- test2.png
- test3.png
- test4.png
- test5.png
- test6.png
- test7.png
- test8.png
- test9.png
- - __init__.py
  - onnxdet.py
  - onnxpose.py
  - util.py
  - wholebody.py
- - engine_model.py
  - framed_models.py
  - onnx_export.py
  - translation.py
- - attention.py
  - attention_processor.py
  - motion_module.py
  - mutual_self_attention.py
  - pose_guider.py
  - resnet.py
  - transformer_2d.py
  - transformer_3d.py
  - unet_2d_blocks.py
  - unet_2d_condition.py
  - unet_3d.py
  - unet_3d_blocks.py
  - unet_3d_explicit_reference.py
- - pipeline_pose2vid.py
  - pipeline_pose2vid_lcm.py
- - scheduler_lcm.py
- - taesdv.py
- morpher.py
- - PLACE_ONNX_WEIGHTS_HERE
- - PLACE_TORCH_WEIGHTS_HERE
.gitignore
gradio_app.py
LICENSE
NOTICE
pipeline_example.py
README.md
requirements_inference.txt

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Introduction
This repository contains source code for [	
RAIN: Real-time Animation Of Infinite Video Stream](https://arxiv.org/abs/2412.19489). A real-time implementation for video generataion on customer-level devices.

Project Page is [here](https://matrixteam-ai.github.io/pages/RAIN).

# Update Plan
- [x] Release Demo code
- [x] Release inference pipeline code
- [ ] Release source code for training
- [ ] Update for a more interactive implementation

# Usage
## Installation
We recommend python `>=3.10`. Install pytorch on the [official website](https://pytorch.org/) firstly (We recommend torch `>=2.3.0`).
```
git clone https://github.com/Pscgylotti/RAIN.git
# clone repository from github
cd RAIN
pip install -r requirements_inference.txt  
# install requirements for inferencing
```

## Weights
You can download original RAIN weights from [Google Drive](https://drive.google.com/drive/folders/1xm8kdjWKgc1xbou63U7OhgSzqxzNUCsQ?usp=drive_link), [Huggingeface Hub](https://huggingface.co/Pscgylotti/RAIN-v0.1/tree/main), and then put them into `weights/torch/`.

You can get `'taesdv.pth'` from https://github.com/madebyollin/taesdv, and put it into `weights/torch/`.

Clone https://huggingface.co/stabilityai/sd-vae-ft-mse into `weights/`.

Download `'image_encoder'` folder and its contents from https://huggingface.co/lambdalabs/sd-image-variations-diffusers and put it into `weights/`.

Download `'dw-mm_ucoco.onnx'` from https://huggingface.co/hr16/UnJIT-DWPose/tree/main and `'yolox_s.onnx'` from https://github.com/Megvii-BaseDetection/YOLOX/releases/tag/0.1.1rc0, and put them into `weights/onnx`. (You can choose to use `'dw-ll_ucoco-384.onnx'` and `'yolox_l.onnx'` from https://huggingface.co/yzd-v/DWPose/tree/main for higher accuracy).

(You can always redirect the weights directory in `configs/rain_morpher.yaml`)

## TensorRT 
In `configs/rain_morpher.yaml`, modify `tensorrt: False` into `tensorrt: True` to enable TensorRT acceleration. In the first launch it will take about ten minutes to compile the model.

## Demo Launch
Simply execute `python gradio_app.py` and open `http://localhost:7860/` in browser (Usually the port is `7860`).

Upload an upper-half-body potrait of any anime characters, fuse reference and turn on the web camera. Click on start to launch face morphing. You may need to adjust some morphing parameters to fit your face with the character face (Especially for the eye-related parameters. The eyes generally fail to synthesize with incompatible parameters).

## Hardware Requirement
It generally takes about `12 GiB` of Device RAM to run the whole inference demo. However, you can unload the reference part after fusing the reference image. Then the synthesis-only model requires about `8 GiB` of Device RAM to run.

# Acknowledgment

Special thanks to [CivitAI Community](https://civit.ai) and [YODOYA](https://www.pixiv.net/users/101922785) for example images. Thanks to [Jianwen Meng](mailto:jwmeng@mail.ustc.edu.cn) for pipeline design.