ComfyUI_RH_Dreamid-Omni

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

ComfyUI_RH_Dreamid-Omni

★ 11 Open GitHub ↗

This is a ComfyUI plugin for https://github.com/Guoxu1233/DreamID-Omni

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ComfyUI_RH_Dreamid-Omni

![License](https://img.shields.io/badge/License-Apache%202.0-green)

A ComfyUI custom node for [DreamID-Omni](https://github.com/Guoxu1233/DreamID-Omni), enabling controllable human-centric audio-video generation with identity preservation. Generate talking-head videos with custom character faces and voices directly within ComfyUI.

## ✨ Features

- **Identity-Preserving Video Generation** — Generate videos where characters retain the face and voice from your reference inputs
- **Single & Multi-Person Support** — Create scenes with one or two characters, each with their own face and voice
- **Memory Optimized** — Runs on 24GB VRAM GPUs with BF16 precision and dynamic FP8 quantization
- **Native ComfyUI Integration** — Outputs standard `VIDEO` type, compatible with ComfyUI's video pipeline

## 🛠️ Installation

### 1. Clone the Repository

```bash
cd ComfyUI/custom_nodes
git clone https://github.com/HM-RunningHub/ComfyUI_RH_Dreamid-Omni.git
```

### 2. Install Dependencies

```bash
cd ComfyUI_RH_Dreamid-Omni
pip install -r requirements.txt
```

> **Note**: `flash_attn` is optional but recommended for better performance:
> ```bash
> pip install flash_attn --no-build-isolation
> ```

## 📦 Model Download & Installation

All models must be placed in `ComfyUI/models/DreamID-Omni/` with the following structure:

```
ComfyUI/
└── models/
    └── DreamID-Omni/
        ├── DreamID_Omni/
        │   └── dreamid_omni_bf16.safetensors   # Main fusion model (BF16)
        ├── Wan2.2-TI2V-5B/
        │   ├── models_t5_umt5-xxl-enc-bf16.pth # T5 text encoder
        │   ├── Wan2.2_VAE.pth                   # Video VAE
        │   └── google/
        │       └── umt5-xxl/                    # T5 tokenizer
        └── MMAudio/
            └── ext_weights/
                ├── v1-16.pth                    # Audio VAE
                └── best_netG.pt                 # BigVGAN vocoder
```

### Download Methods

#### Method 1: Using Download Script (Recommended)

We provide a convenience script that downloads all required models:

```bash
cd ComfyUI/custom_nodes/ComfyUI_RH_Dreamid-Omni
python download_models.py --output-dir ../../models/DreamID-Omni
```

#### Method 2: HuggingFace CLI

```bash
# DreamID-Omni fusion model
huggingface-cli download XuGuo699/DreamID-Omni dreamid_omni.safetensors --local-dir ComfyUI/models/DreamID-Omni/DreamID_Omni

# Wan2.2 T5 + VAE
huggingface-cli download Wan-AI/Wan2.2-TI2V-5B models_t5_umt5-xxl-enc-bf16.pth Wan2.2_VAE.pth --local-dir ComfyUI/models/DreamID-Omni/Wan2.2-TI2V-5B
huggingface-cli download Wan-AI/Wan2.2-TI2V-5B --include "google/*" --local-dir ComfyUI/models/DreamID-Omni/Wan2.2-TI2V-5B

# MMAudio VAE + Vocoder
huggingface-cli download hkchengrex/MMAudio --include "ext_weights/v1-16.pth" "ext_weights/best_netG.pt" --local-dir ComfyUI/models/DreamID-Omni/MMAudio
```

#### Method 3: ModelScope (For China Users)

```bash
pip install modelscope

# DreamID-Omni fusion model
modelscope download --model XuGuo699/DreamID-Omni --local_dir ComfyUI/models/DreamID-Omni/DreamID_Omni

# Wan2.2 T5 + VAE
modelscope download --model Wan-AI/Wan2.2-TI2V-5B models_t5_umt5-xxl-enc-bf16.pth Wan2.2_VAE.pth --local_dir ComfyUI/models/DreamID-Omni/Wan2.2-TI2V-5B

# MMAudio
modelscope download --model hkchengrex/MMAudio --local_dir ComfyUI/models/DreamID-Omni/MMAudio
```

### BF16 Model Conversion (Optional)

If you downloaded the original FP32 `dreamid_omni.safetensors`, you can convert it to BF16 to reduce VRAM usage and improve loading speed:

```bash
python convert_model_to_bf16.py ComfyUI/models/DreamID-Omni/DreamID_Omni/dreamid_omni.safetensors
```

This creates `dreamid_omni_bf16.safetensors` in the same directory. The node will automatically prefer the BF16 version if available.

## 🚀 Usage

### Example Workflow

Download the example workflow from [`workflows/example_workflow.json`](workflows/example_workflow.json) and import it into ComfyUI.

The example demonstrates:

1. **Single-Person Mode** — One reference image + one reference audio → talking-head video
2. **Two-Person Mode** — Two reference images + two reference audios → two-person conversation video

### Prompt Format

DreamID-Omni uses special tags to control characters and speech:

- **Subject Identity**: `<sub1>`, `<sub2>` — Linked to reference images `<img1>`, `<img2>`
- **Speech Tags**: `<S>Your speech content here<E>` — Converted to speech using the character's reference audio

#### Single-Person Example

```
<img1>: In the frame, a woman with black long hair is identified as <sub1>.
**Overall Environment/Scene**: A lively open-kitchen café at night...
**Main Characters/Subjects Appearance**: <sub1> is a young woman with thick dark wavy hair...
**Main Characters/Subjects Actions**: <sub1> tastes the sauce with a spoon...
<sub1> maintains eye contact and says, <S>I keep telling myself I'm fine, but some nights it feels like I'm just performing calm.<E>
```

#### Two-Person Example

```
<img1>: A young man with wavy brown hair, identified as <sub1>.
<img2>: The Mona Lisa painting, a woman, identified as <sub2>.
**Overall Environment/Scene**: A serene Italian Renaissance garden at dusk...
**Main Characters/Subjects Appearance**: <sub1> wears a simple linen shirt. <sub2> is dressed in an elegant gown...
**Main Characters/Subjects Actions**: <sub1> looks at <sub2> admiringly and says, <S>Your smile holds a thousand stories.<E> <sub2> replies, <S>Perhaps one of them is about you.<E>
```

## 📝 Node Reference

### RunningHub DreamID-Omni Loader

Loads the DreamID-Omni pipeline (fusion model, T5, VAEs).

| Output | Type | Description |
|--------|------|-------------|
| pipeline | `RunningHub_DreamID_Omni_Pipeline` | Loaded pipeline for the Sampler node |

### RunningHub DreamID-Omni Sampler

Generates a video from reference images, audio, and a text prompt.

| Input | Type | Default | Description |
|-------|------|---------|-------------|
| pipeline | `RunningHub_DreamID_Omni_Pipeline` | — | Pipeline from Loader node |
| prompt | `STRING` | — | Text prompt with subject/speech tags |
| sample_steps | `INT` | 20 | Denoising steps (more = better quality, slower) |
| seed | `INT` | 100 | Random seed for reproducibility |
| width | `INT` | 992 | Output video width (step of 8) |
| height | `INT` | 512 | Output video height (step of 8) |
| ref_image | `IMAGE` | — | Reference face image for Person 1 |
| ref_image2 | `IMAGE` | — | *(Optional)* Reference face image for Person 2 |
| ref_audio | `AUDIO` | — | Reference voice audio for Person 1 |
| ref_audio2 | `AUDIO` | — | *(Optional)* Reference voice audio for Person 2 |

| Output | Type | Description |
|--------|------|-------------|
| video | `VIDEO` | Generated MP4 video |

> **VRAM Note**: Default resolution `992×512` is optimized for 24GB GPUs. You can try `1280×704` if you have more VRAM.

## 📄 License

This project is licensed under the [Apache License 2.0](LICENSE).

## 🔗 Links

- [RunningHub](https://www.runninghub.cn)
- [Original DreamID-Omni Project](https://github.com/Guoxu1233/DreamID-Omni)
- [DreamID-Omni Paper (arXiv)](https://arxiv.org/abs/2602.12160)
- [DreamID-V](https://github.com/bytedance/DreamID-V)

## 🙏 Acknowledgements

This project is a ComfyUI node wrapper based on [DreamID-Omni](https://github.com/Guoxu1233/DreamID-Omni), developed by Xu Guo, Fulong Ye, Qichao Sun et al. from Tsinghua University and ByteDance.

DreamID-Omni builds upon several outstanding open-source projects: [Ovi](https://github.com/character-ai/Ovi), [Wan2.2](https://github.com/Wan-Video/Wan2.2), [MMAudio](https://github.com/hkchengrex/MMAudio), [Phantom](https://github.com/Phantom-video/Phantom), [HuMo](https://github.com/Phantom-video/HuMo), [OpenHumanVid](https://github.com/fudan-generative-vision/OpenHumanVid).

## ⭐ Citation

```bibtex
@misc{guo2026dreamidomni,
      title={DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation},
      author={Xu Guo and Fulong Ye and Qichao Sun and Liyang Chen and Bingchuan Li and Pengze Zhang and Jiawei Liu and Songtao Zhao and Qian He and Xiangwang Hou},
      year={2026},
      eprint={2602.12160},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2602.12160},
}
```

ComfyUI_RH_Dreamid-Omni

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

ComfyUI_RH_Dreamid-Omni

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README