NeRF-3DTalker-code

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

NeRF-3DTalker-code

★ 11 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis

Official PyTorch implementation for the paper:

> **NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis**.
>
> Xiaoxing Liu, Zhilei Liu,  Chongke Bi

## News
Thank you to all the reviewers for their feedback. Moving forward, we are committed to ongoing research to enhance the efficiency and performance of this project.

## Environment

- python 3.7
- pytorch 1.13.1
- torchvision 0.14.1

Create conda environment:
```bash
conda create -n xxx python=3.7
conda activate xxx
pip install -r requirements.txt
```

## **Dataset**

1. Download the dataset: https://github.com/yerfor/GeneFace

2. We provide preprocessing code for datasets. You can use /DataProcess/ to preprocess datasets.
```bash
python DataProcess/Gen_HeadMask.py --img_dir "/dataset/Obama/png"

python DataProcess/Gen_Landmark.py --img_dir "/dataset/Obama/png"

python Fitting3DMM/FittingNL3DMM.py --img_size 512 \
                                    --intermediate_size 256  \
                                    --batch_size 9 \
                                    --img_dir "/dataset/Obama/png"
```

3. Additionally, you need to download [Openface](https://ieeexplore.ieee.org/document/8373812)
to extract AUs from face images.

## **Train Models**
```bash
python train.py --batch_size 2 --gpu_id 3 --include_eye_gaze False --eye_gaze_dimension 64 --eye_gaze_scale_factor 1 --print_freq 50 --gaze_D6_rotation False --eye_gaze_disentangle False --comment 'gaze_dim 64,disentanglement True, 10 cam views'

python /StandardizedSpace/main/train_vq.py

python /StandardizedSpace/main/train_au.py
```

## **Acknowledgements**
Our codes are based on the following repos:
* [Wav2lip](https://dl.acm.org/doi/10.1145/3394171.3413532) for lipwav
* [SadTalker](https://ieeexplore.ieee.org/document/10204743) for lipwav
* [HeadNeRF](https://ieeexplore.ieee.org/document/9880349)
* [CodeTalker](https://ieeexplore.ieee.org/document/10204746)

## **Contact**
If you have any questions, please contact xiaoxingliu@tju.edu.cn.

NeRF-3DTalker-code

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

NeRF-3DTalker-code

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README