UnconditionalHeadMotion

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

UnconditionalHeadMotion

★ 15 Open GitHub ↗

Code & demo for the animation of still facial landmarks from an initial pose.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Autoregressive GAN for Semantic Unconditional Head Motion Generation (SUHMo)

## Abstract [[Paper](https://arxiv.org/pdf/2211.00987.pdf)]

We address the task of unconditional head motion generation to animate still human faces in a low-dimensional semantic space.
Deviating from talking head generation conditioned on audio that seldom puts emphasis on realistic head motions, we devise a GAN-based architecture that allows obtaining rich head motion sequences while avoiding known caveats associated with GANs.
Namely, the autoregressive generation of incremental outputs ensures smooth trajectories, while a multi-scale discriminator on input pairs drives generation toward better handling of high and low frequency signals and less mode collapse.
We demonstrate experimentally the relevance of the proposed architecture and compare with models that showed state-of-the-art performances on similar tasks.

## Examplar results

In the results presented below **120 frames** are generated from a **single** reference image.

### SUHMo-RNN (Training on CONFER DB)

<img src="media/git_demo_vis/rnn/dem_rnn_cf_8_9_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_cf_8_0_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_cf_8_1_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_cf_8_22_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_base_try3_14_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_cf_8_27_colors.gif" alt="drawing" width="200"/>
<img src="media/git_demo_vis/rnn/dem_rnn_base_try3_33_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_cf_8_41_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_cf_8_45_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_cf_8_49_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_base_try3_38_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/rnn/dem_rnn_cf_8_6_colors.gif" alt="drawing" width="200"/>

### SUHMo-Transformer (Training on VoxCeleb2) 
\~\~ In Vox2 preprocessing _faces are centered_, hence the suppression of head translation \~\~

<!-- ### Comparison with ACTOR model

#### Training on CONFER DB

<img src="media/git_demo_vis/actor_cf/dem_actor_0_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/actor_cf/dem_actor_17_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/actor_cf/dem_actor_25_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/actor_cf/dem_actor_40_colors.gif" alt="drawing" width="200"/>

#### Training on VoxCeleb2

<img src="media/git_demo_vis/actor_vox/dem_actor_21_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/actor_vox/dem_actor_vox4work_22_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/actor_vox/dem_actor_vox4work_3_colors.gif" alt="drawing" width="200"/> <img src="media/git_demo_vis/actor_vox/dem_actor_vox4work_57_colors.gif" alt="drawing" width="200"/>
 -->

### SUHMo in-the-wild

Several outputs can be obtained from the same reference image. See below for an illustration on SUHMo-RNN trained on CONFER DB.

## Architecture overview

SUHMo is a framework that can be implemented in several forms. Below are the proposed LSTM and Transformer variants of our model.

![uncond_head_mot](https://user-images.githubusercontent.com/36541517/197400808-c6094353-4bb7-4e49-8dd8-8f325aa4539a.png)

## Execution & Pre-trained models
Incoming...

## Citation
```
@misc{https://doi.org/10.48550/arxiv.2211.00987,
  doi = {10.48550/ARXIV.2211.00987},
  url = {https://arxiv.org/abs/2211.00987},
  author = {Airale, Louis and Alameda-Pineda, Xavier and Lathuilière, Stéphane and Vaufreydaz, Dominique},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Autoregressive GAN for Semantic Unconditional Head Motion Generation},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
```

## References

#### Face Alignment
_A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks),” in ICCV, 2017._

#### CONFER DB
_C. Georgakis, Y. Panagakis, S. Zafeiriou, and M. Pantic, “The conflict escalation resolution (confer) database,” Image and Vision Computing, vol. 65, 2017._
#### VoxCeleb2
_J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” in INTERSPEECH, 2018._

UnconditionalHeadMotion

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README