DR-Learning-for-3D-Face

KO | EN

gitlite — search

#javascript #python #react #typescript #llm #golang #android #nodejs #java #windows #awesome #vue #pytorch #openai

DR-Learning-for-3D-Face

(★ 231)

Implementation for paper "Disentangled Representation Learning for 3D Face Shape" CVPR 2019

File Explorer

Download Latest Version (.zip)

- - 11510_boundary.txt
  - face_0.dat
  - FWH_adj_matrix.npz
  - max_data.npy
  - Mean_Face.obj
  - min_data.npy
- interpolation.png
- manifold.png
- pipeline.png
- data_utils.py
- front_part_v.txt
- get_dr.cpython-36m-x86_64-linux-gnu.so
- get_fwh_dr.py
- get_mesh.cpython-36m-x86_64-linux-gnu.so
- igltest.py
- measurement.py
- mesh.py
- model.py
- pyigl.so
- VAE.py
- edgelist.pkl
- preprocess_mesh.py
- sted_compute.py
- ve.pkl
- velist.pkl
- - CMakeLists.txt
  - get_dr.cpp
  - LICENSE
  - pybind11
  - README.md
  - util_3drotation_log_exp.cpp
  - util_3drotation_log_exp.h
- - CMakeLists.txt
  - compile.sh
  - get_mesh.cpp
  - LICENSE
  - pybind11
  - README.md
  - test.py
- eigen-eigen-323c052e1731.tar.bz2
- install.sh
- install_get_dr.sh
- install_get_mesh.sh
- README.md
.gitmodules
main.py
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Disentangled Representation Learning for 3D Face Shape

This repository is the implementation of our CVPR 2019 paper <a href="https://arxiv.org/abs/1902.09887">"Disentangled Representation Learning for 3D Face Shape"</a>

Authors: [Zihang Jiang](https://zihangjiang.github.io/) ,[Qianyi Wu](https://qianyiwu.github.io), [Keyu Chen](https://kychern.github.io/) and [Juyong Zhang](http://staff.ustc.edu.cn/~juyong/) .
<p align="center">
<img src = "Pict/manifold.png" height = "400px"/>
</p>

# Our Proposed Framework

# Result Examples
<img src = "Pict/interpolation.png" height = "400px"/> 
We can manipulate 3D face shape in expression and identity code space.

# Usage

### Dataset
Please download [FaceWareHouse](http://kunzhou.net/zjugaps/facewarehouse/) dataset and extract it into the corresponding dir like `data/FaceWarehouse_Data`.

### Requirements
#### 1. Basic Environment
tensorflow-gpu = 1.9.0

Keras = 2.2.2

openmesh (both python version which can be installed through `pip install openmesh`, and [compiled version](https://www.openmesh.org/download/) are required)

##### 1.5 quick start

For convenience, we included repos for  [get_dr](https://github.com/zihangJiang/DR-Learning-for-3D-Face/tree/master/third_party/get_dr_py) and [get_mesh](https://github.com/zihangJiang/DR-Learning-for-3D-Face/tree/master/third_party/get_mesh_py) in third party folder and provided scripts for easy install. See [readme](https://github.com/zihangJiang/DR-Learning-for-3D-Face/blob/master/third_party/README.md) in third party folder for detail.

#### 2. Requirements for Data  Processing (About Deformation Representation Feature)

1.  We provide a python interface for obtaining ***Deformation Representation (DR) Feature***. Code are avaliable at [Here](https://github.com/QianyiWu/get_dr_py) to generate DR feature for each obj file by specific one reference mesh. Replace the `get_dr.cpython-36m-x86_64-linux-gnu.so` with compiled one and run `python get_fwh_dr.py` After that, you can change the data_path and data_format in `src/data_utils.py`.

2.  To recover mesh from DR feature, you need to compile [get_mesh](https://github.com/QianyiWu/get_mesh_py_API), and replace the `get_mesh.cpython-36m-x86_64-linux-gnu.so` in `src` folder.

3.  Also, python version of [libigl](https://github.com/libigl/libigl) is needed for mesh-IO and you need to replace the `pyigl.so` in `src` folder if the current one does not match your environment.

After all requirements are satisfied, you can use following command to train and test the model.
### Training

Run following command to generate training and testing data for 3D face DR learning
```bash
python src/data_utils.py
```
We have provided **DR feature of expression mesh on Meanface** we used in this project on [here](https://drive.google.com/open?id=1GgCKnKRrLR8r51Pw_TBqDHK8vdu6Oj4M).

Run this command to pretrain identity branch
```bash
python main.py -m gcn_vae_id -e 20
```

Run following command to pretrain expression branch
```bash
python main.py -m gcn_vae_exp -e 20
```

Run following command for end_to_end training the whole framework
```bash
python main.py -m fusion_dr -e 20
```

### Testing
You can test on each branch and the whole framework like 
```bash
python main.py -m fusion_dr -l -t
```
Our pretrained model can be found [here](https://drive.google.com/open?id=1LxxNY7wbjMXwrRdYJ4hJfXhg9ETAyIuQ).
### Evaluation
The `measurement.py` and `STED` folder is for computation of numerical result mentioned in our paper, including two reconstruction metrics and two decompostion metrics.

### Notes
1. if you train the model on your own dataset(for which topology is different from FaceWarehouse mesh), you have to recompute `Mean_Face.obj` and expression meshes on mean face as mentioned in our paper and regenerate the `FWH_adj_matrix.npz` in `data/disentagle` folder using `src/igl_test.py`.
2. We will release srcipts for data augmentation method metioned in our paper. You could put the augmented interpolated data in `data/disentangle/Interpolated_results`  
3. Currently we have fully tested this package on Ubuntu 16.04 LTS environment with CUDA 9.0. Windows and MacOS are not ensured working.
4. Errors like `Unknown CMake command "pybind11_add_module"` you may encounter while building [get_mesh](https://github.com/QianyiWu/get_mesh_py_API) and [get_dr](https://github.com/QianyiWu/get_dr_py) can be solved by 
```bash
git submodule init
git submodule update --recursive
```
5. If you have comments or questions, please contact Zihang Jiang (jzh0103@mail.ustc.edu.cn), Qianyi Wu (wqy9619@mail.ustc.edu.cn), Keyu Chen (cky95@mail.ustc.edu.cn), Juyong Zhang (juyong@ustc.edu.cn).
# Citation
Please cite the following papers if it helps your research:

<a href="https://arxiv.org/abs/1902.09887">Disentangled Representation Learning for 3D Face Shape</a>

@inproceedings{Jiang2019Disentangled,
          title={Disentangled Representation Learning for 3D Face Shape},
          author={Jiang, Zi-Hang and Wu, Qianyi and Chen, Keyu and Zhang, Juyong},
          booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
          year={2019}
    }

# Acknowledgement
GCN part code was inspired by https://github.com/tkipf/keras-gcn.
# License
Free for personal or research use, for commercial use please contact us via email.

DR-Learning-for-3D-Face

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

DR-Learning-for-3D-Face

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README