nemo-cvpr2023

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 46)

[CVPR 2023 Highlight] Official implementation of "NeMo: 3D Neural Motion Fields from Multiple Video Instances of the Same Action"

nemo-cvpr2023 Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# NeMo [CVPR2023 Highlight]

[![Project Page](https://img.shields.io/badge/-project%20page-blueviolet)](https://sites.google.com/view/nemo-neural-motion-field)
[![arXiv](https://img.shields.io/badge/arxiv-2212.13660-red)](https://arxiv.org/abs/2212.13660)

This repo contains the official PyTorch implementation of our paper:

**NeMo: 3D Neural Motion Fields from Multiple Video Instances of the Same Action**  
_by_ [Kuan-Chieh (Jackson) Wang](https://wangkua1.github.io/), [Zhenzhen Weng](https://zzweng.github.io/), [Maria Xenochristou](https://mariaxen.github.io/), [Joao Pedro Araujo](https://scholar.google.com/citations?user=fs8k5acAAAAJ&hl=en), [Jeffrey Gu](https://its-gucci.github.io/), [C. Karen Liu](https://tml.stanford.edu/), [Serena Yeung](https://ai.stanford.edu/~syyeung/)

**([Project Page 🌐](https://sites.google.com/view/nemo-neural-motion-field) | [Paper 📄](https://openaccess.thecvf.com/content/CVPR2023/html/Wang_NeMo_Learning_3D_Neural_Motion_Fields_From_Multiple_Video_Instances_CVPR_2023_paper.html) | [Data 📀](https://drive.google.com/drive/folders/141-mBNwaSxYa4TB7HR4emDbMJ3ykc964?usp=share_link))**

![NeMo System Figure](https://raw.githubusercontent.com/wangkua1/nemo-cvpr2023/main/asset/nemo-sys-fig.jpeg)
![NeMo Demo](https://raw.githubusercontent.com/wangkua1/nemo-cvpr2023/main/<asset/baseball%20pitch%20(test%201).gif>)

# Installation

### Environment

1. Clone this repository.

```
git clone git@github.com:wangkua1/nemo-cvpr2023.git
```

2. Create a conda environment using the provided environment file.

```
conda env create -f environment.yml
```

Then, activate the conda environment using

```
conda active nemo
```

3. Pip install the missing packages using the provided requirements file.

```
pip install -r requirements.txt
```

### Other software downloads

1. Create a directory `software`.

2. Download the required components:

- "V02_05" -- requied by human_body_prior. Follow the original instructions at the [VPoser github page](https://github.com/nghorbani/human_body_prior).
- "spin_data" -- Follow the original instructions at the [SPIN github page](https://github.com/nkolot/SPIN).
- "smpl" -- Follow the original instructions at [their website](https://smpl.is.tue.mpg.de/).

Alternatively, download them at this [link](https://drive.google.com/drive/folders/1mEqMgHRndJK_4skVP-3v4mxUIS0FapQk?usp=share_link) (~0.5GB).
Note, we only provide these for the purpose of reproducing our work, _please respect the original instruction, license, and copyright_.

# Dataset: _NeMo-MoCap_

1. Download the dataset from this [Google Drive folder](https://drive.google.com/drive/folders/141-mBNwaSxYa4TB7HR4emDbMJ3ykc964?usp=share_link) (~1GB).
   You should organize your files into the following structure:

```

2. Convert the mp4 videos into frames (takes <3min).

```bash
python -m scripts.video_to_frames
```

# Running NeMo

A example inference script for running NeMo on the "Baseball Pitch" motion on the NeMo-MoCap dataset is provided `run_scripts/examples`.  
You can run it locally using the following command:

```
bash run_scripts_examples/nemomocap-example.sh 0
```

or launching a SLURM job with `sbatch` using

```
bash run_scripts_examples/nemomocap-example.sh 1
```

# Running NeMo on your custom video dataset

If you wish to run NeMo on your own video dataset, refer to [Custom Video README](custom_video/README.md).

# Acknowledgement

NeMo is built on many other great works, including
[VPoser](https://github.com/nghorbani/human_body_prior),
[SPIN](https://github.com/nkolot/SPIN),
[SMPL](https://smpl.is.tue.mpg.de/),
[HMR](https://github.com/akanazawa/hmr),
[VIBE](https://github.com/mkocabas/VIBE),
[DAPA](https://github.com/zzweng/dapa_release),
[GLAMR](https://github.com/NVlabs/GLAMR).

# Citation

If you find this work useful, please consider citing:

```
@inproceedings{wang2022nemo,
  title={NeMo: 3D Neural Motion Fields from Multiple Video Instances of the Same Action},
  author={Wang, Kuan-Chieh and Weng, Zhenzhen and Xenochristou, Maria and Araujo, Joao Pedro and Gu, Jeffrey and Liu, C Karen and Yeung, Serena},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023},
  arxiv={2212.13660}
}
```

nemo-cvpr2023

nemo-cvpr2023 Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

nemo-cvpr2023

nemo-cvpr2023 Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README