egoallo

KO | EN

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #cli #claude-code #windows #codex #developer-tools #dsh

★ 292 Open GitHub ↗

Estimating Body and Hand Motion in an Ego-sensed World

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# egoallo

**[Project page](https://egoallo.github.io/) &bull;
[arXiv](https://arxiv.org/abs/2410.03665)**

Code release for our preprint:

<table><tr><td>
 Brent Yi1, Vickie Ye1, Maya Zheng1, Yunqi Li2, Lea M&uuml;ller1, Georgios Pavlakos3, Yi Ma1, Jitendra Malik1, and Angjoo Kanazawa1.
 Estimating Body and Hand Motion in an Ego-sensed World.
 arXiV, 2024.
</td></tr>
</table>
1UC Berkeley, 2ShanghaiTech, 3UT Austin

---

## Updates

- **Oct 7, 2024:** Initial release. (training code, core implementation details)
- **Oct 14, 2024:** Added model checkpoint, dataset preprocessing, inference, and visualization scripts.
- **May 6, 2025:** Updated scripts + instructions for dataset preprocessing, which is now self-contained in this repository.

## Overview

**TLDR;** We use egocentric SLAM poses and images to estimate 3D human body pose, height, and hands.

https://github.com/user-attachments/assets/7d28e07f-ab83-4749-ac6b-abe692d9ba20

This repository is structured as follows:

```
.
├── download_checkpoint_and_data.sh
│                            - Download model checkpoint and sample data.
├── 0_preprocess_training_data.py
│                            - Preprocessing script for training datasets.
├── 1_train_motion_prior.py
│                            - Training script for motion diffusion model.
├── 2_run_hamer_on_vrs.py
│                            - Run HaMeR on inference data (expects Aria VRS).
├── 3_aria_inference.py
│                            - Run full pipeline on inference data.
├── 4_visualize_outputs.py
│                            - Visualize outputs from inference.
├── 5_eval_body_metrics.py
│                            - Compute and print body estimation accuracy metrics.
│
├── src/egoallo/
│   ├── data/                - Dataset utilities.
│   ├── transforms/          - SO(3) / SE(3) transformation helpers.
│   └── *.py                 - All core implementation.
│
└── pyproject.toml          - Python dependencies/package metadata.
```

## Getting started

EgoAllo requires Python 3.12 or newer.

1. **Clone the repository.**
   ```bash
   git clone https://github.com/brentyi/egoallo.git
   ```
2. **Install general dependencies.**
   ```bash
   cd egoallo
   pip install -e .
   ```
3. **Download+unzip model checkpoint and sample data.**

```bash
   bash download_checkpoint_and_data.sh
   ```

You can also download the zip files manually: here are links to the [checkpoint](https://drive.google.com/file/d/14bDkWixFgo3U6dgyrCRmLoXSsXkrDA2w/view?usp=drive_link) and [example trajectories](https://drive.google.com/file/d/14zQ95NYxL4XIT7KIlFgAYTPCRITWxQqu/view?usp=drive_link).

4. **Download the SMPL-H model file.**

You can find the "Extended SMPL+H model" (16 shape parameters) from the [MANO project webpage](https://mano.is.tue.mpg.de/).
   Our scripts assumes an npz file located at `./data/smplh/neutral/model.npz`, but this can be overridden at the command-line (`--smplh-npz-path {your path}`).

5. **Visualize model outputs.**

The example trajectories directory includes example outputs from our model. You can visualize them with:

```bash
   python 4_visualize_outputs.py --search-root-dir ./egoallo_example_trajectories
   ```

## Running inference

1. **Installing inference dependencies.**

Our guidance optimization uses a Levenberg-Marquardt optimizer that's implemented in JAX. If you want to run this on an NVIDIA GPU, you'll need to install JAX with CUDA support:

```bash
   # Also see: https://jax.readthedocs.io/en/latest/installation.html
   pip install "jax[cuda12]==0.6.1"
   ```

You'll also need [jaxls](https://github.com/brentyi/jaxls):

```bash
   pip install git+https://github.com/brentyi/jaxls.git
   ```

2. **Running inference on example data.**

Here's an example command for running EgoAllo on the "coffeemachine" sequence:

```bash
   python 3_aria_inference.py --traj-root ./egoallo_example_trajectories/coffeemachine
   ```

You can run `python 3_aria_inference.py --help` to see the full list of options.

3. **Running inference on your own data.**

To run inference on your own data, you can copy the structure of the example trajectories. The key files are:

- A VRS file from Project Aria, which contains calibrations and images.
   - SLAM outputs from Project Aria's MPS: `closed_loop_trajectory.csv` and `semidense_points.csv.gz`.
   - (optional) HaMeR outputs, which we save to a `hamer_outputs.pkl`.
   - (optional) Project Aria wrist and palm tracking outputs.

4. **Running HaMeR on your own data.**

To generate the `hamer_outputs.pkl` file, you'll need to install [hamer_helper](https://github.com/brentyi/hamer_helper).

Then, as an example for running on our coffeemachine sequence:

```bash
   python 2_run_hamer_on_vrs.py --traj-root ./egoallo_example_trajectories/coffeemachine
   ```

## Preprocessing Training Data

To train the motion prior model, we use data from the [AMASS dataset](https://amass.is.tue.mpg.de/). Due to licensing constraints, we cannot redistribute the preprocessed data. Instead, we provide two sequential preprocessing scripts:

1. **Download the AMASS dataset.**

Download the AMASS dataset from the [official website](https://amass.is.tue.mpg.de/). We use the following splits:

- **Training**: ACCAD, BioMotionLab_NTroje, BMLhandball, BMLmovi, CMU, DanceDB, DFaust_67EKUT, Eyes_Japan_Dataset, KIT, MPI_Limits, TCD_handMocap, TotalCapture
   - **Validation**: HumanEva, MPI_HDM05, SFU, MPI_mosh
   - **Testing**: Transitions_mocap, SSM_synced

2. **Run the first preprocessing script.**

```bash
   python 0a_preprocess_training_data.py --help
   python 0a_preprocess_training_data.py --data-root /path/to/amass --smplh-root ./data/smplh
   ```

This script, adapted from HuMoR, processes raw AMASS data by:

- Converting to gender-neutral SMPL-H parameters
   - Computing contact labels for feet, hands, and knees
   - Filtering out problematic sequences (treadmill walking, sequences with foot skating)
   - Downsampling to 30fps

3. **Run the second preprocessing script.**

```bash
   python 0b_preprocess_training_data.py --help
   python 0b_preprocess_training_data.py --data-npz-dir ./data/processed_30fps_no_skating/
   ```

This converts the processed NPZ files to a unified HDF5 format for more efficient training, with optimized chunk sizes for reading sequences.

## Status

This repository currently contains:

- `egoallo` package, which contains reference training and sampling implementation details.
- Training script.
- Model checkpoints.
- Dataset preprocessing script.
- Inference script.
- Visualization script.
- Setup instructions.

While we've put effort into cleaning up our code for release, this is research
code and there's room for improvement. If you have questions or comments,
please reach out!

egoallo

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README