KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

motion_tracking

★ 374 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Motion Tracking

[![HEFT Website](https://img.shields.io/badge/Website-heft.axell.top-0A66C2)](https://heft.axell.top/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://raw.githubusercontent.com/Axellwppr/motion_tracking/main/LICENSE)
![Python 3.10](https://img.shields.io/badge/Python-3.10-blue.svg)
![mjlab 1.3.0](https://img.shields.io/badge/mjlab-1.3.0-orange.svg)

This repository contains training, evaluation, export, and deployment-facing assets for
humanoid whole-body motion tracking.

> **Branch guide**
>
> - The **main branch** is the official implementation of
>   [HEFT: Heavy-Payload Full-size Humanoid Teleoperation with Privileged Motion
>   Guidance and Windowed Payload Curriculum](https://heft.axell.top/).
> - The **compliance branch** contains a G1 tracking + compliance framework inspired by
>   Gentle humanoid. [Demo](https://motion-tracking.axell.top/).
> - The **sim2real branch** provides the deployment runtime and available checkpoints.

The simulation and training utilities are based on **mjlab**.

## Repository Structure

To train HEFT G1/L7 policies, clone the default `main` branch:

```bash
git clone https://github.com/Axellwppr/motion_tracking.git
cd motion_tracking
```

To deploy released policies, clone the `sim2real` branch:

```bash
git clone -b sim2real --single-branch https://github.com/Axellwppr/motion_tracking.git motion_tracking_sim2real
cd motion_tracking_sim2real
```

To use the G1 compliance framework, clone the `compliance` branch:

```bash
git clone -b compliance --single-branch https://github.com/Axellwppr/motion_tracking.git motion_tracking_compliance
cd motion_tracking_compliance
```

## Release Status

HEFT consists of an efficient humanoid motion tracking framework, PMG
(Privileged Motion Guidance), and WPC (Windowed Payload Curriculum).

- [x] Efficient motion tracking training framework
- [x] PMG (Privileged Motion Guidance) support
- [x] WPC (Windowed Payload Curriculum) support
- [x] Released checkpoints on the `sim2real` branch
- [x] Deployment runtime on the `sim2real` branch
- [ ] Public training datasets and WPC window-payload labels
- [ ] VR data recording, reference reconstruction, and paired-dataset generation workflow

## Installation

This project uses `uv` for Python dependency and environment management. If you do not
have `uv` installed, follow the
[uv installation guide](https://docs.astral.sh/uv/getting-started/installation/).

```bash
uv sync
```

## Dataset Layout

Training loads motion memdatasets from `dataset/` by default. You can override the
dataset root with the `MEMPATH` environment variable:

```bash
export MEMPATH=/path/to/motion_tracking_dataset
```

The profile files reference paths relative to `MEMPATH`, for example:

```text
dataset/
  g1/
    lafan/
    100style/
    seed/all/
    vr_paired/
      clean/
      raw/
  l7/
    lafan/
    100style/
    seed/all/
    vr_paired/
      clean/
      raw/
```

The paired `vr_paired/{clean,raw}` folders are used by PMG. The clean side is used as
the teacher/reference target, while the raw side represents deployable student input.

We provide some preprocessed G1 memdatasets for smoke-testing the
training pipeline:
[Google Drive dataset](https://drive.google.com/drive/folders/1-FBUxllaYwqGIUSCaWg_4inD-u5Tdvi9?usp=sharing).
After downloading, place the extracted folders under `dataset/g1/` or set `MEMPATH` to
the extracted dataset root. These samples are not the exact HEFT training set, but they
are useful for validating the framework and obtaining similar training behavior. The
full dataset will be released later.

## Build A Motion Dataset

The dataset builder consumes retargeted robot `npz` files and writes a memdataset:

Each `npz` should follow the GMR-style robot motion format at 50 Hz. The required
fields are `root_pos`, `root_rot` in `xyzw` quaternion order, `dof_pos`,
`local_body_pos`, `joint_names`, and `body_names`. You can use the
[modified GMR exporter](https://github.com/Axellwppr/GMR) to generate compatible
retargeted motions.

```bash
uv run python scripts/data_process/generate_dataset.py \
  --dataset-root /path/to/retargeted_npz \
  --mem-path dataset/g1/my_motion_set
```

For paired PMG data, provide both clean teacher motions and raw student motions:

```bash
uv run python scripts/data_process/generate_dataset.py \
  --dataset-root /path/to/clean_npz \
  --mem-path dataset/g1/vr_paired/clean \
  --student-root /path/to/raw_npz \
  --student-mem-path dataset/g1/vr_paired/raw
```

If you write to `dataset/g1/my_motion_set`, use `g1/my_motion_set` in the YAML profile.
If you write under another root, set `MEMPATH` before training.

## Configure Training

Robot-specific training settings live in:

- `cfg/task/profile/G1_tracking.yaml`
- `cfg/task/profile/L7_tracking.yaml`

The most common changes are:

- dataset groups under `command.dataset.groups`
- dataset weights
- whether the `vr_paired` group is enabled
- `student_motion_randomization.enable`
- action/noise/randomization ranges for a specific robot

For PMG training with paired clean/raw data:

1. Enable the `vr_paired` dataset group.
2. Set `student_motion_randomization.enable: false`.

If paired PMG data is unavailable:

1. Leave the paired group commented out.
2. Keep `student_motion_randomization.enable: true`.

L7 WPC support is configured through `cfg/load/l7_wpc.yaml`, but the public WPC payload
labels and datasets are planned for a later release.

## Training

Use `train.sh` as the main entry point. Edit the global configuration at the top of the
file, especially:

- `PROJECT`
- `NPROC`
- the final `run_pipeline` line

For G1:

```bash
run_pipeline "G1_tracking" "g1_track" "release"
```

For L7:

```bash
run_pipeline "L7_tracking" "l7_track" "release"
```

Then run:

```bash
bash train.sh
```

`train.sh` runs the full three-stage pipeline:

1. `+exp=train`
2. `+exp=adapt`, initialized from the train run
3. `+exp=finetune`, initialized from the adapt run

With 8*8192 envs, training takes roughly 6 hours on 8 PRO6000 GPUs. If GPU memory
is constrained, reduce `NPROC` in `train.sh` and `num_envs` in
`cfg/task/common/base.yaml`. Reducing the total number of environments or the training
budget may significantly affect final performance.

## Evaluation And Export

Each stage saves a checkpoint every 150 iterations. During or after training, you can
use `eval.py` to evaluate checkpoint performance or export a deployment policy.

Play a trained policy:

```bash
uv run python scripts/eval.py --run_path ${wandb_run_path} -p
```

Export a deployment policy:

```bash
uv run python scripts/eval.py --run_path ${wandb_run_path} -p --export
```

Evaluate with an explicit profile:

```bash
uv run python scripts/eval.py --run_path ${wandb_run_path} --task tracking --profile L7_tracking -p
```

Exported policies are written under:

```text
scripts/exports/<task-name>-<timestamp>/
```

The exported folder contains the policy artifacts needed by the deployment runtime,
including `policy.onnx`, `policy.pt`, and `policy.json`.

## Deployment

Deployment assets are provided on the `sim2real` branch:

```bash
git clone -b sim2real --single-branch https://github.com/Axellwppr/motion_tracking.git motion_tracking_sim2real
```

That branch contains the deployment runtime, sim2sim/sim2real configuration, and
released policy assets.

motion_tracking

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README