automoma

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 23 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">Scalable Trajectory Generation for Whole-Body Mobile Manipulation</h1>

<h3 align="center">CVPR 2026 Highlight</h3>

<div align="center">
 
 <a href="https://github.com/fi6">Yida Niu</a>1,3,4,5*&nbsp;&nbsp;
 <a href="https://chang-xinhai.github.io/">Xinhai Chang</a>1,3,4,5,6*&nbsp;&nbsp;
 <a href="https://openreview.net/profile?id=~Xin_Liu72">Xin Liu</a>4&nbsp;&nbsp;
 <a href="https://sites.google.com/g.ucla.edu/zyjiao/home">Ziyuan Jiao</a>2,4†&nbsp;&nbsp;
 <a href="https://yzhu.io/">Yixin Zhu</a>3,4,5,7†&nbsp;&nbsp;
 
 
 1Institute for AI, Peking University&nbsp;&nbsp;&nbsp;
 2Institute of Unmanned System, Beihang University&nbsp;&nbsp;&nbsp;
 3School of Psychological and Cognitive Sciences, Peking University&nbsp;&nbsp;&nbsp;
 4State Key Laboratory of General Artificial Intelligence&nbsp;&nbsp;&nbsp;
 5Beijing Key Laboratory of Behavior and Mental Health, Peking University&nbsp;&nbsp;&nbsp;
 6Yuanpei College, Peking University&nbsp;&nbsp;&nbsp;
 7Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence
 
 
 * Equal Contribution &nbsp;&nbsp;&nbsp;
 † Corresponding Author
 
</div>

## Overview

Whole-body mobile manipulation requires robots to coordinate mobile base and arm motion simultaneously. This coupled mobility and dexterity yields a state space that grows combinatorially with scene and object diversity, demanding datasets far larger than those sufficient for fixed-base manipulation.

**AutoMoMa** addresses this bottleneck with a **GPU-accelerated framework** that unifies **Articulated Kinematic Representation (AKR)** modeling with parallelized trajectory optimization. Our approach achieves **5,000 episodes per GPU-hour**—over **80× faster** than CPU-based baselines—producing a dataset of over **500k physically valid trajectories** spanning 330 scenes, diverse articulated objects, and multiple robot embodiments.

## Key Features

- **GPU-Accelerated Planning**: 5,000 episodes per GPU-hour via cuRobo integration
- **500k+ Trajectories**: Large-scale physically valid whole-body motion data
- **330 Diverse Scenes**: From AI2-THOR and Infinigen procedural generation
- **Multi-Robot Support**: Summit+Franka, TIAGo, and R1 embodiments
- **Grasp Switching**: Complex multi-step interactions in confined spaces
- **Imitation Learning Ready**: LeRobot-compatible dataset format

## News

- [2026-04-18] Release dataset [zarr-100k](https://huggingface.co/datasets/automoma/automoma-500k/tree/zarr-100k)!
- [2026-04-09] AutoMoMa has been selected as a highlight paper!
- [2026-01] AutoMoMa accepted to CVPR 2026!

## Pipeline Guide

For the detailed local handoff from asset preparation through planning, recording, conversion, training, and evaluation, see [`docs/pipeline.md`](docs/pipeline.md).

## Installation

### Prerequisites

- NVIDIA GPU with CUDA support
- Linux (Ubuntu 22.04 / 24.04)
- Python 3.11
- CUDA 12.8 recommended for CUDA extensions and PyTorch cu128
- NVIDIA driver R580 is recommended for Isaac Sim 5.1 camera rendering. Driver R590
  can segfault in headless `--enable_cameras` runs during RTX renderer startup.

Isaac Sim 5.1 and IsaacLab require Python 3.11. Do not use Python 3.12 for the full local pipeline.

### Installation Steps

**1) Create and activate conda environment**
```bash
conda create -y -n automoma python=3.11
conda activate automoma

cd <repo-root>
```

**2) Install common build tools and CUDA compiler**
```bash
python -m pip install -U pip wheel
pip install "setuptools>=71,<81"
conda install -y -c conda-forge ninja
conda install -y -c nvidia "cuda-nvcc=12.8.*"

export CUDA_HOME="$CONDA_PREFIX"
export PATH="$CUDA_HOME/bin:$PATH"
export OMNI_KIT_ACCEPT_EULA=YES
```

Optional but recommended: persist the environment variables on every `conda activate automoma`.
```bash
mkdir -p "$CONDA_PREFIX/etc/conda/activate.d"
cat > "$CONDA_PREFIX/etc/conda/activate.d/automoma.sh" <<'EOF'
export CUDA_HOME="$CONDA_PREFIX"
export PATH="$CUDA_HOME/bin:$PATH"
export OMNI_KIT_ACCEPT_EULA=YES
EOF
```

**3) Install CUDA-enabled PyTorch**
```bash
pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128
```

**4) Install AutoMoMa base package**
```bash
pip install -e .
```

**5) Install mode-specific dependencies**

For full local usage (`plan`, `record`, `convert`, `train`, `eval`, and `debug`), run both **B** and **C**.

**A) Plan-only dependencies (curobo)**
```bash
# Initialize submodules (once)
git submodule update --init --recursive third_party/curobo

# Install AutoMoMa plan extras and build/install curobo
pip install -e ".[plan]"
pip install -e "./third_party/curobo[isaacsim]" --no-build-isolation
```

**B) Sim dependencies (curobo + Isaac Sim + IsaacLab)**
```bash
# Initialize submodules (once)
git submodule update --init --recursive third_party/curobo third_party/IsaacLab-Arena

# Install AutoMoMa sim extras and build/install curobo
pip install -e ".[sim]" --extra-index-url https://pypi.nvidia.com
pip install -e "./third_party/curobo[isaacsim]" --no-build-isolation

# Create link for IsaacLab (pip default path)
cd third_party/IsaacLab-Arena/submodules/IsaacLab
ln -sf "$CONDA_PREFIX/lib/python3.11/site-packages/isaacsim" _isaac_sim
# If Isaac Sim is installed elsewhere, update the path above accordingly.

# Install IsaacLab + Arena
# flatdict 4.0.1 needs the non-isolated build env with setuptools<81.
pip install "flatdict==4.0.1" --no-build-isolation
./isaaclab.sh -i
cd <repo-root>
pip install -e ./third_party/IsaacLab-Arena/submodules/IsaacLab/source/isaaclab --no-build-isolation
pip install -e ./third_party/IsaacLab-Arena

# Auto-configure environment hooks
automoma install-hooks

# Keep Isaac Sim and Ray on a compatible click version
pip install "click==8.1.7"
```

**C) Train-only dependencies (no sim, no curobo)**
```bash
# Initialize submodules (once)
git submodule update --init --recursive third_party/lerobot

# Install AutoMoMa train extras and LeRobot
pip install -e ".[train]"
pip install -e ./third_party/lerobot
```

**6) Verify setup**
```bash
python --version
nvcc --version

python - <<'PY'
import torch
print("torch", torch.__version__)
print("torch cuda", torch.version.cuda)
print("cuda available", torch.cuda.is_available())
if torch.cuda.is_available():
 print(torch.cuda.get_device_name(0))
PY

automoma check
```

**Environment Notes**
- `automoma install-hooks` sets env vars on `conda activate` (no manual sourcing).
- `pip install -e ".[sim]" --extra-index-url https://pypi.nvidia.com` installs Isaac Sim 5.1.0 from NVIDIA's PyPI index as declared in `pyproject.toml`.
- `pip install -e ".[train]"` installs conversion/training dependencies, but LeRobot itself is installed from `third_party/lerobot`.

### Mode Dependencies

| Mode | Description |
|------|-------------|
| **plan** | Base + curobo (GPU motion planning) |
| **sim** | plan build deps + Isaac Sim 5.1.0 + IsaacLab + IsaacLab-Arena |
| **train** | LeRobot conversion/training dependencies (ACT, DP, dataset conversion, etc.) |
| **dev** | development tools; combine with other extras as needed |

### Training a Policy

```bash
# Activate environment (env vars auto-configured)
conda activate automoma

# Train a policy
lerobot-train \
    --policy.type=act \
    --dataset.repo_id=automoma/summit_franka_open-microwave_7221-scene_0_seed_0-30 \
    --dataset.root=data/lerobot \
    --batch_size=128 \
    --steps=10000 \
    --output_dir=outputs/train/act_example
```

### AutoMoMa pipeline dependency note

The unified `scripts/run_pipeline.sh` training routes require these extra Python packages in the active environment:

- `zarr>=2.18,<3`
- `fpsample>=1.0,<2`

They are included in `.[train]` in `pyproject.toml`.

### Note

If you encounter `ValueError: mutable default` errors when importing curobo, the included patches in `third_party/curobo/src/curobo/rollout/rollout_base.py` and `third_party/curobo/src/curobo/util/sample_lib.py` fix this for newer Python versions.

If you need to clean curobo build artifacts before rebuilding, use:
```bash
./tools/dev/clean_curobo_build.sh
```

### Isaac Sim Headless Camera Crash

If `record` or `eval` crashes in headless mode with `--enable_cameras` and a
backtrace in `librtx.scenedb.plugin.so` / `libcarb.scenerenderer-rtx.plugin.so`,
check the NVIDIA driver version:

```bash
nvidia-smi
```

Isaac Sim 5.1 camera rendering is known to be unstable with the R590 driver
branch in headless mode. Use one of these workarounds:

```bash
# Preferred long-term fix: use an R580 production driver such as 580.65.06.

# Short-term workaround on a local workstation with an active X session:
export DISPLAY=:1
bash scripts/run_pipeline.sh record microwave_7221 scene_0_seed_0 30 --no-headless
```

Also make sure no old planning or simulation process is still occupying GPU
memory before starting camera recording:

```bash
nvidia-smi
kill <stale-python-pid>
```

## Quick Start

For additional command examples, see `scripts/quickstart.sh` and `docs/workflows.md`. Maintainer utilities are documented in `docs/tools.md`.

### 1. Plan Trajectories

Generate trajectories for a specific object and scene:

```bash
# Default: Microwave 7221, scene_0_seed_0
python scripts/plan.py

# Custom object and scene
python scripts/plan.py object_id=7221 scene_name=scene_0_seed_0

# With collision visualization
python scripts/plan.py object_id=7221 scene_name=scene_0_seed_0 planner.visualize_collision=true
```

### 2. Record Trajectories in IsaacLab Arena

```bash
# Prepare object
python tools/assets/prepare_object.py --object_type Microwave --object_id 7221

# Record with 30 episodes.
# Default replay timing is --interpolated 5 --interpolation_type cubic --decimation 1 --init_steps 5.
# Base actions are absolute by default; add --mobile_base_relative only for relative-delta datasets.
bash scripts/run_pipeline.sh record microwave_7221 scene_0_seed_0 30

# Override interpolation only when doing an ablation/debug run.
bash scripts/run_pipeline.sh record microwave_7221 scene_1_seed_1 30 --interpolated 2
```

### 2.1 Replay Trajectories Without Recording HDF5

Use `replay` when you want to inspect planner trajectories or collect lightweight
metrics without spending time or disk on camera/HDF5 recording.

```bash
bash scripts/run_pipeline.sh replay microwave_7221 scene_0_seed_0 4 \
  --metrics \
  --metrics_file debug/replay_probe/microwave_scene0.csv \
  --episode_indices 0,1,2,3 \
  --interpolated 5 \
  --interpolation_type cubic \
  --decimation 1 \
  --init_steps 5
```

With `--metrics`, replay writes:
- a per-episode summary CSV at `--metrics_file`;
- per-step joint target/actual/error and EEF FK target/actual pose under
 `<metrics_file_parent>/debug_curves/`;
- joint-value, joint-error, EEF-pose, handle-tracking, and one joint/EEF gap
 PNG per replayed trajectory.

For the 5-object replay trace probe:

```bash
python tools/debug/run_grasp_filter_metrics.py \
  --max-objects 5 \
  --scenes-per-object 1 \
  --max-episodes-per-scene 50 \
  --interpolated 5 \
  --interpolation-type cubic \
  --decimation 1 \
  --init-steps 5 \
  --keep-going
```

### 3. Convert to LeRobot Format

```bash
# Convert recorded trajectories to LeRobot dataset
bash scripts/run_pipeline.sh convert lerobot microwave_7221 scene_0_seed_0 30

# Visualize converted LeRobot dataset
lerobot-dataset-viz \
    --repo-id automoma/summit_franka_open-microwave_7221-scene_0_seed_0-30 \
    --root data/lerobot/automoma/summit_franka_open-microwave_7221-scene_0_seed_0-30 \
    --episode-index 0
```

### 3.1 Visualize Raw AutoMoMa HDF5

Use `tools/dataset/viz_hdf5.py` to inspect raw AutoMoMa HDF5 recordings directly in an interactive Rerun viewer.

```bash
# Preview a small dataset (all demos become tabs)
python tools/dataset/viz_hdf5.py \
    data/automoma/summit_franka_open-microwave_7221-scene_0_seed_0-10.hdf5

# Preview specific demos only
python tools/dataset/viz_hdf5.py \
    data/automoma/summit_franka_open-microwave_7221-scene_0_seed_0-10.hdf5 \
    --demo-index 1,3-4

# Large datasets: load only one demo for fast inspection
python tools/dataset/viz_hdf5.py \
    data/automoma/code_validation/code_validation-microwave_7221-scene_0_seed_0-6400-set_state.hdf5 \
    --demo-index 123
```

Notes:
- `--demo-index` accepts a single demo id like `123` or a subset like `1,3-4`.
- If the dataset contains many demos and `--demo-index` is omitted, the viewer auto-limits to `demo_0` for fast startup.
- The viewer shows RGB/depth cameras, action/state time series, and keeps playback on `frame_index`.

### 4. GUI Recording (with Display)

```bash
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
export AUTOMOMA_SCENE_ROOT="$PWD/assets/scene/infinigen/scene_v2"
export AUTOMOMA_OBJECT_ROOT="$PWD/assets/object"
export AUTOMOMA_ROBOT_ROOT="$PWD/assets/robot"

cd third_party/IsaacLab-Arena
python isaaclab_arena/scripts/record_automoma_demos.py \
  --enable_cameras \
  --traj_file data/trajs/summit_franka/microwave_7221/scene_0_seed_0/train/traj_data_train.pt \
  --dataset_file data/automoma/test_gui.hdf5 \
  --num_episodes 1 \
  summit_franka_open_door \
  --object_name microwave_7221 \
  --scene_name scene_1_seed_1 \
  --object_center
```

Recording and eval use absolute mobile-base actions by default. Pass
`--mobile_base_relative` only when working with relative-delta action datasets.

### 5. Evaluation Semantics

Open-door eval now uses a final-time contact-aware rule instead of an "ever touched" summary.

An episode is successful only if both are true:
- `door_open_any=True`: the articulated joint exceeded the configured openness threshold during the episode.
- `final_engaged=True`: at the final timestep, `final_handle_distance <= 0.1` between the handle reference point and the closest robot point (EEF or fingertip).

Each eval run writes a live CSV to `<output_dir>/per_episode_results.csv` with per-episode fields such as:
- `success`
- `max_openness`
- `door_open_any`
- `final_engaged`
- `min_handle_distance`
- `final_handle_distance`

Handle/robot debug markers are optional and remain off by default. Enable them for eval/debug runs with:

```bash
DEBUG_VISUALIZE_HANDLE=true bash scripts/run_pipeline.sh eval lerobot act microwave_7221 scene_7_seed_7 1000 \
  --output_dir=outputs/eval/debug_handle_markers \
  --eval.n_episodes=10 \
  --env.headless=true
```

## Common Workflows

### Dishwasher Manipulation

```bash
# Prepare and record dishwasher trajectories
python tools/assets/prepare_object.py --object_type Dishwasher --object_id 11622
bash scripts/run_pipeline.sh record dishwasher_11622 scene_0_seed_0 30 --set_state --disable_collision
bash scripts/run_pipeline.sh convert lerobot dishwasher_11622 scene_0_seed_0 30
```

### Debugging Trajectories

```bash
# Debug IK solutions for a specific grasp
bash scripts/run_pipeline.sh debug microwave_7221 scene_0_seed_0 \
  --debug_file data/trajs/summit_franka/microwave_7221/scene_0_seed_0/train/grasp_0001/ik_data.pt \
  --num_episodes 1000 \
  --set_state

# Debug a specific per-grasp trajectory
bash scripts/run_pipeline.sh debug microwave_7221 scene_0_seed_0 \
  --debug_file data/trajs/summit_franka/microwave_7221/scene_0_seed_0/train/grasp_0001/traj_data.pt
```

## Dataset

AutoMoMa contains **500k+ physically valid whole-body trajectories** with:

| Property | Value |
|----------|-------|
| Total Episodes | 500,000+ |
| Scenes | 330 (30 curated + 300 procedural) |
| Robots | Summit+Franka, TIAGo, R1 |
| Objects | PartNet-Mobility articulated objects |
| Trajectory Length | 30 waypoints |
| Observations | RGB-D images + point clouds (4,096 points) |
| Render Resolution | 320×240 @ 30Hz |

### Dataset Statistics

The dataset covers diverse:
- **Solutions**: Multiple IK solutions per grasp
- **Scenes**: Increasing spatial confinement
- **Embodiments**: Different kinematic structures
- **Tasks**: Grasp switching, object relocation, chair pushing

### Download

Download the full dataset from [Hugging Face](https://huggingface.co/datasets/automoma/automoma-500k).

```bash
# HuggingFace
huggingface-cli download automoma/automoma-500k --repo-type dataset --local-dir ./data/automoma
```

## Method

### Articulated Kinematic Representation (AKR)

AKR unifies the mobile base, manipulator, and target object into a single serial kinematic chain:

1. **Virtual Base**: Mobile base planar motion modeled via virtual joints
2. **Kinematic Inversion**: Object kinematic tree inverted to re-root at grasp point
3. **Virtual Joint**: Attaches object to robot end-effector

### GPU-Accelerated Planning

AutoMoMa batches trajectory optimization and collision checking on GPU via cuRobo, enabling:
- Parallel IK solving
- Sphere-based collision approximation
- Constrained trajectory optimization

### Pipeline

```
Task Specification (S, O, R)
    ↓
Problem Instantiation (ESDF + AKR construction)
    ↓
Trajectory Generation (GPU-accelerated optimization)
    ↓
Rendering (Isaac Sim RGB-D + point clouds)
```

## Assets Structure

```
assets/
├── scene/
│   └── infinigen/           # Procedurally generated scenes (kitchen layouts)
├── object/                  # Articulated objects (URDF + meshes)
│   ├── Cabinet/
│   ├── Dishwasher/
│   ├── Microwave/
│   ├── Oven/
│   ├── Refrigerator/
│   └── TrashCan/
└── robot/
    └── summit_franka/       # Summit mobile base + Franka arm URDF
```

## Training Policies

AutoMoMa trajectories are compatible with LeRobot for policy training:

```bash
# Example: Train DP3 on AutoMoMa trajectories
lerobot-train \
    --config dp3 \
    --dataset.path automoma/automoma-500k \
    --env.type=isaaclab_arena
```

See [LeRobot documentation](https://github.com/huggingface/lerobot) for full training instructions.

## Citation

If our work assists your research, please cite:

```bibtex
@inproceedings{niu2026scalable,
  title     = {Scalable Trajectory Generation for Whole-Body Mobile Manipulation},
  author    = {Niu, Yida and Chang, Xinhai and Liu, Xin and Jiao, Ziyuan and Zhu, Yixin},
  year      = {2026},
  booktitle = {CVPR},
  url       = {https://yzhu.io/publication/tamp2026cvpr/paper.pdf}
}
```

## Acknowledgments

This work is supported by the National Science and Technology Innovation 2030 Major Program, National Natural Science Foundation of China, PKU-BingJi Joint Laboratory for Artificial Intelligence, and Wuhan East Lake High-Tech Development Zone.

## Related Projects

- [IsaacLab Arena](https://github.com/isaac-sim/IsaacLab-Arena) - GPU-accelerated robot simulation
- [LeRobot](https://github.com/huggingface/lerobot) - Robot learning framework
- [cuRobo](https://github.com/nvidia/curobo) - GPU-accelerated motion planning
- [PartNet-Mobility](https://sapien.ucsd.edu/) - Articulated object dataset

automoma

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README