CoMo

KO | EN

gitlite — search

#python #hacktoberfest #react #typescript #golang #machine-learning #deep-learning #docker #macos #windows #security #mcp #ios #openai

(★ 16)

[CVPR 2026] CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning

CoMo 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## [CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning](https://arxiv.org/pdf/2505.17006?)

[Jiange Yang](https://yangjiangeyjg.github.io/), [Yansong Shi](https://scholar.google.com/citations?user=R7J57vQAAAAJ&hl=zh-CN&oi=sra), [Haoyi Zhu](https://github.com/HaoyiZhu), [Mingyu Liu](https://mingyulau.github.io/), [Kaijing Ma](https://github.com/KaijingOfficial), [Yating Wang](https://scholar.google.com/citations?user=5SuBWh0AAAAJ&hl=zh-CN&oi=sra), [Gangshan Wu](https://mcg.nju.edu.cn/member/gswu/index.html), [Tong He](https://tonghe90.github.io/), [Limin Wang](https://wanglimin.github.io/)

## Overview

![CoMo Overview](https://raw.githubusercontent.com/MCG-NJU/CoMo/master/method.png)

### Training latent motion IDM.

Prepare the training video data by following the list format used in [idm/mask_extracted_40k.txt](idm/mask_extracted_40k.txt), [idm/droid_success_40k.txt](idm/droid_success_40k.txt), and [idm/samv_40k.txt](idm/samv_40k.txt).

Example command (uses 4 GPUs):

```bash
cd idm
CUDA_VISIBLE_DEVICES=0,1,2,3 NUM_PROCESSES=4 MAIN_PROCESS_PORT=18897 bash train.sh configs/train_como_default.json
```

### Extracting latent motion using trained IDM as pseudo action labels.

Example: Download the LIBERO dataset from [ATM](https://github.com/Large-Trajectory-Model/ATM) first. Extract latent motion from ATM LIBERO-10 data

```bash
cd idm
python process.py \
  --model-path /path/to/your/model.pt \
  --data-folder /path/to/ATM/libero/libero_10/*/images/*/ \
  --output-file latent_motion_our \
  --model-config-path configs/process_default.json 
```

### Training and evaluating the unified CoMo policy.

The policy stage jointly trains on robot demonstrations and human-video latent motion labels. The preprocessed HDF5 files should follow the ATM/LIBERO layout under a suite directory such as:

```text
<COMO_LIBERO_DATA_ROOT>/
  libero_10/
    <task_name>_demo/
      demo_0.hdf5
      env_meta.json
      images/
```

For each task, CoMo uses 50 demos:

- `demo_0` to `demo_9`: robot demos with 7-D actions
- `demo_10` to `demo_49`: human-video demos whose `root/actions` are replaced by 256-D latent motion

By default the latent-motion files are expected at:

```text
<COMO_LIBERO_DATA_ROOT>/latent_motion/latent_motion_our/<suite>/<task_name>_demo/demo_*.hdf5
```

`--latent-action-tag` controls the subdirectory name in that path. `--latent-action-root` can be used to point to a different latent-motion directory directly.

If you need to preprocess raw LIBERO demonstrations into the ATM/LIBERO layout:

```bash
cd DP
python scripts/preprocess_libero.py \
  --root /path/to/raw/libero \
  --save /path/to/atm_libero \
  --suite libero_10
```

Train the policy on a target LIBERO suite (example: `libero_10`):

```bash
cd DP
export COMO_LIBERO_DATA_ROOT=/path/to/atm_libero
python scripts/train_libero_policy.py --suite libero_10
```

Or pass the data root explicitly:

```bash
cd DP
python scripts/train_libero_policy.py \
  --suite libero_10 \
  --data-root /path/to/atm_libero
```

If the latent-motion files are stored elsewhere:

```bash
cd DP
python scripts/train_libero_policy.py \
  --suite libero_10 \
  --data-root /path/to/atm_libero \
  --latent-action-root /path/to/latent_motion_our
```

If you only need to switch the latent tag:

```bash
cd DP
python scripts/train_libero_policy.py \
  --suite libero_10 \
  --data-root /path/to/atm_libero \
  --latent-action-tag latent_motion_our
```

Evaluate a trained policy checkpoint directory:

```bash
cd DP
export COMO_LIBERO_DATA_ROOT=/path/to/atm_libero
python scripts/eval_libero_policy.py \
  --suite libero_object \
  --exp-dir /path/to/DP/results/policy/<your_experiment_dir>
```

## Citation

```bibtex
@article{yang2025learning,
  title={Como: Learning continuous latent motion from internet videos for scalable robot learning},
  author={Yang, Jiange and Shi, Yansong and Zhu, Haoyi and Liu, Mingyu and Ma, Kaijing and Wang, Yating and Wu, Gangshan and He, Tong and Wang, Limin},
  journal={arXiv preprint arXiv:2505.17006},
  year={2025}
}
```

## Contact

jiangeyang.jgy@gmail.com

## Acknowledgements

We thank the authors and open-source communities of the following projects:

- [Moto-GPT](https://github.com/TencentARC/Moto)
- [RoboticsDiffusionTransformer](https://github.com/thu-ml/RoboticsDiffusionTransformer)
- [LAPA](https://github.com/LatentActionPretraining/LAPA)
- [deoxys_control](https://github.com/UT-Austin-RPL/deoxys_control)
- [ATM](https://github.com/Large-Trajectory-Model/ATM)

CoMo

CoMo 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

CoMo

CoMo 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README