RoboManipAug

KO | EN

gitlite — search

#javascript #python #react #typescript #go #linux #macos #windows #cli #vue #awesome-list #security #openai #open-source

(★ 15)

Self-Augmented Robot Trajectory: Efficient Imitation Learning via Safe Self-augmentation with Demonstrator-annotated Precision

RoboManipAug Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# [RoboManipAug](https://github.com/isri-aist/RoboManipAug)
Self-Augmented Robot Trajectory: Efficient Imitation Learning via Safe Self-augmentation with Demonstrator-annotated Precision.

[![Project Page](https://img.shields.io/badge/Project-Page-blue?logo=github)](https://sites.google.com/view/sart-il/)
[![LICENSE](https://img.shields.io/github/license/isri-aist/RoboManipAug)](https://github.com/isri-aist/RoboManipAug/blob/main/LICENSE)

## Install
Install RoboManipBaselines with the `act` extra as described [here](https://github.com/isri-aist/RoboManipBaselines/blob/master/doc/install.md#act).

> [!NOTE]
> Operation has been verified with the [RoboManipAug](https://github.com/isri-aist/RoboManipBaselines/tree/RoboManipAug) tag of RoboManipBaselines. Unless you have a specific reason to use the latest features, installing this tag is the more stable option.

Install RoboManipAug:
```console
$ git clone git@github.com:isri-aist/RoboManipAug.git --recursive
$ cd RoboManipAug
$ pip install -e .
```

> [!NOTE]
> Clone both `RoboManipBaselines` and `RoboManipAug` under the same directory, e.g.:
> ```console
> $ ls <workspace_dir>
> RoboManipBaselines  RoboManipAug
> ```

## Examples
### Manual data collection
#### Collect single data by teleoperation
```console
# Go to the top directory of RoboManipBaselines
$ cd robo_manip_baselines
$ python bin/Teleop.py MujocoUR5eInsert --world_idx_list 2
```

https://github.com/user-attachments/assets/789415ee-1b59-48f8-bf3a-fbe5a710edea

Move the saved RMB file to `robo_manip_aug/teleop_data/sample/MujocoUR5eInsert_base_demo.rmb`.  
(The `sample` directory in the following can be an arbitrary name.)

#### Generate environment point cloud from the hand camera RGBD images in the teleoperation data
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/GenerateMergedPointCloud.py ./teleop_data/sample/MujocoUR5eInsert_base_demo.rmb ./env_data/sample/MujocoUR5eInsert.pcd
```

#### [Optional] Visualize environment point cloud with a 3D viewer
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/VisualizePointCloud.py ./env_data/sample/MujocoUR5eInsert.pcd
```

#### Annotate acceptable regions of end-effectors in data augmentation
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/AnnotateAcceptableRegion.py ./teleop_data/sample/MujocoUR5eInsert_base_demo.rmb ./annotation_data/sample/MujocoUR5eInsert_annotation.pkl --point_cloud_path ./env_data/sample/MujocoUR5eInsert.pcd
```
When the `--load_annotation` option is specified, the already saved acceptable regions will be visualized.

### Automatic data augmentation
#### Collect augmented data within the acceptable regions in the simulation
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/CollectAugmentedData.py MujocoUR5eInsert ./teleop_data/sample/MujocoUR5eInsert_base_demo.rmb ./annotation_data/sample/MujocoUR5eInsert_annotation.pkl
```

https://github.com/user-attachments/assets/b1580334-83b7-4009-80a6-ed5e74c9ef3b

The augmented data is stored in `./augmented_data/MujocoUR5eInsert_<data_suffix>`. Rename this directory to `./augmented_data/sample/MujocoUR5eInsert`.

#### [Optional] Visualize the trajectories of the collected data with a 3D viewer
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/VisualizeData3D.py ./augmented_data/sample/MujocoUR5eInsert/ --base_demo_path ./teleop_data/sample/MujocoUR5eInsert_base_demo.rmb --point_cloud_path ./env_data/sample/MujocoUR5eInsert.pcd
```

#### [Optional] Plot base data and augmented data
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./misc/PlotAugmentedData.py ./augmented_data/sample/MujocoUR5eInsert/base_demo.rmb ./augmented_data/sample/MujocoUR5eInsert/region000/MujocoUR5eInsert_augmented_region000_00.rmb
```

You can plot the end-effector pose by specifying the `--data_key eef_pose` option (by default, `joint_pos` is plotted).

#### [Optional] Replay the augmented data
```console
# Go to the top directory of RoboManipBaselines
$ cd robo_manip_baselines
$ python ./bin/Teleop.py MujocoUR5eInsert --replay_log ../../RoboManipAug/robo_manip_aug/augmented_data/sample/MujocoUR5eInsert/region000/MujocoUR5eInsert_augmented_region000_00.rmb --replay_keys command_eef_pose_rel
```

### Policy
#### Compose learning data
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/ComposeDataset.py ./augmented_data/sample/MujocoUR5eInsert ./learning_data/sample/MujocoUR5eInsert
```

By adding the `--num_data_per_region <N>` option, you can specify the number of data per region to be N (by default, all data is used).

#### Train policy
```console
# Go to the top directory of RoboManipBaselines
$ cd robo_manip_baselines
$ python ./bin/Train.py Mlp --dataset_dir ../../RoboManipAug/robo_manip_aug/learning_data/sample/MujocoUR5eInsert/ --checkpoint_dir ./checkpoint/sample/MujocoUR5eInsert/ --state_keys --action_keys command_eef_pose_rel --camera_names hand --train_ratio 1.0 --val_ratio 0.2
```

#### Rollout policy
```console
# Go to the top directory of RoboManipBaselines
$ cd robo_manip_baselines
$ python ./bin/Rollout.py Mlp MujocoUR5eInsert --checkpoint ./checkpoint/sample/MujocoUR5eInsert/policy_last.ckpt --world_idx 2 --world_random_scale 0.05 0.05 0.0
```

By using the option `--world_idx_list 2 2 2 2` instead of `--world_idx 2`, the rollout will be repeated four times.

### [Deprecated] Environment
#### Extract images for learning 3D gaussian splatting from teleoperation data
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/ExtractImagesFromData.py <path_to_rmb> --out_dir ./env_data/MujocoUR5eInsert
```

#### Visualize environment point cloud with a 3D viewer
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/VisualizePointCloud.py ./env_data/MujocoUR5eInsert_SuGaR.ply
```

### Misc scripts
#### Visualize training trajectories
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./misc/VisualizeTrainingData.py <dir_to_RMBs> --base_demo_path <path_to_RMB> --point_cloud_path <path_to_pcd>
```

#### Visualize rollout trajectories
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./misc/VisualizeRollout.py --proposed_path <path_to_RMB> --baseline_path <path_to_RMB> --human_path <path_to_RMB> --replay_path <path_to_RMB> --point_cloud_path <path_to_pcd>
```

#### Add relative gripper joint position data to RMB file
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./misc/AddGripperRelToRmbData.py <rmb_file>
```
`<rmb_file>` can be either the path to an RMB file or the path to a directory containing multiple RMB files.

RoboManipAug

RoboManipAug Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

RoboManipAug

RoboManipAug Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README