RoboManipAug

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 15 Open GitHub ↗

Self-Augmented Robot Trajectory: Efficient Imitation Learning via Safe Self-augmentation with Demonstrator-annotated Precision

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# [RoboManipAug](https://github.com/isri-aist/RoboManipAug)
Self-Augmented Robot Trajectory: Efficient Imitation Learning via Safe Self-augmentation with Demonstrator-annotated Precision.

[![Project Page](https://img.shields.io/badge/Project-Page-blue?logo=github)](https://sites.google.com/view/sart-il/)
[![LICENSE](https://img.shields.io/github/license/isri-aist/RoboManipAug)](https://github.com/isri-aist/RoboManipAug/blob/main/LICENSE)

## Install
Install RoboManipBaselines with the `act` extra as described [here](https://github.com/isri-aist/RoboManipBaselines/blob/master/doc/install.md#act).

> [!NOTE]
> Operation has been verified with the [RoboManipAug](https://github.com/isri-aist/RoboManipBaselines/tree/RoboManipAug) tag of RoboManipBaselines. Unless you have a specific reason to use the latest features, installing this tag is the more stable option.

Install RoboManipAug:
```console
$ git clone git@github.com:isri-aist/RoboManipAug.git --recursive
$ cd RoboManipAug
$ pip install -e .
```

> [!NOTE]
> Clone both `RoboManipBaselines` and `RoboManipAug` under the same directory, e.g.:
> ```console
> $ ls <workspace_dir>
> RoboManipBaselines  RoboManipAug
> ```

## Examples
### Manual data collection
#### Collect single data by teleoperation
```console
# Go to the top directory of RoboManipBaselines
$ cd robo_manip_baselines
$ python bin/Teleop.py MujocoUR5eInsert --world_idx_list 2
```

https://github.com/user-attachments/assets/789415ee-1b59-48f8-bf3a-fbe5a710edea

Move the saved RMB file to `robo_manip_aug/teleop_data/sample/MujocoUR5eInsert_base_demo.rmb`.  
(The `sample` directory in the following can be an arbitrary name.)

#### Generate environment point cloud from the hand camera RGBD images in the teleoperation data
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/GenerateMergedPointCloud.py ./teleop_data/sample/MujocoUR5eInsert_base_demo.rmb ./env_data/sample/MujocoUR5eInsert.pcd
```

#### [Optional] Visualize environment point cloud with a 3D viewer
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/VisualizePointCloud.py ./env_data/sample/MujocoUR5eInsert.pcd
```

#### Annotate acceptable regions of end-effectors in data augmentation
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/AnnotateAcceptableRegion.py ./teleop_data/sample/MujocoUR5eInsert_base_demo.rmb ./annotation_data/sample/MujocoUR5eInsert_annotation.pkl --point_cloud_path ./env_data/sample/MujocoUR5eInsert.pcd
```
When the `--load_annotation` option is specified, the already saved acceptable regions will be visualized.

### Automatic data augmentation
#### Collect augmented data within the acceptable regions in the simulation
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/CollectAugmentedData.py MujocoUR5eInsert ./teleop_data/sample/MujocoUR5eInsert_base_demo.rmb ./annotation_data/sample/MujocoUR5eInsert_annotation.pkl
```

https://github.com/user-attachments/assets/b1580334-83b7-4009-80a6-ed5e74c9ef3b

The augmented data is stored in `./augmented_data/MujocoUR5eInsert_<data_suffix>`. Rename this directory to `./augmented_data/sample/MujocoUR5eInsert`.

#### [Optional] Visualize the trajectories of the collected data with a 3D viewer
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/VisualizeData3D.py ./augmented_data/sample/MujocoUR5eInsert/ --base_demo_path ./teleop_data/sample/MujocoUR5eInsert_base_demo.rmb --point_cloud_path ./env_data/sample/MujocoUR5eInsert.pcd
```

#### [Optional] Plot base data and augmented data
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./misc/PlotAugmentedData.py ./augmented_data/sample/MujocoUR5eInsert/base_demo.rmb ./augmented_data/sample/MujocoUR5eInsert/region000/MujocoUR5eInsert_augmented_region000_00.rmb
```

You can plot the end-effector pose by specifying the `--data_key eef_pose` option (by default, `joint_pos` is plotted).

#### [Optional] Replay the augmented data
```console
# Go to the top directory of RoboManipBaselines
$ cd robo_manip_baselines
$ python ./bin/Teleop.py MujocoUR5eInsert --replay_log ../../RoboManipAug/robo_manip_aug/augmented_data/sample/MujocoUR5eInsert/region000/MujocoUR5eInsert_augmented_region000_00.rmb --replay_keys command_eef_pose_rel
```

### Policy
#### Compose learning data
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/ComposeDataset.py ./augmented_data/sample/MujocoUR5eInsert ./learning_data/sample/MujocoUR5eInsert
```

By adding the `--num_data_per_region <N>` option, you can specify the number of data per region to be N (by default, all data is used).

#### Train policy
```console
# Go to the top directory of RoboManipBaselines
$ cd robo_manip_baselines
$ python ./bin/Train.py Mlp --dataset_dir ../../RoboManipAug/robo_manip_aug/learning_data/sample/MujocoUR5eInsert/ --checkpoint_dir ./checkpoint/sample/MujocoUR5eInsert/ --state_keys --action_keys command_eef_pose_rel --camera_names hand --train_ratio 1.0 --val_ratio 0.2
```

#### Rollout policy
```console
# Go to the top directory of RoboManipBaselines
$ cd robo_manip_baselines
$ python ./bin/Rollout.py Mlp MujocoUR5eInsert --checkpoint ./checkpoint/sample/MujocoUR5eInsert/policy_last.ckpt --world_idx 2 --world_random_scale 0.05 0.05 0.0
```

By using the option `--world_idx_list 2 2 2 2` instead of `--world_idx 2`, the rollout will be repeated four times.

### [Deprecated] Environment
#### Extract images for learning 3D gaussian splatting from teleoperation data
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/ExtractImagesFromData.py <path_to_rmb> --out_dir ./env_data/MujocoUR5eInsert
```

#### Visualize environment point cloud with a 3D viewer
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./bin/VisualizePointCloud.py ./env_data/MujocoUR5eInsert_SuGaR.ply
```

### Misc scripts
#### Visualize training trajectories
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./misc/VisualizeTrainingData.py <dir_to_RMBs> --base_demo_path <path_to_RMB> --point_cloud_path <path_to_pcd>
```

#### Visualize rollout trajectories
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./misc/VisualizeRollout.py --proposed_path <path_to_RMB> --baseline_path <path_to_RMB> --human_path <path_to_RMB> --replay_path <path_to_RMB> --point_cloud_path <path_to_pcd>
```

#### Add relative gripper joint position data to RMB file
```console
# Go to the top directory of RoboManipAug
$ cd robo_manip_aug
$ python ./misc/AddGripperRelToRmbData.py <rmb_file>
```
`<rmb_file>` can be either the path to an RMB file or the path to a directory containing multiple RMB files.

RoboManipAug

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README