RL_arm_under_sparse_reward - GitLite

KO | EN

gitlite — search

#python #hacktoberfest #typescript #llm #golang #machine-learning #deep-learning #macos #java #kubernetes #vue #awesome-list #security #ios

RL_arm_under_sparse_reward

(★ 122)

A reinforcement learning project for robotic arm under sparse reward

파일 탐색기

최종 버전 다운로드 (.zip)

- arguments.cpython-38.pyc
- ddpg_agent.cpython-38.pyc
- get_to_handle.cpython-38.pyc
- her.cpython-38.pyc
- models.cpython-38.pyc
- normalizer.cpython-38.pyc
- replay_buffer.cpython-38.pyc
- utils.cpython-38.pyc
- - bmirobot.cpython-38.pyc
  - bmirobot_env.cpython-38.pyc
  - bmirobot_env_pickandplace_v2.cpython-38.pyc
  - bmirobot_env_push_F.cpython-38.pyc
  - bmirobot_inverse_kinematics.cpython-38.pyc
  - bmirobot_pickandplace_v2.cpython-38.pyc
  - bmirobot_push_F.cpython-38.pyc
- bmirobot.py
- bmirobot_env_pickandplace_v2.py
- bmirobot_env_push_F.py
- bmirobot_inverse_kinematics.py
- bmirobot_joints_info_pybullet.txt
- bmirobot_pickandplace_v2.py
- bmirobot_push_F.py
- algorithm.png
- environment_pick-1620701478861.png
- environment_pick.png
- Honeycam 2021-04-07 09-20-37.gif
- Honeycam 2021-04-10 21-48-08.gif
- Honeycam 2021-04-14 17-04-21.gif
- Honeycam 2021-04-14 17-08-45.gif
- Honeycam 2021-04-14 20-07-58.gif
- Honeycam 2021-05-11 10-39-59.gif
- Honeycam 2021-05-11 10-43-32.gif
- image-20210511102018022.png
- PickandPlace_1_big.png
- push1.gif
- Push_1_big.png
- 图片1.gif
- 图片2.gif
- 图片3.gif
- 图片4.gif
- cube_small_pick.urdf
- cube_small_push.urdf
- cube_small_target_pick.urdf
- cube_small_target_push.urdf
arguments.py
bmirobot_1000_pick_demo.npz
bmirobot_1000_push_demo.npz
compare_clip_curve.py
ddpg_agent.py
demo_push.py
get_demo_data_pick.py
get_demo_data_push.py
her.py
models.py
normalizer.py
README.md
replay_buffer.py
train.py
utils.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<img src="README.assets\environment_pick.png" alt="Honeycam 2021-05-11 10-39-59" style="zoom: 80%;" />

## I. Dependency：

```
gym v 0.18.0: openai RL lib
Mpi4py v 3.0.3：parallel computing  and download the offical MPI plug
      https://www.microsoft.com/en-    us/download/details.aspxid=57467)https://www.microsoft.com/en-us/download/details.aspx?id=57467)
torch v 1.8.1: Deep learning framework
pybullet   v 3.1.7: a physical simulation lib
numpy v1.19.2 
```

## II. Quick Start

This is a RL project for a humanoid arm environment under sparse reward.

```
├── URDF_model          # The robotic and cubes model
|── bmirobot_env        # robotic arm environment using pybullet lib
|   ├── bmirobot.py     # base env
|   ├── bmirobot_env_pickandplace_v2.py #pick and place task
|   ├── bmirobot_env_push_F.py          #Push task
|   ├── bmirobot_inverse_kinematics.py  #arm IK module
|   └── bmirobot_joints_info_pybullet.txt #arm joint and link information
|   └── bmirobot_pickandplace_v2.py #configuration pick and place task
|   └── bmirobot_push.py            #configuration push task
├── arguments                       #train arguments
├── bmirobot_1000_pick_demo.npz     #1000 expert data for pick task
├── bmirobot_1000_push_demo.npz     #1000 expert data for push task
├── ddpg_agent.py                   #DDPG algorithm code
├── demo_push.py                    #load the training model actor and performs
├── get_demo_data_pick.py           #get the expert data for pick and place task by using a programmed trajectory
├── get_demo_data_push.py           #get the expert data for push task by using a programmed trajectory
├── her.py                          #Hindsight experience replay algorithm
├── models.py                       #actor and critic model
├── normalizer.py                   #data normalize
├── train.py                        #train agent
├── utils.py                        #common functions
```

use `train.py` to train the agent

and adjust the arguments in `arguments.py`

```python
class Args:
    def __init__(self):
        self.n_epochs = 200  # epoch num
        self.n_cycles = 50   # one epoch cycle time : 1 epoch: n_cycles*num_rollouts_per_mpi
        self.n_batches = 40  # updating network batches
        self.save_interval = 5
        self.seed = 125  # random seed
        self.num_workers = 19  # no use
        self.replay_strategy = 'future' #futrue sampling for HER algorithm
        self.clip_return = 50   
        self.save_dir = 'saved_models/' # save model path
        self.noise_eps = 0.01           # action selection noise
        self.random_eps = 0.3           # random action selection
        self.buffer_size = 1e6*1/2      # replay buffer size
        self.replay_k = 4  # replay with k random states which come from the same episode as the transition being replayed and were observed after it
        self.clip_obs = 200             #obs=clip(obs,max,min)
        self.batch_size = 256
        self.gamma = 0.98               #discount rate
        self.action_l2 = 1
        self.lr_actor = 0.001   #learing rate
        self.lr_critic = 0.001
        self.polyak = 0.95  # soft learning rate
        self.n_test_rollouts = 25 #testing time in a epoch model training
        self.clip_range = 5
        self.demo_length = 25 
        self.cuda = False # use cuda
        self.num_rollouts_per_mpi = 2 # 1 epoch: n_cycles*num_rollouts_per_mpi
        self.add_demo = True  # add expert data or not
        self.demo_name="bmirobot_1000_push_demo.npz"  #add expert data for push task, if pick, using pick expert data
        #self.demo_name="bmirobot_1000_pick_demo.npz"
        self.train_type = "push" #or "pick"
        self.Use_GUI =True  #GUI is for visualizing the training process
        self.env_name = 'bmirobot_'+str(self.train_type)+" seed"+str( self.seed ) #env_save_path_name
```

## III algorithm

DDPG + HER + Expert Data + action clip 
<img src="README.assets/algorithm.png" alt="总图v2" style="zoom: 33%;" />

## IV. Result

<img src="README.assets/Honeycam 2021-05-11 10-39-59.gif" alt="Honeycam 2021-05-11 10-39-59" style="zoom: 80%;" />

<center>push task</center>

<img src="README.assets/Honeycam 2021-05-11 10-43-32.gif" alt="Honeycam 2021-05-11 10-43-32" style="zoom: 80%;" />

<center>pick and place task</center>

<img src="README.assets/Push_1_big.png" alt="Push_1_big" style="zoom: 50%;" />

<img src="README.assets/PickandPlace_1_big.png" alt="PickandPlace_1_big" style="zoom:50%;" />