GPyOpt-ml-agents

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

GPyOpt-ml-agents

★ 9 Open GitHub ↗

Gaussian process optimization using GPyOpt for Unity ML-Agents Toolkit

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GPyOpt-ml-agents
Gaussian process optimization using [GPyOpt](https://github.com/SheffieldML/GPyOpt) for [Unity ML-Agents Toolkit](https://github.com/Unity-Technologies/ml-agents/).

If you are fed up with tuning your ml-agents training parameters by hand, this repository is made for you!

## Requirements
Follow the ml-agents installation documentation to install it as a python module:
[Link](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md)

If you plan to use a GPU, you need to modify the ml-agents code to prevent tensorflow session to not allocate all the memory. Set this `gpu_options.allow_growth = True` to the options when creating the session. For more info see [this](https://github.com/Unity-Technologies/ml-agents/issues/612)

And install repository dependencies:
```
pip install -r requirements.txt
```

## Usage
Since Unity ML-Agents is using grpc, make sure you do not have any set proxy.

The parameters contained in the config `trainer_config.yaml` is used as default when the parameters are not specified.
Make sure your `max_steps` is greater than `summary_freq` and the steps required to finish an episode.

The code uses a slightly modified ml-agents `learn.py` which accepts an additional parameter to specify the trainer config file.

###  Gaussian process optimization
Modify the config file `hyperopt_conf.py` to fit your environment and run
```
python hyperopt.py <env>
```
The optimization algorithm will explore the parameters space and try to maximize the final reward at `max_steps`. The reward is read in the tensorboard summary event file.

The `batch_size` variable specifies the number of instances launched in parallel.
The total number of trains will be  `batch_size` * `max_iter` + `batch_size` as the algorithm will run an initial batch to collect points before running the optimization.

For more details, see [GPyOpt](https://github.com/SheffieldML/GPyOpt) documentation

### Grid search
If your training is too long to run multiple iterations, you can always use the grid search.

Specify your hyperparameter space dict as `params_grid` in `grid_search_conf.py` and run
```
python grid_search.py <env>
```

The grid search currently runs in parallel all the combinations of the parameters space without taking into account the number of CPU core, so beware to not specify a too large search space.

You can stop the training to generate the graph for each instance whenever you want using a simple `CTRL+C`.
It is also possible to reload the saved model to continue training using the `--load` option.

## Example

### ml-agents 3DBall environment

We are searching for the best hyperparameters that maximize the reward at 10k steps.

Config and Parameters space:
```
definition = [{
  'name': 'learning_rate',
  'type': 'continuous',
  'domain': [1e-5, 1e-3]
}, {
  'name': 'epsilon',
  'type': 'continuous',
  'domain': [0.1, 0.3]
}, {
  'name': 'gamma',
  'type': 'continuous',
  'domain': [0.8, 0.995]
}, {
  'name': 'lambd',
  'type': 'continuous',
  'domain': [0.9, 0.95]
}, {
  'name': 'num_epoch',
  'type': 'discrete',
  'domain': [3, 10]
}, {
  'name': 'beta',
  'type': 'continuous',
  'domain': [1e-4, 1e-2]
}, {
  'name': 'num_layers',
  'type': 'discrete',
  'domain': [1, 3]
}, {
  'name': 'hidden_units',
  'type': 'discrete',
  'domain': [32, 64, 128, 256, 512]
}]

batch_size = 8
num_cores = 32
max_iter = 16
```

Optimal params found achieving `96.80000305175781` as mean reward:
```
learning_rate: 1.0e-03
epsilon: 0.3
gamma: 0.995
lambd: 0.95
num_epoch: 3
beta: 1.0e-02
num_layers: 3
hidden_units: 256
```

![screenshot](https://raw.githubusercontent.com/Coac/GPyOpt-ml-agents/master/docs/3dball_hyperopt_10ksteps.png)

GPyOpt-ml-agents

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README