batch-ppo - GitLite

KO | EN

gitlite — search

#python #react #ai #llm #go #golang #rust #deep-learning #windows #vue #awesome-list #security #openai #open-source

batch-ppo

(★ 979)

Efficient Batched Reinforcement Learning in TensorFlow

파일 탐색기

최종 버전 다운로드 (.zip)

- - __init__.py
- - __init__.py
  - iterate_sequences.py
  - memory.py
  - normalize.py
- - __init__.py
  - configs.py
  - networks.py
  - train.py
  - train_ppo_test.py
  - utility.py
  - visualize.py
- - __init__.py
  - attr_dict.py
  - attr_dict_test.py
  - batch_env.py
  - count_weights.py
  - count_weights_test.py
  - in_graph_batch_env.py
  - in_graph_env.py
  - loop.py
  - loop_test.py
  - mock_algorithm.py
  - mock_environment.py
  - nested.py
  - nested_test.py
  - simulate.py
  - simulate_test.py
  - streaming_mean.py
  - wrappers.py
  - wrappers_test.py
- __init__.py
.gitignore
.pylintrc
.travis.yml
AUTHORS
CONTRIBUTING.md
LICENSE
README.md
setup.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/google-research/batch-ppo

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd batch-ppo

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

python3 -m agents.scripts.train --logdir=/path/to/logdir --config=pendulum

파이썬 스크립트(또는 모듈)를 실행합니다.

python3 -m agents.scripts.visualize --logdir=/path/to/logdir/<time>-<config> --outdir=/path/to/outdir/

파이썬 스크립트(또는 모듈)를 실행합니다.

python3 -m unittest discover -p "*_test.py"

파이썬 스크립트(또는 모듈)를 실행합니다.

python3 -m pylint agents

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<img src="https://www.tensorflow.org/images/tf_logo_transp.png" width=25% align="right">

Batch PPO
=========

This project provides optimized infrastructure for reinforcement learning. It
extends the [OpenAI gym interface][post-gym] to multiple parallel environments
and allows agents to be implemented in TensorFlow and perform batched
computation. As a starting point, we provide BatchPPO, an optimized
implementation of [Proximal Policy Optimization][post-ppo].

Please cite the [TensorFlow Agents paper][paper-agents] if you use code from
this project in your research:

```bibtex
@article{hafner2017agents,
  title={TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow},
  author={Hafner, Danijar and Davidson, James and Vanhoucke, Vincent},
  journal={arXiv preprint arXiv:1709.02878},
  year={2017}
}
```

Dependencies: Python 2/3, TensorFlow 1.3+, Gym, ruamel.yaml

[paper-agents]: https://arxiv.org/pdf/1709.02878.pdf
[post-gym]: https://blog.openai.com/openai-gym-beta/
[post-ppo]: https://blog.openai.com/openai-baselines-ppo/

Instructions
------------

Clone the repository and run the PPO algorithm by typing:

```shell
python3 -m agents.scripts.train --logdir=/path/to/logdir --config=pendulum
```

The algorithm to use is defined in the configuration and `pendulum` started
here uses the included PPO implementation. Check out more pre-defined
configurations in `agents/scripts/configs.py`.

If you want to resume a previously started run, add the `--timestamp=<time>`
flag to the last command and provide the timestamp in the directory name of
your run.

To visualize metrics start TensorBoard from another terminal, then point your
browser to `http://localhost:2222`:

```shell
tensorboard --logdir=/path/to/logdir --port=2222
```

To render videos and gather OpenAI Gym statistics to upload to the scoreboard,
type:

```shell
python3 -m agents.scripts.visualize --logdir=/path/to/logdir/<time>-<config> --outdir=/path/to/outdir/
```

Modifications
-------------

We release this project as a starting point that makes it easy to implement new
reinforcement learning ideas. These files are good places to start when
modifying the code:

| File | Content |
| ---- | ------- |
| `scripts/configs.py` | Experiment configurations specifying the tasks and algorithms. |
| `scripts/networks.py` | Neural network models. |
| `scripts/train.py` | The executable file containing the training setup. |
| `algorithms/ppo/ppo.py` | The TensorFlow graph for the PPO algorithm. |

To run unit tests and linting, type:

```shell
python2 -m unittest discover -p "*_test.py"
python3 -m unittest discover -p "*_test.py"
python3 -m pylint agents
```

For further questions, please open an issue on Github.

Implementation
--------------

We include a batched interface for OpenAI Gym environments that fully integrates
with TensorFlow for efficient algorithm implementations. This is achieved
through these core components:

- **`agents.tools.wrappers.ExternalProcess`** is an environment wrapper that
  constructs an OpenAI Gym environment inside of an external process. Calls to
  `step()` and `reset()`, as well as attribute access, are forwarded to the
  process and wait for the result. This allows to run multiple environments in
  parallel without being restricted by Python's global interpreter lock.
- **`agents.tools.BatchEnv`** extends the OpenAI Gym interface to batches of
  environments. It combines multiple OpenAI Gym environments, with `step()`
  accepting a batch of actions and returning a batch of observations, rewards,
  done flags, and info objects. If the individual environments live in external
  processes, they will be stepped in parallel.
- **`agents.tools.InGraphBatchEnv`** integrates a batch environment into the
  TensorFlow graph and makes its `step()` and `reset()` functions accessible as
  operations. The current batch of observations, last actions, rewards, and done
  flags is stored in variables and made available as tensors.
- **`agents.tools.simulate()`** fuses the step of an in-graph batch environment
  and a reinforcement learning algorithm together into a single operation to be
  called inside the training loop. This reduces the number of session calls and
  provides a simple way to train future algorithms.

To understand all the code, please make yourself familiar with TensorFlow's
control flow operations, especially [`tf.cond()`][tf-cond],
[`tf.scan()`][tf-scan], and
[`tf.control_dependencies()`][tf-control-dependencies].

[tf-cond]: https://www.tensorflow.org/api_docs/python/tf/cond
[tf-scan]: https://www.tensorflow.org/api_docs/python/tf/scan
[tf-control-dependencies]: https://www.tensorflow.org/api_docs/python/tf/control_dependencies

Disclaimer
----------

This is not an official Google product.