fqf-iqn-qrdqn.pytorch

KO | EN

gitlite — search

#javascript #hacktoberfest #go #golang #deep-learning #linux #docker #windows #awesome #kubernetes #awesome-list #pytorch #security #open-source

fqf-iqn-qrdqn.pytorch

(★ 191)

PyTorch implementation of FQF, IQN and QR-DQN.

파일 탐색기

최종 버전 다운로드 (.zip)

- fqf.yaml
- iqn-rainbow.yaml
- iqn.yaml
- qrdqn.yaml
- - __init__.py
  - base_agent.py
  - fqf_agent.py
  - iqn_agent.py
  - qrdqn_agent.py
- - __init__.py
  - base.py
  - per.py
  - segment_tree.py
- - __init__.py
  - base_model.py
  - fqf.py
  - iqn.py
  - qrdqn.py
- __init__.py
- env.py
- network.py
- utils.py
.gitignore
LICENSE
README.md
requirements.txt
train_fqf.py
train_iqn.py
train_qrdqn.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# FQF, IQN and QR-DQN in PyTorch

This is a PyTorch implementation of Fully parameterized Quantile Function(FQF)[[1]](#references), Implicit Quantile Networks(IQN)[[2]](#references) and Quantile Regression DQN(QR-DQN)[[3]](#references). I tried to make it easy for readers to understand algorithms. Please let me know if you have any questions. Also, any pull requests are welcomed.

**UPDATE**
- 2020.6.9
    - Bump torch up to 1.5.0.
- 2020.5.10
    - Refactor codes.
    - Fix Prioritized Experience Replay and Noisy Networks.
    - Test IQN with Rainbow's components.
- 2020.6.9
    - Bump Torch up to 1.5.0.

## Setup
If you are using Anaconda, first create the virtual environment.

```bash
conda create -n fqf python=3.8 -y
conda activate fqf
```

You can install Python liblaries using pip.

```bash
pip install --upgrade pip
pip install -r requirements.txt
```

If you're using other than CUDA 10.2, you may need to install PyTorch. See [instructions](https://pytorch.org/get-started/locally/) for more details.

## Examples
You can train FQF agent using hyperparameters [here](https://github.com/ku2482/fqf-iqn-qrdqn.pytorch/blob/master/config/fqf.yaml).

```
python train_fqf.py --cuda --env_id PongNoFrameskip-v4 --seed 0 --config config/fqf.yaml
```

You can also train IQN or QR-DQN agent in the same way. Note that we log results with the number of frames, which equals to the number of agent's steps multiplied by 4 (e.g. 100M frames means 25M agent's steps).

## Results
Results of examples (without n-step rewards, double q-learning, dueling network nor noisy net) are shown below, which is comparable (if no better) with the paper. Scores below are evaluated arfer every 1M frames (250k agent's steps). Result are averaged over 2 seeds and visualized with min/max.

**Note that I reported the "mean" score, not the "best" score as in the paper.** Also, I only trained a limited number of frames due to limited resources (e.g. 100M frames instead of 200M).

### BreakoutNoFrameskip-v4
I tested FQF, IQN and QR-DQN on `BreakoutNoFrameskip-v4` for 30M frames to see algorithms worked.

### BerzerkNoFrameskip-v4
I also tested FQF and IQN on `BerzerkNoFrameskip-v4` for 100M frames to see the difference between FQF's performance and IQN's, which is quite obvious on this task.

### IQN-Rainbow
I also tested IQN with Rainbow's components on `PongNoFrameskip-v4` (just 1 seed). Note that I decreased `num_steps` to 7500000(30M frames), but kept `start_steps` as the same.

## TODO

- [ ] Implement risk-averse policies for IQN.
- [ ] Test FQF-Rainbow agent.

## References
[[1]](https://arxiv.org/abs/1911.02140) Yang, Derek, et al. "Fully Parameterized Quantile Function for Distributional Reinforcement Learning." Advances in Neural Information Processing Systems. 2019.

[[2]](https://arxiv.org/abs/1806.06923) Dabney, Will, et al. "Implicit quantile networks for distributional reinforcement learning." arXiv preprint. 2018.

[[3]](https://arxiv.org/abs/1710.10044) Dabney, Will, et al. "Distributional reinforcement learning with quantile regression." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.

fqf-iqn-qrdqn.pytorch

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

fqf-iqn-qrdqn.pytorch

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README