ppo - GitLite

KO | EN

gitlite — search

#python #hacktoberfest #ai #rust #deep-learning #linux #nodejs #windows #awesome #kubernetes #ios #mcp #openai #open-source

ppo

(★ 126)

An implementation of PPO in Pytorch

파일 탐색기

최종 버전 다운로드 (.zip)

.gitignore
LICENSE
lunar.gif
README.md
requirements.txt
train.py
train_action_chunking.py
train_action_chunking_antmaze.py
train_binary_mapper.py
train_continuous_actions.py
train_critic_past_action.py
train_diayn.py
train_epo.py
train_epo_miikkulainen.py
train_internal_actions.py
train_internal_actions_ant_maze.py
train_pmpo.py
train_probabilistic_value_network.py
train_with_evo_strat.py
train_wm_with_evo.py
train_world_model.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/lucidrains/ppo

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd ppo

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

$ pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<img src="./lunar.gif" width="400px"></img>

*1k steps*

## PPO

An implementation of PPO with recent random improvements

The phasic part has been removed, repository to be renamed. I do not think it does anything

## Install

```bash
$ pip install -r requirements.txt
```

You may need to install `swig`

```bash
$ apt install swig
```

## Use

```bash
$ python train.py
```

## Citations

```bibtex
@article{Schulman2017ProximalPO,
    title   = {Proximal Policy Optimization Algorithms},
    author  = {John Schulman and Filip Wolski and Prafulla Dhariwal and Alec Radford and Oleg Klimov},
    journal = {ArXiv},
    year    = {2017},
    volume  = {abs/1707.06347},
    url     = {https://api.semanticscholar.org/CorpusID:28695052}
}
```

```bibtex
@article{Zhang2024ReLU2WD,
    title   = {ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs},
    author  = {Zhengyan Zhang and Yixin Song and Guanghui Yu and Xu Han and Yankai Lin and Chaojun Xiao and Chenyang Song and Zhiyuan Liu and Zeyu Mi and Maosong Sun},
    journal = {ArXiv},
    year    = {2024},
    volume  = {abs/2402.03804},
    url     = {https://api.semanticscholar.org/CorpusID:267499856}
}
```

```bibtex
@inproceedings{Lee2024SimBaSB,
    title  = {SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning},
    author = {Hojoon Lee and Dongyoon Hwang and Donghu Kim and Hyunseung Kim and Jun Jet Tai and Kaushik Subramanian and Peter R. Wurman and Jaegul Choo and Peter Stone and Takuma Seno},
    year   = {2024},
    url    = {https://api.semanticscholar.org/CorpusID:273346233}
}
```

```bibtex
@inproceedings{anonymous2024the,
    title   = {The Complexity Dynamics of Grokking},
    author  = {Anonymous},
    booktitle = {Submitted to The Thirteenth International Conference on Learning Representations},
    year    = {2024},
    url     = {https://openreview.net/forum?id=07N9jCfIE4},
    note    = {under review}
}
```

```bibtex
@article{Yang2020LearningLD,
    title   = {Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification},
    author  = {Huanrui Yang and Minxue Tang and Wei Wen and Feng Yan and Daniel Hu and Ang Li and Hai Helen Li and Yiran Chen},
    journal = {2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
    year    = {2020},
    pages   = {2899-2908},
    url     = {https://api.semanticscholar.org/CorpusID:213940794}
}
```

```bibtex
@article{Farebrother2024StopRT,
    title   = {Stop Regressing: Training Value Functions via Classification for Scalable Deep RL},
    author  = {Jesse Farebrother and Jordi Orbay and Quan Ho Vuong and Adrien Ali Taiga and Yevgen Chebotar and Ted Xiao and Alex Irpan and Sergey Levine and Pablo Samuel Castro and Aleksandra Faust and Aviral Kumar and Rishabh Agarwal},
    journal = {ArXiv},
    year   = {2024},
    volume = {abs/2403.03950},
    url    = {https://api.semanticscholar.org/CorpusID:268253088}
}
```

```bibtex
@article{Lee2024AnalysisClippedCritic,
    title   = {On Analysis of Clipped Critic Loss in Proximal Policy Gradient},
    author  = {Yongjin Lee, Moonyoung Chung},
    journal = {Authorea},
    year    = {2024}
}
```

```bibtex
@inproceedings{Felizardo2025ARL,
    title   = {A Reinforcement Learning Method for Environments with Stochastic Variables: Post-Decision Proximal Policy Optimization with Dual Critic Networks},
    author  = {Leonardo Kanashiro Felizardo and Edoardo Fadda and Paolo Brandimarte and Emilio Del-Moral-Hernandez and Mari'a Cristina Vasconcelos Nascimento},
    year    = {2025},
    url     = {https://api.semanticscholar.org/CorpusID:277621941}
}
```

```bibtex
@misc{xie2025simplepolicyoptimization,
    title   = {Simple Policy Optimization},
    author  = {Zhengpeng Xie and Qiang Zhang and Fan Yang and Marco Hutter and Renjing Xu},
    year    = {2025},
    eprint  = {2401.16025},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url = {https://arxiv.org/abs/2401.16025},
}
```

```bibtex
@inproceedings{anonymous2025flow,
    title   = {Flow Policy Gradients for Legged Robots},
    author  = {Anonymous},
    booktitle = {Submitted to The Fourteenth International Conference on Learning Representations},
    year    = {2025},
    url     = {https://openreview.net/forum?id=BA6n0nmagi},
    note    = {under review}
}
```

```bibtex
@inproceedings{Seitzer2022Pitfalls,
    title   = {On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks},
    author  = {Maximilian Seitzer and Abdul-Saboor Sheikh and Georg Martius},
    booktitle = {International Conference on Learning Representations},
    year    = {2022},
    url     = {https://openreview.net/forum?id=9O_xF9y7A-}
}
```

```bibtex
@inproceedings{Wang2025EvolutionaryPO,
    title   = {Evolutionary Policy Optimization},
    author  = {Jianren Wang and Yifan Su and Abhinav Gupta and Deepak Pathak},
    booktitle = {Advances in Neural Information Processing Systems},
    year    = {2025},
    url     = {https://arxiv.org/abs/2503.19037}
}
```

```bibtex
@misc{osband2026delightfulpolicygradient,
    title   = {Delightful Policy Gradient},
    author  = {Ian Osband},
    year    = {2026},
    eprint  = {2603.14608},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url     = {https://arxiv.org/abs/2603.14608},
}
```

```bibtex
@misc{eysenbach2018diversityneedlearningskills,
    title   = {Diversity is All You Need: Learning Skills without a Reward Function},
    author  = {Benjamin Eysenbach and Abhishek Gupta and Julian Ibarz and Sergey Levine},
    year    = {2018},
    eprint  = {1802.06070},
    archivePrefix = {arXiv},
    primaryClass = {cs.AI},
    url     = {https://arxiv.org/abs/1802.06070},
}
```

```bibtex
@misc{mustafaoglu2025evolutionarypolicyoptimization,
    title   = {Evolutionary Policy Optimization},
    author  = {Zelal Su "Lain" Mustafaoglu and Keshav Pingali and Risto Miikkulainen},
    year    = {2025},
    eprint  = {2504.12568},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url     = {https://arxiv.org/abs/2504.12568},
}
```

```bibtex
@misc{han2026firefrobeniusisometryreinitializationbalancing,
    title   = {FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff},
    author  = {Isaac Han and Sangyeon Park and Seungwon Oh and Donghu Kim and Hojoon Lee and Kyung-Joong Kim},
    year    = {2026},
    eprint  = {2602.08040},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url     = {https://arxiv.org/abs/2602.08040},
}
```

```bibtex
@misc{tian2026chunkingcritic,
    title   = {Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns},
    author  = {Dong Tian and Onur Celik and Gerhard Neumann},
    year    = {2026},
    eprint  = {2503.03660},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url     = {https://arxiv.org/abs/2503.03660},
}
```

```bibtex
@misc{hahn2025action,
    title   = {Action Chunking Proximal Policy Optimization for Universal Robotic Dexterous Grasping},
    author  = {Sanghyun Hahn and Jonghyun Choi},
    year    = {2025},
    url     = {https://openreview.net/forum?id=WFQnqY1c39}
}
```

```bibtex
@misc{shin2026adaptiveactionchunkingmultichunk,
    title   = {Adaptive Action Chunking via Multi-Chunk Q Value Estimation},
    author  = {Yongjae Shin and Jongseong Chae and Seongmin Kim and Jongeui Park and Youngchul Sung},
    year    = {2026},
    eprint  = {2605.10044},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url     = {https://arxiv.org/abs/2605.10044},
}
```