minirllab

KO | EN

gitlite — search

#hacktoberfest #ai #llm #go #rust #linux #macos #awesome #kubernetes #awesome-list #vue #mcp #openai #open-source

(★ 16)

Mini RL Lab

minirllab 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Mini RL Lab

### Easy Agent Experiments for Beginners

I wrote this set of scripts to help me research and experiement with the latest concepts in RL, as well as a way to learn Python and PyTorch.

It is a setup and workflow that works well for me to debug and experiment with concepts like agent algorithms, world models, planning, plasticity, transformers etc, and other beginners might find it a useful starting point for their own experiments.

The focus of Mini RL Lab is on continous control gym-like environments aimed at physical systems or expensive-to-sample simulations - my personal research interest, and it makes the problem and architecture space tractable.

The basis are CleanRL's PPO and SAC agents [https://github.com/vwxyzjn/cleanrl] which I modified to:

1. Separate the environment rollout and logging from the agent code. CleanRL's single file approach is great but I find this arrangement easier for experiments
2. Simplify the code, improve performance where possible
3. Use different specialised training scripts
4. Include algorithms + variants as baselines with which to compare

### Benefits

1. Agents based on established, tried and tested baselines from CleanRL
2. Agents are structured for easy experimentation, whilst staying ~"one file"
3. Various performance considerations such as minimising cpu<>gpu syncs, data transfers from buffer etc
   1. Helpful for those of us limited to one workstation and a midrange GPU 
4. Inline comments document design choices and links to source papers
5. Learn scripts implement a lot best practices I discovered as I went, minor (data logging structure) to major (multiprocessing allows running a number of parallel agents with different seeds, essential in RL)

### Prerequisites

* Pytorch 2 (though 1.x will work with small changes)
* Numpy (1.25 though older should work)
* Tensorboard
* Gymnasium[Box2D] and/or [Mujoco] (https://gymnasium.farama.org)
  * Or other gym compatible environment of choice
* Bayesian Optimisation (https://github.com/bayesian-optimization/BayesianOptimization)

### Quickstart

Test a change quickly for major errors:

`Python learn_simple.py`

Training run with multiple random seeds logging to tensorboard:

`Python learn_simple.py --log --seed 8 --name "testing X new feature"`

Run a vectorised environment with cuda and log:

`Python learn_vectorised.py --log --cuda`

Use bayesian optimisation to optimise hyperparameter(s):

`Python hypertune.py`

### Usage Notes

* ppo_baseline
  * Based on CleanRL's continuous PPO agent 
  * Simplified for easy modifictions
  * Improved samples per second through small optimisations

* sac_baseline
  * Based on CleanRL's continuous SAC agent
  * Simplified for easy modification
  * Removed CUDA <> CPU synchronisations for better performance
  * Variants: DroQ and CrossQ
    * CrossQ in particular is great, see the paper: https://arxiv.org/abs/1902.05605

* Novel agents
  * crossq_cem
    * Based on sac_crossq with actor replaced by a cross entropy method optimiser
    * Inspired by QT-Opt https://arxiv.org/abs/1806.10293 and TD-MPC2 https://arxiv.org/abs/2310.16828
    * Question: Can QT-Opt's performance improve to match TD-MPC2 using CrossQ's improvements to the Q functions?
    * Result: maybe, but CEM actor is so compute intensive it is not clear this is a direction worth pursuing
    * WIP, could be improved
  
  * **sac_crossq_bro**
    * Inspired by various papers showing that SAC can be improved with (a) more compute (b) regularisation (c) simple design changes
    * **Promising results**, first agent in Mini RL Lab that seems to reliably solve WalkerHardcore in <500k steps
    * WIP, not tuned or optimised yet
     
* learn_simple.py
  * Multiple training runs in parallel using multiprocessing (the processes have independent agents and environments)
  * Few assumptions about environments, more easily compatible with rl envs approximating the open ai gym api
  * Easy to edit and modify, design choices in comments
  * Use case: test performance of new feature on multiple environments with many random seeds in parallel

* learn_vectorised.py
  * No multiprocessing, runs a single process
  * PPO seems to really need different hyperparameters when vectorised
  * Use case: check performance when vectorised

* hypertune.py
  * uses bayesian optimisation to tune selected hyperparameters
  * uses multiprocessing to run multiple evaluations in parallel
  * implements a median pruner to stop badly performing runs early
  * Use case: optimise a new hyperparameter
  * Hyperparameters in RL https://arxiv.org/abs/2306.01324 is a good reference for this

minirllab

minirllab 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

minirllab

minirllab 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README