# Technical Documentation: vwxyzjn/cleanrl

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/vwxyzjn/cleanrl)

### Overview (README.md)

# CleanRL (Clean Implementation of RL Algorithms)


[<img src="https://img.shields.io/badge/license-MIT-blue">](https://github.com/vwxyzjn/cleanrl)
[](https://github.com/vwxyzjn/cleanrl/actions/workflows/tests.yaml)
[](https://docs.cleanrl.dev/)
[<img src="https://img.shields.io/discord/767863440248143916?label=discord">](https://discord.gg/D6RCjA6sVT)
[<img src="https://img.shields.io/youtube/channel/views/UCDdC6BIFRI0jvcwuhi3aI6w?style=social">](https://www.youtube.com/channel/UCDdC6BIFRI0jvcwuhi3aI6w/videos)
[](https://github.com/psf/black)
[](https://pycqa.github.io/isort/)
[<img src="https://img.shields.io/badge/%F0%9F%A4%97%20Models-Huggingface-F8D521">](https://huggingface.co/cleanrl)
[](https://colab.research.google.com/github/vwxyzjn/cleanrl/blob/master/docs/get-started/CleanRL_Huggingface_Integration_Demo.ipynb)


CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:



* 📜 Single-file implementation
   * *Every detail about an algorithm variant is put into a single standalone file.* 
   * For example, our `ppo_atari.py` only has 340 lines of code but contains all implementation details on how PPO works with Atari games, **so it is a great reference implementation to read for folks who do not wish to read an entire modular library**.
* 📊 Benchmarked Implementation (7+ algorithms and 34+ games at https://benchmark.cleanrl.dev)
* 📈 Tensorboard Logging
* 🪛 Local Reproducibility via Seeding
* 🎮 Videos of Gameplay Capturing
* 🧫 Experiment Management with [Weights and Biases](https://wandb.ai/site)
* 💸 Cloud Integration with docker and AWS 

You can read more about CleanRL in our [JMLR paper](https://www.jmlr.org/papers/volume23/21-1342/21-1342.pdf) and [documentation](https://docs.cleanrl.dev/).

Notable CleanRL-related projects:

* [corl-team/CORL](https://github.com/corl-team/CORL): Offline RL algorithm implemented in CleanRL style
* [pytorch-labs/LeanRL](https://github.com/pytorch-labs/LeanRL): Fast optimized PyTorch implementation of CleanRL RL algorithms using CUDAGraphs.


> ℹ️ **Support for Gymnasium**: [Farama-Foundation/Gymnasium](https://github.com/Farama-Foundation/Gymnasium) is the next generation of [`openai/gym`](https://github.com/openai/gym) that will continue to be maintained and introduce new features. Please see their [announcement](https://farama.org/Announcing-The-Farama-Foundation) for further detail. We are migrating to `gymnasium` and the progress can be tracked in [vwxyzjn/cleanrl#277](https://github.com/vwxyzjn/cleanrl/pull/277). 


> ⚠️ **NOTE**: CleanRL is *not* a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's variant or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).

## Get started

Prerequisites:
* Python >=3.7.1,<3.11
* [uv 0.7.9+](https://docs.astral.sh/uv/)

To run experiments locally, give the following a try:

```bash
git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .

# alternatively, you could use `uv venv` and do
# `python run cleanrl/ppo.py`
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000

# open another terminal and enter `cd cleanrl/cleanrl`
tensorboard --logdir runs
```

To use experiment tracking with wandb, run
```bash
wandb login # only required for the first time
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000 \
    --track \
    --wandb-project-name cleanrltest
```

If you are not using `uv`, you can install CleanRL with `requirements.txt`:

```bash
# core dependencies
pip install -r requirements/requirements.txt

# optional dependencies
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-mujoco_py.txt
pip install -r requirements/requirements-procgen.txt
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-jax.txt
pip install -r requirements/requirements-docs.txt
pip install -r requirements/requirements-cloud.txt
pip install -r requirements/requirements-memory_gym.txt
```

To run training scripts in other games:
```
uv venv

# classic control
python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1

# atari
uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
uv pip install ".[envpool]"
python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
uv run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3

# procgen
uv pip install ".[procgen]"
python cleanrl/ppo_procgen.py --env-id starpilot
python cleanrl/ppg_procgen.py --env-id starpilot

# ppo + lstm
uv pip install ".[atari]"
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4
```

You may also use a prebuilt development environment hosted in Gitpod:

[](https://gitpod.io/#https://github.com/vwxyzjn/cleanrl)

## Algorithms Implemented


| Algorithm      | Variants Implemented |
| ----------- | ----------- |
| ✅ [Proximal Policy Optimization (PPO)](https://arxiv.org/pdf/1707.06347.pdf)  |  [`ppo.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py),   [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppopy) |
| |  [`ppo_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari.py),   [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_ataripy)
| |  [`ppo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py),   [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_continuous_actionpy)
| |  [`ppo_atari_lstm.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_lstm.py),   [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_lstmpy)
| |  [`ppo_atari_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool.py),   [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_envpoolpy)
| | [`ppo_atari_envpool_xla_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_envpool_xla_jaxpy)
| | [`ppo_atari_envpool_xla_jax_scan.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax_scan.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_envpool_xla_jax_scanpy))
| |  [`ppo_procgen.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_procgen.py),   [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_procgenpy)
| |  [`ppo_atari_multigpu.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_multigpu.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_multigpupy)
| | [`ppo_pettingzoo_ma_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_pettingzoo_ma_atari.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_pettingzoo_ma_ataripy)
| | [`ppo_continuous_action_isaacgym.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_continuous_action_isaacgympy)
| | [`ppo_trxl.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_trxl/ppo_trxl.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/ppo-trxl/)
| ✅ [Deep Q-Learning (DQN)](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf) |  [`dqn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/dqn/#dqnpy) |
| | [`dqn_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/dqn/#dqn_ataripy) |
| | [`dqn_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_jax.py), [docs](https://docs.cleanrl.dev/rl-algorithms/dqn/#dqn_jaxpy) |
| | [`dqn_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py), [docs](https://docs.cleanrl.dev/rl-algorithms/dqn/#dqn_atari_jaxpy) |
| ✅ [Categorical DQN (C51)](https://arxiv.org/pdf/1707.06887.pdf) |  [`c51.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/c51/#c51py) |
| |  [`c51_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/c51/#c51_ataripy) |
| | [`c51_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_jax.py), [docs](https://docs.cleanrl.dev/rl-algorithms/c51/#c51_jaxpy) |
| | [`c51_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari_jax.py), [docs](https://docs.cleanrl.dev/rl-algorithms/c51/#c51_atari_jaxpy) |
| ✅ [Soft Actor-Critic (SAC)](https://arxiv.org/pdf/1812.05905.pdf) |  [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/sac/#sac_continuous_actionpy) |
| |  [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/sac/#sac_atarinpy) |
| ✅ [Deep Deterministic Policy Gradient (DDPG)](https://arxiv.org/pdf/1509.02971.pdf) |  [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/ddpg/#ddpg_continuous_actionpy) |
| | [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/ddpg/#ddpg_continuous_action_jaxpy)
| ✅ [Twin Delayed Deep Deterministic Policy Gradient (TD3)](https://arxiv.org/pdf/1802.09477.pdf) |  [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/td3/#td3_continuous_actionpy) |
|  | [`td3_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/td3/#td3_continuous_action_jaxpy) |
| ✅ [Phasic Policy Gradient (PPG)](https://arxiv.org/abs/2009.04416) |  [`ppg_procgen.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppg_procgen.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/ppg/#ppg_procgenpy) |
| ✅ [Random Network Distillation (RND)](https://arxiv.org/abs/1810.12894) |  [`ppo_rnd_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_rnd_envpool.py),  [docs](/rl-algorithms/ppo-rnd/#ppo_rnd_envpoolpy) |
| ✅ [Qdagger](https://arxiv.org/abs/2206.01626)  |  [`qdagger_dqn_atari_impalacnn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_impalacnn.py),  [docs](https://docs.cleanrl.dev/rl-algorithms/qdagger/#qdagger_dqn_atari_impalacnnpy) |
|  | [`qdagger_dqn_atari_jax_impalacnn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_jax_impalacnn.py), [docs](https://docs.cleanrl.dev/rl-algorithms/qdagger/#qdagger_dqn_atari_jax_impalacnnpy) |


## Open RL Benchmark

To make our experimental data transparent, CleanRL participates in a related project called [Open RL Benchmark](https://github.com/openrlbenchmark/openrlbenchmark), which contains tracked experiments from popular DRL libraries such as ours, [Stable-baselines3](https://github.com/DLR-RM/stable-baselines3), [openai/baselines](https://github.com/openai/baselines), [jaxrl](https://github.com/ikostrikov/jaxrl), and others. 

Check out https://benchmark.cleanrl.dev/ for a collection of Weights and Biases reports showcasing tracked DRL experiments. The reports are interactive, and researchers can easily query information such as GPU utilization and videos of an agent's gameplay that are normally hard to acquire in other RL benchmarks. In the future, Open RL Benchmark will likely provide an dataset API for researchers to easily access the data (see [repo](https://github.com/openrlbenchmark/openrlbenchmark)).






## Support and get involved

We have a [Discord Community](https://discord.gg/D6RCjA6sVT) for support. Feel free to ask questions. Posting in [Github Issues](https://github.com/vwxyzjn/cleanrl/issues) and PRs are also welcome. Also our past video recordings are available at [YouTube](https://www.youtube.com/watch?v=dm4HdGujpPs&list=PLQpKd36nzSuMynZLU2soIpNSMeXMplnKP&index=2)

## Citing CleanRL

If you use CleanRL in your work, please cite our technical [paper](https://www.jmlr.org/papers/v23/21-1342.html):

```bibtex
@article{huang2022cleanrl,
  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {http://jmlr.org/papers/v23/21-1342.html}
}
```


## Acknowledgement

CleanRL is a community-powered by project and our contributors run experiments on a variety of hardware.

* We thank many contributors for using their own computers to run experiments
* We thank Google's [TPU research cloud](https://sites.research.google/trc/about/) for providing TPU resources.
* We thank [Hugging Face](https://huggingface.co/)'s cluster for providing GPU resources.

---

### Index (`docs/index.md`)

# CleanRL - Overview


[<img src="https://img.shields.io/badge/license-MIT-blue">](https://github.com/vwxyzjn/cleanrl)
[](https://github.com/vwxyzjn/cleanrl/actions/workflows/tests.yaml)
[](https://docs.cleanrl.dev/)
[<img src="https://img.shields.io/discord/767863440248143916?label=discord">](https://discord.gg/D6RCjA6sVT)
[<img src="https://img.shields.io/youtube/channel/views/UCDdC6BIFRI0jvcwuhi3aI6w?style=social">](https://www.youtube.com/channel/UCDdC6BIFRI0jvcwuhi3aI6w/videos)
[](https://github.com/psf/black)
[](https://pycqa.github.io/isort/)
[<img src="https://img.shields.io/badge/%F0%9F%A4%97%20Models-Huggingface-F8D521">](https://huggingface.co/cleanrl)
[](https://colab.research.google.com/github/vwxyzjn/cleanrl/blob/master/docs/get-started/CleanRL_Huggingface_Integration_Demo.ipynb)



CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:

* 📜 Single-file implementation
   * *Every detail about an algorithm variant is put into a single standalone file.* 
   * For example, our `ppo_atari.py` only has 340 lines of code but contains all implementation details on how PPO works with Atari games, **so it is a great reference implementation to read for folks who do not wish to read an entire modular library**.
* 📊 Benchmarked Implementation (7+ algorithms and 34+ games at https://benchmark.cleanrl.dev)
* 📈 Tensorboard Logging
* 🪛 Local Reproducibility via Seeding
* 🎮 Videos of Gameplay Capturing
* 🧫 Experiment Management with [Weights and Biases](https://wandb.ai/site)
* 💸 Cloud Integration with docker and AWS 

You can read more about CleanRL in our [technical paper](https://arxiv.org/abs/2111.08819) and [documentation](https://docs.cleanrl.dev/).

CleanRL only contains implementations of **online** deep reinforcement learning algorithms. If you are looking for **offline** algorithms, please check out [corl-team/CORL](https://github.com/corl-team/CORL), which shares a similar design philosophy as CleanRL.

!!! info
    
    **Support for Gymnasium**: [Farama-Foundation/Gymnasium](https://github.com/Farama-Foundation/Gymnasium) is the next generation of [`openai/gym`](https://github.com/openai/gym) that will continue to be maintained and introduce new features. Please see their [announcement](https://farama.org/Announcing-The-Farama-Foundation) for further detail. We are migrating to `gymnasium` and the progress can be tracked in [vwxyzjn/cleanrl#277](https://github.com/vwxyzjn/cleanrl/pull/277).



!!! warning
    
    CleanRL is *not* a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's variant or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).

## Citing CleanRL

If you use CleanRL in your work, please cite our technical [paper](https://www.jmlr.org/papers/volume23/21-1342/21-1342.pdf):

```bibtex
@article{huang2022cleanrl,
  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {http://jmlr.org/papers/v23/21-1342.html}
}
```

---

### Rl Algorithms Overview (`docs/rl-algorithms/overview.md`)

# Overview

| Algorithm      | Variants Implemented |
| ----------- | ----------- |
| ✅ [Proximal Policy Optimization (PPO)](https://arxiv.org/pdf/1707.06347.pdf)  | :material-github: [`ppo.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppopy) |
| | :material-github: [`ppo_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_ataripy)
| | :material-github: [`ppo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_continuous_actionpy)
| | :material-github: [`ppo_atari_lstm.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_lstm.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_lstmpy)
| | :material-github: [`ppo_atari_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_envpoolpy)
| | :material-github: [`ppo_atari_envpool_xla_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_envpool_xla_jaxpy)
| | :material-github: [`ppo_atari_envpool_xla_jax_scan.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax_scan.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_envpool_xla_jax_scanpy)
| | :material-github: [`ppo_procgen.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_procgen.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_procgenpy)
| | :material-github: [`ppo_atari_multigpu.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_multigpu.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_multigpupy)
| | :material-github: [`ppo_pettingzoo_ma_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_pettingzoo_ma_atari.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_pettingzoo_ma_ataripy)
| | :material-github: [`ppo_continuous_action_isaacgym.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_continuous_action_isaacgympy)
| ✅ [Deep Q-Learning (DQN)](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf) | :material-github: [`dqn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqnpy) |
| | :material-github: [`dqn_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_ataripy) |
| | :material-github: [`dqn_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_jax.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_jaxpy) |
| | :material-github: [`dqn_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_atari_jaxpy) |
| ✅ [Categorical DQN (C51)](https://arxiv.org/pdf/1707.06887.pdf) | :material-github: [`c51.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51.py), :material-file-document: [docs](/rl-algorithms/c51/#c51py) |
| | :material-github: [`c51_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_ataripy) |
| | :material-github: [`c51_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_jax.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_jaxpy) |
| | :material-github: [`c51_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari_jax.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_atari_jaxpy) |
| ✅ [Soft Actor-Critic (SAC)](https://arxiv.org/pdf/1812.05905.pdf) | :material-github: [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py), :material-file-document: [docs](/rl-algorithms/sac/#sac_continuous_actionpy) |
| | :material-github: [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py), :material-file-document: [docs](https://docs.cleanrl.dev/rl-algorithms/sac/#sac_atarinpy) |
| ✅ [Deep Deterministic Policy Gradient (DDPG)](https://arxiv.org/pdf/1509.02971.pdf) | :material-github: [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py), :material-file-document: [docs](/rl-algorithms/ddpg/#ddpg_continuous_actionpy) |
| | :material-github: [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py),  :material-file-document: [docs](/rl-algorithms/ddpg/#ddpg_continuous_action_jaxpy)
| ✅ [Twin Delayed Deep Deterministic Policy Gradient (TD3)](https://arxiv.org/pdf/1802.09477.pdf) | :material-github: [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py), :material-file-document: [docs](/rl-algorithms/td3/#td3_continuous_actionpy) |
|  | :material-github: [`td3_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py), :material-file-document: [docs](/rl-algorithms/td3/#td3_continuous_action_jaxpy) |
| ✅ [Phasic Policy Gradient (PPG)](https://arxiv.org/abs/2009.04416) | :material-github: [`ppg_procgen.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppg_procgen.py), :material-file-document: [docs](/rl-algorithms/ppg/#ppg_procgenpy) |
| ✅ [Random Network Distillation (RND)](https://arxiv.org/abs/1810.12894) | :material-github: [`ppo_rnd_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_rnd_envpool.py), :material-file-document: [docs](/rl-algorithms/ppo-rnd/#ppo_rnd_envpoolpy) |
| ✅ [Qdagger](https://arxiv.org/abs/2206.01626) | :material-github: [`qdagger_dqn_atari_impalacnn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_impalacnn.py), :material-file-document: [docs](/rl-algorithms/qdagger/#qdagger_dqn_atari_impalacnnpy) |
|  | :material-github: [`qdagger_dqn_atari_jax_impalacnn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_jax_impalacnn.py), :material-file-document: [docs](/rl-algorithms/qdagger/#qdagger_dqn_atari_jax_impalacnnpy) |

---

### Cleanrl Supported Papers Projects (`docs/cleanrl-supported-papers-projects.md`)

# CleanRL-supported Papers / Projects

CleanRL has become an increasingly popular deep reinforcement learning library, especially among practitioners who prefer more customizable code. Since its debut in July 2019, CleanRL has supported many open source projects and publications. Below are some CleanRL-supported projects and publications.

**Feel free to edit this list if your project or paper has used CleanRL.**

## Publications

* Md Masudur Rahman and Yexiang Xue. "Bootstrap Advantage Estimation for Policy Optimization in Reinforcement Learning." In Proceedings of the IEEE International Conference on Machine Learning and Applications (ICMLA), 2022. [https://arxiv.org/pdf/2210.07312.pdf](https://arxiv.org/pdf/2210.07312.pdf)

* Centa, Matheus, and Philippe Preux. "Soft Action Priors: Towards Robust Policy Transfer." arXiv preprint arXiv:2209.09882 (2022). [https://arxiv.org/pdf/2209.09882.pdf](https://arxiv.org/pdf/2209.09882.pdf)

* Weng, Jiayi, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu et al. "Envpool: A highly parallel reinforcement learning environment execution engine." In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [https://openreview.net/forum?id=BubxnHpuMbG](https://openreview.net/forum?id=BubxnHpuMbG)

* Huang, Shengyi, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. "The 37 Implementation Details of Proximal Policy Optimization." International Conference on Learning Representations 2022 Blog Post Track, [https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)

* Huang, Shengyi, and Santiago Ontañón. "A closer look at invalid action masking in policy gradient algorithms." The International FLAIRS Conference Proceedings, 35. [https://journals.flvc.org/FLAIRS/article/view/130584](https://journals.flvc.org/FLAIRS/article/view/130584)

* Schmidt, Dominik, and Thomas Schmied. "Fast and Data-Efficient Training of Rainbow: an Experimental Study on Atari." Deep Reinforcement Learning Workshop at the 35th Conference on Neural Information Processing Systems, [https://arxiv.org/abs/2111.10247](https://arxiv.org/abs/2111.10247)


* Dossa, Rousslan Fernand Julien, Shengyi Huang, Santiago Ontañón, and Takashi Matsubara. "An Empirical Investigation of Early Stopping Optimizations in Proximal Policy Optimization." IEEE Access 9 (2021): 117981-117992. [https://ieeexplore.ieee.org/abstract/document/9520424](https://ieeexplore.ieee.org/abstract/document/9520424)

* Huang, Shengyi, Santiago Ontañón, Chris Bamford, and Lukasz Grela. "Gym-µRTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning." In 2021 IEEE Conference on Games (CoG), pp. 1-8. IEEE, 2021. [https://ieeexplore.ieee.org/abstract/document/9619076](https://ieeexplore.ieee.org/abstract/document/9619076)

* Huang, Shengyi, and Santiago Ontañón. "Measuring Generalization of Deep Reinforcement Learning Applied to Real-time Strategy Games", AAAI 2021 Reinforcement Learning in Games Workshop, http://aaai-rlg.mlanctot.info/papers/AAAI21-RLG_paper_33.pdf

* Bamford, Chris, Huang, Shengyi, and Lucas, Simon, "Griddly: A platform for AI research in games", *AAAI 2021 Reinforcement Learning in Games Workshop*, [https://arxiv.org/abs/2011.](https://arxiv.org/abs/2011.)06363

* Huang, Shengyi, and Santiago Ontañón. "Action guidance: Getting the best of sparse rewards and shaped rewards for real-time strategy games." AIIDE Workshop on Artificial Intelligence for Strategy Games, [https://arxiv.org/abs/2010.03956](https://arxiv.org/abs/2010.03956)

* Huang, Shengyi, and Santiago Ontañón. "Comparing Observation and Action Representations for Deep Reinforcement Learning in $\mu $ RTS." AIIDE Workshop on Artificial Intelligence for Strategy Gamee, October 2019 [https://arxiv.org/abs/1910.12134](https://arxiv.org/abs/1910.12134)

---

### Contribution (`docs/contribution.md`)

👍🎉 Thank you for taking the time to contribute! 🎉👍

Feel free to open an issue or a Pull Request if you have any questions or suggestions. You can also [join our Discord](https://discord.gg/D6RCjA6sVT) and ask questions there. If you plan to work on an issue, let us know in the issue thread to avoid duplicate work.

Good luck and have fun!

## Development Environment Setup

To setup the development environment, please clone the repository and follow the [installation docs](/get-started/installation/) and [usage docs](/get-started/basic-usage). They should help you set a working uv environment, so you have the same set-up as other contributors. Additionally, you may want to run the following command to install dev-dependencies for documentation:

```bash
uv pip install ".[docs]"
```


To build the documentation, you can run the following command:

```bash
uv run mkdocs serve
```

For testing, we generally recommend making a PR and let GitHub run the tests in [`.github/workflows/tests.yaml`](https://github.com/vwxyzjn/cleanrl/blob/master/.github/workflows/tests.yaml) automatically. These tests are cross-platform and run on Linux, macOS, and Windows. However, if you want to run the tests locally, you can run the following command:

```bash
uv run install --all-extras
uv run pytest tests/.
```



Also, we use [pre-commit](https://pre-commit.com/) to helps us automate a sequence of short tasks (called pre-commit "hooks") such as code formatting. In particular, we always use the following hooks when submitting code to the main repository.

* [**pyupgrade**](https://github.com/asottile/pyupgrade): pyupgrade upgrades syntax for newer versions of the language. 
* [**isort**](https://github.com/PyCQA/isort): isort sorts imported dependencies according to their type (e.g, standard library vs third-party library) and name.
* [**black**](https://black.readthedocs.io/en/stable/): black enforces an uniform code style across the codebase.
* [**codespell**](https://github.com/codespell-project/codespell): codespell helps avoid common incorrect spelling.

You can run the following command to run the following hooks:

```bash
uv run pre-commit run --all-files
```

which in most cases should automatically fix things as shown below: 





## Contribution Process


!!! warning

    **Before opening a pull request**, please open an issue first to discuss with us since this is likely a sizable effort. Once we agree on the plan, feel free to make a PR to include the new algorithm.

To ensure the validity of new features or bug fixes and prevent regressions, we use the "RLops" process. Deep Reinforcement Learning (DRL) is brittle and suffers from various reproducibility issues. Even bug fixes can sometimes lead to performance regressions (e.g., see [how a bug fix of contact force in MuJoCo results in worse performance for PPO](https://github.com/openai/gym/pull/2762#discussion_r853488897)).


Therefore, it is essential to understand how the proposed changes impact the performance of the algorithms. Broadly, we categorize contributions into two types: 1) non-performance-impacting changes and 2) performance-impacting changes.

* **non-performance-impacting changes**: this type of change does *not* impact the performance of the algorithm, such as documentation fixes ([:material-github: #282](https://github.com/vwxyzjn/cleanrl/pull/282)), renaming variables ([:material-github: #257](https://github.com/vwxyzjn/cleanrl/pull/257)), and removing unused code ([:material-github: #287](https://github.com/vwxyzjn/cleanrl/pull/287)). For this type of change, we can easily merge them without worrying too much about the consequences.
* **performance-impacting changes**: this type of change influences the algorithm's performance. Examples include making a slight modification to the `gamma` parameter in PPO ([:material-github: #209](https://github.com/vwxyzjn/cleanrl/pull/209)), properly handling action bounds in DDPG ([:material-github: #211](https://github.com/vwxyzjn/cleanrl/pull/211)), and fixing bugs ([:material-github: #281](https://github.com/vwxyzjn/cleanrl/pull/281))


**Non-performance-impacting changes** are relatively straightforward — we just need to make sure the tests pass and the code is formatted correctly. However, **performance-impacting changes** are more complicated. We need to make sure that the proposed changes do not lead to performance regressions. To do so, we use the "RLops" process detailed below, which is a set of procedures to help us run experiments and understand the impacts of the proposed changes.


## RLops for Performance-impacting Changes:


Importantly, **regardless of the slight difference in performance-impacting changes, we need to re-run the benchmark to ensure there is no regression**. To do so, we are going to leverage the `openrlbenchmark.rlops` CLI utility. This utility is a command-line interface that helps us run experiments and compare the performance of different versions of the codebase.

### (Step 1) Run the benchmark

Given a new feature, we create a PR and then run the benchmark experiments through [`benchmark.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/benchmark.py), such as the following:

```bash
uv pip install ".[docs, mujoco]"
xvfb-run -a python -m cleanrl_utils.benchmark \
    --env-ids HalfCheetah-v4 Walker2d-v4 Hopper-v4 \
    --command "uv run python cleanrl/ddpg_continuous_action.py --track --capture_video" \
    --num-seeds 3 \
    --workers 1
```

under the hood, this script by default invokes an `--autotag` feature that tries to tag the the experiments with version control information, such as the git tag (e.g., `v1.0.0b2-8-g6081d30`) and the github PR number (e.g., `pr-299`). This is useful for us to compare the performance of the same algorithm across different versions.



### (Step 2) Regression check

Let's say our latest experiments is tagged with `pr-299`. We can then run the following command to compare its performance with our pilot experiments `rlops-pilot`. Note that the pilot experiments include all experiments before we started using RLops (i.e., `rlops-pilot` is the baseline).


```bash
python -m openrlbenchmark.rlops \
    --filters '?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return' \
        'ddpg_continuous_action?tag=pr-299' \
        'ddpg_continuous_action?tag=rlops-pilot' \
    --output-filename compare \
    --scan-history \
    --report
```

Here, we created a filter `'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return'`, which is a query string that specifies the following:

* `we`: the W&B entity name
* `wpn`: the W&B project name
* `ceik`: the custom key for the environment id
* `cen`: the custom key for the experiment name
* `metric`: the metric we are interested in

So we are fetching metrics from [https://wandb.ai/openrlbenchmark/cleanrl](https://wandb.ai/openrlbenchmark/cleanrl). The environment id is stored in the `env` key, and the experiment name is stored in the `exp_name` key. The metric we are interested in is `charts/episodic_return`.


<script async id="asciicast-570481" src="https://asciinema.org/a/570481.js"></script>


It generates a table comparing the runtime (in minutes):

|                |   openrlbenchmark/cleanrl/ddpg_continuous_action ({'tag': ['pr-299']}) |   openrlbenchmark/cleanrl/ddpg_continuous_action ({'tag': ['rlops-pilot']}) |
|:---------------|-----------------------------------------------------------------------:|----------------------------------------------------------------------------:|
| Hopper-v2      |                                                                49.596  |                                                                     64.7033 |
| Walker2d-v2    |                                                                50.1181 |                                                                     64.8198 |
| HalfCheetah-v2 |                                                                50.97   |                                                                     65.8884 |

It also generates a table comparing the episodic return in the last 100 episodes:

|                | openrlbenchmark/cleanrl/ddpg_continuous_action ({'tag': ['pr-299']})   | openrlbenchmark/cleanrl/ddpg_continuous_action ({'tag': ['rlops-pilot']})   |
|:---------------|:-----------------------------------------------------------------------|:----------------------------------------------------------------------------|
| Hopper-v2      | 1007.44 ± 148.29                                                       | 1126.37 ± 278.02                                                            |
| Walker2d-v2    | 1661.14 ± 250.01                                                       | 1447.09 ± 260.24                                                            |
| HalfCheetah-v2 | 10210.57 ± 196.22                                                      | 9205.65 ± 1093.88                                                           |


!!! info

    For more documentation on `openrlbenchmark.rlops`, please refer to its [documentation](https://github.com/openrlbenchmark/openrlbenchmark). 

!!! tip

    To make the script run faster, we can choose not to use `--scan-history` which allows wandb to sample 500 data points from the training data. This is the default behavior and is much faster.


It also generates the following image and a wandb report.



<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl-cache/reports/Regression-Report-ddpg_continuous_action-pr-299-rlops-pilot---VmlldzoyODkyNTcy" style="width:100%; height:500px" title="Regression Report: ddpg_continuous_action (['pr-299', 'rlops-pilot'])"></iframe>



### (Step 3) Update the documentation

Once we confirm there is no regression in the performance, we can update the documentation to display the new benchmark results. Run the previous command without comparing previous tags:

```bash
python -m cleanrl_utils.rlops --exp-name ddpg_continuous_action \
    --wandb-project-name cleanrl \
    --wandb-entity openrlbenchmark \
    --tags 'pr-299' \
    --env-ids HalfCheetah-v2 Walker2d-v2 Hopper-v2 \
    --output-filename compare.png \
    --scan-history \
    --metric-last-n-average-window 100
```

which gives us a table like below and a `compare.png` as the learning curve.

```
               CleanRL's ddpg_continuous_action (pr-299)
HalfCheetah-v2                         10210.57 ± 196.22
Walker2d-v2                             1661.14 ± 250.01
Hopper-v2                               1007.44 ± 148.29
```

We will use them to update the [experimental result section](https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ddpg.md#experiment-results) in the docs and replace the learning curves with the new ones. 




### (Step 4) Update test cases

In CleanRL, we have end-to-end test cases that at least ensure the code can run without crashing. When applicable, you should update the test cases in `tests` and CI setting at [`.github/workflows/tests.yaml`](https://github.com/vwxyzjn/cleanrl/blob/master/.github/workflows/tests.yaml).



### (Step 5) Merge the PR

Finally, we can merge the PR.


## Checklist

Here is a checklist of the contribution process. See [:material-github: #331](https://github.com/vwxyzjn/cleanrl/pull/331/files) as an example for the list of deliverables.

- [ ] I've read the [CONTRIBUTION](https://docs.cleanrl.dev/contribution/) guide (**required**).
- [ ] I have ensured `pre-commit run --all-files` passes (**required**).
- [ ] I have updated the tests accordingly (if applicable).
- [ ] I have updated the documentation and previewed the changes via `mkdocs serve`.
    - [ ] I have explained note-worthy implementation details.
    - [ ] I have explained the logged metrics.
    - [ ] I have added links to the original paper and related papers.


If you need to run benchmark experiments for a performance-impacting changes:

- [ ] I have contacted @vwxyzjn to obtain access to the [openrlbenchmark W&B team](https://wandb.ai/openrlbenchmark).
- [ ] I have used the [benchmark utility](/get-started/benchmark-utility/) to submit the tracked experiments to the [openrlbenchmark/cleanrl](https://wandb.ai/openrlbenchmark/cleanrl) W&B project, optionally with `--capture_video`.
- [ ] I have performed RLops with `python -m openrlbenchmark.rlops`.
    - For new feature or bug fix:
        - [ ] I have used the RLops utility to understand the performance impact of the changes and confirmed there is no regression.
    - For new algorithm:
        - [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
    - [ ] I have added the learning curves generated by the `python -m openrlbenchmark.rlops` utility to the documentation.
    - [ ] I have added links to the tracked experiments in W&B, generated by `python -m openrlbenchmark.rlops ....your_args... --report`,  to the documentation.

---

### Advanced Hyperparameter Tuning (`docs/advanced/hyperparameter-tuning.md`)

# Hyperparameter Tuning

CleanRL comes with a lightweight hyperparameter tuning utility `Tuner`, with a primary purpose of helping researchers find **a single set of  hyperparameters that works well with multiple tasks of similar type** (e.g., one set of parameters for MuJoCo, another set for Atari). We try to avoid tuning hyperparameters per task, which is usually adhoc and do not generalize.

## Get started

Consider the [:material-github: `tuner_example.py`](https://github.com/vwxyzjn/cleanrl/blob/master/tuner_example.py) in the CleanRL's root directory, which has the following content:


```python title="tuner_example.py"
import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": [0, 500],
        "Acrobot-v1": [-500, 0],
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)
```

You can run the tuner with the following command:

```bash
uv pip install ".[optuna]"
python tuner_example.py
```

<script id="asciicast-517226" src="https://asciinema.org/a/517226.js" async></script>


Often it is useful to find a single set of hyper parameters that work well with multiple environments, but this is challenging **because each environment may have different reward scales**, so we need to normalize the reward scales. 

This is where `target_scores` comes in. We can use it to specify an upper and lower threshold of rewards for each environment (they don't have to be exact boundaries and can be just ballpark estimates). For example, if we want to find a set of hyper parameters that work well with `CartPole-v1` and `Acrobot-v1`, we can set the `target_scores` and `aggregation_type` as follows:

```python
tuner = Tuner(
    ...,
    target_scores={
        "CartPole-v1": [0, 500],
        "Acrobot-v1": [-500, 0],
    }
    aggregation_type="average",
)
```


Here is what happened when running `python tuner_example.py`:

1. The `tuner_example.py` launches `num_trials=100` *trials* to find the best single set of hyperparameters for `CartPole-v1` and `Acrobot-v1` in `script="cleanrl/ppo.py"`.
1. Each *trial* samples a set of hyperparameters from the `params_fn` to run `num_seeds=3` *experiments* with different random seeds, mitigating the impact of randomness on the results. 
    * In each *experiment*, `tuner_example.py` averages the last `metric_last_n_average_window=50` reported `metric="charts/episodic_return"` to a number $x_i$ and calculate a normalized score $z_i$ according to the `target_scores`. In this case, $z_{i_0} = (x_{i_0} - 0) / (500 - 0)$ and $z_{i_1} = (x_{i_1} - -500) / (0 - -500)$. Then the tuner will `aggregation_type="average"` the two scores to get the normalized score $z_i = (z_{i_0} + z_{i_1}) / 2$.
1. Each *trial* then averages the normalized scores $z_i$ of the `num_seeds=3` *experiments* to a number $z$ and the tuner optimizes $z$ according `direction="maximize"`.


Note that we are using 3 random seeds for each environment in `["CartPole-v1","Acrobot-v1"]`, totalling `2*3=6` experiments per trial.


???+ info

    When optimizing Atari games, you can put the human normalized scores in `target_scores` (Mnih et al., 2015, Extended Data Table 2)[^1], as done in the following example. The first number for each environment is the score obtained by random play and the second number is the score obtained by professional game testers. Note here we using `aggregation_type="median"`, this means we will optimize for the **median** of the human normalized scores **averaged** over the `num_seeds=3` experiments for each trial (pay close attention to the phrasing).

    ```python
    tuner = Tuner(
        script="cleanrl/ppo_atari_envpool.py",
        metric="charts/episodic_return",
        metric_last_n_average_window=50,
        direction="maximize",
        aggregation_type="median",
        target_scores={
            "Alien-v5": [227.8, 6875],
            "Amidar-v5": [5.8, 1676],
            'Assault-v5': (222.4, 1496),
            'Asterix-v5': (210.0, 8503),
            'Asteroids-v5': (719.1, 13157),
            ...
        },
        num_seeds=3,
        ...
    )
    ```

## Visualization

Running `python tuner_example.py` will create a sqlite database containing all of the hyperparameter trials in `./cleanrl_hpopt.db`  We can use [optuna-dashboard](https://github.com/optuna/optuna-dashboard) to visualize the process.

```bash
uv run optuna-dashboard sqlite:///cleanrl_hpopt.db
```



In the panel above, the y-axis shows the normalized score of the two environments (`CartPole-v1` and `Acrobot-v1`) and the x-axis shows the number of trials.



The panel above shows the parallel coordinates plot of the normalized score — for example, we can see having a lower learning rate and a higher number of minibatches is a good choice.


???+ info

    You can use a different database by passing `Tuner(..., storage="mysql://root@localhost/example")` to use a more persistent storage.

## Work w/o knowing the reward scales

What if we don't know the reward scales for the environments? We can optionally set the value `target_scores` to `None`, but this will only work with one environment at a time because it's difficult to optimize without knowing the scale of the rewards in each environment.


```python title="tuner_example.py" hl_lines="11-13"
import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": None,
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)
```

In the example above, we will set the normalized score $z_i$ to be the average of the last `metric_last_n_average_window=50` reported `metric="charts/episodic_return"` for `CartPole-v1`.


## Work w/ pruners and samplers

You can use `Tuner` with any [pruner](https://optuna.readthedocs.io/en/stable/reference/pruners.html) from `optuna` to prune less promising experiments or [samplers](https://optuna.readthedocs.io/en/stable/reference/samplers.html) to sample new hyperparameters. If you don't specify them explicitly, the script will use the [default ones](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.create_study.html).

```python title="tuner_example.py" hl_lines="1 24 25"
import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": None,
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)
```


## Track experiments w/ Weights and Biases

The `Tuner` can track all the experiments into [Weights and Biases](https://wandb.ai) to help you visualize the progress of the tuning.


```python title="tuner_example.py" hl_lines="26"
import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": None,
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
    wandb_kwargs={"project": "cleanrl"},
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)
```





[^1]:Mnih, V., Kavukcuoglu, K., Silver, D. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236

---

### Advanced Resume Training (`docs/advanced/resume-training.md`)

# Resume Training


A common question we get asked is how to set up model checkpoints to continue training. In this document, we take this [PPO example](https://github.com/vwxyzjn/gym-microrts/blob/master/experiments/ppo_gridnet.py) to explain that question.

## Save model checkpoints

The first step is to save models periodically. By default, we save the model to `wandb`.

```python linenums="1" hl_lines="3 4 6 9-14"
num_updates = args.total_timesteps // args.batch_size

CHECKPOINT_FREQUENCY = 50
starting_update = 1

for update in range(starting_update, num_updates + 1):
    # ... do rollouts and train models

    if args.track:
        # make sure to tune `CHECKPOINT_FREQUENCY` 
        # so models are not saved too frequently
        if update % CHECKPOINT_FREQUENCY == 0:
            torch.save(agent.state_dict(), f"{wandb.run.dir}/agent.pt")
            wandb.save(f"{wandb.run.dir}/agent.pt", policy="now")
```

Then we could run the following to train our agents

```
python ppo_gridnet.py --prod-mode --capture_video
```

If the training was terminated early, we can still see the last updated model `agent.pt` in W&B like in this URL [https://wandb.ai/costa-huang/cleanRL/runs/21421tda/files](https://wandb.ai/costa-huang/cleanRL/runs/21421tda/files) or as follows

<iframe src="https://wandb.ai/costa-huang/cleanRL/runs/21421tda/files" style="width:100%; height:500px" title="CleanRL CartPole-v1 Example"></iframe>


## Resume training

The second step is to automatically download the `agent.pt` from the URL above and resume training as follows:


```python linenums="1" hl_lines="6-16"
num_updates = args.total_timesteps // args.batch_size

CHECKPOINT_FREQUENCY = 50
starting_update = 1

if args.track and wandb.run.resumed:
    starting_update = run.summary.get("charts/update") + 1
    global_step = starting_update * args.batch_size
    api = wandb.Api()
    run = api.run(f"{run.entity}/{run.project}/{run.id}")
    model = run.file("agent.pt")
    model.download(f"models/{experiment_name}/")
    agent.load_state_dict(torch.load(
        f"models/{experiment_name}/agent.pt", map_location=device))
    agent.eval()
    print(f"resumed at update {starting_update}")

for update in range(starting_update, num_updates + 1):
    # ... do rollouts and train models

    if args.track:
        # make sure to tune `CHECKPOINT_FREQUENCY` 
        # so models are not saved too frequently
        if update % CHECKPOINT_FREQUENCY == 0:
            torch.save(agent.state_dict(), f"{wandb.run.dir}/agent.pt")
            wandb.save(f"{wandb.run.dir}/agent.pt", policy="now")
```

To resume training, note the ID of the experiment is `21421tda` as in the URL [https://wandb.ai/costa-huang/cleanRL/runs/21421tda](https://wandb.ai/costa-huang/cleanRL/runs/21421tda), so we need to pass in the ID via environment variable to trigger the resume mode of W&B:

```
WANDB_RUN_ID=21421tda WANDB_RESUME=must python ppo_gridnet.py --prod-mode --capture_video
```

---

### Benchmark Ddpg (`docs/benchmark/ddpg.md`)

|                     | openrlbenchmark/cleanrl/ddpg_continuous_action ({'tag': ['pr-424']})   | openrlbenchmark/cleanrl/ddpg_continuous_action_jax ({'tag': ['pr-424']})   |
|:--------------------|:-----------------------------------------------------------------------|:---------------------------------------------------------------------------|
| HalfCheetah-v4      | 10374.07 ± 157.37                                                      | 8638.60 ± 1954.46                                                          |
| Walker2d-v4         | 1240.16 ± 390.10                                                       | 1427.23 ± 104.91                                                           |
| Hopper-v4           | 1576.78 ± 818.98                                                       | 1208.52 ± 659.22                                                           |
| InvertedPendulum-v4 | 642.68 ± 69.56                                                         | 804.30 ± 87.60                                                             |
| Humanoid-v4         | 1699.56 ± 694.22                                                       | 1513.61 ± 248.60                                                           |
| Pusher-v4           | -77.30 ± 38.78                                                         | -38.56 ± 4.47                                                              |

---

### Benchmark Ppo (`docs/benchmark/ppo.md`)

|                | openrlbenchmark/cleanrl/ppo ({'tag': ['pr-424']})   |
|:---------------|:----------------------------------------------------|
| CartPole-v1    | 490.04 ± 6.12                                       |
| Acrobot-v1     | -86.36 ± 1.32                                       |
| MountainCar-v0 | -200.00 ± 0.00                                      |

---

### Benchmark Ppo Atari (`docs/benchmark/ppo_atari.md`)

|                         | openrlbenchmark/cleanrl/ppo_atari ({'tag': ['pr-424']})   |
|:------------------------|:----------------------------------------------------------|
| PongNoFrameskip-v4      | 20.36 ± 0.20                                              |
| BeamRiderNoFrameskip-v4 | 1915.93 ± 484.58                                          |
| BreakoutNoFrameskip-v4  | 414.66 ± 28.09                                            |

---

### Benchmark Ppo Atari Envpool (`docs/benchmark/ppo_atari_envpool.md`)

|              | openrlbenchmark/cleanrl/ppo_atari_envpool ({'tag': ['pr-424']})   | openrlbenchmark/cleanrl/ppo_atari ({'tag': ['pr-424']})   |
|:-------------|:------------------------------------------------------------------|:----------------------------------------------------------|
| Pong-v5      | 20.45 ± 0.09                                                      | 20.36 ± 0.20                                              |
| BeamRider-v5 | 2501.85 ± 210.52                                                  | 1915.93 ± 484.58                                          |
| Breakout-v5  | 211.24 ± 151.84                                                   | 414.66 ± 28.09                                            |

---

### Benchmark Ppo Atari Envpool Runtimes (`docs/benchmark/ppo_atari_envpool_runtimes.md`)

|              |   openrlbenchmark/cleanrl/ppo_atari_envpool ({'tag': ['pr-424']}) |   openrlbenchmark/cleanrl/ppo_atari ({'tag': ['pr-424']}) |
|:-------------|------------------------------------------------------------------:|----------------------------------------------------------:|
| Pong-v5      |                                                           178.375 |                                                   281.071 |
| BeamRider-v5 |                                                           182.944 |                                                   284.941 |
| Breakout-v5  |                                                           151.384 |                                                   264.077 |

---

### Benchmark Ppo Atari Envpool Xla Jax (`docs/benchmark/ppo_atari_envpool_xla_jax.md`)

|                     | openrlbenchmark/envpool-atari/ppo_atari_envpool_xla_jax ({})   | openrlbenchmark/baselines/baselines-ppo2-cnn ({})   |
|:--------------------|:---------------------------------------------------------------|:----------------------------------------------------|
| Alien-v5            | 1736.39 ± 68.65                                                | 1705.80 ± 439.74                                    |
| Amidar-v5           | 653.53 ± 44.06                                                 | 585.99 ± 52.92                                      |
| Assault-v5          | 6791.74 ± 420.03                                               | 4878.67 ± 815.64                                    |
| Asterix-v5          | 4820.33 ± 1091.83                                              | 3738.50 ± 745.13                                    |
| Asteroids-v5        | 1633.67 ± 247.21                                               | 1556.90 ± 151.20                                    |
| Atlantis-v5         | 3778458.33 ± 117680.68                                         | 2036749.00 ± 95929.75                               |
| BankHeist-v5        | 1195.44 ± 18.54                                                | 1213.47 ± 14.46                                     |
| BattleZone-v5       | 24283.75 ± 1841.94                                             | 19980.00 ± 1355.21                                  |
| BeamRider-v5        | 2478.44 ± 336.55                                               | 2835.71 ± 387.92                                    |
| Berzerk-v5          | 992.88 ± 196.90                                                | 1049.77 ± 144.58                                    |
| Bowling-v5          | 51.62 ± 13.53                                                  | 59.66 ± 0.62                                        |
| Boxing-v5           | 92.68 ± 1.41                                                   | 93.32 ± 0.36                                        |
| Breakout-v5         | 430.09 ± 8.12                                                  | 405.73 ± 11.47                                      |
| Centipede-v5        | 3309.34 ± 325.05                                               | 3688.54 ± 412.24                                    |
| ChopperCommand-v5   | 5642.83 ± 802.34                                               | 816.33 ± 114.14                                     |
| CrazyClimber-v5     | 118763.04 ± 4915.34                                            | 119344.67 ± 4902.83                                 |
| Defender-v5         | 48558.98 ± 4466.76                                             | 50161.67 ± 4477.49                                  |
| DemonAttack-v5      | 29283.83 ± 7007.31                                             | 13788.43 ± 1313.44                                  |
| DoubleDunk-v5       | -6.81 ± 0.24                                                   | -12.96 ± 0.31                                       |
| Enduro-v5           | 1297.23 ± 143.71                                               | 986.69 ± 25.28                                      |
| FishingDerby-v5     | 21.21 ± 6.73                                                   | 26.23 ± 2.76                                        |
| Freeway-v5          | 33.10 ± 0.31                                                   | 32.97 ± 0.37                                        |
| Frostbite-v5        | 1137.34 ± 1192.05                                              | 933.60 ± 885.92                                     |
| Gopher-v5           | 6505.29 ± 7655.20                                              | 3672.53 ± 1749.20                                   |
| Gravitar-v5         | 1099.33 ± 603.06                                               | 881.67 ± 33.73                                      |
| Hero-v5             | 26429.65 ± 924.74                                              | 24746.88 ± 3530.10                                  |
| IceHockey-v5        | -4.33 ± 0.43                                                   | -4.12 ± 0.20                                        |
| Jamesbond-v5        | 496.08 ± 24.60                                                 | 536.50 ± 82.33                                      |
| Kangaroo-v5         | 6582.12 ± 5395.44                                              | 5325.33 ± 3464.80                                   |
| Krull-v5            | 9718.09 ± 649.15                                               | 8737.10 ± 294.58                                    |
| KungFuMaster-v5     | 26000.25 ± 1965.22                                             | 30451.67 ± 5515.45                                  |
| MontezumaRevenge-v5 | 0.08 ± 0.12                                                    | 1.00 ± 1.41                                         |
| MsPacman-v5         | 2345.67 ± 185.94                                               | 2152.83 ± 152.80                                    |
| NameThisGame-v5     | 5750.00 ± 181.32                                               | 6815.63 ± 1098.95                                   |
| Phoenix-v5          | 14474.11 ± 1794.83                                             | 9517.73 ± 1176.62                                   |
| Pitfall-v5          | 0.00 ± 0.00                                                    | -0.76 ± 0.55                                        |
| Pong-v5             | 20.39 ± 0.24                                                   | 20.45 ± 0.81                                        |
| PrivateEye-v5       | 100.00 ± 0.00                                                  | 31.83 ± 43.74                                       |
| Qbert-v5            | 17246.27 ± 605.40                                              | 15228.25 ± 920.95                                   |
| Riverraid-v5        | 8275.25 ± 256.63                                               | 9023.57 ± 1386.85                                   |
| RoadRunner-v5       | 33040.38 ± 16488.95                                            | 40125.33 ± 7249.13                                  |
| Robotank-v5         | 14.43 ± 4.98                                                   | 16.45 ± 3.37                                        |
| Seaquest-v5         | 1240.30 ± 419.36                                               | 1518.33 ± 400.35                                    |
| Skiing-v5           | -18483.46 ± 8684.71                                            | -22978.48 ± 9894.25                                 |
| Solaris-v5          | 2198.36 ± 147.23                                               | 2365.33 ± 157.75                                    |
| SpaceInvaders-v5    | 1188.82 ± 80.52                                                | 1019.75 ± 49.08                                     |
| StarGunner-v5       | 43519.12 ± 4709.23                                             | 44457.67 ± 3031.86                                  |
| Surround-v5         | -2.58 ± 2.31                                                   | -4.97 ± 0.99                                        |
| Tennis-v5           | -17.64 ± 4.60                                                  | -16.44 ± 1.46                                       |
| TimePilot-v5        | 6476.46 ± 993.30                                               | 6346.67 ± 663.31                                    |
| Tutankham-v5        | 249.05 ± 16.56                                                 | 190.73 ± 12.00                                      |
| UpNDown-v5          | 487495.41 ± 39751.49                                           | 156143.70 ± 70620.88                                |
| Venture-v5          | 0.00 ± 0.00                                                    | 109.33 ± 61.57                                      |
| VideoPinball-v5     | 43133.94 ± 6362.12                                             | 53121.26 ± 2580.70                                  |
| WizardOfWor-v5      | 6353.58 ± 116.59                                               | 5346.33 ± 277.11                                    |
| YarsRevenge-v5      | 55757.68 ± 7467.49                                             | 9394.97 ± 2743.74                                   |
| Zaxxon-v5           | 3689.67 ± 2477.25                                              | 5532.67 ± 2607.65                                   |

---

### Benchmark Ppo Atari Envpool Xla Jax Runtimes (`docs/benchmark/ppo_atari_envpool_xla_jax_runtimes.md`)

|                     |   openrlbenchmark/envpool-atari/ppo_atari_envpool_xla_jax ({}) |   openrlbenchmark/baselines/baselines-ppo2-cnn ({}) |
|:--------------------|---------------------------------------------------------------:|----------------------------------------------------:|
| Alien-v5            |                                                        50.3275 |                                             117.397 |
| Amidar-v5           |                                                        42.8176 |                                             114.093 |
| Assault-v5          |                                                        35.9245 |                                             108.094 |
| Asterix-v5          |                                                        37.7117 |                                             113.386 |
| Asteroids-v5        |                                                        39.9731 |                                             114.409 |
| Atlantis-v5         |                                                        40.1527 |                                             123.05  |
| BankHeist-v5        |                                                        38.7443 |                                             137.308 |
| BattleZone-v5       |                                                        45.0654 |                                             138.489 |
| BeamRider-v5        |                                                        42.0778 |                                             119.437 |
| Berzerk-v5          |                                                        38.7173 |                                             135.316 |
| Bowling-v5          |                                                        35.0156 |                                             131.365 |
| Boxing-v5           |                                                        48.8149 |                                             151.607 |
| Breakout-v5         |                                                        42.3547 |                                             122.828 |
| Centipede-v5        |                                                        43.6886 |                                             150.112 |
| ChopperCommand-v5   |                                                        45.9308 |                                             131.192 |
| CrazyClimber-v5     |                                                        36.0841 |                                             127.942 |
| Defender-v5         |                                                        35.1029 |                                             132.29  |
| DemonAttack-v5      |                                                        35.41   |                                             128.476 |
| DoubleDunk-v5       |                                                        41.4521 |                                             108.028 |
| Enduro-v5           |                                                        44.9909 |                                             142.046 |
| FishingDerby-v5     |                                                        51.6075 |                                             151.286 |
| Freeway-v5          |                                                        50.7103 |                                             154.163 |
| Frostbite-v5        |                                                        47.5474 |                                             146.092 |
| Gopher-v5           |                                                        36.2977 |                                             139.496 |
| Gravitar-v5         |                                                        41.9322 |                                             138.746 |
| Hero-v5             |                                                        50.5106 |                                             152.413 |
| IceHockey-v5        |                                                        43.0228 |                                             144.455 |
| Jamesbond-v5        |                                                        38.8264 |                                             137.321 |
| Kangaroo-v5         |                                                        44.4304 |                                             142.436 |
| Krull-v5            |                                                        47.7748 |                                             147.313 |
| KungFuMaster-v5     |                                                        43.1534 |                                             141.903 |
| MontezumaRevenge-v5 |                                                        44.8838 |                                             146.777 |
| MsPacman-v5         |                                                        42.6463 |                                             138.382 |
| NameThisGame-v5     |                                                        43.8473 |                                             136.264 |
| Phoenix-v5          |                                                        36.7586 |                                             129.716 |
| Pitfall-v5          |                                                        44.6369 |                                             137.36  |
| Pong-v5             |                                                        36.7657 |                                             118.745 |
| PrivateEye-v5       |                                                        43.3399 |                                             143.957 |
| Qbert-v5            |                                                        40.1475 |                                             135.255 |
| Riverraid-v5        |                                                        44.2555 |                                             142.627 |
| RoadRunner-v5       |                                                        46.1059 |                                             145.451 |
| Robotank-v5         |                                                        48.3364 |                                             149.681 |
| Seaquest-v5         |                                                        38.3639 |                                             136.942 |
| Skiing-v5           |                                                        38.6402 |                                             132.061 |
| Solaris-v5          |                                                        50.2944 |                                             136.9   |
| SpaceInvaders-v5    |                                                        39.4931 |                                             125.83  |
| StarGunner-v5       |                                                        33.7096 |                                             119.18  |
| Surround-v5         |                                                        33.923  |                                             132.017 |
| Tennis-v5           |                                                        39.6194 |                                              97.019 |
| TimePilot-v5        |                                                        37.0124 |                                             130.693 |
| Tutankham-v5        |                                                        36.9677 |                                             139.694 |
| UpNDown-v5          |                                                        52.9895 |                                             140.876 |
| Venture-v5          |                                                        37.9828 |                                             144.236 |
| VideoPinball-v5     |                                                        47.1716 |                                             179.866 |
| WizardOfWor-v5      |                                                        37.5751 |                                             142.086 |
| YarsRevenge-v5      |                                                        36.5889 |                                             127.358 |
| Zaxxon-v5           |                                                        41.9785 |                                             133.922 |

---

### Benchmark Ppo Atari Envpool Xla Jax Scan (`docs/benchmark/ppo_atari_envpool_xla_jax_scan.md`)

|              | openrlbenchmark/cleanrl/ppo_atari_envpool_xla_jax ({'tag': ['pr-424']})   | openrlbenchmark/cleanrl/ppo_atari_envpool_xla_jax_scan ({'tag': ['pr-424']})   |
|:-------------|:--------------------------------------------------------------------------|:-------------------------------------------------------------------------------|
| Pong-v5      | 20.82 ± 0.21                                                              | 20.52 ± 0.32                                                                   |
| BeamRider-v5 | 2678.73 ± 426.42                                                          | 2860.61 ± 801.30                                                               |
| Breakout-v5  | 420.92 ± 16.75                                                            | 423.90 ± 5.49                                                                  |

---

### Benchmark Ppo Atari Envpool Xla Jax Scan Runtimes (`docs/benchmark/ppo_atari_envpool_xla_jax_scan_runtimes.md`)

|              |   openrlbenchmark/cleanrl/ppo_atari_envpool_xla_jax ({'tag': ['pr-424']}) |   openrlbenchmark/cleanrl/ppo_atari_envpool_xla_jax_scan ({'tag': ['pr-424']}) |
|:-------------|--------------------------------------------------------------------------:|-------------------------------------------------------------------------------:|
| Pong-v5      |                                                                   34.3237 |                                                                        34.701  |
| BeamRider-v5 |                                                                   37.1076 |                                                                        37.2449 |
| Breakout-v5  |                                                                   39.576  |                                                                        39.775  |

---

### Benchmark Ppo Atari Lstm (`docs/benchmark/ppo_atari_lstm.md`)

|                         | openrlbenchmark/cleanrl/ppo_atari_lstm ({'tag': ['pr-424']})   |
|:------------------------|:---------------------------------------------------------------|
| PongNoFrameskip-v4      | 19.81 ± 0.62                                                   |
| BeamRiderNoFrameskip-v4 | 1299.25 ± 509.90                                               |
| BreakoutNoFrameskip-v4  | 113.42 ± 5.85                                                  |

---

### Benchmark Ppo Atari Lstm Runtimes (`docs/benchmark/ppo_atari_lstm_runtimes.md`)

|                         |   openrlbenchmark/cleanrl/ppo_atari_lstm ({'tag': ['pr-424']}) |
|:------------------------|---------------------------------------------------------------:|
| PongNoFrameskip-v4      |                                                        317.607 |
| BeamRiderNoFrameskip-v4 |                                                        314.864 |
| BreakoutNoFrameskip-v4  |                                                        383.724 |

---

### Benchmark Ppo Atari Multigpu (`docs/benchmark/ppo_atari_multigpu.md`)

|                         | openrlbenchmark/cleanrl/ppo_atari_multigpu ({'tag': ['pr-424']})   | openrlbenchmark/cleanrl/ppo_atari ({'tag': ['pr-424']})   |
|:------------------------|:-------------------------------------------------------------------|:----------------------------------------------------------|
| PongNoFrameskip-v4      | 20.34 ± 0.43                                                       | 20.36 ± 0.20                                              |
| BeamRiderNoFrameskip-v4 | 2414.65 ± 643.74                                                   | 1915.93 ± 484.58                                          |
| BreakoutNoFrameskip-v4  | 414.94 ± 20.60                                                     | 414.66 ± 28.09                                            |

---

### Benchmark Ppo Atari Multigpu Runtimes (`docs/benchmark/ppo_atari_multigpu_runtimes.md`)

|                         |   openrlbenchmark/cleanrl/ppo_atari_multigpu ({'tag': ['pr-424']}) |
|:------------------------|-------------------------------------------------------------------:|
| PongNoFrameskip-v4      |                                                            276.599 |
| BeamRiderNoFrameskip-v4 |                                                            280.902 |
| BreakoutNoFrameskip-v4  |                                                            270.532 |

---

### Benchmark Ppo Atari Runtimes (`docs/benchmark/ppo_atari_runtimes.md`)

|                         |   openrlbenchmark/cleanrl/ppo_atari ({'tag': ['pr-424']}) |
|:------------------------|----------------------------------------------------------:|
| PongNoFrameskip-v4      |                                                   281.071 |
| BeamRiderNoFrameskip-v4 |                                                   284.941 |
| BreakoutNoFrameskip-v4  |                                                   264.077 |

---

### Benchmark Ppo Continuous Action (`docs/benchmark/ppo_continuous_action.md`)

|                                      | openrlbenchmark/cleanrl/ppo_continuous_action ({'tag': ['pr-424']})   |
|:-------------------------------------|:----------------------------------------------------------------------|
| HalfCheetah-v4                       | 1442.64 ± 46.03                                                       |
| Walker2d-v4                          | 2287.95 ± 571.78                                                      |
| Hopper-v4                            | 2382.86 ± 271.74                                                      |
| InvertedPendulum-v4                  | 963.09 ± 22.20                                                        |
| Humanoid-v4                          | 716.11 ± 49.08                                                        |
| Pusher-v4                            | -40.38 ± 7.15                                                         |
| dm_control/acrobot-swingup-v0        | 25.60 ± 6.30                                                          |
| dm_control/acrobot-swingup_sparse-v0 | 1.35 ± 0.27                                                           |
| dm_control/ball_in_cup-catch-v0      | 619.26 ± 278.67                                                       |

---

### Benchmark Ppo Continuous Action Runtimes (`docs/benchmark/ppo_continuous_action_runtimes.md`)

|                                      |   openrlbenchmark/cleanrl/ppo_continuous_action ({'tag': ['pr-424']}) |
|:-------------------------------------|----------------------------------------------------------------------:|
| HalfCheetah-v4                       |                                                               25.3589 |
| Walker2d-v4                          |                                                               24.3157 |
| Hopper-v4                            |                                                               25.7066 |
| InvertedPendulum-v4                  |                                                               23.7672 |
| Humanoid-v4                          |                                                               49.5592 |
| Pusher-v4                            |                                                               28.8162 |
| dm_control/acrobot-swingup-v0        |                                                               26.5793 |
| dm_control/acrobot-swingup_sparse-v0 |                                                               25.1265 |
| dm_control/ball_in_cup-catch-v0      |                                                               26.1947 |

---

### Benchmark Ppo Envpool (`docs/benchmark/ppo_envpool.md`)

|              | openrlbenchmark/cleanrl/ppo_atari_envpool_xla_jax ({'tag': ['pr-424']})   | openrlbenchmark/cleanrl/ppo_atari_envpool_xla_jax_scan ({'tag': ['pr-424']})   | openrlbenchmark/cleanrl/ppo_atari_envpool ({'tag': ['pr-424']})   |
|:-------------|:--------------------------------------------------------------------------|:-------------------------------------------------------------------------------|:------------------------------------------------------------------|
| Pong-v5      | 20.82 ± 0.21                                                              | 20.52 ± 0.32                                                                   | 20.45 ± 0.09                                                      |
| BeamRider-v5 | 2678.73 ± 426.42                                                          | 2860.61 ± 801.30                                                               | 2501.85 ± 210.52                                                  |
| Breakout-v5  | 420.92 ± 16.75                                                            | 423.90 ± 5.49                                                                  | 211.24 ± 151.84                                                   |

---

### Benchmark Ppo Envpool Runtimes (`docs/benchmark/ppo_envpool_runtimes.md`)

|              |   openrlbenchmark/cleanrl/ppo_atari_envpool_xla_jax ({'tag': ['pr-424']}) |   openrlbenchmark/cleanrl/ppo_atari_envpool_xla_jax_scan ({'tag': ['pr-424']}) |   openrlbenchmark/cleanrl/ppo_atari_envpool ({'tag': ['pr-424']}) |
|:-------------|--------------------------------------------------------------------------:|-------------------------------------------------------------------------------:|------------------------------------------------------------------:|
| Pong-v5      |                                                                   34.3237 |                                                                        34.701  |                                                           178.375 |
| BeamRider-v5 |                                                                   37.1076 |                                                                        37.2449 |                                                           182.944 |
| Breakout-v5  |                                                                   39.576  |                                                                        39.775  |                                                           151.384 |

---

### Benchmark Ppo Procgen (`docs/benchmark/ppo_procgen.md`)

|           | openrlbenchmark/cleanrl/ppo_procgen ({'tag': ['pr-424']})   |
|:----------|:------------------------------------------------------------|
| starpilot | 30.99 ± 1.96                                                |
| bossfight | 8.85 ± 0.33                                                 |
| bigfish   | 16.46 ± 2.71                                                |

---

### Benchmark Ppo Procgen Runtimes (`docs/benchmark/ppo_procgen_runtimes.md`)

|           |   openrlbenchmark/cleanrl/ppo_procgen ({'tag': ['pr-424']}) |
|:----------|------------------------------------------------------------:|
| starpilot |                                                     114.649 |
| bossfight |                                                     128.679 |
| bigfish   |                                                     107.788 |

---

### Benchmark Ppo Runtimes (`docs/benchmark/ppo_runtimes.md`)

|                |   openrlbenchmark/cleanrl/ppo ({'tag': ['pr-424']}) |
|:---------------|----------------------------------------------------:|
| CartPole-v1    |                                            10.4737  |
| Acrobot-v1     |                                            15.4606  |
| MountainCar-v0 |                                             6.95995 |

---

### Benchmark Sac (`docs/benchmark/sac.md`)

|                     | openrlbenchmark/cleanrl/sac_continuous_action ({'tag': ['pr-424']})   |
|:--------------------|:----------------------------------------------------------------------|
| HalfCheetah-v4      | 9634.89 ± 1423.73                                                     |
| Walker2d-v4         | 3591.45 ± 911.33                                                      |
| Hopper-v4           | 2310.46 ± 342.82                                                      |
| InvertedPendulum-v4 | 909.37 ± 55.66                                                        |
| Humanoid-v4         | 4996.29 ± 686.40                                                      |
| Pusher-v4           | -22.45 ± 0.51                                                         |

---

### Benchmark Sac Runtimes (`docs/benchmark/sac_runtimes.md`)

|                     |   openrlbenchmark/cleanrl/sac_continuous_action ({'tag': ['pr-424']}) |
|:--------------------|----------------------------------------------------------------------:|
| HalfCheetah-v4      |                                                               174.778 |
| Walker2d-v4         |                                                               161.161 |
| Hopper-v4           |                                                               173.242 |
| InvertedPendulum-v4 |                                                               179.042 |
| Humanoid-v4         |                                                               177.31  |
| Pusher-v4           |                                                               172.123 |

---

### Benchmark Td3 (`docs/benchmark/td3.md`)

|                     | openrlbenchmark/cleanrl/td3_continuous_action ({'tag': ['pr-424']})   | openrlbenchmark/cleanrl/td3_continuous_action_jax ({'tag': ['pr-424']})   |
|:--------------------|:----------------------------------------------------------------------|:--------------------------------------------------------------------------|
| HalfCheetah-v4      | 9583.22 ± 126.09                                                      | 9345.93 ± 770.54                                                          |
| Walker2d-v4         | 4057.59 ± 658.78                                                      | 3686.19 ± 141.23                                                          |
| Hopper-v4           | 3134.61 ± 360.18                                                      | 2940.10 ± 655.63                                                          |
| InvertedPendulum-v4 | 968.99 ± 25.80                                                        | 988.94 ± 8.86                                                             |
| Humanoid-v4         | 5035.36 ± 21.67                                                       | 5033.22 ± 122.14                                                          |
| Pusher-v4           | -30.92 ± 1.05                                                         | -29.18 ± 1.02                                                             |

---

### Benchmark Td3 Runtimes (`docs/benchmark/td3_runtimes.md`)

|                     |   openrlbenchmark/cleanrl/td3_continuous_action ({'tag': ['pr-424']}) |   openrlbenchmark/cleanrl/td3_continuous_action_jax ({'tag': ['pr-424']}) |
|:--------------------|----------------------------------------------------------------------:|--------------------------------------------------------------------------:|
| HalfCheetah-v4      |                                                               87.353  |                                                                   39.5119 |
| Walker2d-v4         |                                                               80.8592 |                                                                   34.0497 |
| Hopper-v4           |                                                               90.9921 |                                                                   33.4079 |
| InvertedPendulum-v4 |                                                               70.4218 |                                                                   30.2624 |
| Humanoid-v4         |                                                               79.1624 |                                                                   70.2437 |
| Pusher-v4           |                                                               95.2208 |                                                                   39.6051 |

---

### Cloud Installation (`docs/cloud/installation.md`)

# Installation

The rough idea behind the cloud integration is to package our code into a docker container and use AWS Batch to
run thousands of experiments concurrently. 

## Prerequisites

* Terraform (see installation tutorial [here](https://learn.hashicorp.com/tutorials/terraform/install-cli))

We use Terraform to define our infrastructure with AWS Batch, which you can spin up as follows

```bash
# assuming you are at the root of the CleanRL project
uv pip install ".[cloud]"
cd cloud
python -m awscli configure
terraform init
export AWS_DEFAULT_REGION=$(aws configure get region --profile default)
terraform apply
```

<script id="asciicast-445048" src="https://asciinema.org/a/445048.js" async></script>

!!! note
    Don't worry about the cost of spinning up these AWS Batch compute environments and job queues. They are completely free and you are only charged when you submit experiments.


Then your AWS Batch console should look like




### Clean Up
Uninstalling/Deleting the infrastructure is pretty straightforward:
```
export AWS_DEFAULT_REGION=$(aws configure get region --profile default)
terraform destroy
```

---

### Cloud Submit Experiments (`docs/cloud/submit-experiments.md`)

# Submit Experiments

### Inspection

Dry run to inspect the generated docker command
```
uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --num-seed 1
```

The generated docker command should look like
```
docker run -d --cpuset-cpus="0" -e WANDB_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx vwxyzjn/cleanrl:latest /bin/bash -c "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video --seed 1"
```

### Run on AWS

Submit a job using AWS's compute-optimized spot instances 
```
uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --job-queue c5a-large-spot \
    --num-seed 1 \
    --num-vcpu 1 \
    --num-memory 2000 \
    --num-hours 48.0 \
    --provider aws
```

Submit a job using AWS's accelerated-computing spot instances 
```
uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4 --track --capture_video" \
    --job-queue g4dn-xlarge-spot \
    --num-seed 1 \
    --num-vcpu 1 \
    --num-gpu 1 \
    --num-memory 4000 \
    --num-hours 48.0 \
    --provider aws
```

Submit a job using AWS's compute-optimized on-demand instances 
```
uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --job-queue c5a-large \
    --num-seed 1 \
    --num-vcpu 1 \
    --num-memory 2000 \
    --num-hours 48.0 \
    --provider aws
```

Submit a job using AWS's accelerated-computing on-demand instances 
```
uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4 --track --capture_video" \
    --job-queue g4dn-xlarge \
    --num-seed 1 \
    --num-vcpu 1 \
    --num-gpu 1 \
    --num-memory 4000 \
    --num-hours 48.0 \
    --provider aws
```

<script id="asciicast-445050" src="https://asciinema.org/a/445050.js" async></script>

Then you should see:






## Customize the Docker Container

Set up docker's `buildx` and login in to your preferred registry.

```
docker buildx create --use
docker login
```

Then you could build a container using the `--build` flag based on the `Dockerfile` in the current directory. Also, `--push` will auto-push to the docker registry.

```
uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --build --push
```

To build a multi-arch image using `--archs linux/arm64,linux/amd64`:

```
uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --archs linux/arm64,linux/amd64
    --build --push
```

!!! note
    Building an multi-arch image is quite slow but will allow you to use ARM instances such as `m6gd.medium` that is 20-70% cheaper than X86 instances.
    However, note there is no cloud providers that give ARM instances with Nvidia's GPU (to my knowledge), so this effort might not be worth it.

    If you still wants to pursue multi-arch, you can speed things up by using a native ARM server and connect it to your `buildx` instance:

    ```
    docker -H ssh://costa@gpu info
    docker buildx create --name remote --use
    docker buildx create --name remote --append ssh://costa@gpu
    docker buildx inspect --bootstrap
    python -m cleanrl_utils.submit_exp -b --archs linux/arm64,linux/amd64
    ```

---

### Get Started Basic Usage (`docs/get-started/basic-usage.md`)

# Basic Usage

## Two Ways to Run
After the dependencies have been installed, there are **two ways** to run 
the CleanRL script under the uv virtual environments.


1. Using `uv run`:

    ```bash
    uv run python cleanrl/ppo.py \
        --seed 1 \
        --env-id CartPole-v0 \
        --total-timesteps 50000
    ```
    <script id="asciicast-443649" src="https://asciinema.org/a/443649.js" async></script>


2. Using `uv venv`:

    1. We first activate the virtual environment by using
    `uv venv`
    2. Then, run any desired CleanRL script
   
    Attention: Each step must be executed separately!


    ```bash
    uv venv
    ```
    ```bash
    python cleanrl/ppo.py \
        --seed 1 \
        --env-id CartPole-v0 \
        --total-timesteps 50000
    ```
    <script id="asciicast-JL1FR00I2JNklAhMd2dwEAQuz" src="https://asciinema.org/a/JL1FR00I2JNklAhMd2dwEAQuz.js" async></script>

!!! note

    We recommend `uv venv` workflow for development. When the shell is activated, you should
    be seeing a prefix like `(cleanrl-iXg02GqF-py3.9)` in your shell's prompt, which is the name
    of the poetry's virtual environment.
    **We will assume to run other commands (e.g. `tensorboard`) in the documentation within the poetry's shell.**


!!! warning

    If you are using NVIDIA ampere GPUs (e.g., 3060 TI), you might meet the following error

    ```bash
    NVIDIA GeForce RTX 3060 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation.
    The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
    If you want to use the NVIDIA GeForce RTX 3060 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

    warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
    Traceback (most recent call last):
    File "ppo_atari_envpool.py", line 240, in <module>
        action, logprob, _, value = agent.get_action_and_value(next_obs)
    File "ppo_atari_envpool.py", line 156, in get_action_and_value
        hidden = self.network(x / 255.0)
    RuntimeError: CUDA error: no kernel image is available for execution on the device
    CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
    For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
    ```

    This is because the `torch` wheel on PyPi is built with cuda 10.2. You would need to manually install the cuda 11.3 wheel like this:
    ```bash
    uv pip install torch==1.12.1 --upgrade --extra-index-url https://download.pytorch.org/whl/cu113
    ```
    Then, you can run the script again.

## Visualize Training Metrics

By default, the CleanRL scripts record all the training metrics via Tensorboard
into the `runs` folder. So, after running the training script above, feel free to run

```bash
tensorboard --logdir runs
```




## Visualize the Agent's Gameplay Videos

CleanRL helps record the agent's gameplay videos with a `--capture_video` flag,
which will save the videos in the `videos/{$run_name}` folder.

```bash linenums="1" hl_lines="5"
python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000 \
    --capture_video
```




## Get Documentation

You can directly obtained the documentation by using the `--help` flag.

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Get Started Benchmark Utility (`docs/get-started/benchmark-utility.md`)

# Benchmark Utility

CleanRL comes with a utility module `cleanrl_utils.benchmark` to help schedule and run benchmark experiments on your local machine.

## Usage

Try running `python -m cleanrl_utils.benchmark --help` to get the help text.

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Examples

The following example demonstrates how to run classic control benchmark experiments.

```bash
OMP_NUM_THREADS=1 xvfb-run -a python -m cleanrl_utils.benchmark \
    --env-ids CartPole-v1 Acrobot-v1 MountainCar-v0 \
    --command "uv run python cleanrl/ppo.py --no_cuda --track --capture_video" \
    --num-seeds 3 \
    --workers 5
```

What just happened here? In principle the helps run the following commands in 5 subprocesses:

```bash
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id CartPole-v1 --seed 1
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id Acrobot-v1 --seed 1
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id MountainCar-v0 --seed 1
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id CartPole-v1 --seed 2
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id Acrobot-v1 --seed 2
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id MountainCar-v0 --seed 2
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id CartPole-v1 --seed 3
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id Acrobot-v1 --seed 3
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id MountainCar-v0 --seed 3
```

More specifically:

1. `--env-ids CartPole-v1 Acrobot-v1 MountainCar-v0` specifies that running experiments against these three environments
1. `--command "uv run python cleanrl/ppo.py --no_cuda --track --capture_video"` suggests running `ppo.py` with these settings:
    * turn off GPU usage via `--no_cuda`: because `ppo.py` has such as small neural network it often runs faster on CPU only
    * track the experiments via `--track`
    * render the agent gameplay videos via `--capture_video`; these videos algo get saved to the tracked experiments
        * ` xvfb-run -a` virtualizes a display for video recording, enabling these commands on a headless linux system
1. `--num-seeds 3` suggests running the the command with 3 random seeds for each `env-id`
1. `--workers 5` suggests at maximum using 5 subprocesses to run the experiments
    * `OMP_NUM_THREADS=1` suggests `torch` to use only 1 thread for each subprocesses; this way we don't have processes fighting each other.
1. `--autotag` tries to tag the the experiments with version control information, such as the git tag (e.g., `v1.0.0b2-8-g6081d30`) and the github PR number (e.g., `pr-299`). This is useful for us to compare the performance of the same algorithm across different versions.


Note that when you run with high-throughput environments such as `envpool` or `procgen`, it's recommended to set `--workers 1` to maximuize SPS (steps per second), such as

```bash
xvfb-run -a python -m cleanrl_utils.benchmark \
    --env-ids Pong-v5 BeamRider-v5 Breakout-v5 \
    --command "uv run python cleanrl/ppo_atari_envpool.py --track --capture_video" \
    --num-seeds 3 \
    --workers 1
```

For more example usage, see [https://github.com/vwxyzjn/cleanrl/blob/master/benchmark](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark)


## Slurm integration

If you have access to a slurm cluster, you can use `cleanrl_utils.benchmark` to schedule jobs on the cluster. The following example demonstrates how to run classic control benchmark experiments on a slurm cluster.

``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:3:12"
```

```
poetry install
OMP_NUM_THREADS=1 xvfb-run -a uv run python -m cleanrl_utils.benchmark \
    --env-ids CartPole-v1 Acrobot-v1 MountainCar-v0 \
    --command "uv run python cleanrl/ppo.py --no_cuda --track --capture_video" \
    --num-seeds 3 \
    --workers 9 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 10 \
    --slurm-template-path benchmark/cleanrl_1gpu.slurm_template
```

Here, we have
* `--slurm-gpus-per-task 1` suggests that each slurm job should use 1 GPU
* `--slurm-ntasks 1` suggests that each slurm job should use 1 CPU
* `--slurm-total-cpus 10` suggests that each slurm job should use 10 CPUs in total
* `--slurm-template-path benchmark/cleanrl_1gpu.slurm_template` suggests that we should use the template file `benchmark/cleanrl_1gpu.slurm_template` to generate the slurm job scripts. The template file looks like this:

``` title="benchmark/cleanrl_1gpu.slurm_template" linenums="1"
--8<-- "benchmark/cleanrl_1gpu.slurm_template"
```

The utility will generate a slurm script based on the template file and submit the job to the cluster. The generated slurm script looks like this:

```
#!/bin/bash
#SBATCH --job-name=low-priority
#SBATCH --partition=production-cluster
#SBATCH --gpus-per-task=1
#SBATCH --cpus-per-gpu=10
#SBATCH --ntasks=1
#SBATCH --output=slurm/logs/%x_%j.out
#SBATCH --array=0-8%9
#SBATCH --mem-per-cpu=12G
#SBATCH --exclude=ip-26-0-147-[245,247],ip-26-0-156-239
##SBATCH --nodelist=ip-26-0-156-13


env_ids=(CartPole-v1 Acrobot-v1 MountainCar-v0)
seeds=(1 2 3)
env_id=${env_ids[$SLURM_ARRAY_TASK_ID / 3]}
seed=${seeds[$SLURM_ARRAY_TASK_ID % 3]}

echo "Running task $SLURM_ARRAY_TASK_ID with env_id: $env_id and seed: $seed"

srun uv run python cleanrl/ppo.py --no_cuda --track --env-id $env_id --seed $seed # 
```

---

### Get Started Examples (`docs/get-started/examples.md`)

# Examples

## Atari
```
uv venv

uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
uv pip install ".[envpool]"
python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
uv run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3
```
### Demo

<script id="asciicast-443625" src="https://asciinema.org/a/443625.js" async></script>

You can also run training scripts in other games, such as:

## Classic Control
```
uv venv

python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1
```

## Procgen 
```
uv venv

uv pip install ".[procgen]"
python cleanrl/ppo_procgen.py --env-id starpilot
python cleanrl/ppg_procgen.py --env-id starpilot
```


## PPO + LSTM
```
uv venv

uv pip install ".[atari]"
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4
```

---

### Get Started Experiment Tracking (`docs/get-started/experiment-tracking.md`)

# Experiment tracking

To use experiment tracking with wandb, run with the `--track` flag, which will also
upload the videos recorded by the `--capture_video` flag.
```bash
uv venv
wandb login # only required for the first time
python cleanrl/ppo.py --track --capture_video
```


<script id="asciicast-443626" src="https://asciinema.org/a/443626.js" async></script>

The console will output the url for the tracked experiment like the following

```bash
wandb:  View project at https://wandb.ai/costa-huang/cleanRL                            
wandb:  View run at https://wandb.ai/costa-huang/cleanRL/runs/10dwbgeh
```

When you open the URL, it's going to look like the following page:

<iframe src="https://wandb.ai/costa-huang/cleanRL/runs/10dwbgeh" style="width:100%; height:1000px" title="CleanRL CartPole-v1 Example"></iframe>

---

### Get Started Installation (`docs/get-started/installation.md`)

# Installation

## Prerequisites

* Python >=3.7.1,<3.11
* [uv 0.7.19+](https://docs.astral.sh/uv/)

Simply run the following command for a quick start

```bash
git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .
```

<script id="asciicast-443647" src="https://asciinema.org/a/443647.js" async></script>


!!! note "Working with different CUDA versions for `torch`"

    By default, the `torch` wheel is built with CUDA 10.2. If you are using newer NVIDIA GPUs (e.g., 3060 TI), you may need to specifically install CUDA 11.3 wheels by overriding the `torch` dependency with `pip`:

    ```bash
    uv pip install "torch==1.12.1" --upgrade --extra-index-url https://download.pytorch.org/whl/cu113
    ```


## Install via `pip`

While we recommend using `uv` to manage environments and dependencies, the traditional `requirements.txt` are available:

```bash
# core dependencies
pip install -r requirements/requirements.txt

# optional dependencies
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-mujoco_py.txt
pip install -r requirements/requirements-procgen.txt
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-jax.txt
pip install -r requirements/requirements-docs.txt
pip install -r requirements/requirements-cloud.txt
```


## Optional Dependencies

CleanRL makes it easy to install optional dependencies for common RL environments
and various development utilities. These optional dependencies are defined at the
[`pyproject.toml`](https://github.com/vwxyzjn/cleanrl/blob/6afb51624a6fd51775b8351dd25099bd778cb1b1/pyproject.toml#L22-L37) as optional dependencies

You can install them using the following command

```bash
uv pip install ".[atari]"
uv pip install ".[mujoco]"
uv pip install ".[dm_control]"
uv pip install ".[procgen]"
uv pip install ".[envpool]"
uv pip install ".[pettingzoo]"
uv pip install ".[jax]"
uv pip install ".[optuna]"
uv pip install ".[docs]"
uv pip install ".[cloud]"
```

---

### Get Started Zoo (`docs/get-started/zoo.md`)

# 🤗 Model Zoo

[<img src="https://img.shields.io/badge/%F0%9F%A4%97%20Models-Huggingface-F8D521">](https://huggingface.co/cleanrl)
[](https://colab.research.google.com/github/vwxyzjn/cleanrl/blob/master/docs/get-started/CleanRL_Huggingface_Integration_Demo.ipynb)

CleanRL now has 🧪 experimental support for saving and loading models from 🤗 HuggingFace's [Model Hub](https://huggingface.co/models). We are rolling out this feature in phases, and currently only support saving and loading models from the following algorithm variants:


| Algorithm      | Variants Implemented |
| ----------- | ----------- |
| ✅ [Deep Q-Learning (DQN)](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf) | :material-github: [`dqn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqnpy) |
| | :material-github: [`dqn_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_ataripy) |
| | :material-github: [`dqn_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_jax.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_jaxpy) |
| | :material-github: [`dqn_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_atari_jaxpy) |
| ✅ [Categorical DQN (C51)](https://arxiv.org/pdf/1707.06887.pdf) | :material-github: [`c51.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51.py), :material-file-document: [docs](/rl-algorithms/c51/#c51py) |
| | :material-github: [`c51_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_ataripy) |
| | :material-github: [`c51_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_jax.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_jaxpy) |
| | :material-github: [`c51_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari_jax.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_atari_jaxpy) |
| ✅ [Deep Deterministic Policy Gradient (DDPG)](https://arxiv.org/pdf/1509.02971.pdf) | :material-github: [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py), :material-file-document: [docs](/rl-algorithms/ddpg/#ddpg_continuous_actionpy) |
| | :material-github: [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py),  :material-file-document: [docs](/rl-algorithms/ddpg/#ddpg_continuous_action_jaxpy)
| ✅ [Twin Delayed Deep Deterministic Policy Gradient (TD3)](https://arxiv.org/pdf/1802.09477.pdf) | :material-github: [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py), :material-file-document: [docs](/rl-algorithms/td3/#td3_continuous_actionpy) |
|  | :material-github: [`td3_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py), :material-file-document: [docs](/rl-algorithms/td3/#td3_continuous_action_jaxpy) |


## Load models from the Model Hub

We have a simple utility `enjoy.py` to load models from the hub and run them in an environment. We currently support the following commands:

```bash
uv pip install ".[dqn]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn --env-id CartPole-v1
uv pip install ".[dqn, jax]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_jax --env-id CartPole-v1

uv pip install ".[atari]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_atari --env-id BreakoutNoFrameskip-v4
uv pip install ".[atari, jax]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_atari_jax --env-id BreakoutNoFrameskip-v4
```

To see a list of supported models, please visit 🤗 [https://huggingface.co/cleanrl](https://huggingface.co/cleanrl).


???+ info "What happens under the hood?"
    
    The `cleanrl_utils.enjoy` is a simple wrapper to load the models from the hub and run them in an environment. A minimal version of the script can be found at [cleanrl_utils/evals/dqn_eval.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/evals/dqn_eval.py), which may give you a more fine-grained control and access to the model.
    
    <script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fcleanrl_utils%2Fevals%2Fdqn_eval.py&style=github&type=code&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script>

## Save model to Model Hub

In the supported algorithm variants, you can run the script with the `--save-model` flag, which saves a model to the `runs` folder, and the `--upload-model` flag, which upload the model to huggingface under your default entity (username). Optionally, you may override the default entity with `--hf-entity` flag.

```bash
uv run python cleanrl/dqn_jax.py --env-id CartPole-v1 --save-model --upload-model # --hf-entity cleanrl
uv run python cleanrl/dqn_atari_jax.py --env-id SeaquestNoFrameskip-v4  --save-model --upload-model # --hf-entity cleanrl
```

---

### Rl Algorithms C51 (`docs/rl-algorithms/c51.md`)

# Categorical DQN (C51)

## Overview

C51 introduces a distributional perspective for DQN: instead of learning a single value for an action, C51 learns to predict a distribution of values for the action. Empirically, C51 demonstrates impressive performance in ALE.


Original papers: 

* [A Distributional Perspective on Reinforcement Learning](https://arxiv.org/abs/1707.06887)

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`c51_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_ataripy) |  For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques. |
| :material-github: [`c51.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51.py), :material-file-document: [docs](/rl-algorithms/c51/#c51py) | For classic control tasks like `CartPole-v1`. |
| :material-github: [`c51_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari_jax.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_atari_jaxpy) |  For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques. |
| :material-github: [`c51_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_jax.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_jaxpy) | For classic control tasks like `CartPole-v1`. |

Below are our single-file implementations of C51:


## `c51_atari.py`

The [c51_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py) has the following features:

* For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

### Usage


=== "poetry"

    ```bash
    uv pip install ".[atari]"
    uv run python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
    uv run python cleanrl/c51_atari.py --env-id PongNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
    python cleanrl/c51_atari.py --env-id PongNoFrameskip-v4
    ```


### Explanation of the logged metrics

Running `python cleanrl/c51_atari.py` will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

* `charts/episodic_return`: episodic return of the game
* `charts/SPS`: number of steps per second
* `losses/loss`: the cross entropy loss between the $t$ step state value distribution and the projected $t+1$ step state value distribution
* `losses/q_values`: implemented as `(old_pmfs * q_network.atoms).sum(1)`, which is the sum of the probability of getting returns $x$ (`old_pmfs`) multiplied by $x$ (`q_network.atoms`), averaged over the sample obtained from the replay buffer; useful when gauging if under or over estimation happens


### Implementation details

[c51_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py) is based on (Bellemare et al., 2017)[^1] but presents a few implementation differences:

1. (Bellemare et al., 2017)[^1] injects stochaticity by doing "on each frame the environment rejects the agent’s selected action with probability $p = 0.25$", but `c51_atari.py` does not do this
1. `c51_atari.py` use a self-contained evaluation scheme: `c51_atari.py` reports the episodic returns obtained throughout training, whereas (Bellemare et al., 2017)[^1] is trained with `--end-e=0.01` but reported episodic returns using a separate evaluation process with `--end-e=0.001` (See "5.2. State-of-the-Art Results" on page 7).


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/c51.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/c51.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fc51.sh%23L8-L13&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>

Below are the average episodic returns for `c51_atari.py`. 


| Environment      | `c51_atari.py` 10M steps | (Bellemare et al., 2017, Figure 14)[^1] 50M steps | (Hessel et al., 2017, Figure 5)[^3] 
| ----------- | ----------- | ----------- | ---- |
| BreakoutNoFrameskip-v4      | 461.86 ± 69.65      | 748  | ~500 at 10M steps, ~600 at 50M steps
| PongNoFrameskip-v4  | 19.46 ± 0.70    |  20.9 |  ~20 10M steps, ~20 at 50M steps 
| BeamRiderNoFrameskip-v4   | 9592.90 ± 2270.15        | 14,074 | ~12000 10M steps, ~14000 at 50M steps 


Note that we save computational time by reducing timesteps from 50M to 10M, but our `c51_atari.py` scores the same or higher than (Mnih et al., 2015)[^1] in 10M steps.


Learning curves:

<div class="grid-container">
<img src="../c51/BeamRiderNoFrameskip-v4.png">

<img src="../c51/BreakoutNoFrameskip-v4.png">

<img src="../c51/PongNoFrameskip-v4.png">
</div>


Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-C51--VmlldzoxNzI0NzQ0" style="width:100%; height:500px" title="CleanRL C51 Tracked Experiments"></iframe>



## `c51.py`

The [c51.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51.py) has the following features:

* Works with the `Box` observation space of low-level features
* Works with the `Discrete` action space
* Works with envs like `CartPole-v1`


### Usage

=== "poetry"

    ```bash
    uv run python cleanrl/c51.py --env-id CartPole-v1
    ```

=== "pip"

    ```bash
    python cleanrl/c51.py --env-id CartPole-v1
    ```


### Explanation of the logged metrics

See [related docs](/rl-algorithms/c51/#explanation-of-the-logged-metrics) for `c51_atari.py`.

### Implementation details

The [c51.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51.py) shares the same implementation details as [`c51_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py) except the `c51.py` runs with different hyperparameters and neural network architecture. Specifically,

1. `c51.py` uses a simpler neural network as follows:
        ```python
        self.network = nn.Sequential(
            nn.Linear(np.array(env.single_observation_space.shape).prod(), 120),
            nn.ReLU(),
            nn.Linear(120, 84),
            nn.ReLU(),
            nn.Linear(84, env.single_action_space.n),
        )
        ```
2. `c51.py` runs with different hyperparameters:

    ```bash
    python c51.py --total-timesteps 500000 \
        --learning-rate 2.5e-4 \
        --buffer-size 10000 \
        --gamma 0.99 \
        --target-network-frequency 500 \
        --max-grad-norm 0.5 \
        --batch-size 128 \
        --start-e 1 \
        --end-e 0.05 \
        --exploration-fraction 0.5 \
        --learning-starts 10000 \
        --train-frequency 10
    ```


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/c51.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/c51.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fc51.sh%23L2-L6&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>


Below are the average episodic returns for `c51.py`. 


| Environment      | `c51.py`  | 
| ----------- | ----------- | 
| CartPole-v1      | 481.20 ± 20.53      |
| Acrobot-v1  | -87.70 ± 5.52     | 
| MountainCar-v0   | -166.38 ± 27.94        | 


Note that the C51 has no official benchmark on classic control environments, so we did not include a comparison. That said, our `c51.py` was able to achieve near perfect scores in `CartPole-v1` and `Acrobot-v1`; further, it can obtain successful runs in the sparse environment `MountainCar-v0`.


Learning curves:

<div class="grid-container">
<img src="../c51/CartPole-v1.png">

<img src="../c51/Acrobot-v1.png">

<img src="../c51/MountainCar-v0.png">
</div>


Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Classic-Control-CleanRL-s-C51--VmlldzoxODIwMTE4" style="width:100%; height:500px" title="CleanRL C51 Tracked Experiments"></iframe>


## `c51_atari_jax.py`

The [c51_atari_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari_jax.py) has the following features:

* Uses [Jax](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Optax](https://github.com/deepmind/optax) instead of `torch`.  [c51_atari_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari_jax.py) is roughly 25% faster than  [c51_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py)
* For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

### Usage


=== "poetry"

    ```bash
    uv pip install ".[atari, jax]"
    uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    uv run python cleanrl/c51_atari_jax.py --env-id BreakoutNoFrameskip-v4
    uv run python cleanrl/c51_atari_jax.py --env-id PongNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    pip install -r requirements/requirements-jax.txt
    pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    python cleanrl/c51_atari_jax.py --env-id BreakoutNoFrameskip-v4
    python cleanrl/c51_atari_jax.py --env-id PongNoFrameskip-v4
    ```


### Explanation of the logged metrics
See [related docs](/rl-algorithms/c51/#explanation-of-the-logged-metrics) for `c51_atari.py`.


### Implementation details
See [related docs](/rl-algorithms/c51/#implementation-details) for `c51_atari.py`.


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/c51.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/c51.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fc51.sh%23L23-L29&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>

Below are the average episodic returns for `c51_atari_jax.py`. 


| Environment             | `c51_atari_jax.py` 10M steps | `c51_atari.py` 10M steps | (Bellemare et al., 2017, Figure 14)[^1] 50M steps | (Hessel et al., 2017, Figure 5)[^3]   |
| ----------------------- | ---------------------------- | ------------------------ | ------------------------------------------------- | ------------------------------------- |
| BreakoutNoFrameskip-v4  | 448.56 ± 17.02               | 461.86 ± 69.65           | 748                                               | ~500 at 10M steps, ~600 at 50M steps  |
| PongNoFrameskip-v4      | 19.88 ± 0.31                 | 19.46 ± 0.70             | 20.9                                              | ~20 10M steps, ~20 at 50M steps       |
| BeamRiderNoFrameskip-v4 | 9504.91 ± 709.69             | 9592.90 ± 2270.15        | 14,074                                            | ~12000 10M steps, ~14000 at 50M steps |



Learning curves:
<div class="grid-container">
<img src="../c51/jax/BeamRiderNoFrameskip-v4.png">
<img src="../c51/jax/BeamRiderNoFrameskip-v4-time.png">

<img src="../c51/jax/BreakoutNoFrameskip-v4.png">
<img src="../c51/jax/BreakoutNoFrameskip-v4-time.png">

<img src="../c51/jax/PongNoFrameskip-v4.png">
<img src="../c51/jax/PongNoFrameskip-v4-time.png">
</div>

Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-C51-JAX--VmlldzozMjM4MTIy" style="width:100%; height:500px" title="CleanRL C51 Tracked Experiments"></iframe>


## `c51_jax.py`

The [c51_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_jax.py) has the following features:

* Uses [Jax](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Optax](https://github.com/deepmind/optax) instead of `torch`.  [c51_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_jax.py) is roughly 55% faster than  [c51.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51.py)
* Works with the `Box` observation space of low-level features
* Works with the `Discrete` action space
* Works with envs like `CartPole-v1`


### Usage


=== "poetry"

    ```bash
    uv pip install ".[jax]"
    uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    uv run python cleanrl/c51_jax.py --env-id CartPole-v1
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-jax.txt
    pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    python cleanrl/c51_jax.py --env-id CartPole-v1
    ```


### Explanation of the logged metrics

See [related docs](/rl-algorithms/c51/#explanation-of-the-logged-metrics) for `c51_atari.py`.

### Implementation details

See [related docs](/rl-algorithms/c51/#implementation-details_1) for `c51.py`.


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/c51.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/c51.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fc51.sh%23L15-L21&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>


Below are the average episodic returns for `c51_jax.py`. 


| Environment    | `c51_jax.py`    | `c51.py`        |
| -------------- | --------------- | --------------- |
| CartPole-v1    | 491.07 ± 9.70   | 481.20 ± 20.53  |
| Acrobot-v1     | -86.74 ± 2.19   | -87.70 ± 5.52   |
| MountainCar-v0 | -174.30 ± 36.35 | -166.38 ± 27.94 |


Learning curves:

<div class="grid-container">
<img src="../c51/jax/CartPole-v1.png">
<img src="../c51/jax/CartPole-v1-time.png">

<img src="../c51/jax/Acrobot-v1.png">
<img src="../c51/jax/Acrobot-v1-time.png">

<img src="../c51/jax/MountainCar-v0.png">
<img src="../c51/jax/MountainCar-v0-time.png">
</div>


Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Classic-Control-CleanRL-s-C51-JAX--VmlldzozMjM4MTM4" style="width:100%; height:500px" title="CleanRL C51 Tracked Experiments"></iframe>


[^1]:Bellemare, M.G., Dabney, W., & Munos, R. (2017). A Distributional Perspective on Reinforcement Learning. ICML.
[^2]:\[Proposal\] Formal API handling of truncation vs termination. https://github.com/openai/gym/issues/2510
[^3]: Hessel, M., Modayil, J., Hasselt, H.V., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.G., & Silver, D. (2018). Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI.

---

### Rl Algorithms Ddpg (`docs/rl-algorithms/ddpg.md`)

# Deep Deterministic Policy Gradient (DDPG)


## Overview

DDPG is a popular DRL algorithm for continuous control. It extends DQN to work with the continuous action space by introducing a deterministic actor that directly outputs continuous actions. DDPG also combines techniques from DQN, such as the replay buffer and target network.


Original paper: 

* [Continuous control with deep reinforcement learning](https://arxiv.org/abs/1509.02971)

Reference resources:

* :material-github: [sfujim/TD3](https://github.com/sfujim/TD3)
* [Deep Deterministic Policy Gradient | Spinning Up in Deep RL](https://spinningup.openai.com/en/latest/algorithms/ddpg.html)
* :material-github: [ikostrikov/jaxrl](https://github.com/ikostrikov/jaxrl) (helpful reference when implemented [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py))

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py), :material-file-document: [docs](/rl-algorithms/ddpg/#ddpg_continuous_actionpy) | For continuous action space |
| :material-github: [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py), :material-file-document: [docs](/rl-algorithms/ddpg/#ddpg_continuous_action_jaxpy) | For continuous action space |

Below is our single-file implementation of DDPG:

## `ddpg_continuous_action.py`

The [ddpg_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) has the following features:

* For continuous action space
* Works with the `Box` observation space of low-level features
* Works with the `Box` (continuous) action space

### Usage

=== "poetry"

    ```bash
    uv pip install .
    uv run python cleanrl/ddpg_continuous_action.py --help
    uv pip install ".[mujoco]"
    uv run python cleanrl/ddpg_continuous_action.py --env-id Hopper-v4
    ```

=== "pip"

    ```bash
    python cleanrl/ddpg_continuous_action.py --help
    pip install -r requirements/requirements-mujoco.txt
    python cleanrl/ddpg_continuous_actions.py --env-id Hopper-v4
    ```

### Explanation of the logged metrics

Running `python cleanrl/ddpg_continuous_action.py` will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

* `charts/episodic_return`: episodic return of the game
* `charts/SPS`: number of steps per second
* `losses/qf1_loss`: the mean squared error (MSE) between the Q values at timestep $t$ and the Bellman update target estimated using the reward $r_t$ and the Q values at timestep $t+1$, thus minimizing the *one-step* temporal difference. Formally, it can be expressed by the equation below.
$$
    J(\theta^{Q}) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \big[ (Q(s, a) - y)^2 \big],
$$
with the Bellman update target $y = r + \gamma \, Q^{'}(s', a')$, where $a' \sim \mu^{'}(s')$, and the replay buffer $\mathcal{D}$.

* `losses/actor_loss`: implemented as `-qf1(data.observations, actor(data.observations)).mean()`; it is the *negative* average Q values calculated based on the 1) observations and the 2) actions computed by the actor based on these observations. By minimizing `actor_loss`, the optimizer updates the actors parameter using the following gradient (Lillicrap et al., 2016, Algorithm 1)[^1]:

$$ \nabla_{\theta^{\mu}} J \approx  \frac{1}{N}\sum_i\left.\left.\nabla_{a} Q\left(s, a \mid \theta^{Q}\right)\right|_{s=s_{i}, a=\mu\left(s_{i}\right)} \nabla_{\theta^{\mu}} \mu\left(s \mid \theta^{\mu}\right)\right|_{s_{i}} $$

* `losses/qf1_values`: implemented as `qf1(data.observations, data.actions).view(-1)`, it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimation happens.


### Implementation details

Our [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) is based on the [`OurDDPG.py`](https://github.com/sfujim/TD3/blob/master/OurDDPG.py) from :material-github: [sfujim/TD3](https://github.com/sfujim/TD3), which presents the the following implementation difference from (Lillicrap et al., 2016)[^1]:

1. [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) uses a gaussian exploration noise $\mathcal{N}(0, 0.1)$, while (Lillicrap et al., 2016)[^1] uses Ornstein-Uhlenbeck process with $\theta=0.15$ and $\sigma=0.2$.

1. [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) runs the experiments using the `openai/gym` MuJoCo environments, while (Lillicrap et al., 2016)[^1] uses their proprietary MuJoCo environments.

1. [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) uses the following architecture:
    ```python
    class QNetwork(nn.Module):
        def __init__(self, env):
            super().__init__()
            self.fc1 = nn.Linear(np.array(env.single_observation_space.shape).prod() + np.prod(env.single_action_space.shape), 256)
            self.fc2 = nn.Linear(256, 256)
            self.fc3 = nn.Linear(256, 1)

        def forward(self, x, a):
            x = torch.cat([x, a], 1)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x


    class Actor(nn.Module):
        def __init__(self, env):
            super().__init__()
            self.fc1 = nn.Linear(np.array(env.single_observation_space.shape).prod(), 256)
            self.fc2 = nn.Linear(256, 256)
            self.fc_mu = nn.Linear(256, np.prod(env.single_action_space.shape))
            # action rescaling
            self.register_buffer(
                "action_scale", torch.tensor((env.action_space.high - env.action_space.low) / 2.0, dtype=torch.float32)
            )
            self.register_buffer(
                "action_bias", torch.tensor((env.action_space.high + env.action_space.low) / 2.0, dtype=torch.float32)
            )

        def forward(self, x):
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = torch.tanh(self.fc_mu(x))
            return x * self.action_scale + self.action_bias
    ```
    while (Lillicrap et al., 2016, see Appendix 7 EXPERIMENT DETAILS)[^1] uses the following architecture (difference highlighted):

    ```python hl_lines="4-6 9-11 19-21"
    class QNetwork(nn.Module):
        def __init__(self, env):
            super(QNetwork, self).__init__()
            self.fc1 = nn.Linear(p.array(env.single_observation_space.shape).prod() + np.prod(env.single_action_space.shape), 400)
            self.fc2 = nn.Linear(400 + np.prod(env.single_action_space.shape), 300)
            self.fc3 = nn.Linear(300, 1)

        def forward(self, x, a):
            x = F.relu(self.fc1(x))
            x = torch.cat([x, a], 1)
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x


    class Actor(nn.Module):
        def __init__(self, env):
            super(Actor, self).__init__()
            self.fc1 = nn.Linear(np.array(env.single_observation_space.shape).prod(), 400)
            self.fc2 = nn.Linear(400, 300)
            self.fc_mu = nn.Linear(300, np.prod(env.single_action_space.shape))
            # action rescaling
            self.register_buffer(
                "action_scale", torch.tensor((env.action_space.high - env.action_space.low) / 2.0, dtype=torch.float32)
            )
            self.register_buffer(
                "action_bias", torch.tensor((env.action_space.high + env.action_space.low) / 2.0, dtype=torch.float32)
            )

        def forward(self, x):
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = torch.tanh(self.fc_mu(x))
            return x * self.action_scale + self.action_bias
    ```

1. [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) uses the following learning rates:

    ```python
    q_optimizer = optim.Adam(list(qf1.parameters()), lr=3e-4)
    actor_optimizer = optim.Adam(list(actor.parameters()), lr=3e-4)
    ```
    while (Lillicrap et al., 2016, see Appendix 7 EXPERIMENT DETAILS)[^1] uses the following learning rates:

    ```python
    q_optimizer = optim.Adam(list(qf1.parameters()), lr=1e-4)
    actor_optimizer = optim.Adam(list(actor.parameters()), lr=1e-3)
    ```

1. [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) uses `--batch-size=256 --tau=0.005`, while (Lillicrap et al., 2016, see Appendix 7 EXPERIMENT DETAILS)[^1] uses `--batch-size=64 --tau=0.001`

1. [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py)  also adds support for handling continuous environments where the lower and higher bounds of the action space are not $[-1,1]$, or are asymmetric.
The case where the bounds are not $[-1,1]$ is handled in [`DDPG.py`](https://github.com/sfujim/TD3/blob/385b33ac7de4767bab17eb02ade4a268d3e4e24f/DDPG.py#L15) (Fujimoto et al., 2018)[^2] as follows:
```python
class Actor(nn.Module):

    ...

	def forward(self, state):
		a = F.relu(self.l1(state))
		a = F.relu(self.l2(a))
		return self.max_action * torch.tanh(self.l3(a)) # Scale from [-1,1] to [-action_high, action_high]
```
 On the other hand, in [`CleanRL's ddpg_continuous_action.py`](https://github.com/dosssman/cleanrl/blob/10b606e7bd9bd1b06e455e8ef542df2b7699a20c/cleanrl/ddpg_continuous_action.py#L98), the mean and the scale of the the action space are computed as `action_bias` and `action_scale` respectively.
 Those scalars are in turn used to scale the output of a `tanh` activation function in the actor to the original action space range:
```python
class Actor(nn.Module):
    def __init__(self, env):
        ...
        # action rescaling
        self.register_buffer(
            "action_scale", torch.tensor((env.action_space.high - env.action_space.low) / 2.0, dtype=torch.float32)
        )
        self.register_buffer(
            "action_bias", torch.tensor((env.action_space.high + env.action_space.low) / 2.0, dtype=torch.float32)
        )

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = torch.tanh(self.fc_mu(x))
        return x * self.action_scale + self.action_bias # Scale from [-1,1] to [-action_low, action_high]
```

Additionally, when drawing exploration noise that is added to the actions produced by the actor, [`CleanRL's ddpg_continuous_action.py`](https://github.com/dosssman/cleanrl/blob/10b606e7bd9bd1b06e455e8ef542df2b7699a20c/cleanrl/ddpg_continuous_action.py#L175) centers the distribution the sampled from at `action_bias`, and the scale of the distribution is set to `action_scale * exploration_noise`.

???+ info

    Note that `Humanoid-v2`, `InvertedPendulum-v2`, `Pusher-v2` have action space bounds that are not the standard `[-1, 1]`. See below.
    
    ```
    Ant-v2 Observation space: Box(-inf, inf, (111,), float64) Action space: Box(-1.0, 1.0, (8,), float32)
    HalfCheetah-v2 Observation space: Box(-inf, inf, (17,), float64) Action space: Box(-1.0, 1.0, (6,), float32)
    Hopper-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (3,), float32)
    Humanoid-v2 Observation space: Box(-inf, inf, (376,), float64) Action space: Box(-0.4, 0.4, (17,), float32)
    InvertedDoublePendulum-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (1,), float32)
    InvertedPendulum-v2 Observation space: Box(-inf, inf, (4,), float64) Action space: Box(-3.0, 3.0, (1,), float32)
    Pusher-v2 Observation space: Box(-inf, inf, (23,), float64) Action space: Box(-2.0, 2.0, (7,), float32)
    Reacher-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (2,), float32)
    Swimmer-v2 Observation space: Box(-inf, inf, (8,), float64) Action space: Box(-1.0, 1.0, (2,), float32)
    Walker2d-v2 Observation space: Box(-inf, inf, (17,), float64) Action space: Box(-1.0, 1.0, (6,), float32)
    ```


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ddpg.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ddpg.sh). Specifically, execute the following command:


``` title="benchmark/ddpg.sh" linenums="1"
--8<-- "benchmark/ddpg.sh::7"
```


Below are the average episodic returns for [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) (3 random seeds). To ensure the quality of the implementation, we compared the results against (Fujimoto et al., 2018)[^2].

| Environment      | [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) | [`OurDDPG.py`](https://github.com/sfujim/TD3/blob/master/OurDDPG.py) (Fujimoto et al., 2018, Table 1)[^2]  | [`DDPG.py`](https://github.com/sfujim/TD3/blob/master/DDPG.py) using settings from (Lillicrap et al., 2016)[^1] in (Fujimoto et al., 2018, Table 1)[^2]    |
| ----------- | ----------- | ----------- | ----------- |
| HalfCheetah-v4       | 10374.07 ± 157.37      |8577.29  | 3305.60|
| Walker2d-v4    | 1240.16 ± 390.10     |  3098.11 | 1843.85 |
| Hopper-v4    | 1576.78 ± 818.98         |  1860.02 | 2020.46 |
| InvertedPendulum-v4  |    642.68 ± 69.56    | 1000.00 ± 0.00  | 
| Humanoid-v4  |  1699.56 ± 694.22      |  not available | 
| Pusher-v4  |  -77.30 ± 38.78      |  not available | 



???+ info

    Note that [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) uses gym MuJoCo v4 environments while [`OurDDPG.py`](https://github.com/sfujim/TD3/blob/master/OurDDPG.py) (Fujimoto et al., 2018)[^2] uses the gym MuJoCo v1 environments.

    Also note the performance of our `ddpg_continuous_action.py` seems to be worse than the reference implementation on Walker2d and Hopper. This is likely due to :material-github: [openai/gym#938](https://github.com/openai/baselines/issues/938). We would have a hard time reproducing gym MuJoCo v1 environments because they have been long deprecated.

    One other thing could cause the performance difference: the original code reported the average episodic return using determinisitc evaluation (i.e., without exploration noise), see [`sfujim/TD3/main.py#L15-L32`](https://github.com/sfujim/TD3/blob/385b33ac7de4767bab17eb02ade4a268d3e4e24f/main.py#L15-L32), whereas we reported the episodic return during training and the policy gets updated between environments steps.

Learning curves:

``` title="benchmark/ddpg_plot.sh" linenums="1"
--8<-- "benchmark/ddpg_plot.sh::9"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ddpg.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ddpg-time.png">

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/MuJoCo-CleanRL-s-DDPG--VmlldzoxNjkyMjc1" style="width:100%; height:500px" title="MuJoCo: CleanRL's DDPG"></iframe>



## `ddpg_continuous_action_jax.py`

The [ddpg_continuous_action_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) has the following features:

* Uses [Jax](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Optax](https://github.com/deepmind/optax) instead of `torch`.  [ddpg_continuous_action_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) is roughly 2.5-4x faster than  [ddpg_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py)
* For continuous action space
* Works with the `Box` observation space of low-level features
* Works with the `Box` (continuous) action space

### Usage

=== "poetry"

    ```bash
    uv pip install ".[mujoco, jax]"
    uv run python cleanrl/ddpg_continuous_action_jax.py --help
    uv run python cleanrl/ddpg_continuous_action_jax.py --env-id Hopper-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-mujoco.txt
    pip install -r requirements/requirements-jax.txt
    python cleanrl/ddpg_continuous_action_jax.py --help
    python cleanrl/ddpg_continuous_action_jax.py --env-id Hopper-v4
    ```


???+ warning

    Note that JAX does not work in Windows :fontawesome-brands-windows:. The official [docs](https://github.com/google/jax#installation) recommends using Windows Subsystem for Linux (WSL) to install JAX.

### Explanation of the logged metrics

See [related docs](/rl-algorithms/ddpg/#explanation-of-the-logged-metrics) for `ddpg_continuous_action.py`.


### Implementation details

See [related docs](/rl-algorithms/ddpg/#implementation-details) for `ddpg_continuous_action.py`.


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ddpg.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ddpg.sh). Specifically, execute the following command:


``` title="benchmark/ddpg.sh" linenums="1"
--8<-- "benchmark/ddpg.sh:12:19"
```

Below are the average episodic returns for [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) (3 random seeds).


{!benchmark/ddpg.md!}

Learning curves:


``` title="benchmark/ddpg_plot.sh" linenums="1"
--8<-- "benchmark/ddpg_plot.sh:11:20"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ddpg_jax.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ddpg_jax-time.png">


???+ info

    These are some previous experiments with TPUs. Note the results are very similar to the ones above, but the runtime can be different due to different hardware used.


    Note that the experiments were conducted on different hardwares, so your mileage might vary. This inconsistency is because 1) re-running expeirments on the same hardware is computationally expensive and 2) requiring the same hardware is not inclusive nor feasible to other contributors who might have different hardwares.

    That said, we roughly expect to see a 2-4x speed improvement from using [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) under the same hardware. And if you disable the `--capture_video` overhead, the speed improvement will be even higher.


    | Environment      | [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) (RTX 3060 TI) | [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) (VM w/ TPU) | [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) (RTX 3060 TI) | [`OurDDPG.py`](https://github.com/sfujim/TD3/blob/master/OurDDPG.py) (Fujimoto et al., 2018, Table 1)[^2]    |
    | ----------- | ----------- | ----------- | ----------- | ----------- |
    | HalfCheetah |  9592.25 ± 135.10 | 9125.06 ± 1477.58  | 10210.57 ± 196.22      |8577.29  |
    | Walker2d |  1083.15 ± 567.65  | 1303.82 ± 448.41 | 1661.14 ± 250.01     |  3098.11 | 
    | Hopper |  1275.28 ± 209.60  | 1145.05 ± 41.95 | 1007.44 ± 148.29         |  1860.02 |    

    Learning curves:

    <div class="grid-container">
    <img loading="lazy" src="../ddpg-jax/HalfCheetah-v2.png">
    <img loading="lazy" src="../ddpg-jax/HalfCheetah-v2-time.png">

    <img loading="lazy" src="../ddpg-jax/Walker2d-v2.png">
    <img loading="lazy" src="../ddpg-jax/Walker2d-v2-time.png">

    <img loading="lazy" src="../ddpg-jax/Hopper-v2.png">
    <img loading="lazy" src="../ddpg-jax/Hopper-v2-time.png">
    </div> 

    Tracked experiments and game play videos:

    <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/MuJoCo-CleanRL-s-DDPG-JAX--VmlldzoyMjQxMjE2" style="width:100%; height:500px" title="MuJoCo: CleanRL's DDPG + JAX"></iframe>


[^1]:Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N.M., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2016). Continuous control with deep reinforcement learning. CoRR, abs/1509.02971. https://arxiv.org/abs/1509.02971

[^2]:Fujimoto, S., Hoof, H.V., & Meger, D. (2018). Addressing Function Approximation Error in Actor-Critic Methods. ArXiv, abs/1802.09477. https://arxiv.org/abs/1802.09477

---

### Rl Algorithms Dqn (`docs/rl-algorithms/dqn.md`)

# Deep Q-Learning (DQN)

## Overview

As an extension of the Q-learning, DQN's main technical contribution is the use of replay buffer and target network, both of which would help improve the stability of the algorithm.


Original papers: 

* [Human-level control through deep reinforcement learning
](https://www.nature.com/articles/nature14236)

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`dqn_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_ataripy) |  For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques. |
| :material-github: [`dqn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqnpy) | For classic control tasks like `CartPole-v1`. |
| :material-github: [`dqn_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_atari_jaxpy) |  For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques. |
| :material-github: [`dqn_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_jax.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_jaxpy) | For classic control tasks like `CartPole-v1`. |


Below are our single-file implementations of DQN:


## `dqn_atari.py`

The [dqn_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py) has the following features:

* For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

### Usage

```bash
uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4
```

=== "poetry"

    ```bash
    uv pip install ".[atari]"
    uv run python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
    uv run python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
    python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4
    ```


### Explanation of the logged metrics

Running `python cleanrl/dqn_atari.py` will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

* `charts/episodic_return`: episodic return of the game
* `charts/SPS`: number of steps per second
* `losses/td_loss`: the mean squared error (MSE) between the Q values at timestep $t$ and the Bellman update target estimated using the reward $r_t$ and the Q values at timestep $t+1$, thus minimizing the *one-step* temporal difference. Formally, it can be expressed by the equation below.
$$
    J(\theta^{Q}) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \big[ (Q(s, a) - y)^2 \big],
$$
with the Bellman update target is $y = r + \gamma \, Q^{'}(s', a')$ and the replay buffer is $\mathcal{D}$.
* `losses/q_values`: implemented as `qf1(data.observations, data.actions).view(-1)`, it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimation happens.


### Implementation details

[dqn_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py) is based on (Mnih et al., 2015)[^1] but presents a few implementation differences:

1. `dqn_atari.py` use slightly different hyperparameters. Specifically,
    - `dqn_atari.py` uses the more popular Adam Optimizer with the `--learning-rate=1e-4` as follows:
        ```python
        optim.Adam(q_network.parameters(), lr=1e-4)
        ```
       whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses the RMSProp optimizer with `--learning-rate=2.5e-4`, gradient momentum `0.95`, squared gradient momentum `0.95`, and min squared gradient `0.01` as follows:
        ```python
        optim.RMSprop(
            q_network.parameters(),
            lr=2.5e-4,
            momentum=0.95,
            # ... PyTorch's RMSprop does not directly support
            # squared gradient momentum and min squared gradient
            # so we are not sure what to put here.
        )
        ``` 
    - `dqn_atari.py` uses `--learning-starts=80000` whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses `--learning-starts=50000`.
    - `dqn_atari.py` uses `--target-network-frequency=1000` whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses `--target-network-frequency=10000`.
    - `dqn_atari.py` uses `--total-timesteps=10000000` (i.e., 10M timesteps = 40M frames because of frame-skipping) whereas (Mnih et al., 2015)[^1] uses `--total-timesteps=50000000` (i.e., 50M timesteps = 200M frames) (See "Training details" under "METHODS" on page 6 and the related source code [run_gpu#L32](https://github.com/deepmind/dqn/blob/9d9b1d13a2b491d6ebd4d046740c511c662bbe0f/run_gpu#L32), [dqn/train_agent.lua#L81-L82](https://github.com/deepmind/dqn/blob/9d9b1d13a2b491d6ebd4d046740c511c662bbe0f/dqn/train_agent.lua#L81-L82), and [dqn/train_agent.lua#L165-L169](https://github.com/deepmind/dqn/blob/9d9b1d13a2b491d6ebd4d046740c511c662bbe0f/dqn/train_agent.lua#L165-L169)).
    - `dqn_atari.py` uses `--end-e=0.01` (the final exploration epsilon) whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses `--end-e=0.1`.
    - `dqn_atari.py` uses `--exploration-fraction=0.1` whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses `--exploration-fraction=0.02` (all corresponds to 250000 steps or 1M frames being the frame that epsilon is annealed to `--end-e=0.1` ).
    - `dqn_atari.py` handles truncation and termination properly like (Mnih et al., 2015)[^1] by using SB3's replay buffer's `handle_timeout_termination=True`.
2. `dqn_atari.py` use a self-contained evaluation scheme: `dqn_atari.py` reports the episodic returns obtained throughout training, whereas (Mnih et al., 2015)[^1] is trained with `--end-e=0.1` but reported episodic returns using a separate evaluation process with `--end-e=0.01` (See "Evaluation procedure" under "METHODS" on page 6).
3. `dqn_atari.py` implements target network updates as Polyak updates. Compared to the original implementation in (Mnih et al., 2015)[^1], this version allows soft updates of the target network weights with `--tau` (update coefficient) values of less than 1 (i.e. `--tau=0.9`). Note that by default `--tau=1.0` is used to be consistent with (Mnih et al., 2015)[^1].
4. `dqn_atari.py` uses the standard MSE loss function, whereas (Mnih et al., 2015)[^1] "...found it helpful to clip the error term from the update $r + \gamma \max_{a'}Q(s', a'; \theta^{-}_{i}) - Q(s, a; \theta_{i})$ to be between -1 and 1" (See "Training algorithm for deep Q-networks" under "METHODS" on page 7).

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/dqn.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/dqn.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fdqn.sh%23L8-L13&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>

Below are the average episodic returns for `dqn_atari.py`. 


| Environment      | `dqn_atari.py` 10M steps | (Mnih et al., 2015)[^1] 50M steps | (Hessel et al., 2017, Figure 5)[^3] 
| ----------- | ----------- | ----------- | ---- |
| BreakoutNoFrameskip-v4      | 366.928 ± 39.89      |401.2 ± 26.9  | ~230 at 10M steps, ~300 at 50M steps
| PongNoFrameskip-v4  | 20.25 ± 0.41     |  18.9 ± 1.3 |  ~20 10M steps, ~20 at 50M steps 
| BeamRiderNoFrameskip-v4   | 6673.24 ± 1434.37        | 6846 ± 1619 | ~6000 10M steps, ~7000 at 50M steps 


Note that we save computational time by reducing timesteps from 50M to 10M, but our `dqn_atari.py` scores the same or higher than (Mnih et al., 2015)[^1] in 10M steps.


Learning curves:

<div class="grid-container">
<img src="../dqn/BeamRiderNoFrameskip-v4.png">

<img src="../dqn/BreakoutNoFrameskip-v4.png">

<img src="../dqn/PongNoFrameskip-v4.png">
</div>


Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-DQN--VmlldzoxNjk3NjYx" style="width:100%; height:500px" title="CleanRL DQN + Atari Tracked Experiments"></iframe>


## `dqn.py`

The [dqn.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn.py) has the following features:

* Works with the `Box` observation space of low-level features
* Works with the `Discrete` action space
* Works with envs like `CartPole-v1`


### Usage



=== "poetry"

    ```bash
    uv run python cleanrl/dqn.py --env-id CartPole-v1
    ```

=== "pip"

    ```bash
    python cleanrl/dqn.py --env-id CartPole-v1
    ```


### Explanation of the logged metrics

See [related docs](/rl-algorithms/dqn/#explanation-of-the-logged-metrics) for `dqn_atari.py`.

### Implementation details

The [dqn.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn.py) shares the same implementation details as [`dqn_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py) except the `dqn.py` runs with different hyperparameters and neural network architecture. Specifically,

1. `dqn.py` uses a simpler neural network as follows:
        ```python
        self.network = nn.Sequential(
            nn.Linear(np.array(env.single_observation_space.shape).prod(), 120),
            nn.ReLU(),
            nn.Linear(120, 84),
            nn.ReLU(),
            nn.Linear(84, env.single_action_space.n),
        )
        ```
2. `dqn.py` runs with different hyperparameters:

    ```bash
    python dqn.py --total-timesteps 500000 \
        --learning-rate 2.5e-4 \
        --buffer-size 10000 \
        --gamma 0.99 \
        --target-network-frequency 500 \
        --max-grad-norm 0.5 \
        --batch-size 128 \
        --start-e 1 \
        --end-e 0.05 \
        --exploration-fraction 0.5 \
        --learning-starts 10000 \
        --train-frequency 10
    ```


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/dqn.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/dqn.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fdqn.sh%23L2-L6&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>

Below are the average episodic returns for `dqn.py`. 


| Environment      | `dqn.py`  | 
| ----------- | ----------- | 
| CartPole-v1      | 488.69 ± 16.11      |
| Acrobot-v1  | -91.54 ± 7.20     | 
| MountainCar-v0   | -194.95 ± 8.48        | 


Note that the DQN has no official benchmark on classic control environments, so we did not include a comparison. That said, our `dqn.py` was able to achieve near perfect scores in `CartPole-v1` and `Acrobot-v1`; further, it can obtain successful runs in the sparse environment `MountainCar-v0`.


Learning curves:

<div class="grid-container">
<img src="../dqn/CartPole-v1.png">

<img src="../dqn/Acrobot-v1.png">

<img src="../dqn/MountainCar-v0.png">
</div>

Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Classic-Control-CleanRL-s-DQN--VmlldzoxODE4Mjg1" style="width:100%; height:500px" title="CleanRL DQN Tracked Experiments"></iframe>



## `dqn_atari_jax.py`


The [dqn_atari_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py) has the following features:

* Uses [Jax](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Optax](https://github.com/deepmind/optax) instead of `torch`.  [dqn_atari_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py) is roughly 25%-50% faster than  [dqn_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py)
* For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

### Usage


=== "poetry"

    ```bash
    uv pip install ".[atari, jax]"
    uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    uv run python cleanrl/dqn_atari_jax.py --env-id BreakoutNoFrameskip-v4
    uv run python cleanrl/dqn_atari_jax.py --env-id PongNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    pip install -r requirements/requirements-jax.txt
    pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    python cleanrl/dqn_atari_jax.py --env-id BreakoutNoFrameskip-v4
    python cleanrl/dqn_atari_jax.py --env-id PongNoFrameskip-v4
    ```


???+ warning

    Note that JAX does not work in Windows :fontawesome-brands-windows:. The official [docs](https://github.com/google/jax#installation) recommends using Windows Subsystem for Linux (WSL) to install JAX.

### Explanation of the logged metrics

See [related docs](/rl-algorithms/dqn/#explanation-of-the-logged-metrics) for `dqn_atari.py`.

### Implementation details

See [related docs](/rl-algorithms/dqn/#implementation-details) for `dqn_atari.py`.

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/dqn.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/dqn.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fdqn.sh%23L23-L29&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>


Below are the average episodic returns for [`dqn_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py) (3 random seeds).


| Environment             | `dqn_atari_jax.py` 10M steps | `dqn_atari.py` 10M steps | (Mnih et al., 2015)[^1] 50M steps | (Hessel et al., 2017, Figure 5)[^3]  |
| ----------------------- | ---------------------------- | ------------------------ | --------------------------------- | ------------------------------------ |
| BreakoutNoFrameskip-v4  | 377.82 ± 34.91               | 366.928 ± 39.89          | 401.2 ± 26.9                      | ~230 at 10M steps, ~300 at 50M steps |
| PongNoFrameskip-v4      | 20.43 ± 0.34                 | 20.25 ± 0.41             | 18.9 ± 1.3                        | ~20 10M steps, ~20 at 50M steps      |
| BeamRiderNoFrameskip-v4 | 5938.13 ± 955.84             | 6673.24 ± 1434.37        | 6846 ± 1619                       | ~6000 10M steps, ~7000 at 50M steps  |


???+ info
    
    We observe a speedup of `~25%` in ['dqn_atari_jax.py'](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py) compared to ['dqn_atari.py'](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py). This could be because the training loop is tightly integrated with the experience collection loop. We run a training loop every `4` environment steps by default. So more time is utilised in collecting experience than training the network. We observe much more speed-ups in algorithms which run a training step for each environment step. E.g., [DDPG](/rl-algorithms/ddpg/#experiment-results_1)

Learning curves:

<div class="grid-container">
<img src="../dqn/jax/BeamRiderNoFrameskip-v4.png">
<img src="../dqn/jax/BeamRiderNoFrameskip-v4-time.png">

<img src="../dqn/jax/BreakoutNoFrameskip-v4.png">
<img src="../dqn/jax/BreakoutNoFrameskip-v4-time.png">

<img src="../dqn/jax/PongNoFrameskip-v4.png">
<img src="../dqn/jax/PongNoFrameskip-v4-time.png">
</div>

Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-DQN-JAX--VmlldzoyMzM3MDg1" style="width:100%; height:500px" title="CleanRL DQN + JAX + Atari Tracked Experiments"></iframe>



## `dqn_jax.py`
* Uses [Jax](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Optax](https://github.com/deepmind/optax) instead of `torch`.  [dqn_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_jax.py) is roughly 50% faster than  [dqn.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn.py)
* Works with the `Box` observation space of low-level features
* Works with the `Discrete` action space
* Works with envs like `CartPole-v1`

### Usage

```bash
python cleanrl/dqn_jax.py --env-id CartPole-v1
```

=== "poetry"

    ```bash
    uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    uv run python cleanrl/dqn_jax.py --env-id CartPole-v1
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-jax.txt
    pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    python cleanrl/dqn_jax.py --env-id CartPole-v1
    ```


### Explanation of the logged metrics

See [related docs](/rl-algorithms/dqn/#explanation-of-the-logged-metrics) for `dqn_atari.py`.

### Implementation details

See [related docs](/rl-algorithms/dqn/#implementation-details_1) for `dqn.py`.

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/dqn.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/dqn.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fdqn.sh%23L15-L21&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>

Below are the average episodic returns for [`dqn_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_jax.py) (3 random seeds).



| Environment    | `dqn_jax.py`   | `dqn.py`  | 
| ----------- | ----------- | ----------- | 
| CartPole-v1    |  498.38 ± 2.29 | 488.69 ± 16.11      |
| Acrobot-v1 | -88.89 ± 1.56 | -91.54 ± 7.20     | 
| MountainCar-v0 | -188.90 ± 11.78  | -194.95 ± 8.48        | 



<div class="grid-container">
<img src="../dqn/jax/CartPole-v1.png">
<img src="../dqn/jax/CartPole-v1-time.png">

<img src="../dqn/jax/Acrobot-v1.png">
<img src="../dqn/jax/Acrobot-v1-time.png">

<img src="../dqn/jax/MountainCar-v0.png">
<img src="../dqn/jax/MountainCar-v0-time.png">
</div>


Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Classic-Control-CleanRL-s-DQN-JAX--VmlldzozMjM5Mjgx" style="width:100%; height:500px" title="CleanRL DQN + JAX Tracked Experiments"></iframe>




[^1]:Mnih, V., Kavukcuoglu, K., Silver, D. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236
[^2]:\[Proposal\] Formal API handling of truncation vs termination. https://github.com/openai/gym/issues/2510
[^3]: Hessel, M., Modayil, J., Hasselt, H.V., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.G., & Silver, D. (2018). Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI.

---

### Rl Algorithms Ppg (`docs/rl-algorithms/ppg.md`)

# Phasic Policy Gradient (PPG)

## Overview

PPG is a DRL algorithm that separates policy and value function training by introducing an auxiliary phase. The training proceeds by running PPO during the policy phase, saving all the experience in a replay buffer. Then the replay buffer is used to train the value function. This makes the algorithm considerably slower than PPO, but improves sample efficiency on Procgen benchmark.

Original paper: 

* [Phasic Policy Gradient](https://arxiv.org/abs/2009.04416)

Reference resources:

* [Code for the paper "Phasic Policy Gradient"](https://github.com/openai/phasic-policy-gradient) - by original authors from OpenAI

The original code has multiple code level details that are not mentioned in the paper. We found these changes to be important for reproducing the results claimed by the paper.

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`ppg_procgen.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppg_procgen.py), :material-file-document: [docs](/rl-algorithms/ppg/#ppg_procgenpy) | For classic control tasks like `CartPole-v1`. |

Below are our single-file implementations of PPG:

## `ppg_procgen.py`

`ppg_procgen.py` works with the Procgen benchmark, which uses 64x64 RGB image observations, and discrete actions

### Usage

=== "uv"

    ```bash
    uv pip install ".[procgen]"
    uv run python cleanrl/ppg_procgen.py --help
    uv run python cleanrl/ppg_procgen.py --env-id starpilot
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-procgen.txt
    python cleanrl/ppg_procgen.py --help
    python cleanrl/ppg_procgen.py --env-id starpilot
    ```

### Explanation of the logged metrics

Running `python cleanrl/ppg_procgen.py` will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

Same as PPO:

* `charts/episodic_return`: episodic return of the game
* `charts/episodic_length`: episodic length of the game
* `charts/SPS`: number of steps per second (this is initially high but drops off after the auxiliary phase)
* `charts/learning_rate`: the current learning rate (annealing is not done by default)
* `losses/value_loss`: the mean value loss across all data points
* `losses/policy_loss`: the mean policy loss across all data points
* `losses/entropy`: the mean entropy value across all data points
* `losses/old_approx_kl`: the approximate Kullback–Leibler divergence, measured by `(-logratio).mean()`, which corresponds to the k1 estimator in John Schulman’s blog post on [approximating KL](http://joschu.net/blog/kl-approx.html)
* `losses/approx_kl`: better alternative to `old_approx_kl` measured by `(logratio.exp() - 1) - logratio`, which corresponds to the k3 estimator in [approximating KL](http://joschu.net/blog/kl-approx.html)
* `losses/clipfrac`: the fraction of the training data that triggered the clipped objective
* `losses/explained_variance`: the explained variance for the value function

PPG specific:

* `losses/aux/kl_loss`: the mean value of the KL divergence when distilling the latest policy during the auxiliary phase.
* `losses/aux/aux_value_loss`: the mean value loss on the auxiliary value head
* `losses/aux/real_value_loss`: the mean value loss on the detached value head used to calculate the GAE returns during policy phase

### Implementation details

`ppg_procgen.py` includes the <TODO> level implementation details that are different from PPO:

1. Full rollout sampling during auxiliary phase - (:material-github: [phasic_policy_gradient/ppg.py#L173](https://github.com/openai/phasic-policy-gradient/blob/c789b00be58aa704f7223b6fc8cd28a5aaa2e101/phasic_policy_gradient/ppg.py#L173)) - Instead of randomly sampling observations over the entire auxiliary buffer, PPG samples full rullouts from the buffer (Sets of 256 steps). This full rollout sampling is only done during the auxiliary phase. Note that the rollouts will still be at random starting points because PPO truncates the rollouts per env. This change gives a decent performance boost.

1. Batch level advantage normalization - PPG normalizes the full batch of advantage values before PPO updates instead of advantage normalization on each minibatch. (:material-github: [phasic_policy_gradient/ppo.py#L70](https://github.com/openai/phasic-policy-gradient/blob/c789b00be58aa704f7223b6fc8cd28a5aaa2e101/phasic_policy_gradient/ppo.py#L70))

1. Normalized network initialization - (:material-github: [phasic_policy_gradient/impala_cnn.py#L64](https://github.com/openai/phasic-policy-gradient/blob/c789b00be58aa704f7223b6fc8cd28a5aaa2e101/phasic_policy_gradient/impala_cnn.py#L64)) - PPG uses normalized initialization for all layers, with different scales. 
    * Original PPO used orthogonal initialization of only the Policy head and Value heads with scale of 0.01 and 1. respectively.
    * For PPG
        * All weights are initialized with the default torch initialization (Kaiming Uniform)
        * Each layer’s weights are divided by the L2 norm of the weights such that the weights of `input_channels` axis are individually normalized (axis 1 for linear layers and 1,2,3 for convolutional layers). Then the weights are multiplied by a scale factor.
        * Scale factors for different layers
            * Value head, Policy head, Auxiliary value head - 0.1
            * Fully connected layer after last conv later - 1.4
            * Convolutional layers - Approximately 0.638
1. The Adam Optimizer's Epsilon Parameter -(:material-github: [phasic_policy_gradient/ppg.py#L239](https://github.com/openai/phasic-policy-gradient/blob/c789b00be58aa704f7223b6fc8cd28a5aaa2e101/phasic_policy_gradient/ppg.py#L239)) - Set to torch default of 1e-8 instead of 1e-5 which is used in PPO.
1. Use the same `gamma` parameter in the `NormalizeReward` wrapper. Note that the original implementation from [openai/train-procgen](https://github.com/openai/train-procgen) uses the default `gamma=0.99` in [the `VecNormalize` wrapper](https://github.com/openai/train-procgen/blob/1a2ae2194a61f76a733a39339530401c024c3ad8/train_procgen/train.py#L43) but `gamma=0.999` as PPO's parameter. The mismatch between the `gamma`s is technically incorrect. See [#209](https://github.com/vwxyzjn/cleanrl/pull/209)

Here are some additional notes:

- All the default hyperparameters from the original PPG implementation are used. Except setting 64 for the number of environments.
- The original PPG paper does not report results on easy environments, hence more hyperparameter tuning can give better results.
- Skipping every alternate auxiliary phase gives similar performance on easy environments while saving compute.
- Normalized network initialization scheme seems to matter a lot, but using layernorm with orthogonal initialization also works.
- Using mixed precision for auxiliary phase also works well to save compute, but using on policy phase makes training unstable.


Also, `ppg_procgen.py` differs from the original  `openai/phasic-policy-gradient` implementation in the following ways.

- The original PPG code supports LSTM whereas the CleanRL code does not.
- The original PPG code uses separate optimizers for policy and auxiliary phase, but we do not implement this as we found it to not make too much difference.
- The original PPG code utilizes multiple GPUs but our implementation does not


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ppg.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppg.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fppg.sh%23L3-L8&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>


Below are the average episodic returns for `ppg_procgen.py`, and comparison with `ppg_procgen.py` on 25M timesteps.

| Environment      | `ppg_procgen.py` | `ppo_procgen.py` | `openai/phasic-policy-gradient` (easy) |
|------------------|------------------|------------------|----------------------------------------|
| Starpilot (easy) | 34.82 ± 13.77    | 32.47 ± 11.21    | 42.01 ± 9.59                           |
| Bossfight (easy) | 10.78 ± 1.90     | 9.63 ± 2.35      | 10.71 ± 2.05                           |
| Bigfish (easy)   | 24.23 ± 10.73    | 16.80 ± 9.49     | 15.94 ± 10.80                          |


???+ warning

    Note that we have run the procgen experiments using the `easy` distribution for reducing the computational cost. However, the original paper's results were condcuted with the `hard` distribution mode. For convenience, in the learning curves below, we compared the performance of the original code base (`openai/phasic-policy-gradient` the purple curve) in the `easy` distribution. 

Learning curves:

<div class="grid-container">
<img src="../ppg/StarPilot.png">
<img src="../ppg/comparison/StarPilot.png">

<img src="../ppg/BossFight.png">
<img src="../ppg/comparison/BossFight.png">

<img src="../ppg/BigFish.png">
<img src="../ppg/comparison/BigFish.png">
</div>


???+ info

    Also note that our `ppo_procgen.py` which closely matches implementation details of `openai/baselines`' PPO which might not be the same as `openai/phasic-policy-gradient`'s PPO. We take the reported results from (Cobbe et al., 2020)[^1] and (Cobbe et al., 2021)[^2] and compared them in a [google sheet](https://docs.google.com/spreadsheets/d/1ZC_D2WPL6-PzhecM4ZFQWQ6nY6dkXeQDOIgRHVp1BNU/edit?usp=sharing) (screenshot shown below). As shown, the performance seems to diverge a bit. We also note that (Cobbe et al., 2020)[^1] used [`procgen==0.9.2`](https://github.com/openai/train-procgen/blob/1a2ae2194a61f76a733a39339530401c024c3ad8/environment.yml#L10) and (Cobbe et al., 2021)[^2] used [`procgen==0.10.4`](https://github.com/openai/phasic-policy-gradient/blob/7295473f0185c82f9eb9c1e17a373135edd8aacc/environment.yml#L10), which also could cause performance difference. It is for this reason, we ran our own `openai/phasic-policy-gradient` experiments on the `easy` distribution for comparison, but this does mean it's challenging to compare our results against those in the original PPG paper (Cobbe et al., 2021)[^2].

    

Tracked experiments and game play videos:


<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Procgen-CleanRL-s-PPG--VmlldzoyMDc1MDMz" style="width:100%; height:500px" title="Procgen-CleanRL-s-PPG"></iframe>

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Procgen-CleanRL-s-PPG-vs-PPO-vs-openai-phasic-policy-gradient--VmlldzoyMDc1MDc3" style="width:100%; height:500px" title="Procgen-CleanRL-s-PPG-PPO-openai-phasic-policy-gradient"></iframe>


[^1]: Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2020, November). Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning (pp. 2048-2056). PMLR.
[^2]: Cobbe, K. W., Hilton, J., Klimov, O., & Schulman, J. (2021, July). Phasic policy gradient. In International Conference on Machine Learning (pp. 2020-2027). PMLR.

---

### Rl Algorithms Ppo Isaacgymenvs (`docs/rl-algorithms/ppo-isaacgymenvs.md`)

## `ppo_continuous_action_isaacgym.py`


???+ warning

    `ppo_continuous_action_isaacgym.py` is temporarily deprecated. Please checkout the code in [https://github.com/vwxyzjn/cleanrl/releases/tag/v1.0.0](https://github.com/vwxyzjn/cleanrl/releases/tag/v1.0.0)


The [ppo_continuous_action_isaacgym.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py) has the following features:

- Works with IsaacGymEnvs.
- Works with the `Box` observation space of low-level features
- Works with the `Box` (continuous) action space

[IsaacGymEnvs](https://github.com/NVIDIA-Omniverse/IsaacGymEnvs) is a hardware-accelerated (or GPU-accelerated) robotics simulation environment based on `torch`, which allows us to run thousands of simulation environments at the same time, empowering RL agents to learn many MuJoCo-style robotics tasks in minutes instead of hours. When creating an environment with IsaacGymEnvs via `isaacgymenvs.make("Ant")`, it creates a vectorized environment which produces GPU tensors as observations and take GPU tensors as actions to execute.

???+ info

    Note that **Isaac Gym** is the underlying core physics engine, and **IssacGymEnvs** is a collection of environments built on Isaac Gym.

???+ info

    `ppo_continuous_action_isaacgym.py` works with most environments in IsaacGymEnvs but it does not work with the following environments yet:

    * AnymalTerrain
    * FrankaCabinet
    * ShadowHandOpenAI_FF
    * ShadowHandOpenAI_LSTM
    * Trifinger
    * Ingenuity Quadcopter

    🔥 we need contributors to work on supporting and tuning our PPO implementation in these envs. If you are interested, please read our [contribution guide](https://github.com/vwxyzjn/cleanrl/blob/master/CONTRIBUTING.md) and reach out!

### Usage

The installation of `isaacgym` requires a bit of work since it's not a standard Python package.

Please go to [https://developer.nvidia.com/isaac-gym](https://developer.nvidia.com/isaac-gym) to download and install the latest version of Issac Gym which should look like `IsaacGym_Preview_4_Package.tar.gz`. Put this `IsaacGym_Preview_4_Package.tar.gz` into the `~/Downloads/` folder. Make sure your python version is either 3.7, or 3.8 (3.9 _not_ supported yet).

```bash
# extract and move the content in `python` folder in the IsaacGym_Preview_4_Package.tar.gz
# into the `cleanrl/ppo_continuous_action_isaacgym/isaacgym/` folder
cp ~/Downloads/IsaacGym_Preview_4_Package.tar.gz IsaacGym_Preview_4_Package.tar.gz 
stat IsaacGym_Preview_4_Package.tar.gz
mkdir temp_isaacgym
tar -xf IsaacGym_Preview_4_Package.tar.gz -C temp_isaacgym
mv temp_isaacgym/isaacgym/python/* cleanrl/ppo_continuous_action_isaacgym/isaacgym
rm -rf temp_isaacgym

# if your global python version is not either 3.7 nor 3.8, you need to tell poetry specifically to use a 3.7 or 3.8 python
# e.g., `poetry env use /home/costa/.pyenv/versions/3.7.8/bin/python`
poetry install --with isaacgym
# if you are using NVIDIA's 30xx GPU, you need to specifically install cuda 11.3 wheels
# `uv pip install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu113`
uv run python cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py --help
uv run python cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py --env-id Ant
```

<script id="asciicast-510341" src="https://asciinema.org/a/510341.js" async></script>


???+ warning

    If you encounter the following installation error
    
    ```bash
    Python.h: No such file or directory
    #include <Python.h>
    ```

    or 

    ```bash
    libpython3.8.so.1.0: cannot open shared object file: No such file or directory
    ```

    It usually means your python distribution does not include the shared library files. If you are ubuntu, you can install the following packages:
    
    ```bash
    sudo apt-get install libpython3.8-dev # or sudo apt-get install libpython3.7-dev
    ```

    If you are using [pyenv](https://github.com/pyenv/pyenv), you may try the following:
    
    ```bash
    env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.7.8
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

Additionally, `charts/consecutive_successes` means the number of consecutive episodes that the agent has successfully manipulating the rubix cube to the desired state.

### Implementation details

[ppo_continuous_action_isaacgym.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py) is based on `ppo_continuous_action.py` (see related [docs](/rl-algorithms/ppo/#ppo_continuous_actionpy)), with a few modifications:

1. **Different set of hyperparameters**: `ppo_continuous_action_isaacgym.py` uses hyperparameters primarily derived from [rl-games](https://github.com/Denys88/rl_games)' configuration (see [example](https://github.com/NVIDIA-Omniverse/IsaacGymEnvs/blob/main/isaacgymenvs/cfg/train/AntPPO.yaml)). The basic spirit is to run more `total_timesteps`, with larger `num_envs` and smaller `num_steps`.

| arguments         | `ppo_continuous_action.py` | `ppo_continuous_action_isaacgym.py` | `ppo_continuous_action_isaacgym.py` (for `ShadowHand` and `AllegroHand`) |
| ----------------- | -------------------------- | ----------------------------------- | ----------------------------------- |
| --total-timesteps | 1000000                    | 30000000                            | 600000000                           |
| --learning-rate   | 3e-4                       | 0.0026                              | 0.0026                              |
| --num-envs        | 1                          | 4096                                | 8192                                |
| --num-steps       | 2048                       | 16                                  | 8                                   |
| --anneal-lr       | True                       | False                               | False                               |
| --num-minibatches | 32                         | 2                                   | 4                                   |
| --update-epochs   | 10                         | 4                                   | 5                                   |
| --clip-vloss      | True                       | False                               | False                               |
| --vf-coef         | 0.5                        | 2                                   | 2                                   |
| --max-grad-norm   | 0.5                        | 1                                   | 1                                   |
| --reward-scaler   | N/A                        | 1                                   | 0.01                                |

1. **Slightly larger NN**: `ppo_continuous_action.py` uses the following NN:
   ```python
   self.critic = nn.Sequential(
       layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
       nn.Tanh(),
       layer_init(nn.Linear(64, 64)),
       nn.Tanh(),
       layer_init(nn.Linear(64, 1), std=1.0),
   )
   self.actor_mean = nn.Sequential(
       layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
       nn.Tanh(),
       layer_init(nn.Linear(64, 64)),
       nn.Tanh(),
       layer_init(nn.Linear(64, np.prod(envs.single_action_space.shape)), std=0.01),
   )
   ```
   while `ppo_continuous_action_isaacgym.py` uses the following NN:
   ```python
   self.critic = nn.Sequential(
       layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 256)),
       nn.Tanh(),
       layer_init(nn.Linear(256, 256)),
       nn.Tanh(),
       layer_init(nn.Linear(256, 1), std=1.0),
   )
   self.actor_mean = nn.Sequential(
       layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 256)),
       nn.Tanh(),
       layer_init(nn.Linear(256, 256)),
       nn.Tanh(),
       layer_init(nn.Linear(256, np.prod(envs.single_action_space.shape)), std=0.01),
   )
   ```
1. **No normalization and clipping**: `ppo_continuous_action_isaacgym.py` does _not_ do observation and reward normalization and clipping for simplicity. It does however optionally offer an option to scale the rewards via `--reward-scaler x`, which multiplies all the rewards obtained by `x` as an example.
1. **Remove all CPU-related code**: `ppo_continuous_action_isaacgym.py` needs to remove all CPU-related code (e.g. `action.cpu().numpy()`). This is because almost everything in IsaacGymEnvs happens in GPU. To do this, the major modifications include the following:
   1. Create a custom `RecordEpisodeStatisticsTorch` wrapper that records statstics using GPU tensors instead of `numpy` arrays.
   1. Avoid transferring the tensors to CPU. The related code in `ppo_continuous_action.py` looks like
   ```python
   next_obs, reward, done, info = envs.step(action.cpu().numpy())
   rewards[step] = torch.tensor(reward).to(device).view(-1)
   next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(done).to(device)
   ```
   and the related code in `ppo_continuous_action_isaacgym.py` looks like
   ```python
   next_obs, rewards[step], next_done, info = envs.step(action)
   ```

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fppo.sh%23L61-L73&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>

Below are the average episodic returns for `ppo_continuous_action_isaacgym.py`. To ensure the quality of the implementation, we compared the results against [Denys88/rl_games](https://github.com/Denys88/rl_games)' PPO and present the training time (units being `s (seconds), m (minutes)`). The hardware used is a NVIDIA RTX A6000 in a 24 core machine.

| Environment (training time) | `ppo_continuous_action_isaacgym.py` | [Denys88/rl_games](https://github.com/Denys88/rl_games) |
| --------------------------- | ----------------------------------- | ------------------------------------------------------- |
| Cartpole (40s)              | 413.66 ± 120.93                     | 417.49 (30s)                                            |
| Ant (240s)                  | 3953.30 ± 667.086                   | 5873.05                                                 |
| Humanoid (350s)             | 2987.95 ± 257.60                    | 6254.73                                                 |
| Anymal (317s)               | 29.34 ± 17.80                       | 62.76                                                   |
| BallBalance (160s)          | 161.92 ± 89.20                      | 319.76                                                  |
| AllegroHand (200m)          | 762.93 ± 427.92                     | 3479.85                                                 |
| ShadowHand (130m)           | 427.16 ± 161.79                     | 5713.74                                                 |

Learning curves:

<div class="grid-container">
<img src="../ppo/isaacgymenvs/Cartpole.png">
<img src="../ppo/isaacgymenvs/Cartpole-time.png">
<img src="../ppo/isaacgymenvs/Ant.png">
<img src="../ppo/isaacgymenvs/Ant-time.png">
<img src="../ppo/isaacgymenvs/Humanoid.png">
<img src="../ppo/isaacgymenvs/Humanoid-time.png">
<img src="../ppo/isaacgymenvs/BallBalance.png">
<img src="../ppo/isaacgymenvs/BallBalance-time.png">
<img src="../ppo/isaacgymenvs/Anymal.png">
<img src="../ppo/isaacgymenvs/Anymal-time.png">
<img src="../ppo/isaacgymenvs/AllegroHand.png">
<img src="../ppo/isaacgymenvs/AllegroHand-time.png">
<img src="../ppo/isaacgymenvs/AllegroHand-c.png">
<img src="../ppo/isaacgymenvs/AllegroHand-c-time.png">
<img src="../ppo/isaacgymenvs/ShadowHand.png">
<img src="../ppo/isaacgymenvs/ShadowHand-time.png">
<img src="../ppo/isaacgymenvs/ShadowHand-c.png">
<img src="../ppo/isaacgymenvs/ShadowHand-c-time.png">
</div>

???+ info

    Note `ppo_continuous_action_isaacgym.py`'s performance seems poor compared to [Denys88/rl_games](https://github.com/Denys88/rl_games)' PPO. This is likely due to a few reasons.

    1. [Denys88/rl_games](https://github.com/Denys88/rl_games)' PPO uses different sets of tuned hyperparameters  and neural network architecture configuration for different tasks, whereas `ppo_continuous_action_isaacgym.py` only uses one neural network architecture and 2 set of hyperparameters (ignoring `--total-timesteps`).
    1. `ppo_continuous_action_isaacgym.py` does not use observation normalization (because in my preliminary testing for some reasons it did not help).

    While it should be possible to obtain higher scores with more tuning, the purpose of `ppo_continuous_action_isaacgym.py` is to hit a balance between simplicity and performance. I think `ppo_continuous_action_isaacgym.py` has relatively good performance with a concise codebase, which should be easy to modify and extend for practitioners.


Tracked experiments and game play videos:


<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Isaac-Gym-CleanRL-s-PPO--VmlldzoyMzQzNzMz" style="width:100%; height:500px" title="Isaac-Gym-CleanRL-s-PPO"></iframe>


Old Learning curves w/ Isaac Gym Preview 3 (no longer available in Nvidia's website for download):

<div class="grid-container">
<img src="../ppo/isaacgymenvs/old/Cartpole.png">
<img src="../ppo/isaacgymenvs/old/Ant.png">
<img src="../ppo/isaacgymenvs/old/Humanoid.png">
<img src="../ppo/isaacgymenvs/old/BallBalance.png">
<img src="../ppo/isaacgymenvs/old/Anymal.png">
<img src="../ppo/isaacgymenvs/old/AllegroHand.png">
<img src="../ppo/isaacgymenvs/old/ShadowHand.png">
</div>

???+ info

    Note the `AllegroHand` and `ShadowHand` experiments used the following command `ppo_continuous_action_isaacgym.py --track --capture_video --num-envs 16384 --num-steps 8 --update-epochs 5 --reward-scaler 0.01 --total-timesteps 600000000 --record-video-step-frequency 3660`. Costa: I was able to run this during my internship at NVIDIA, but in my home setup, the computer has less GPU memory which makes it hard to replicate the results w/ `--num-envs 16384`.

---

### Rl Algorithms Ppo Rnd (`docs/rl-algorithms/ppo-rnd.md`)

# Random Network Distillation (RND)


## Overview

RND is an exploration bonus for RL methods that's easy to implement and enables significant progress in some hard exploration Atari games such as Montezuma's Revenge. We use [Proximal Policy Optimization](/rl-algorithms/ppo/#ppopy) as our RL method as used by original paper's [implementation](https://github.com/openai/random-network-distillation)


Original paper: 

* [Exploration by Random Network Distillation](https://arxiv.org/abs/1810.12894)

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`ppo_rnd_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_rnd_envpool.py), :material-file-document: [docs](/rl-algorithms/ppo-rnd/#ppo_rnd_envpoolpy) | For Atari games, uses EnvPool. |


Below are our single-file implementations of RND:

## `ppo_rnd_envpool.py`

The [ppo_rnd_envpool.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_rnd_envpool.py) has the following features:

* Uses the blazing fast [Envpool](https://github.com/sail-sg/envpool) vectorized environment.
* For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discerete` action space

???+ warning

    Note that `ppo_rnd_envpool.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

    EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PPO implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

    * Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
    * Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

    This causes the $s_{last}$ to be off by one. 
    See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.


### Usage

=== "poetry"

    ```bash
    uv pip install ".[envpool]"
    uv run python cleanrl/ppo_rnd_envpool.py --help
    uv run python cleanrl/ppo_rnd_envpool.py --env-id MontezumaRevenge-v5
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-envpool.txt
    python cleanrl/ppo_rnd_envpool.py --help
    python cleanrl/ppo_rnd_envpool.py --env-id MontezumaRevenge-v5
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.
Below is the additional metric for RND:

* `charts/episode_curiosity_reward`: episodic intrinsic rewards.
* `losses/fwd_loss`: the prediction error between predict network and target network, can also be viewed as a proxy of the curiosity reward in that batch.

### Implementation details

[ppo_rnd_envpool.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_rnd_envpool.py) uses a customized `RecordEpisodeStatistics` to work with envpool but has the same other implementation details as `ppo_atari.py` (see [related docs](/rl-algorithms/ppo/#implementation-details_1)). Additionally, it has the following additional details:

1. We initialize the normalization parameters by stepping a random agent in the environment by `args.num_steps * args.num_iterations_obs_norm_init`. `args.num_iterations_obs_norm_init=50` comes from the [original implementation](https://github.com/openai/random-network-distillation/blob/f75c0f1efa473d5109d487062fd8ed49ddce6634/run_atari.py#L69).
1. We uses sticky action from [envpool](https://envpool.readthedocs.io/en/latest/env/atari.html?highlight=repeat_action_probability%20#options) to facilitate the exploration like done in the [original implementation](https://github.com/openai/random-network-distillation/blob/f75c0f1efa473d5109d487062fd8ed49ddce6634/atari_wrappers.py#L204).

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/rnd.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/rnd.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Frnd.sh%23L3-L8&style=github&type=code&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script>

Below are the average episodic returns for `ppo_rnd_envpool.py`. To ensure the quality of the implementation, we compared the results against `openai/random-network-distillation`' PPO.

| Environment      | `ppo_rnd_envpool.py` | (Burda et al., 2019, Figure 7)[^1] 2000M steps
| ----------- | ----------- | ----------- |
| MontezumaRevengeNoFrameSkip-v4      | 7100 (1 seed)    | 8152 (3 seeds)  |

Note the MontezumaRevengeNoFrameSkip-v4 has same setting to MontezumaRevenge-v5.
Our benchmark has one seed due to limited compute resource and extreme long run time (~250 hours).


Learning curves:

<div class="grid-container">
    <img src="../ppo-rnd/MontezumaRevenge-v5.png">
    <img src="../ppo-rnd/MontezumaRevenge-v5-time.png">
</div>

<div></div>


Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/-MontezumaRevenge-CleanRL-s-PPO-RND--VmlldzoyNTIyNjc5" style="width:100%; height:1200px" title="MontezumaRevenge: CleanRL's PPO + RND"></iframe>


[^1]:Burda, Yuri, et al. "Exploration by random network distillation." Seventh International Conference on Learning Representations. 2019.

---

### Rl Algorithms Ppo Trxl (`docs/rl-algorithms/ppo-trxl.md`)

# Tranformer-XL (PPO-TrXL)

## Overview

Real-world tasks may expose imperfect information (e.g. partial observability). Such tasks require an agent to leverage memory capabilities. One way to do this is to use recurrent neural networks (e.g. LSTM) as seen in :material-github: [`ppo_atari_lstm.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_lstm.py), :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_lstmpy). Here, Transformer-XL is used as episodic memory in Proximal Policy Optimization (PPO).

Original Paper and Implementation

* :material-file-document: [Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents](https://arxiv.org/abs/2309.17207)
* :material-github: [neroRL](https://github.com/MarcoMeter/neroRL)
* :material-github:  [Episodic Transformer Memory PPO](https://github.com/MarcoMeter/episodic-transformer-memory-ppo)
* :material-github: [Endless Memory Gym](https://github.com/MarcoMeter/endless-memory-gym)
* :material-play-circle: [Interactive Visualizations of Trained Agents](https://marcometer.github.io/)

Related Publications and Repositories

* :material-file-document: [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860)
* :material-file-document: [Stabilizing Transformers for Reinforcement Learning](https://arxiv.org/abs/1910.06764)
* :material-file-document: [Towards mental time travel: a hierarchical memory for reinforcement learning agents](https://arxiv.org/abs/2105.14039)
* :material-file-document: [Grounded Language Learning Fast and Slow](https://arxiv.org/abs/2009.01719)
* :material-github: [transformerXL_PPO_JAX](https://github.com/Reytuag/transformerXL_PPO_JAX)

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`ppo_trxl.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_trxl/ppo_trxl.py), :material-file-document: [docs](/rl-algorithms/ppo-trxl#ppo_trxlpy) | For training on tasks like `Endless-MortarMayhem-v0`. |

Below is our single-file implementation of PPO-TrXL:

## `ppo_trxl.py`

`ppo_trxl.py` has the following features:

* Works with Memory Gym's environments (84x84 RGB image observation).
* Works with Minigrid Memory (84x84 RGB image observation).
* Works also with environments exposing only game state vector observations (e.g. Proof of Memory Environment).
* Works with just single or multi-discrete action spaces.

### Usage

As the recommended way, the requirements default to PyTorch's CUDA packages.

=== "poetry"

    ```bash
    cd cleanrl/ppo_trxl
    poetry install
    uv run python ppo_trxl.py --help
    uv run python ppo_trxl.py --env-id Endless-MortarMayhem-v0
    ```
  
=== "pip"

    ```bash
    pip install -r requirements/requirements-memory_gym.txt
    python cleanrl/ppo_trxl/ppo_trxl.py --help
    python cleanrl/ppo_trxl/ppo_trxl.py --env-id Endless-MortarMayhem-v0
    ```

### Explanation of the logged metrics

* `episode/r_mean`: mean of the episodic return of the game
* `episode/l_mean`: mean of the episode length of the game in steps
* `episode/t_mean`: mean of the episode duration of the game in seconds
* `episode/advantage_mean`: mean of all computed advantages
* `episode/value_mean`: mean of all approximated values
* `charts/SPS`: number of steps per second
* `charts/learning_rate`: the current learning rate
* `charts/entropy_coefficient`: the current entropy coefficient
* `losses/value_loss`: the mean value loss across all data points
* `losses/policy_loss`: the mean policy loss across all data points
* `losses/entropy`: the mean entropy value across all data points
* `losses/reconstruction_loss`: the mean observation reconstruction loss value across all data points
* `losses/loss`: the mean of all summed losses across all data points
* `losses/old_approx_kl`: the approximate Kullback–Leibler divergence, measured by `(-logratio).mean()`, which corresponds to the k1 estimator in John Schulman’s blog post on [approximating KL](http://joschu.net/blog/kl-approx.html)
* `losses/approx_kl`: better alternative to `old_approx_kl` measured by `(logratio.exp() - 1) - logratio`, which corresponds to the k3 estimator in [approximating KL](http://joschu.net/blog/kl-approx.html)
* `losses/clipfrac`: the fraction of the training data that triggered the clipped objective
* `losses/explained_variance`: the explained variance for the value function

### Implementation details

Most details are derived from [`ppo.py`](/rl-algorithms/ppo#ppopy). These are additional or differing details:

1. The policy and value function share parameters.
2. Multi-head attention is implemented so that all heads share parameters.
3. Absolute positional encoding is used as default. Learned positional encodings are supported.
4. Previously computed hidden states of the TrXL layers are cached and reused for up to `trxl_memory_length`. Only 1 hidden state is computed anew.
5. TrXL layers adhere to pre-layer normalization.
6. Support for multi-discrete action spaces.
7. Support for an auxiliary observation reconstruction loss, which reconstructs TrXL's output to the fed visual observation.
8. The learning rate and the entropy bonus coefficient linearly decay until reaching a lower threshold.

### Experiment results

Note: When training on potentially endless episodes, the cached hidden states demand a large GPU memory. To reproduce the following experiments a minimum of 40GB is required. One workaround is to cache the hidden states in the buffer with lower precision as bfloat16. This is under examination for future updates.

|                              | PPO-TrXL    |
|:-----------------------------|:------------|
| MortarMayhem-Grid-v0         | 0.99 ± 0.00 |
| MortarMayhem-v0              | 0.99 ± 0.00 |
| Endless-MortarMayhem-v0      | 1.50 ± 0.02 |
| MysteryPath-Grid-v0          | 0.97 ± 0.01 |
| MysteryPath-v0               | 1.67 ± 0.02 |
| Endless-MysteryPath-v0       | 1.84 ± 0.06 |
| SearingSpotlights-v0         | 1.11 ± 0.08 |
| Endless-SearingSpotlights-v0 | 1.60 ± 0.03 |

Learning curves:


<img src="../ppo-trxl/compare.png">


Tracked experiments:

<iframe src="https://api.wandb.ai/links/m-pleines/wo9m43hv" style="width:100%; height:500px" title="CleanRL-s-PPO-TrXL"></iframe>


### Hyperparameters

Memory Gym Environments

Please refer to the defaults in [`ppo_trxl.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_trxl/ppo_trxl.py) and the single modifications as found in [`benchmark/ppo_trxl.sh`](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo_trxl.sh)

ProofofMemory-v0
```bash
uv run python ppo_trxl.py \
  --env_id ProofofMemory-v0 \
  --total_timesteps 25000 \
  --num_envs 16 \
  --num_steps 128 \
  --num_minibatches 8 \
  --update_epochs 4 \
  --trxl_num_layers 4 \
  --trxl_num_heads 1 \
  --trxl_dim 64 \
  --trxl_memory_length 16 \
  --trxl_positional_encoding none \
  --vf_coef 0.1 \
  --max_grad_norm 0.5 \
  --init_lr 3.0e-4 \
  --init_ent_coef 0.001 \
  --clip_coef 0.2
```

MiniGrid-MemoryS9-v0
```bash
uv run python ppo_trxl.py \
  --env_id MiniGrid-MemoryS9-v0 \
  --total_timesteps 2048000 \
  --num_envs 16 \
  --num_steps 256 \
  --trxl_num_layers 2 \
  --trxl_num_heads 4 \
  --trxl_dim 256 \
  --trxl_memory_length 64 \
  --max_grad_norm 0.25 \
  --anneal_steps 4096000
  --clip_coef 0.2
```

### Enjoy pre-trained models

Use [`cleanrl/ppo_trxl/enjoy.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_trxl/enjoy.py) to watch pre-trained agents.
You can retrieve pre-trained models from [huggingface](https://huggingface.co/LilHairdy/cleanrl_memory_gym).
Note that Memory Gym environments are usually rendered using the `debug_rgb_array` render mode, which shows ground truth information about the current task that the agent cannot observe.


Run models from the hub:
```bash
python cleanrl/ppo_trxl/enjoy.py --hub --name Endless-MortarMayhem-v0_12.nn
python cleanrl/ppo_trxl/enjoy.py --hub --name Endless-MysterPath-v0_11.nn
python cleanrl/ppo_trxl/enjoy.py --hub --name Endless-SearingSpotlights-v0_30.nn
python cleanrl/ppo_trxl/enjoy.py --hub --name MiniGrid-MemoryS9-v0_10.nn
python cleanrl/ppo_trxl/enjoy.py --hub --name ProofofMemory-v0_1.nn
```


Run local models (or download them from the hub manually):
```bash
python cleanrl/ppo_trxl/enjoy.py --name Your.cleanrl_model
```

---

### Rl Algorithms Ppo (`docs/rl-algorithms/ppo.md`)

# Proximal Policy Optimization (PPO)


## Overview

PPO is one of the most popular DRL algorithms. It runs reasonably fast by leveraging vector (parallel) environments and naturally works well with different action spaces, therefore supporting a variety of games. It also has good sample efficiency compared to algorithms such as DQN.


Original paper: 

* [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347)

Reference resources:

* [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://arxiv.org/abs/2005.12729)
* [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://arxiv.org/abs/2006.05990)
* ⭐ [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)

All our PPO implementations below are augmented with the same code-level optimizations presented in `openai/baselines`'s [PPO](https://github.com/openai/baselines/tree/master/baselines/ppo2). To achieve this, see how we matched the implementation details in our blog post [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/).

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`ppo.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py), :material-file-document: [docs](/rl-algorithms/ppo/#ppopy) | For classic control tasks like `CartPole-v1`. |
| :material-github: [`ppo_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari.py), :material-file-document: [docs](/rl-algorithms/ppo/#ppo_ataripy) |  For Atari games. It uses convolutional layers and common atari-based pre-processing techniques. |
| :material-github: [`ppo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py), :material-file-document: [docs](/rl-algorithms/ppo/#ppo_continuous_actionpy) | For continuous action space. Also implemented Mujoco-specific code-level optimizations. |
| :material-github: [`ppo_atari_lstm.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_lstm.py), :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_lstmpy) | For Atari games using LSTM without stacked frames. |
| :material-github: [`ppo_atari_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool.py), :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_envpoolpy) | Uses the blazing fast Envpool Atari vectorized environment. |
| :material-github: [`ppo_atari_envpool_xla_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_envpool_xla_jaxpy) | Uses the blazing fast Envpool Atari vectorized environment with EnvPool's XLA interface and JAX. |
| :material-github: [`ppo_atari_envpool_xla_jax_scan.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax_scan.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_envpool_xla_jax_scanpy) | Uses native `jax.scan` as opposed to python loops for faster compilation time. |
| :material-github: [`ppo_procgen.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_procgen.py), :material-file-document: [docs](/rl-algorithms/ppo/#ppo_procgenpy) | For the procgen environments. |
| :material-github: [`ppo_atari_multigpu.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_multigpu.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_multigpupy)| For Atari environments leveraging multi-GPUs. |
| :material-github: [`ppo_pettingzoo_ma_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_pettingzoo_ma_atari.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_pettingzoo_ma_ataripy)| For Pettingzoo's multi-agent Atari environments. |

Below are our single-file implementations of PPO:

## `ppo.py`

The [ppo.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py) has the following features:

* Works with the `Box` observation space of low-level features
* Works with the `Discrete` action space
* Works with envs like `CartPole-v1`

### Usage

=== "uv"

    ```bash
    uv pip install .
    uv run python cleanrl/ppo.py --help
    uv run python cleanrl/ppo.py --env-id CartPole-v1
    ```

=== "pip"

    ```bash
    python cleanrl/ppo.py --help
    python cleanrl/ppo.py --env-id CartPole-v1
    ```

### Explanation of the logged metrics

Running `python cleanrl/ppo.py` will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

* `charts/episodic_return`: episodic return of the game
* `charts/episodic_length`: episodic length of the game
* `charts/SPS`: number of steps per second
* `charts/learning_rate`: the current learning rate
* `losses/value_loss`: the mean value loss across all data points
* `losses/policy_loss`: the mean policy loss across all data points
* `losses/entropy`: the mean entropy value across all data points
* `losses/old_approx_kl`: the approximate Kullback–Leibler divergence, measured by `(-logratio).mean()`, which corresponds to the k1 estimator in John Schulman’s blog post on [approximating KL](http://joschu.net/blog/kl-approx.html)
* `losses/approx_kl`: better alternative to `old_approx_kl` measured by `(logratio.exp() - 1) - logratio`, which corresponds to the k3 estimator in [approximating KL](http://joschu.net/blog/kl-approx.html)
* `losses/clipfrac`: the fraction of the training data that triggered the clipped objective
* `losses/explained_variance`: the explained variance for the value function


### Implementation details

[ppo.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py) is based on the "13 core implementation details" in [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/), which are as follows:

1. Vectorized architecture (:material-github: [common/cmd_util.py#L22](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L22))
1. Orthogonal Initialization of Weights and Constant Initialization of biases (:material-github: [a2c/utils.py#L58)](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L58))
1. The Adam Optimizer's Epsilon Parameter (:material-github: [ppo2/model.py#L100](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L100))
1. Adam Learning Rate Annealing (:material-github: [ppo2/ppo2.py#L133-L135](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/ppo2.py#L133-L135))
1. Generalized Advantage Estimation (:material-github: [ppo2/runner.py#L56-L65](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py#L56-L65))
1. Mini-batch Updates (:material-github: [ppo2/ppo2.py#L157-L166](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/ppo2.py#L157-L166))
1. Normalization of Advantages (:material-github: [ppo2/model.py#L139](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L139))
1. Clipped surrogate objective (:material-github: [ppo2/model.py#L81-L86](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L81-L86))
1. Value Function Loss Clipping (:material-github: [ppo2/model.py#L68-L75](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L68-L75))
1. Overall Loss and Entropy Bonus (:material-github: [ppo2/model.py#L91](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L91))
1. Global Gradient Clipping (:material-github: [ppo2/model.py#L102-L108](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L102-L108))
1. Debug variables (:material-github: [ppo2/model.py#L115-L116](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L115-L116))
1. Separate MLP networks for policy and value functions (:material-github: [common/policies.py#L156-L160](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L156-L160), [baselines/common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103))

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:

``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:3:8"
```

Below are the average episodic returns for `ppo.py`. To ensure the quality of the implementation, we compared the results against `openai/baselines`' PPO.

| Environment      | `ppo.py` | `openai/baselines`' PPO (Huang et al., 2022)[^1]
| ----------- | ----------- | ----------- |
| CartPole-v1      | 490.04 ± 6.12     |497.54 ± 4.02  |
| Acrobot-v1       | -86.36 ± 1.32     |  -81.82 ± 5.58 |
| MountainCar-v0   | -200.00 ± 0.00         | -200.00 ± 0.00 |


Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh::9"
```


<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo-time.png">


Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Classic-Control-CleanRL-s-PPO--VmlldzoxODU5MDY1" style="width:100%; height:500px" title="Classic-Control-CleanRL-s-PPO"></iframe>

### Video tutorial

If you'd like to learn `ppo.py` in-depth, consider checking out the following video tutorial:


<div style="text-align: center;"><iframe width="560" height="315" src="https://www.youtube.com/embed/MEt6rrxH8W4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>


## `ppo_atari.py`

The [ppo_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari.py) has the following features:

* For Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space


### Usage

=== "poetry"

    ```bash
    uv pip install ".[atari]"
    uv run python cleanrl/ppo_atari.py --help
    uv run python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/ppo_atari.py --help
    python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

### Implementation details

[ppo_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari.py) is based on the "9 Atari implementation details" in [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/), which are as follows:

1. The Use of `NoopResetEnv` (:material-github: [common/atari_wrappers.py#L12](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L12)) 
1. The Use of `MaxAndSkipEnv` (:material-github: [common/atari_wrappers.py#L97](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L97)) 
1. The Use of `EpisodicLifeEnv` (:material-github: [common/atari_wrappers.py#L61](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L61)) 
1. The Use of `FireResetEnv` (:material-github: [common/atari_wrappers.py#L41](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L41)) 
1. The Use of `WarpFrame` (Image transformation) [common/atari_wrappers.py#L134](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L134) 
1. The Use of `ClipRewardEnv` (:material-github: [common/atari_wrappers.py#L125](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L125)) 
1. The Use of `FrameStack` (:material-github: [common/atari_wrappers.py#L188](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L188)) 
1. Shared Nature-CNN network for the policy and value functions (:material-github: [common/policies.py#L157](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L157), [common/models.py#L15-L26](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L15-L26))
1. Scaling the Images to Range [0, 1] (:material-github: [common/models.py#L19](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/models.py#L19))

### Experiment results



To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:

``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:14:19"
```


Below are the average episodic returns for `ppo_atari.py`. To ensure the quality of the implementation, we compared the results against `openai/baselines`' PPO.

| Environment      | `ppo_atari.py` | `openai/baselines`' PPO (Huang et al., 2022)[^1]
| ----------- | ----------- | ----------- |
| BreakoutNoFrameskip-v4      | 414.66 ± 28.09     | 406.57 ± 31.554  |
| PongNoFrameskip-v4   | 20.36 ± 0.20    |  20.512 ± 0.50 |
| BeamRiderNoFrameskip-v4   | 1915.93 ± 484.58         | 2642.97 ± 670.37 |


Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh:11:19"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari-time.png">


Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-PPO--VmlldzoxNjk3NjYy" style="width:100%; height:500px" title="Atari-CleanRL-s-PPO"></iframe>

### Video tutorial

If you'd like to learn `ppo_atari.py` in-depth, consider checking out the following video tutorial:


<div style="text-align: center;"><iframe width="560" height="315" src="https://www.youtube.com/embed/05RMTj-2K_Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>


## `ppo_continuous_action.py`

The [ppo_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py) has the following features:

* For continuous action space. Also implemented Mujoco-specific code-level optimizations
* Works with the `Box` observation space of low-level features
* Works with the `Box` (continuous) action space
* adding experimental support for [Gymnasium](https://gymnasium.farama.org/)
* 🧪 support `dm_control` environments via [Shimmy](https://github.com/Farama-Foundation/Shimmy)


???+ warning

    We are now recommending users to use [`rpo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/rpo_continuous_action.py) instead of [`ppo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py) because `rpo_continuous_action.py` empirically performs better than `ppo_continuous_action.py` in 93% of the environments we tested. Please see [experiment results](/rl-algorithms/rpo/#experiment-results) for detailed analysis.

### Usage

=== "poetry"

    ```bash
    # mujoco v4 environments
    uv pip install ".[mujoco]"
    python cleanrl/ppo_continuous_action.py --help
    python cleanrl/ppo_continuous_action.py --env-id Hopper-v4
    # dm_control environments
    uv pip install ".[mujoco, dm_control]"
    python cleanrl/ppo_continuous_action.py --env-id dm_control/cartpole-balance-v0
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-mujoco.txt
    python cleanrl/ppo_continuous_action.py --help
    python cleanrl/ppo_continuous_action.py --env-id Hopper-v4
    pip install -r requirements/requirements-dm_control.txt
    python cleanrl/ppo_continuous_action.py --env-id dm_control/cartpole-balance-v0
    ```

???+ warning "dm_control installation issue"

    If you run into error like `AttributeError: 'GLFWContext' object has no attribute '_context'` in Linux, it's because the rendering dependencies are not installed properly. To fix it, try running

    ```
    sudo apt-get update && sudo apt-get -y install libgl1-mesa-glx libosmesa6 libglfw3 
    ```

    See [https://github.com/deepmind/dm_control#rendering](https://github.com/deepmind/dm_control#rendering) for more detail.


### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

### Implementation details

[ppo_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py) is based on the "9 details for continuous action domains (e.g. Mujoco)" in [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/), which are as follows:


1. Continuous actions via normal distributions (:material-github: [common/distributions.py#L103-L104](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L103-L104))
2. State-independent log standard deviation (:material-github: [common/distributions.py#L104](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L104))
3. Independent action components (:material-github: [common/distributions.py#L238-L246](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/distributions.py#L238-L246))
4. Separate MLP networks for policy and value functions (:material-github: [common/policies.py#L160](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L160), [baselines/common/models.py#L75-L103](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L75-L103)
5. Handling of action clipping to valid range and storage (:material-github: [common/cmd_util.py#L99-L100](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L99-L100)) 
6. Normalization of Observation (:material-github: [common/vec_env/vec_normalize.py#L4](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L4))
7. Observation Clipping (:material-github: [common/vec_env/vec_normalize.py#L39](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L39))
8. Reward Scaling (:material-github: [common/vec_env/vec_normalize.py#L28](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L28))
9. Reward Clipping (:material-github: [common/vec_env/vec_normalize.py#L32](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/vec_env/vec_normalize.py#L32))



### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:


MuJoCo v4

``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:25:30"
```

{!benchmark/ppo_continuous_action.md!}

Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh:11:19"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_continuous_action.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_continuous_action-time.png">

Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/costa-huang/cleanRL/reports/MuJoCo-v4-CleanRL-s-PPO--VmlldzozMTIxOTI5" style="width:100%; height:500px" title="MuJoCo-CleanRL-s-PPO"></iframe>



``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:36:41"
```

Below are the average episodic returns for `ppo_continuous_action.py` in `dm_control` environments.

|                                       | ppo_continuous_action ({'tag': ['v1.0.0-13-gcbd83f6']})   |
|:--------------------------------------|:----------------------------------------------------------|
| dm_control/acrobot-swingup-v0         | 27.84 ± 9.25                                              |
| dm_control/acrobot-swingup_sparse-v0  | 1.60 ± 1.17                                               |
| dm_control/ball_in_cup-catch-v0       | 900.78 ± 5.26                                             |
| dm_control/cartpole-balance-v0        | 855.47 ± 22.06                                            |
| dm_control/cartpole-balance_sparse-v0 | 999.93 ± 0.10                                             |
| dm_control/cartpole-swingup-v0        | 640.86 ± 11.44                                            |
| dm_control/cartpole-swingup_sparse-v0 | 51.34 ± 58.35                                             |
| dm_control/cartpole-two_poles-v0      | 203.86 ± 11.84                                            |
| dm_control/cartpole-three_poles-v0    | 164.59 ± 3.23                                             |
| dm_control/cheetah-run-v0             | 432.56 ± 82.54                                            |
| dm_control/dog-stand-v0               | 307.79 ± 46.26                                            |
| dm_control/dog-walk-v0                | 120.05 ± 8.80                                             |
| dm_control/dog-trot-v0                | 76.56 ± 6.44                                              |
| dm_control/dog-run-v0                 | 60.25 ± 1.33                                              |
| dm_control/dog-fetch-v0               | 34.26 ± 2.24                                              |
| dm_control/finger-spin-v0             | 590.49 ± 171.09                                           |
| dm_control/finger-turn_easy-v0        | 180.42 ± 44.91                                            |
| dm_control/finger-turn_hard-v0        | 61.40 ± 9.59                                              |
| dm_control/fish-upright-v0            | 516.21 ± 59.52                                            |
| dm_control/fish-swim-v0               | 87.91 ± 6.83                                              |
| dm_control/hopper-stand-v0            | 2.72 ± 1.72                                               |
| dm_control/hopper-hop-v0              | 0.52 ± 0.48                                               |
| dm_control/humanoid-stand-v0          | 6.59 ± 0.18                                               |
| dm_control/humanoid-walk-v0           | 1.73 ± 0.03                                               |
| dm_control/humanoid-run-v0            | 1.11 ± 0.04                                               |
| dm_control/humanoid-run_pure_state-v0 | 0.98 ± 0.03                                               |
| dm_control/humanoid_CMU-stand-v0      | 4.79 ± 0.18                                               |
| dm_control/humanoid_CMU-run-v0        | 0.88 ± 0.05                                               |
| dm_control/manipulator-bring_ball-v0  | 0.50 ± 0.29                                               |
| dm_control/manipulator-bring_peg-v0   | 1.80 ± 1.58                                               |
| dm_control/manipulator-insert_ball-v0 | 35.50 ± 13.04                                             |
| dm_control/manipulator-insert_peg-v0  | 60.40 ± 21.76                                             |
| dm_control/pendulum-swingup-v0        | 242.81 ± 245.95                                           |
| dm_control/point_mass-easy-v0         | 273.95 ± 362.28                                           |
| dm_control/point_mass-hard-v0         | 143.25 ± 38.12                                            |
| dm_control/quadruped-walk-v0          | 239.03 ± 66.17                                            |
| dm_control/quadruped-run-v0           | 180.44 ± 32.91                                            |
| dm_control/quadruped-escape-v0        | 28.92 ± 11.21                                             |
| dm_control/quadruped-fetch-v0         | 193.97 ± 22.20                                            |
| dm_control/reacher-easy-v0            | 626.28 ± 15.51                                            |
| dm_control/reacher-hard-v0            | 443.80 ± 9.64                                             |
| dm_control/stacker-stack_2-v0         | 75.68 ± 4.83                                              |
| dm_control/stacker-stack_4-v0         | 68.02 ± 4.02                                              |
| dm_control/swimmer-swimmer6-v0        | 158.19 ± 10.22                                            |
| dm_control/swimmer-swimmer15-v0       | 131.94 ± 0.88                                             |
| dm_control/walker-stand-v0            | 564.46 ± 235.22                                           |
| dm_control/walker-walk-v0             | 392.51 ± 56.25                                            |
| dm_control/walker-run-v0              | 125.92 ± 10.01                                            |

Note that the dm_control/lqr-lqr_2_1-v0 dm_control/lqr-lqr_6_2-v0 environments are never terminated or truncated. See https://wandb.ai/openrlbenchmark/cleanrl/runs/3tm00923 and https://wandb.ai/openrlbenchmark/cleanrl/runs/1z9us07j as an example.

Learning curves:



Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/dm_control-CleanRL-s-PPO-part-1---VmlldzozMTI2MjE2" style="width:100%; height:500px" title="dm_control-CleanRL-s-PPO-part-1"></iframe>
<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/dm_control-CleanRL-s-PPO-part-2---VmlldzozMTI2MjI1" style="width:100%; height:500px" title="dm_control-CleanRL-s-PPO-part-2"></iframe>


???+ info

    In the gymnasium environments, we use the v4 mujoco environments, which roughly results in the same performance as the v2 mujoco environments.

    


### Video tutorial

If you'd like to learn `ppo_continuous_action.py` in-depth, consider checking out the following video tutorial:


<div style="text-align: center;"><iframe width="560" height="315" src="https://www.youtube.com/embed/BvZvx7ENZBw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>


## `ppo_atari_lstm.py`

The [ppo_atari_lstm.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_lstm.py) has the following features:

* For Atari games using LSTM without stacked frames. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

### Usage


=== "poetry"

    ```bash
    uv pip install ".[atari]"
    uv run python cleanrl/ppo_atari_lstm.py --help
    uv run python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/ppo_atari_lstm.py --help
    python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4
    ```


### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

### Implementation details

[ppo_atari_lstm.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_lstm.py) is based on the "5 LSTM implementation details" in [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/), which are as follows:

1. Layer initialization for LSTM layers (:material-github: [a2c/utils.py#L84-L86](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L84-L86))
2. Initialize the LSTM states to be zeros (:material-github: [common/models.py#L179](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L179))
3. Reset LSTM states at the end of the episode (:material-github: [common/models.py#L141](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L141))
4. Prepare sequential rollouts in mini-batches (:material-github: [a2c/utils.py#L81](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L81))
5. Reconstruct LSTM states during training (:material-github: [a2c/utils.py#L81](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L81))

To help test out the memory, we remove the 4 stacked frames from the observation (i.e., using `env = gym.wrappers.FrameStack(env, 1)` instead of `env = gym.wrappers.FrameStack(env, 4)` like in `ppo_atari.py` )



### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:

``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:47:52"
```

Below are the average episodic returns for `ppo_atari_lstm.py`. To ensure the quality of the implementation, we compared the results against `openai/baselines`' PPO.


| Environment      | `ppo_atari_lstm.py` | `openai/baselines`' PPO (Huang et al., 2022)[^1]
| ----------- | ----------- | ----------- |
| BreakoutNoFrameskip-v4      | 128.92 ± 31.10    | 138.98 ± 50.76  |
| PongNoFrameskip-v4   | 19.78 ± 1.58    | 19.79 ± 0.67 |
| BeamRiderNoFrameskip-v4   | 1536.20 ± 612.21         | 1591.68 ± 372.95|


Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh:11:19"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_lstm.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_lstm-time.png">

Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-PPO-LSTM--VmlldzoxODcxMzE4" style="width:100%; height:500px" title="Atari-CleanRL-s-PPO-LSTM"></iframe>



## `ppo_atari_envpool.py`

The [ppo_atari_envpool.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool.py) has the following features:

* Uses the blazing fast [Envpool](https://github.com/sail-sg/envpool) vectorized environment.
* For Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

???+ warning

    Note that `ppo_atari_envpool.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

    EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PPO implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

    * Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
    * Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

    This causes the $s_{last}$ to be off by one. 
    See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.


### Usage

=== "poetry"

    ```bash
    uv pip install ".[envpool]"
    uv run python cleanrl/ppo_atari_envpool.py --help
    uv run python cleanrl/ppo_atari_envpool.py --env-id Breakout-v5
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-envpool.txt
    python cleanrl/ppo_atari_envpool.py --help
    python cleanrl/ppo_atari_envpool.py --env-id Breakout-v5
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

### Implementation details

[ppo_atari_envpool.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool.py) uses a customized `RecordEpisodeStatistics` to work with envpool but has the same other implementation details as `ppo_atari.py` (see [related docs](/rl-algorithms/ppo/#implementation-details_1)).

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:


``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:58:63"
```

{!benchmark/ppo_atari_envpool.md!}


Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh:51:62"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_envpool.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_envpool-time.png">


Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-PPO-Envpool--VmlldzoxODcxMzI3" style="width:100%; height:500px" title="Atari-CleanRL-s-PPO-Envpool"></iframe>


## `ppo_atari_envpool_xla_jax.py`

The [ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py) has the following features:

* Uses the blazing fast [Envpool](https://github.com/sail-sg/envpool) vectorized environment.
    * Uses EnvPool's experimental [XLA interface](https://envpool.readthedocs.io/en/latest/content/xla_interface.html).
* Uses [Jax](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Optax](https://github.com/deepmind/optax) instead of `torch`.  
* For Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

???+ warning

    Note that `ppo_atari_envpool_xla_jax.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)


???+ bug

    EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PPO implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

    * Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
    * Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

    This causes the $s_{last}$ to be off by one. 
    See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.



### Usage

=== "poetry"

    ```bash
    uv pip install ".[envpool, jax]"
    uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    uv run python cleanrl/ppo_atari_envpool_xla_jax.py --help
    uv run python cleanrl/ppo_atari_envpool_xla_jax.py --env-id Breakout-v5
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-envpool.txt
    pip install -r requirements/requirements-jax.txt
    pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    python cleanrl/ppo_atari_envpool_xla_jax.py --help
    python cleanrl/ppo_atari_envpool_xla_jax.py --env-id Breakout-v5
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`. In [ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py) we omit logging `losses/old_approx_kl` and `losses/clipfrac` for brevity.

Additionally, we record the following metric:

* `charts/avg_episodic_return`: the average value of the *latest* episodic returns of `args.num_envs=8` envs
* `charts/avg_episodic_length`: the average value of the *latest* episodic lengths of `args.num_envs=8` envs

???+ info

    Note that we use `charts/avg_episodic_return` and `charts/avg_episodic_length` in place of `charts/episodic_return` and `charts/episodic_length` because under the EnvPool's XLA interface, we can only record fixed-shape metrics where as there could be a variable number of raw episodic returns / lengths. To resolve this challenge, we create variables (e.g., `returned_episode_returns`, `returned_episode_lengths`) to keep track of the *latest* episodic returns / lengths of each environment and average them for reporting purposes.

### Implementation details

[ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py) uses the same other implementation details as `ppo_atari.py` (see [related docs](/rl-algorithms/ppo/#implementation-details_1)), with two differences

1. [ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py) does not use the value function clipping by default, because there is no sufficient evidence that value function clipping actually improves performance.
1. [ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py) uses a customized `EpisodeStatistics` to record episode statistics instead of the `RecordEpisodeStatistics` used in other variants. `RecordEpisodeStatistics` is a *stateful* python wrapper which is incompatible with EnvPool's *stateless* XLA interface. To address this issue, we used a `EpisodeStatistics` dataclass and simply implement the logic of `RecordEpisodeStatistics`. However, `EpisodeStatistics` comes with a major limitation: its storage has a fixed shape and can only record the *latest* episodic return of the sub-environments. Furthermore, the default episodic return values in `EpisodeStatistics` are set to zeros, which does not necessarily correspond to the episodic return obtained by a random policy. For example, we would report `charts/avg_episodic_return=0` for `Pong-v5`, even if they should have been `charts/avg_episodic_return=-21`. That said, this issue goes away as soon as the sub-environments finished their first episodes, therefore not impacting the reported results.


???+ info
    
    We benchmarked the PPO implementation w/ and w/o value function clipping, finding no significant difference in performance, which is consistent with the findings in Andrychowicz et al.[^2]. See the related report [part 1](https://wandb.ai/costa-huang/cleanRL/reports/CleanRL-PPO-JAX-EnvPool-s-XLA-w-and-w-o-value-loss-clipping-vs-openai-baselins-PPO-part-1---VmlldzoyNzQ3MzQ1) and [part 2](https://wandb.ai/costa-huang/cleanRL/reports/CleanRL-PPO-JAX-EnvPool-s-XLA-w-and-w-o-value-loss-clipping-vs-openai-baselins-PPO-part-2---VmlldzoyNzQ3MzUw).

    


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:


``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:69:74"
```


{!benchmark/ppo_atari_envpool_xla_jax.md!}


Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh:64:85"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_envpool_xla_jax_sample_walltime_efficiency.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_envpool_xla_jax.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_envpool_xla_jax-time.png">



???+ info

    Note the original openai/baselines uses `atari-py==0.2.6` which hangs on `gym.make("DefenderNoFrameskip-v4")` and does not support SurroundNoFrameskip-v4 (see issue [:material-github: openai/atari-py#73](https://github.com/openai/atari-py/issues/73)). To get results on these environments, we use `gym==0.23.1 ale-py==0.7.4 "AutoROM[accept-rom-license]==0.4.2` and [manually register `SurroundNoFrameskip-v4` in our fork](https://github.com/vwxyzjn/baselines/blob/e2cb1c938a62fa8d7fe98187246cde08dfd57bd1/baselines/common/register_all_atari_envs.py#L2). 


Median Human Normalized Score (HNS) compared to SEEDRL's R2D2 (data available [here](https://github.com/google-research/seed_rl/blob/66e8890261f09d0355e8bf5f1c5e41968ca9f02b/docs/seed_r2d2_atari_graphs.csv)). 



???+ info

    Note the SEEDRL's R2D2's median HNS data does not include learning curves for `Defender` and `Surround` (see [google-research/seed_rl#78](https://github.com/google-research/seed_rl/issues/78)). Also note the SEEDRL's R2D2 uses slightly different Atari preprocessing than our `ppo_atari_envpool_xla_jax.py`, so we may be comparing apples and oranges; however, the results are still informative at the scale of 57 Atari games — we would be at least comparing similar apples.



Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-PPO-JAX-EnvPool-s-XLA-vs-openai-baselins-PPO-part-1---VmlldzoyNjE2ODMz" style="width:100%; height:500px" title="Atari-CleanRL-s-PPO-Envpool-vs-openai-baselines-Part-1"></iframe>

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-PPO-JAX-EnvPool-s-XLA-vs-openai-baselins-PPO-part-2---VmlldzoyNjE2ODM1" style="width:100%; height:500px" title="Atari-CleanRL-s-PPO-Envpool-vs-openai-baselines-Part-2"></iframe>



## `ppo_atari_envpool_xla_jax_scan.py`

The [ppo_atari_envpool_xla_jax_scan.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax_scan.py) has the following features:

* Replaces python loops in `compute_gae`, `update_ppo`, and `rollout` functions of [ppo_atari_envpool_xla_jax.py](/rl-algorithms/ppo/#ppo_atari_envpool_xla_jaxpy) with native `jax.scan`
* Warnings and caveats from [ppo_atari_envpool_xla_jax.py](/rl-algorithms/ppo/#ppo_atari_envpool_xla_jaxpy) also apply here

### Usage

=== "poetry"

    ```bash
    uv pip install ".[envpool, jax]"
    uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    uv run python cleanrl/ppo_atari_envpool_xla_jax_scan.py --help
    uv run python cleanrl/ppo_atari_envpool_xla_jax_scan.py --env-id Breakout-v5
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-envpool.txt
    pip install -r requirements/requirements-jax.txt
    pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    python cleanrl/ppo_atari_envpool_xla_jax_scan.py --help
    python cleanrl/ppo_atari_envpool_xla_jax_scan.py --env-id Breakout-v5
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`. The metrics are the same as those in [ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py).

### Implementation details

[ppo_atari_envpool_xla_jax_scan.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax_scan.py) is a clone of [ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py) that replaces the python loops with native `jax.scan`.

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:


``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:80:85"
```


{!benchmark/ppo_atari_envpool_xla_jax_scan.md!}


Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh:87:96"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_envpool_xla_jax_scan.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_envpool_xla_jax_scan-time.png">

Learning curves:

???+ info

    The training time of this variant and that of [ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py) are very similar but the compilation time is reduced significantly (see [vwxyzjn/cleanrl#328](https://github.com/vwxyzjn/cleanrl/pull/328#issuecomment-1340474894)). Note that the hardware also affects the speed in the learning curve below. Runs from [`costa-huang`](https://github.com/vwxyzjn/) (red) are slower from those of [`51616`](https://github.com/51616/) (blue and orange) because of hardware differences.

    
    


Tracked experiments:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Regression-Report-ppo_atari_envpool_xla_jax_scan--VmlldzozMTk2MzM2" style="width:100%; height:500px" title="Atari-CleanRL-s-PPO-Envpool-Jax-scan"></iframe>


## `ppo_procgen.py`

The [ppo_procgen.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_procgen.py) has the following features:

* For the procgen environments
* Uses IMPALA-style neural network
* Works with the `Discrete` action space


### Usage

=== "poetry"

    ```bash
    uv pip install ".[procgen]"
    uv run python cleanrl/ppo_procgen.py --help
    uv run python cleanrl/ppo_procgen.py --env-id starpilot
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-procgen.txt
    python cleanrl/ppo_procgen.py --help
    python cleanrl/ppo_procgen.py --env-id starpilot
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

### Implementation details

[ppo_procgen.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_procgen.py) is based on the details in "Appendix" in [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/), which are as follows:

1. IMPALA-style Neural Network (:material-github: [common/models.py#L28](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L28))
1. Use the same `gamma` parameter in the `NormalizeReward` wrapper. Note that the original implementation from [openai/train-procgen](https://github.com/openai/train-procgen) uses the default `gamma=0.99` in [the `VecNormalize` wrapper](https://github.com/openai/train-procgen/blob/1a2ae2194a61f76a733a39339530401c024c3ad8/train_procgen/train.py#L43) but `gamma=0.999` as PPO's parameter. The mismatch between the `gamma`s is technically incorrect. See [#209](https://github.com/vwxyzjn/cleanrl/pull/209)

### Experiment results



To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:


``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:91:100"
```

We try to match the default setting in [openai/train-procgen](https://github.com/openai/train-procgen) except that we use the `easy` distribution mode and `total_timesteps=25e6` to save compute. Notice [openai/train-procgen](https://github.com/openai/train-procgen) has the following settings:

1. Learning rate annealing is turned off by default
1. Reward scaling and reward clipping is used


Below are the average episodic returns for `ppo_procgen.py`. To ensure the quality of the implementation, we compared the results against `openai/baselines`' PPO.

| Environment      | `ppo_procgen.py` | `openai/baselines`' PPO (Huang et al., 2022)[^1]
| ----------- | ----------- | ----------- |
| StarPilot (easy)      | 30.99 ± 1.96      | 33.97 ± 7.86  |
| BossFight (easy)   | 8.85 ± 0.33    |  9.35 ± 2.04 |
| BigFish  (easy)  | 16.46 ± 2.71         | 20.06 ± 5.34 |



Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh:98:106"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_procgen.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_procgen-time.png">


???+ info

    Note that we have run the procgen experiments using the `easy` distribution for reducing the computational cost.


Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Procgen-CleanRL-s-PPO--VmlldzoxODcxMzUy" style="width:100%; height:500px" title="Procgen-CleanRL-s-PPO"></iframe>



## `ppo_atari_multigpu.py`

The [ppo_atari_multigpu.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_multigpu.py) leverages data parallelism to speed up training time *at no cost of sample efficiency*. 

`ppo_atari_multigpu.py` has the following features:

* Allows the users to use do training leveraging data parallelism
* For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

???+ warning

    Note that `ppo_atari_multigpu.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. It will error out with `NOTE: Redirects are currently not supported in Windows or MacOs.` See [pytorch/pytorch#20380](https://github.com/pytorch/pytorch/issues/20380)

### Usage


=== "poetry"

    ```bash
    uv pip install ".[atari]"
    uv run python cleanrl/ppo_atari_multigpu.py --help

    # `--nproc_per_node=2` specifies how many subprocesses we spawn for training with data parallelism
    # note it is possible to run this with a *single GPU*: each process will simply share the same GPU
    uv run torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4

    # by default we use the `gloo` backend, but you can use the `nccl` backend for better multi-GPU performance
    uv run torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --backend nccl

    # it is possible to spawn more processes than the amount of GPUs you have via `--device-ids`
    # e.g., the command below spawns two processes using GPU 0 and two processes using GPU 1
    uv run torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --device-ids 0 0 1 1
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/ppo_atari_multigpu.py --help

    # `--nproc_per_node=2` specifies how many subprocesses we spawn for training with data parallelism
    # note it is possible to run this with a *single GPU*: each process will simply share the same GPU
    torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4

    # by default we use the `gloo` backend, but you can use the `nccl` backend for better multi-GPU performance
    torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --backend nccl

    # it is possible to spawn more processes than the amount of GPUs you have via `--device-ids`
    # e.g., the command below spawns two processes using GPU 0 and two processes using GPU 1
    torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --device-ids 0 0 1 1
    ```


### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

### Implementation details

[ppo_atari_multigpu.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_multigpu.py) is based on `ppo_atari.py` (see its [related docs](/rl-algorithms/ppo/#implementation-details_1)).

We use [Pytorch's distributed API](https://pytorch.org/tutorials/intermediate/dist_tuto.html) to implement the data parallelism paradigm. The basic idea is that the user can spawn $N$ processes each running a copy of `ppo_atari.py`,  holding a copy of the model, stepping the environments, and averaging their gradients together for the backward pass. Here are a few note-worthy implementation details.

1. **Local versus global parameters**: All of the parameters in `ppo_atari.py` are global (such as batch size), but in `ppo_atari_multigpu.py` we have local parameters as well. Say we run `torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --local-num-envs=4`; here are how all multi-gpu related parameters are adjusted:
    * **number of environments**: `num_envs = local_num_envs * world_size = 4 * 2 = 8`
    * **batch size**: `local_batch_size = local_num_envs * num_steps = 4 * 128 = 512`, `batch_size = num_envs * num_steps) = 8 * 128 = 1024`
    * **minibatch size**:  `local_minibatch_size = int(args.local_batch_size // args.num_minibatches) = 512 // 4 = 128`, `minibatch_size = int(args.batch_size // args.num_minibatches) = 1024 // 4 = 256`
    * **number of updates**: `num_iterations = args.total_timesteps // args.batch_size = 10000000 // 1024 = 9765`
1. **Adjust seed per process**: we need be very careful with seeding: we could have used the exact same seed for each subprocess. To ensure this does not happen, we do the following

    ```python hl_lines="2 5 16"
    # CRUCIAL: note that we needed to pass a different seed for each data parallelism worker
    args.seed += local_rank
    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed - local_rank)
    torch.backends.cudnn.deterministic = args.torch_deterministic

    # ...

    envs = gym.vector.SyncVectorEnv(
        [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)]
    )
    assert isinstance(envs.single_action_space, gym.spaces.Discrete), "only discrete action space is supported"

    agent = Agent(envs).to(device)
    torch.manual_seed(args.seed)
    optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5)
    ```

    Notice that we adjust the seed with `args.seed += local_rank` (line 2), where `local_rank` is the index of the subprocesses. This ensures we seed packages and envs with uncorrealted seeds. However, we do need to use the same `torch` seed for all process to initialize same weights for the `agent` (line 5), after which we can use a different seed for `torch` (line 16).
1. **Efficient gradient averaging**: PyTorch recommends to average the gradient across the whole world via the following (see [docs](https://pytorch.org/tutorials/intermediate/dist_tuto.html#distributed-training))

    ```python
    for param in agent.parameters():
        dist.all_reduce(param.grad.data, op=dist.ReduceOp.SUM)
        param.grad.data /= world_size
    ```

    However, [@cswinter](https://github.com/cswinter) introduces a more efficient gradient averaging scheme with proper batching (see :material-github: [entity-neural-network/incubator#220](https://github.com/entity-neural-network/incubator/pull/220)), which looks like:

    ```python
    all_grads_list = []
    for param in agent.parameters():
        if param.grad is not None:
            all_grads_list.append(param.grad.view(-1))
    all_grads = torch.cat(all_grads_list)
    dist.all_reduce(all_grads, op=dist.ReduceOp.SUM)
    offset = 0
    for param in agent.parameters():
        if param.grad is not None:
            param.grad.data.copy_(
                all_grads[offset : offset + param.numel()].view_as(param.grad.data) / world_size
            )
            offset += param.numel()
    ```

    In our previous empirical testing (see :material-github: [vwxyzjn/cleanrl#162](https://github.com/vwxyzjn/cleanrl/pull/162#issuecomment-1107909696)), we have found [@cswinter](https://github.com/cswinter)'s implementation to be faster, hence we adopt it in our implementation.




### Experiment results



To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:


``` title="benchmark/ppo.sh" linenums="1"
--8<-- "benchmark/ppo.sh:102:107"
```

Below are the average episodic returns for `ppo_atari_multigpu.py`. To ensure no loss of sample efficiency, we compared the results against `ppo_atari.py`.


{!benchmark/ppo_atari_multigpu.md!}


Learning curves:

``` title="benchmark/ppo_plot.sh" linenums="1"
--8<-- "benchmark/ppo_plot.sh:108:117"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_multigpu.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/ppo_atari_multigpu-time.png">




Under the same hardware, we see that `ppo_atari_multigpu.py` is about **30% faster** than `ppo_atari.py` with no loss of sample efficiency. 


???+ info

    The experiments above is to show correctness -- we show that by aligning the same hyperparameters of `ppo_atari.py` and `ppo_atari_multigpu.py`, we can achieve the same sample efficiency. However, we can train even faster by simply running a much larger batch size. For example, we can run `torchrun --standalone --nnodes=1 --nproc_per_node=8 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --local-num-envs=8`, which will run 8 x 8 = 64 environments in parallel and achieve a batch size of 64 x 128 = 8192. This will likely result in a sample efficiency but should increase the wall time efficiency.


???+ info

    Although `ppo_atari_multigpu.py` is 30% faster than `ppo_atari.py`, `ppo_atari_multigpu.py` is still slower than `ppo_atari_envpool.py`, as shown below.  This comparison really highlights the different kinds of optimization possible.


    <div class="grid-container">
        <img src="../ppo/Breakout-a.png">
        <img src="../ppo/Breakout-time-a.png">
    </div>

    The purpose of `ppo_atari_multigpu.py` is not (yet) to achieve the fastest PPO + Atari example. Rather, its purpose is to *rigorously validate data parallelism does provide performance benefits*. We could do something like `ppo_atari_multigpu_envpool.py` to possibly obtain the fastest PPO + Atari possible, but that is for another day. Note we may need `numba` to pin the threads `envpool` is using in each subprocess to avoid threads fighting each other and lowering the throughput.


Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-PPO-MultiGPU--VmlldzoxOTM2NDUx" style="width:100%; height:500px" title="Atari-CleanRL-s-PPO"></iframe>





## `ppo_pettingzoo_ma_atari.py`
[ppo_pettingzoo_ma_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_pettingzoo_ma_atari.py) trains an agent to learn playing Atari games via selfplay. The selfplay environment is implemented as a vectorized environment from [PettingZoo.ml](https://www.pettingzoo.ml/atari). The basic idea is to create vectorized environment $E$ with `num_envs = N`, where $N$ is the number of players in the game. Say $N = 2$, then the 0-th sub environment of $E$ will return the observation for player 0 and 1-th sub environment will return the observation of player 1. Then the two environments takes a batch of 2 actions and execute them for player 0 and player 1, respectively. See "Vectorized architecture" in [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/) for more detail.

`ppo_pettingzoo_ma_atari.py` has the following features:

* For playing the pettingzoo's multi-agent Atari game.
* Works with the pixel-based observation space
* Works with the `Box` action space

???+ warning

    Note that `ppo_pettingzoo_ma_atari.py` does not work in Windows :fontawesome-brands-windows:. See [https://pypi.org/project/multi-agent-ale-py/#files](https://pypi.org/project/multi-agent-ale-py/#files)

### Usage

=== "poetry"

    ```bash
    uv pip install ".[pettingzoo, atari]"
    uv run AutoROM --accept-license
    uv run  cleanrl/ppo_pettingzoo_ma_atari.py --help
    uv run  cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3
    uv run  cleanrl/ppo_pettingzoo_ma_atari.py --env-id surround_v2
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-pettingzoo.txt
    pip install -r requirements/requirements-atari.txt
    AutoROM --accept-license
    python cleanrl/ppo_pettingzoo_ma_atari.py --help
    python cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3
    python cleanrl/ppo_pettingzoo_ma_atari.py --env-id surround_v2
    ```

See [https://www.pettingzoo.ml/atari](https://www.pettingzoo.ml/atari) for a full-list of supported environments such as `basketball_pong_v3`. Notice pettingzoo sometimes introduces breaking changes, so make sure to install the pinned dependencies via `poetry`.

### Explanation of the logged metrics

Additionally, it logs the following metrics

* `charts/episodic_return-player0`: episodic return of the game for player 0
* `charts/episodic_return-player1`: episodic return of the game for player 1
* `charts/episodic_length-player0`: episodic length of the game for player 0
* `charts/episodic_length-player1`: episodic length of the game for player 1

See other logged metrics in the [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

### Implementation details

[ppo_pettingzoo_ma_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_pettingzoo_ma_atari.py) is based on `ppo_atari.py` (see its [related docs](/rl-algorithms/ppo/#implementation-details_1)).

`ppo_pettingzoo_ma_atari.py` additionally has the following implementation details:

1. **`supersuit` wrappers**:  uses preprocessing wrappers from `supersuit` instead of from `stable_baselines3`, which looks like the following. In particular note that the `supersuit` does not offer a wrapper similar to `NoopResetEnv`, and that it uses the `agent_indicator_v0` to add two channels indicating the which player the agent controls.

    ```diff
    -env = gym.make(env_id)
    -env = NoopResetEnv(env, noop_max=30)
    -env = MaxAndSkipEnv(env, skip=4)
    -env = EpisodicLifeEnv(env)
    -if "FIRE" in env.unwrapped.get_action_meanings():
    -    env = FireResetEnv(env)
    -env = ClipRewardEnv(env)
    -env = gym.wrappers.ResizeObservation(env, (84, 84))
    -env = gym.wrappers.GrayScaleObservation(env)
    -env = gym.wrappers.FrameStack(env, 4)
    +env = importlib.import_module(f"pettingzoo.atari.{args.env_id}").parallel_env()
    +env = ss.max_observation_v0(env, 2)
    +env = ss.frame_skip_v0(env, 4)
    +env = ss.clip_reward_v0(env, lower_bound=-1, upper_bound=1)
    +env = ss.color_reduction_v0(env, mode="B")
    +env = ss.resize_v1(env, x_size=84, y_size=84)
    +env = ss.frame_stack_v1(env, 4)
    +env = ss.agent_indicator_v0(env, type_only=False)
    +env = ss.pettingzoo_env_to_vec_env_v1(env)
    +envs = ss.concat_vec_envs_v1(env, args.num_envs // 2, num_cpus=0, base_class="gym")
    ```
1. **A more detailed note on the `agent_indicator_v0` wrapper**: let's dig deeper into how `agent_indicator_v0` works. We do `print(envs.reset(), envs.reset().shape)`
    ```python
    [  0.,   0.,   0., 236.,   1,   0.]],

    [[  0.,   0.,   0., 236.,   0.,   1.],
    [  0.,   0.,   0., 236.,   0.,   1.],
    [  0.,   0.,   0., 236.,   0.,   1.],
    ...,
    [  0.,   0.,   0., 236.,   0.,   1.],
    [  0.,   0.,   0., 236.,   0.,   1.],
    [  0.,   0.,   0., 236.,   0.,   1.]]]]) torch.Size([16, 84, 84, 6])
    ```
    
    So the `agent_indicator_v0` adds the last two columns, where `[  0.,   0.,   0., 236.,   1,   0.]]` means this observation is for player 0, and `[  0.,   0.,   0., 236.,   0.,   1.]` is for player 1. Notice the observation still has the range of $[0, 255]$ but the agent indicator channel has the range of $[0,1]$, so we need to be careful when dividing the observation by 255. In particular, we would only divide the first four channels by 255 and leave the agent indicator channels untouched as follows:

    ```py
    def get_action_and_value(self, x, action=None):
        x = x.clone()
        x[:, :, :, [0, 1, 2, 3]] /= 255.0
        hidden = self.network(x.permute((0, 3, 1, 2)))
    ```


### Experiment results



To run benchmark experiments, see :material-github: [benchmark/ppo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/ppo.sh). Specifically, execute the following command:

<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Fppo.sh%23L53-L59&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>

???+ info

    Note that evaluation is usually tricker in in selfplay environments. The usual episodic return is not a good indicator of the agent's performance in zero-sum games because the episodic return converges to zero. To evaluate the agent's ability, an intuitive approach is to take a look at the videos of the agents playing the game (included below), visually inspect the agent's behavior. The best scheme, however, is rating systems like [Trueskill](https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/) or [ELO scores](https://en.wikipedia.org/wiki/Elo_rating_system). However, they are more difficult to implement and are outside the scode of `ppo_pettingzoo_ma_atari.py`. 
    
    
    For simplicity, we measure the **episodic length** instead, which in a sense measures how many "back and forth" the agent can create. In other words, the longer the agent can play the game, the better the agent can play. Empirically, we have found episodic length to be a good indicator of the agent's skill, especially in `pong_v3` and `surround_v2`. However, it is not the case for `tennis_v3` and we'd need to visually inspect the agents' game play videos.


Below are the average **episodic length** for `ppo_pettingzoo_ma_atari.py`. To ensure no loss of sample efficiency, we compared the results against `ppo_atari.py`.

| Environment      | `ppo_pettingzoo_ma_atari.py`  | 
| ----------- | ----------- | 
| pong_v3 | 4153.60 ± 190.80      | 
| surround_v2 | 3055.33 ± 223.68  | 
| tennis_v3 | 14538.02 ± 7005.54   | 


Learning curves:

<div class="grid-container">
<img src="../ppo/pong_v3.png">

<img src="../ppo/surround_v2.png">

<img src="../ppo/tennis_v3.png">
</div>



Tracked experiments and game play videos:

<iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Pettingzoo-s-Multi-agent-Atari-CleanRL-s-PPO--VmlldzoyMDkxNTE5" style="width:100%; height:500px" title="Atari-CleanRL-s-PPO"></iframe>



{!rl-algorithms/ppo-isaacgymenvs.md!}

[^1]: Huang, Shengyi; Dossa, Rousslan Fernand Julien; Raffin, Antonin; Kanervisto, Anssi; Wang, Weixun (2022). The 37 Implementation Details of Proximal Policy Optimization. ICLR 2022 Blog Track https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/

[^2]: Andrychowicz, Marcin, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot et al. "What matters in on-policy reinforcement learning? a large-scale empirical study." International Conference on Learning Representations 2021, https://openreview.net/forum?id=nIAxjsniDzg

---

### Rl Algorithms Pqn (`docs/rl-algorithms/pqn.md`)

# Parallel Q Network (PQN)


## Overview

PQN is a parallelized version of the Deep Q-learning algorithm. It is designed to be more efficient than DQN by using multiple agents to interact with the environment in parallel. PQN can be thought of as DQN (1) without replay buffer and target networks, and (2) with layer normalizations and parallel environments.

Original paper: 

* [Simplifying Deep Temporal Difference Learning](https://arxiv.org/html/2407.04811v2)

Reference resources:

* :material-github: [purejaxql](https://github.com/mttga/purejaxql)

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`pqn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/pqn.py), :material-file-document: [docs](/rl-algorithms/pqn/#pqnpy) | For classic control tasks like `CartPole-v1`. |
| :material-github: [`pqn_atari_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/pqn_atari_envpool.py), :material-file-document: [docs](/rl-algorithms/pqn/#pqn_atari_envpoolpy) |  For Atari games. Uses the blazing fast Envpool Atari vectorized environment. It uses convolutional layers and common atari-based pre-processing techniques. |
| :material-github: [`pqn_atari_envpool_lstm.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/pqn_atari_envpool_lstm.py), :material-file-document: [docs](/rl-algorithms/pqn/#pqn_atari_envpool_lstmpy) | For Atari games. Uses the blazing fast Envpool Atari vectorized environment. Using LSTM without stacked frames. |

Below are our single-file implementations of PQN:

## `pqn.py`

The [pqn.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/pqn.py) has the following features:

* Works with the `Box` observation space of low-level features
* Works with the `Discrete` action space
* Works with envs like `CartPole-v1`

### Usage

=== "poetry"

    ```bash
    poetry install
    uv run python cleanrl/pqn.py --help
    uv run python cleanrl/pqn.py --env-id CartPole-v1
    ```

=== "pip"

    ```bash
    python cleanrl/pqn.py --help
    python cleanrl/pqn.py --env-id CartPole-v1
    ```

### Explanation of the logged metrics

Running `python cleanrl/pqn.py` will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

* `charts/episodic_return`: episodic return of the game
* `charts/episodic_length`: episodic length of the game
* `charts/SPS`: number of steps per second
* `charts/learning_rate`: the current learning rate
* `losses/td_loss`: the mean squared error (MSE) between the Q values at timestep $t$ and the Bellman update target estimated using the $Q(\lambda)$ returns.
* `losses/q_values`: it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimation happens.

### Implementation details

1. Vectorized architecture (:material-github: [common/cmd_util.py#L22](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/cmd_util.py#L22))
2. Orthogonal Initialization of Weights and Constant Initialization of biases (:material-github: [a2c/utils.py#L58)](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L58))
3. Normalized Q Network (:material-github: [purejaxql/pqn_atari.py#L200](https://github.com/mttga/purejaxql/blob/2205ae5308134d2cedccd749074bff2871832dc8/purejaxql/pqn_atari.py#L200))
4. Uses the RAdam Optimizer with the default epsilon parameter(:material-github: [purejaxql/pqn_atari.py#L362](https://github.com/mttga/purejaxql/blob/2205ae5308134d2cedccd749074bff2871832dc8/purejaxql/pqn_atari.py#L362))
5. Adam Learning Rate Annealing (:material-github: [pqn2/pqn2.py#L133-L135](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/pqn2/pqn2.py#L133-L135))
6. Q Lambda Returns (:material-github: [purejaxql/pqn_atari.py#L446](https://github.com/mttga/purejaxql/blob/2205ae5308134d2cedccd749074bff2871832dc8/purejaxql/pqn_atari.py#L446))
7. Mini-batch Updates (:material-github: [pqn2/pqn2.py#L157-L166](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/pqn2/pqn2.py#L157-L166))
8. Global Gradient Clipping (:material-github: [purejaxql/pqn_atari.py#L360](https://github.com/mttga/purejaxql/blob/2205ae5308134d2cedccd749074bff2871832dc8/purejaxql/pqn_atari.py#L360))

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/pqn.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/pqn.sh). Specifically, execute the following command:

``` title="benchmark/pqn.sh" linenums="1"
--8<-- "benchmark/pqn.sh:0:6"
```

Episode Rewards:

|                | CleanRL PQN    |
|:---------------|:---------------|
| CartPole-v1    | 495.13 ± 6.89  |
| Acrobot-v1     | -95.63 ± 5.73  |
| MountainCar-v0 | -200.00 ± 0.00 |

Runtime:

|                |   CleanRL PQN |
|:---------------|--------------:|
| CartPole-v1    |      0.833548 |
| Acrobot-v1     |      1.35797  |
| MountainCar-v0 |      1.02083  |

Learning curves:

``` title="benchmark/pqn_plot.sh" linenums="1"
--8<-- "benchmark/pqn_plot.sh:1:9"
```

<img src="../pqn/pqn_state.png">

Tracked experiments: 

<iframe src="https://api.wandb.ai/links/rogercreus/9s50xw0j" style="width:100%; height:500px" title="CleanRL-s-PPO-TrXL"></iframe>

## `pqn_atari_envpool.py`

The [pqn_atari_envpool.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/pqn_atari_envpool.py) has the following features:

* Uses the blazing fast [Envpool](https://github.com/sail-sg/envpool) vectorized environment.
* For Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

???+ warning

    Note that `pqn_atari_envpool.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

    EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PQN implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

    * Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
    * Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

    This causes the $s_{last}$ to be off by one. 
    See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.


### Usage

=== "poetry"

    ```bash
    uv pip install ".[envpool]"
    uv run python cleanrl/pqn_atari_envpool.py --help
    uv run python cleanrl/pqn_atari_envpool.py --env-id Breakout-v5
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-envpool.txt
    python cleanrl/pqn_atari_envpool.py --help
    python cleanrl/pqn_atari_envpool.py --env-id Breakout-v5
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/pqn/#explanation-of-the-logged-metrics) for `pqn.py`.

### Implementation details

[pqn_atari_envpool.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/pqn_atari_envpool.py) uses a customized `RecordEpisodeStatistics` to work with envpool but has the same other implementation details as `ppo_atari.py`.

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/pqn.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/pqn.sh). Specifically, execute the following command:

``` title="benchmark/pqn.sh" linenums="1"
--8<-- "benchmark/pqn.sh:12:17"
```

Episode Rewards:

|                  | CleanRL PQN       |
|:-----------------|:------------------|
| Breakout-v5      | 384.85 ± 12.39    |
| SpaceInvaders-v5 | 1325.20 ± 78.49   |
| BeamRider-v5     | 5753.03 ± 2394.70 |
| Pong-v5          | 20.49 ± 0.11      |
| MsPacman-v5      | 2298.83 ± 128.24  |

Runtime:

|                  |   CleanRL PQN |
|:-----------------|--------------:|
| Breakout-v5      |       42.8203 |
| SpaceInvaders-v5 |       41.2196 |
| BeamRider-v5     |       43.0951 |
| Pong-v5          |       40.7316 |
| MsPacman-v5      |       43.7812 |


Learning curves:

``` title="benchmark/pqn_plot.sh" linenums="1"
--8<-- "benchmark/pqn_plot.sh:11:29"
```

<img src="../pqn/pqn.png">

Tracked experiments: 

<iframe src="https://wandb.ai/rogercreus/cleanRL/reports/PQN-PR-472--Vmlldzo5ODg1NTkx" style="width:100%; height:500px" title="CleanRL-s-PPO-TrXL"></iframe>

## `pqn_atari_envpool_lstm.py`

The [pqn_atari_envpool_lstm.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/pqn_atari_envpool_lstm.py) has the following features:

* Uses the blazing fast [Envpool](https://github.com/sail-sg/envpool) vectorized environment.
* For Atari games using LSTM without stacked frames. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

???+ warning

    Note that `pqn_atari_envpool.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

    EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PQN implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

    * Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
    * Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

    This causes the $s_{last}$ to be off by one. 
    See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.

### Usage


=== "poetry"

    ```bash
    uv pip install ".[atari]"
    uv run python cleanrl/pqn_atari_envpool_lstm.py --help
    uv run python cleanrl/pqn_atari_envpool_lstm.py --env-id Breakout-v5
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/pqn_atari_envpool_lstm.py --help
    python cleanrl/pqn_atari_envpool_lstm.py --env-id Breakout-v5
    ```


### Explanation of the logged metrics

See [related docs](/rl-algorithms/pqn/#explanation-of-the-logged-metrics) for `pqn.py`.

### Implementation details

[pqn_atari_envpool_lstm.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/pqn_atari_envpool_lstm.py) is based on the "5 LSTM implementation details" in [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/), which are as follows:

1. Layer initialization for LSTM layers (:material-github: [a2c/utils.py#L84-L86](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L84-L86))
2. Initialize the LSTM states to be zeros (:material-github: [common/models.py#L179](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L179))
3. Reset LSTM states at the end of the episode (:material-github: [common/models.py#L141](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L141))
4. Prepare sequential rollouts in mini-batches (:material-github: [a2c/utils.py#L81](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L81))
5. Reconstruct LSTM states during training (:material-github: [a2c/utils.py#L81](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/a2c/utils.py#L81))

To help test out the memory, we remove the 4 stacked frames from the observation (i.e., using `env = gym.wrappers.FrameStack(env, 1)` instead of `env = gym.wrappers.FrameStack(env, 4)` like in `ppo_atari.py` )

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/pqn.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/pqn.sh). Specifically, execute the following command:

``` title="benchmark/pqn.sh" linenums="1"
--8<-- "benchmark/pqn.sh:23:28"
```


Episode Rewards:

|                  | CleanRL PQN       |
|:-----------------|:------------------|
| Breakout-v5      | 400.35 ± 9.08     |
| SpaceInvaders-v5 | 813.47 ± 58.14    |
| BeamRider-v5     | 11161.43 ± 579.88 |
| Pong-v5          | 20.43 ± 0.11      |
| MsPacman-v5      | 1649.63 ± 135.80  |

Runtime:

|                  |   CleanRL PQN |
|:-----------------|--------------:|
| Breakout-v5      |       178.144 |
| SpaceInvaders-v5 |       209.603 |
| BeamRider-v5     |       174.153 |
| Pong-v5          |       160.462 |
| MsPacman-v5      |       162.222 |
Learning curves:

``` title="benchmark/pqn_plot.sh" linenums="1"
--8<-- "benchmark/pqn_plot.sh:32:50"
```

<img src="../pqn/pqn_lstm.png">

Tracked experiments: 

<iframe src="https://wandb.ai/rogercreus/cleanRL/reports/PQN-PR-472--Vmlldzo5ODg1NTkx" style="width:100%; height:500px" title="CleanRL-s-PPO-TrXL"></iframe>

---

### Rl Algorithms Qdagger (`docs/rl-algorithms/qdagger.md`)

# QDagger

## Overview

QDagger is an extension of the DQN algorithm that uses previously computed results, like teacher policy and teacher replay buffer, to help train student policy. This method eliminates the need for learning from scratch, improving sample efficiency and reducing computational effort in training new policy.

Original paper:

* [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://arxiv.org/abs/2206.01626)

Reference resources:

* :material-github: [google-research/reincarnating_rl](https://github.com/google-research/reincarnating_rl)
* [Original Paper's Website](https://agarwl.github.io/reincarnating_rl/)

## Implemented Variants

| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`qdagger_dqn_atari_impalacnn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_impalacnn.py), :material-file-document: [docs](/rl-algorithms/qdagger/#qdagger_dqn_atari_impalacnnpy) | For playing Atari games. It uses Impala-CNN from RainbowDQN and common atari-based pre-processing techniques. |
| :material-github: [`qdagger_dqn_atari_jax_impalacnn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_jax_impalacnn.py), :material-file-document: [docs](/rl-algorithms/qdagger/#qdagger_dqn_atari_jax_impalacnnpy) | For playing Atari games. It uses Impala-CNN from RainbowDQN and common atari-based pre-processing techniques. |


Below are our single-file implementations of QDagger:


## `qdagger_dqn_atari_impalacnn.py`

The [qdagger_dqn_atari_impalacnn.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_impalacnn.py) has the following features:

* For playing Atari games. It uses Impala-CNN from RainbowDQN and common atari-based pre-processing techniques.
* Its teacher policy uses CleanRL's `dqn_atari` policy from the [huggingface/cleanrl](https://huggingface.co/cleanrl) repository.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

### Usage

```bash
uv pip install ".[atari]"
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4
```

=== "poetry"

    ```bash
    uv pip install ".[atari]"
    uv run python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
    uv run python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
    python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4
    ```


### Explanation of the logged metrics

Running `python cleanrl/qdagger_dqn_atari_impalacnn.py` will automatically record various metrics such as value or distillation losses in Tensorboard. Below is the documentation for these metrics:

* `charts/episodic_return`: episodic return of the game
* `charts/SPS`: number of steps per second
* `losses/td_loss`: the mean squared error (MSE) between the Q values at timestep $t$ and the Bellman update target estimated using the reward $r_t$ and the Q values at timestep $t+1$, thus minimizing the *one-step* temporal difference. Formally, it can be expressed by the equation below.
$$
    J(\theta^{Q}) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \big[ (Q(s, a) - y)^2 \big],
$$
with the Bellman update target is $y = r + \gamma \, Q^{'}(s', a')$ and the replay buffer is $\mathcal{D}$.
* `losses/q_values`: implemented as `qf1(data.observations, data.actions).view(-1)`, it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimation happens.
* `losses/distill_loss`: the distillation loss, which is the KL divergence between the teacher policy $\pi_T$ and the student policy $\pi$. Formally, it can be expressed by the equation below.
$$
    L_{\text{distill}} = \lambda_t \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[ \sum_a \pi_T(a|s)\log\pi(a|s)\right]
$$
* `Charts/distill_coeff`: the coefficient $\lambda_t$ for the distillation loss, which is a function of the ratio between the teacher policy $\pi_T$ and the student policy $\pi$. Formally, it can be expressed by the equation below.
$$
\lambda_t = 1_{t<t_0}\max(1 - G^\pi/G^{\pi_T}, 0)
$$
* `losses/loss`: the total loss, which is the sum of the TD loss and the distillation loss.
$$
    L_{\text{qdagger}} = J(\theta^{Q}) + L_{\text{distill}}
$$
* `charts/teacher/avg_episodic_return`: average episodic return of teacher policy evaluation
* `charts/offline/avg_episodic_return`: average episodic return of policy evaluation in offline training phase


### Implementation details

`qdagger_dqn_atari_impalacnn.py` is based on (Agarwal et al., 2022)[^1] but presents a few implementation differences:

* (Agarwal et al., 2022)[^1] uses the teacher replay buffer data saved during the training of the teacher policy, but our teacher policy, which uses CleanRL's `dqn_atari` from the [huggingface/cleanrl](https://huggingface.co/cleanrl), does not contain replay buffer data, so we populate the teacher replay buffer with the teacher policy before training. For more details, see section A.5 "Additional ablations for QDagger" in the original paper.
* (Agarwal et al., 2022)[^1] uses DQN (Adam) @ 400M frames, but we use CleanRL's `dqn_atari`, which is DQN (Adam) @ 10M steps(40M frames).
* We have used an old set of Atari preprocessing techniques that doesn't use sticky action, but the original paper does.

### Experiment results

Below are the average episodic returns for `qdagger_dqn_atari_impalacnn.py`. 


| Environment | `qdagger_dqn_atari_impalacnn.py` 10M steps(40M frames) | (Agarwal et al., 2022)[^1] 10M frames |
| ----------- | ----------- | ----------- |
| BreakoutNoFrameskip-v4 | 295.55 ± 12.30 | 275.15 ± 20.65 |
| PongNoFrameskip-v4  | 19.72 ± 0.20 | - |
| BeamRiderNoFrameskip-v4 | 9284.99 ± 242.28 | 6514.25 ± 411.10 |


Learning curves:

<div class="grid-container">
<img src="../qdagger/BreakoutNoFrameskip-v4.png">

<img src="../qdagger/PongNoFrameskip-v4.png">

<img src="../qdagger/BeamRiderNoFrameskip-v4.png">
</div>

Learning curve comparison with `dqn_atari`:

<img src="../qdagger/compare.png">

Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/Atari-CleanRL-s-Qdagger--Vmlldzo0NTg1ODY5" style="width:100%; height:500px" title="CleanRL DQN + Atari Tracked Experiments"></iframe>


## `qdagger_dqn_atari_jax_impalacnn.py`


The [qdagger_dqn_atari_jax_impalacnn.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_jax_impalacnn.py) has the following features:

* Uses [Jax](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Optax](https://github.com/deepmind/optax) instead of `torch`.  [qdagger_dqn_atari_jax_impalacnn.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_jax_impalacnn.py) is roughly 25%-50% faster than  [qdagger_dqn_atari_impalacnn.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/qdagger_dqn_atari_impalacnn.py)
* For playing Atari games. It uses Impala-CNN from RainbowDQN and common atari-based pre-processing techniques.
* Its teacher policy uses CleanRL's `dqn_atari_jax` policy from the [huggingface/cleanrl](https://huggingface.co/cleanrl) repository.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

### Usage


=== "poetry"

    ```bash
    uv pip install ".[atari, jax]"
    uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    uv run python cleanrl/qdagger_dqn_atari_jax_impalacnn.py --env-id BreakoutNoFrameskip-v4
    uv run python cleanrl/qdagger_dqn_atari_jax_impalacnn.py --env-id PongNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    pip install -r requirements/requirements-jax.txt
    pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
    python cleanrl/qdagger_dqn_atari_jax_impalacnn.py --env-id BreakoutNoFrameskip-v4
    python cleanrl/qdagger_dqn_atari_jax_impalacnn.py --env-id PongNoFrameskip-v4
    ```


???+ warning

    Note that JAX does not work in Windows :fontawesome-brands-windows:. The official [docs](https://github.com/google/jax#installation) recommends using Windows Subsystem for Linux (WSL) to install JAX.

### Explanation of the logged metrics

See [related docs](/rl-algorithms/qdagger/#explanation-of-the-logged-metrics) for `qdagger_dqn_atari_impalacnn.py`.

### Implementation details

See [related docs](/rl-algorithms/qdagger/#implementation-details) for `qdagger_dqn_atari_impalacnn.py`.

### Experiment results

Below are the average episodic returns for `qdagger_dqn_atari_jax_impalacnn.py`.


| Environment | `qdagger_dqn_atari_jax_impalacnn.py` 10M steps(40M frames) | (Agarwal et al., 2022)[^1] 10M frames |
| ----------- | ----------- | ----------- |
| BreakoutNoFrameskip-v4 | 335.08 ± 19.12 | 275.15 ± 20.65 |
| PongNoFrameskip-v4  | 18.75 ± 0.19 | - |
| BeamRiderNoFrameskip-v4 | 8024.75 ± 579.02 | 6514.25 ± 411.10 |


Learning curves:

<div class="grid-container">
<img src="../qdagger/jax/BreakoutNoFrameskip-v4.png">

<img src="../qdagger/jax/PongNoFrameskip-v4.png">

<img src="../qdagger/jax/BeamRiderNoFrameskip-v4.png">
</div>

Learning curve comparison with `dqn_atari_jax`:

<img src="../qdagger/jax/compare.png">


[^1]:Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. “Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress.” arXiv, October 4, 2022. http://arxiv.org/abs/2206.01626.

---

### Rl Algorithms Rainbow (`docs/rl-algorithms/rainbow.md`)

# Rainbow

## Overview

The Rainbow algorithm is an extension of DQN that combines multiple improvements:

* Prioritized Experience Replay
* Dueling Network Architecture
* Noisy Networks
* Distributional Q-Learning
* N-step Learning
* Double Q-Learning

Original papers: 

* [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://arxiv.org/abs/1710.02298)

Reference resources:

* :material-github: [Dopamine](https://github.com/google/dopamine)

* :material-github: [Kaixhin](https://github.com/Kaixhin/Rainbow)

## Implemented Variants

| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`rainbow_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/rainbow_atari.py), :material-file-document: [docs](/rl-algorithms/rainbow/#rainbow_ataripy) |  For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques. |


## `rainbow_atari.py`

The [rainbow_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/rainbow_atari.py) has the following features:

* For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
* Works with the Atari's pixel `Box` observation space of shape `(210, 160, 3)`
* Works with the `Discrete` action space

### Usage

```bash
poetry install -E atari
python cleanrl/rainbow_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/rainbow_atari.py --env-id PongNoFrameskip-v4
```

=== "poetry"

    ```bash
    poetry install -E atari
    poetry run python cleanrl/rainbow_atari.py --env-id BreakoutNoFrameskip-v4
    poetry run python cleanrl/rainbow_atari.py --env-id PongNoFrameskip-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/rainbow_atari.py --env-id BreakoutNoFrameskip-v4
    python cleanrl/rainbow_atari.py --env-id PongNoFrameskip-v4
    ```


### Explanation of the logged metrics

Running `python cleanrl/rainbow_atari.py` will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

* `charts/episodic_return`: episodic return of the game
* `charts/SPS`: number of steps per second
* `losses/td_loss`: the n-step distributional TD loss
* `losses/q_values`: the mean Q values of the sampled data in the replay buffer
* `charts/beta`: the beta value of the prioritized experience replay

### Implementation details

[rainbow_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/rainbow_atari.py) is based on (Hessel et al., 2018)[^1], and uses the same hyperparameters as (Hessel et al., 2018)[^1]. See Table 1 in (Hessel et al., 2018)[^1] for the hyperparameters. However, there are a few implementation differences:

1. `rainbow_atari.py` uses the more popular Adam Optimizer with the `--learning-rate=0.0000625` as follows:
    ```python
    optim.Adam(q_network.parameters(), lr=0.0000625)
    ```
    whereas (Hessel et al., 2018)[^1] uses the RMSProp optimizer with `--learning-rate=0.0000625`, gradient momentum `0.95`, squared gradient momentum `0.95`, and min squared gradient `0.01` as follows:
    ```python
    optim.RMSprop(
        q_network.parameters(),
        lr=2.5e-4,
        momentum=0.95,
        # ... PyTorch's RMSprop does not directly support
        # squared gradient momentum and min squared gradient
        # so we are not sure what to put here.
    )
    ``` 

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/rainbow.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/rainbow.sh). Specifically, execute the following command:

``` title="benchmark/rainbow.sh" linenums="1"
--8<-- "benchmark/rainbow.sh:0:6"
```

Below are the average episodic returns for `rainbow_atari.py`. 

|                             | Rainbow   | C51   | DQN   |
|:----------------------------|:----------------------------------------|:------------------------------------|:------------------------------------|
| AlienNoFrameskip-v4         | 2907.03 ± 355.53                        | 1831.00 ± 98.23                     | 1275.77 ± 65.41                     |
| AssaultNoFrameskip-v4       | 7661.11 ± 226.51                        | 3322.54 ± 94.46                     | 3845.70 ± 443.31                    |
| GopherNoFrameskip-v4        | 8111.07 ± 300.60                        | 8715.60 ± 492.23                    | 10415.53 ± 3438.12                  |
| YarsRevengeNoFrameskip-v4   | 63536.39 ± 5432.22                      | 11010.99 ± 904.27                   | 15290.12 ± 8010.56                  |
| SpaceInvadersNoFrameskip-v4 | 1835.52 ± 205.10                        | 2009.05 ± 226.96                    | 1441.68 ± 23.92                     |
| MsPacmanNoFrameskip-v4      | 3113.30 ± 393.00                        | 2445.13 ± 30.16                     | 2109.43 ± 49.85                     |


Learning curves:

Rainbow shows better performance than C51 and DQN.
<div class="grid-container">
<img src="../rainbow/rainbow_env_curves.png">
</div>

Rainbow is also more sample efficient than C51 and DQN.

<div class="grid-container">
<img src="../rainbow/rainbow_sample_eff.png">
</div>

Rainbow obtains better aggregated performance than C51 and DQN.
<div class="grid-container">
<img src="../rainbow/rainbow_c51_dqn_bars.png">
</div>




[^1]: Hessel, M., Modayil, J., Hasselt, H.V., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.G., & Silver, D. (2018). Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI.

---

### Rl Algorithms Rpo (`docs/rl-algorithms/rpo.md`)

# Robust Policy Optimization (RPO)

## Overview

RPO leverages a method of perturbing the distribution representing actions. The goal is to encourage high-entropy actions and provide a better representation of the action space. The method consists of a simple modification on top of the objective of the PPO algorithm. In the RPO algorithm, the mean of the action distribution is perturbed using a random number drawn from a Uniform distribution.

Original paper: 

* [Robust Policy Optimization in Deep Reinforcement Learning](https://arxiv.org/abs/2212.07536)

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`rpo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/rpo_continuous_action.py), :material-file-document: [docs](/rl-algorithms/rpo/#rpo_continuous_actionpy) | For classic control tasks like Gym `Pendulum-v1`, and dm_control. |

Below are our single-file implementations of RPO:

## `rpo_continuous_action.py`

`rpo_continuous_action.py` works with Gym (Gymnasium), dm_control, Mujoco environments with continuous action and vector observations.

The [rpo_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/rpo_continuous_action.py) has the following features (similar to [ppo_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py)):

* For continuous action space. Also implemented Mujoco-specific code-level optimizations
* Works with the `Box` observation space of low-level features
* Works with the `Box` (continuous) action space
* adding experimental support for [Gymnasium](https://gymnasium.farama.org/)
* 🧪 support `dm_control` environments via [Shimmy](https://github.com/Farama-Foundation/Shimmy)

### Usage

```bash
# mujoco v4 environments
uv pip install ".[mujoco]"
python cleanrl/rpo_continuous_action.py --help
python cleanrl/rpo_continuous_action.py --env-id Walker2d-v4
# NOTE: we recommend using --rpo-alpha 0.01 for Ant Hopper InvertedDoublePendulum Reacher Pusher
python cleanrl/rpo_continuous_action.py --env-id Ant-v4 --rpo-alpha 0.01
# dm_control v4 environments
uv pip install ".[mujoco, dm_control]"
python cleanrl/rpo_continuous_action.py --env-id dm_control/cartpole-balance-v0
# BipedalWalker-v3 experiment (hack)
uv pip install .
uv pip install box2d-py==2.3.5
python cleanrl/rpo_continuous_action.py --env-id BipedalWalker-v3
```


=== "poetry"

    ```bash
    # mujoco v4 environments
    uv pip install ".[mujoco]"
    python cleanrl/rpo_continuous_action.py --help
    python cleanrl/rpo_continuous_action.py --env-id Hopper-v4
    # NOTE: we recommend using --rpo-alpha 0.01 for Ant Hopper InvertedDoublePendulum Reacher Pusher
    python cleanrl/rpo_continuous_action.py --env-id Ant-v4 --rpo-alpha 0.01
    # dm_control environments
    uv pip install ".[mujoco, dm_control]"
    python cleanrl/rpo_continuous_action.py --env-id dm_control/cartpole-balance-v0
    # BipedalWalker-v3 experiment (hack)
    uv pip install box2d-py==2.3.5
    uv run python cleanrl/rpo_continuous_action.py --env-id BipedalWalker-v3
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-mujoco.txt
    python cleanrl/rpo_continuous_action.py --help
    python cleanrl/rpo_continuous_action.py --env-id Hopper-v4
    # NOTE: we recommend using --rpo-alpha 0.01 for Ant Hopper InvertedDoublePendulum Reacher Pusher
    python cleanrl/rpo_continuous_action.py --env-id Ant-v4 --rpo-alpha 0.01
    pip install -r requirements/requirements-dm_control.txt
    python cleanrl/rpo_continuous_action.py --env-id dm_control/cartpole-balance-v0
    pip install box2d-py==2.3.5
    python cleanrl/rpo_continuous_action.py --env-id BipedalWalker-v3
    ```


### Explanation of the logged metrics

See [related docs](/rl-algorithms/ppo/#explanation-of-the-logged-metrics) for `ppo.py`.

### Implementation details
[rpo_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/rpo_continuous_action.py) has the same implementation details as `ppo_continuous_action.py` (see related [docs](/rl-algorithms/ppo/#ppo_continuous_actionpy)) but with a few lines of code differences.

```python hl_lines="30-34"
class Agent(nn.Module):
    def __init__(self, envs):
        super().__init__()
        self.critic = nn.Sequential(
            layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
            nn.Tanh(),
            layer_init(nn.Linear(64, 64)),
            nn.Tanh(),
            layer_init(nn.Linear(64, 1), std=1.0),
        )
        self.actor_mean = nn.Sequential(
            layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
            nn.Tanh(),
            layer_init(nn.Linear(64, 64)),
            nn.Tanh(),
            layer_init(nn.Linear(64, np.prod(envs.single_action_space.shape)), std=0.01),
        )
        self.actor_logstd = nn.Parameter(torch.zeros(1, np.prod(envs.single_action_space.shape)))

    def get_value(self, x):
        return self.critic(x)

    def get_action_and_value(self, x, action=None):
        action_mean = self.actor_mean(x)
        action_logstd = self.actor_logstd.expand_as(action_mean)
        action_std = torch.exp(action_logstd)
        probs = Normal(action_mean, action_std)
        if action is None:
            action = probs.sample()
        else: # new to RPO
            # sample again to add stochasticity, for the policy update
            z = torch.FloatTensor(action_mean.shape).uniform_(-self.rpo_alpha, self.rpo_alpha)
            action_mean = action_mean + z
            probs = Normal(action_mean, action_std)
        
```

???+ note 

    RPO usages the same PPO-specific hyperparameters. In benchmarking results, we run both algorithms for 8M timesteps.
    RPO has one additional hyperparameter, `rpo_alpha`, which determines the amount of random perturbation on the action mean. 
    We set a default value of `rpo_alpha=0.5` at which RPO is strictly equal to or better than the default PPO in 93% of environments tested (all 48/48 dm_control, 2/2 Gym, 7/11 mujoco_v4. Total 57 out of 61 environments tested.).
    With finetuning `rpo_alpha=0.01` on four mujoco environments, namely, Ant, InvertedDoublePendulum, Reacher, and Pusher, RPO is strictly equal to or better than the default PPO in all tested environments.

### Experiment results

To run benchmark experiments, see  [benchmark/rpo.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/rpo.sh). Specifically, execute the following command:
<script src="https://emgithub.com/embed.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fbenchmark%2Frpo.sh%23L1-L6&style=github&showBorder=on&showLineNumbers=on&showFileMeta=on&showCopy=on"></script>

???+ note "Result tables, learning curves"

    === "dm_control"

        Results on all dm_control environments. The PPO and RPO run for 8M timesteps, and results are computed over 10 random seeds.

        |                                       | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
        |:--------------------------------------|:-------------------------------------------------------------|:----------------------------------------------|
        | dm_control/acrobot-swingup-v0         | 26.87 ± 7.93                                                 | 42.97 ± 2.71                                  |
        | dm_control/acrobot-swingup_sparse-v0  | 1.70 ± 0.88                                                  | 3.31 ± 0.84                                   |
        | dm_control/ball_in_cup-catch-v0       | 935.26 ± 12.57                                               | 939.75 ± 10.18                                |
        | dm_control/cartpole-balance-v0        | 790.36 ± 17.64                                               | 795.12 ± 10.49                                |
        | dm_control/cartpole-balance_sparse-v0 | 986.04 ± 9.93                                                | 988.56 ± 4.28                                 |
        | dm_control/cartpole-swingup-v0        | 590.21 ± 16.72                                               | 613.46 ± 10.91                                |
        | dm_control/cartpole-swingup_sparse-v0 | 240.14 ± 299.93                                              | 525.49 ± 185.96                               |
        | dm_control/cartpole-two_poles-v0      | 216.31 ± 4.03                                                | 218.31 ± 7.30                                 |
        | dm_control/cartpole-three_poles-v0    | 160.03 ± 2.52                                                | 159.97 ± 2.28                                 |
        | dm_control/cheetah-run-v0             | 472.14 ± 99.62                                               | 565.51 ± 58.03                                |
        | dm_control/dog-stand-v0               | 332.06 ± 23.66                                               | 501.22 ± 131.98                               |
        | dm_control/dog-walk-v0                | 124.92 ± 23.13                                               | 166.39 ± 44.65                                |
        | dm_control/dog-trot-v0                | 79.89 ± 12.30                                                | 115.39 ± 29.68                                |
        | dm_control/dog-run-v0                 | 69.07 ± 8.17                                                 | 104.27 ± 24.44                                |
        | dm_control/dog-fetch-v0               | 28.34 ± 4.87                                                 | 43.58 ± 6.88                                  |
        | dm_control/finger-spin-v0             | 630.06 ± 252.99                                              | 848.67 ± 25.21                                |
        | dm_control/finger-turn_easy-v0        | 237.76 ± 78.10                                               | 450.88 ± 133.54                               |
        | dm_control/finger-turn_hard-v0        | 83.76 ± 28.96                                                | 259.99 ± 144.83                               |
        | dm_control/fish-upright-v0            | 559.72 ± 65.79                                               | 803.21 ± 28.36                                |
        | dm_control/fish-swim-v0               | 80.42 ± 9.18                                                 | 140.33 ± 49.10                                |
        | dm_control/hopper-stand-v0            | 3.26 ± 1.65                                                  | 404.39 ± 198.17                               |
        | dm_control/hopper-hop-v0              | 6.48 ± 18.92                                                 | 62.60 ± 87.29                                 |
        | dm_control/humanoid-stand-v0          | 20.76 ± 29.35                                                | 140.43 ± 57.27                                |
        | dm_control/humanoid-walk-v0           | 8.92 ± 18.01                                                 | 77.00 ± 53.00                                 |
        | dm_control/humanoid-run-v0            | 5.44 ± 9.16                                                  | 24.00 ± 19.54                                 |
        | dm_control/humanoid-run_pure_state-v0 | 1.13 ± 0.11                                                  | 3.24 ± 2.41                                   |
        | dm_control/humanoid_CMU-stand-v0      | 4.64 ± 0.37                                                  | 4.32 ± 0.33                                   |
        | dm_control/humanoid_CMU-run-v0        | 0.88 ± 0.09                                                  | 0.80 ± 0.09                                   |
        | dm_control/manipulator-bring_ball-v0  | 0.42 ± 0.14                                                  | 0.44 ± 0.23                                   |
        | dm_control/manipulator-bring_peg-v0   | 0.95 ± 0.43                                                  | 1.07 ± 1.01                                   |
        | dm_control/manipulator-insert_ball-v0 | 41.24 ± 27.27                                                | 43.63 ± 12.77                                 |
        | dm_control/manipulator-insert_peg-v0  | 40.72 ± 15.95                                                | 44.87 ± 26.55                                 |
        | dm_control/pendulum-swingup-v0        | 472.19 ± 385.47                                              | 774.30 ± 21.03                                |
        | dm_control/point_mass-easy-v0         | 534.23 ± 264.35                                              | 653.73 ± 23.14                                |
        | dm_control/point_mass-hard-v0         | 129.75 ± 61.18                                               | 185.81 ± 36.25                                |
        | dm_control/quadruped-walk-v0          | 247.29 ± 90.48                                               | 602.64 ± 223.23                               |
        | dm_control/quadruped-run-v0           | 171.50 ± 37.90                                               | 367.98 ± 117.18                               |
        | dm_control/quadruped-escape-v0        | 23.11 ± 10.48                                                | 68.50 ± 27.81                                 |
        | dm_control/quadruped-fetch-v0         | 183.71 ± 25.14                                               | 216.32 ± 17.44                                |
        | dm_control/reacher-easy-v0            | 773.01 ± 56.70                                               | 716.89 ± 50.07                                |
        | dm_control/reacher-hard-v0            | 637.84 ± 81.15                                               | 576.81 ± 48.25                                |
        | dm_control/stacker-stack_2-v0         | 58.02 ± 11.04                                                | 70.95 ± 16.84                                 |
        | dm_control/stacker-stack_4-v0         | 73.84 ± 14.48                                                | 65.54 ± 18.06                                 |
        | dm_control/swimmer-swimmer6-v0        | 164.22 ± 18.44                                               | 159.60 ± 39.52                                |
        | dm_control/swimmer-swimmer15-v0       | 161.02 ± 24.56                                               | 153.91 ± 28.08                                |
        | dm_control/walker-stand-v0            | 439.24 ± 210.22                                              | 734.74 ± 142.52                               |
        | dm_control/walker-walk-v0             | 305.74 ± 92.15                                               | 787.11 ± 125.97                               |
        | dm_control/walker-run-v0              | 128.18 ± 91.52                                               | 391.56 ± 119.75                               |
                
        Learning curves:
        

        Tracked experiments: 
        
        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-on-dm_control-Part-1--VmlldzozMjU4NTE4" style="width:100%; height:500px" title="dm_control-CleanRL-s-RPO-part-1"></iframe>
        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-on-dm_control-Part-2--VmlldzozMjU4NjYy" style="width:100%; height:500px" title="dm_control-CleanRL-s-RPO-part-2"></iframe>


    
    
    === "MuJoCo v4"

        |                     | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
        |:--------------------|:-------------------------------------------------------------|:----------------------------------------------|
        | HumanoidStandup-v4  | 109325.87 ± 16161.71                                         | 150972.11 ± 6926.19                           |
        | Humanoid-v4         | 583.17 ± 27.88                                               | 799.44 ± 170.85                               |
        | InvertedPendulum-v4 | 888.83 ± 34.66                                               | 879.81 ± 35.52                                |
        | Walker2d-v4         | 2872.92 ± 690.53                                             | 3665.48 ± 278.61                              |

        Learning curves:
        


        Tracked experiments:     

        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-on-Mujoco_v4-Part-1--VmlldzozMjU3ODIz" style="width:100%; height:500px" title="RPO-mujoco-v4-part1"></iframe>

        
        The following environments require tuning of `alpha` (Algorithm 1, line 13, paper: https://arxiv.org/pdf/2212.07536.pdf). As described in the paper, this variable should be tuned for environments tested. A larger value means more randomness, whereas a smaller value indicates less randomness. Some mujoco environments require a smaller `alpha=0.01` value to achieve a reasonable performance compared to `alpha=0.5` for the rest of the environments. This version (`alpha=0.01`) of runs is indicated as `rpo_continuous_action_alpha_0_01` in the table and learning curves.

        |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action_alpha_0_01 ({'tag': ['pr-331']})   |
        |:--------------------------|:-------------------------------------------------------------|:---------------------------------------------------------|
        | Ant-v4                    | 1824.17 ± 905.78                                             | 2702.91 ± 683.53                                         |
        | HalfCheetah-v4            | 2637.19 ± 1068.49                                            | 2716.51 ± 1314.93                                        |
        | Hopper-v4                 | 2741.42 ± 269.11                                             | 2334.22 ± 441.89                                         |
        | InvertedDoublePendulum-v4 | 5626.22 ± 289.23                                             | 5409.03 ± 318.68                                         |
        | Reacher-v4                | -4.65 ± 0.96                                                 | -3.93 ± 0.19                                             |
        | Swimmer-v4                | 124.88 ± 22.24                                               | 129.97 ± 12.02                                           |
        | Pusher-v4                 | -30.35 ± 6.43                                                | -31.48 ± 9.83                                            |

        Learning curves:
        

        Tracked experiments:     

        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-on-Mujoco_v4-Part-2--VmlldzozMjU3OTM4" style="width:100%; height:500px" title="RPO-mujoco-v4-part2"></iframe>
        

        Results with `rpo_alpha=0.5` (not tuned) on the tuned environments:

        |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
        |:--------------------------|:-------------------------------------------------------------|:----------------------------------------------|
        | Ant-v4                    | 1774.42 ± 819.08                                             | -7.99 ± 2.47                                  |
        | HalfCheetah-v4            | 2667.34 ± 1109.99                                            | 2163.57 ± 790.16                              |
        | Hopper-v4                 | 2761.77 ± 286.88                                             | 1557.18 ± 206.74                              |
        | InvertedDoublePendulum-v4 | 5644.00 ± 353.46                                             | 296.97 ± 15.95                                |
        | Reacher-v4                | -4.67 ± 0.88                                                 | -66.35 ± 0.66                                 |
        | Swimmer-v4                | 124.52 ± 22.10                                               | 117.82 ± 10.07                                |
        | Pusher-v4                 | -30.62 ± 6.80                                                | -276.32 ± 26.99                               |

        Learning curves:
        


        Tracked experiments:     

        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-alpha-0-5-on-Mujoco_v4-Part-2--VmlldzozMjU4MTM1" style="width:100%; height:500px" title="RPO-mujoco-v4-0-5-part-2"></iframe>

 
    
    === "MuJoCo v2"

        |                     | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
        |:--------------------|:-------------------------------------------------------------|:----------------------------------------------|
        | HumanoidStandup-v2  | 109118.07 ± 19422.20                                         | 156848.90 ± 11414.50                          |
        | Humanoid-v2         | 588.22 ± 43.80                                               | 717.37 ± 97.18                                |
        | InvertedPendulum-v2 | 867.64 ± 19.97                                               | 866.60 ± 27.06                                |
        | Walker2d-v2         | 3220.99 ± 923.84                                             | 4150.51 ± 348.03                              |

        Learning curves:
        

        Tracked experiments:     

        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-on-Mujoco_v2-Part-1--VmlldzozMjU4Mjc5" style="width:100%; height:500px" title="RPO-mujoco-v2-part1"></iframe>

        The following environments require tuning of `alpha` (Algorithm 1, line 13, paper: https://arxiv.org/pdf/2212.07536.pdf). As described in the paper, this variable should be tuned for environments tested. A larger value means more randomness, whereas a smaller value indicates less randomness. Some mujoco environments require a smaller `alpha=0.01` value to achieve a reasonable performance compared to `alpha=0.5` for the rest of the environments. This version (`alpha=0.01`) of runs is indicated as `rpo_continuous_action_alpha_0_01` in the table and learning curves.

        |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action_alpha_0_01 ({'tag': ['pr-331']})   |
        |:--------------------------|:-------------------------------------------------------------|:---------------------------------------------------------|
        | Ant-v2                    | 2412.35 ± 949.44                                             | 3084.95 ± 759.51                                         |
        | HalfCheetah-v2            | 2717.27 ± 1269.85                                            | 2707.91 ± 1215.21                                        |
        | Hopper-v2                 | 2387.39 ± 645.41                                             | 2272.78 ± 588.66                                         |
        | InvertedDoublePendulum-v2 | 5630.91 ± 377.93                                             | 5661.29 ± 316.04                                         |
        | Reacher-v2                | -4.61 ± 0.53                                                 | -4.24 ± 0.25                                             |
        | Swimmer-v2                | 132.07 ± 9.92                                                | 141.37 ± 8.70                                            |
        | Pusher-v2                 | -33.93 ± 8.55                                                | -26.22 ± 2.52                                            |
                
        
        Learning curves:
        

        Tracked experiments:     

        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-on-Mujoco_v2-Part-2--VmlldzozMjU4MzI1" style="width:100%; height:500px" title="RPO-mujoco-v2-part1"></iframe>

        Results with `rpo_alpha=0.5` (not tuned) on the tuned environments:

        |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
        |:--------------------------|:-------------------------------------------------------------|:----------------------------------------------|
        | Ant-v2                    | 2495.65 ± 991.65                                             | -7.81 ± 3.57                                  |
        | HalfCheetah-v2            | 2722.03 ± 1231.28                                            | 2605.06 ± 1183.30                             |
        | Hopper-v2                 | 2356.83 ± 650.91                                             | 1609.79 ± 164.16                              |
        | InvertedDoublePendulum-v2 | 5675.31 ± 244.34                                             | 274.78 ± 16.40                                |
        | Reacher-v2                | -4.67 ± 0.48                                                 | -66.55 ± 0.20                                 |
        | Swimmer-v2                | 131.53 ± 9.94                                                | 114.34 ± 3.95                                 |
        | Pusher-v2                 | -33.46 ± 8.41                                                | -275.09 ± 15.65                               |

        Learning curves:
        


        Tracked experiments:     

        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-alpha-0-5-on-Mujoco_v2-Part-2--VmlldzozMjU4MzQ0" style="width:100%; height:500px" title="RPO-mujoco-v2-0-5-part-2s"></iframe>

        

    
    === "Gym(Gymnasium)"

        Results on two continuous gym environments. The PPO and RPO run for 8M timesteps, and results are computed over 10 random seeds.


        |                  | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
        |:-----------------|:-------------------------------------------------------------|:----------------------------------------------|
        | Pendulum-v1      | -1141.98 ± 135.55                                            | -151.08 ± 3.73                                |
        | BipedalWalker-v3 | 172.12 ± 96.05                                               | 227.11 ± 18.23                                |

        Learning curves:
        

        Tracked experiments:     

        <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-on-Gym-Gymnasium---VmlldzozMjU3NzUy" style="width:100%; height:500px" title="RPO-gymnasium"></iframe>



???+ failure 

    Failure case of `rpo_alpha=0.5`:

    Overall, we observed that `rpo_alpha=0.5` is strictly equal to or better than the default PPO in 93% of environments tested (all 48/48 dm_control, 2/2 Gym, 7/11 mujoco_v4. Total 57 out of 61 environments tested). 

    Here are the failure cases:
    `Mujoco v4 and v2: Ant InvertedDoublePendulum  Reacher Pusher`

    |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
    |:--------------------------|:-------------------------------------------------------------|:----------------------------------------------|
    | Ant-v4                    | 1831.63 ± 867.71                                             | -10.43 ± 8.16                                 |
    | InvertedDoublePendulum-v4 | 5490.71 ± 261.50                                             | 303.36 ± 13.39                                |
    | Reacher-v4                | -4.58 ± 0.73                                                 | -66.62 ± 0.56                                 |
    | Pusher-v4                 | -30.63 ± 6.42                                                | -276.11 ± 26.52                               |

    Learning curves:
    

    Tracked experiments:     

    <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-alpha-0-5-s-failure-cases-on-Mujoco_v4--VmlldzozMjU4MTYy" style="width:100%; height:500px" title="RPO-mujoco-v4-failure-cases"></iframe>



    |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
    |:--------------------------|:-------------------------------------------------------------|:----------------------------------------------|
    | Ant-v2                    | 2493.50 ± 993.24                                             | -7.26 ± 2.28                                  |
    | InvertedDoublePendulum-v2 | 5568.37 ± 401.65                                             | 278.94 ± 15.34                                |
    | Reacher-v2                | -4.62 ± 0.47                                                 | -66.61 ± 0.23                                 |
    | Pusher-v2                 | -33.51 ± 8.47                                                | -276.01 ± 15.93                               |


    Learning curves:
    


    Tracked experiments:     

    <iframe loading="lazy" src="https://wandb.ai/openrlbenchmark/cleanrl/reports/RPO-alpha-0-5-s-failure-cases-on-Mujoco_v2--VmlldzozMjU4MjQ1" style="width:100%; height:500px" title="RPO-mujoco-v4-failure-cases"></iframe>


    However, tuning of `rpo_alpha` (`rpo_alpha=0.01` on failed cases) helps RPO to overcome the failure, and it performs strictly equal to or better than the default PPO in all (100%) of tested environments.

---

### Rl Algorithms Sac (`docs/rl-algorithms/sac.md`)

# Soft Actor-Critic (SAC)

## Overview

The Soft Actor-Critic (SAC) algorithm extends the DDPG algorithm by 1) using a stochastic policy, which in theory can express multi-modal optimal policies.
This also enables the use of 2) *entropy regularization* based on the stochastic policy's entropy. It serves as a built-in, state-dependent exploration heuristic for the agent, instead of relying on non-correlated noise processes as in [DDPG](/rl-algorithms/ddpg/), or [TD3](/rl-algorithms/td3/)
Additionally, it incorporates the 3) usage of two *Soft Q-network* to reduce the overestimation bias issue in Q-network-based methods.

Original papers:
The SAC algorithm's initial proposal, and later updates and improvements can be chronologically traced through the following publications:

* [Reinforcement Learning with Deep Energy-Based Policies](https://arxiv.org/abs/1702.08165)
* [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://arxiv.org/abs/1801.01290)
* [Composable Deep Reinforcement Learning for Robotic Manipulation](https://arxiv.org/abs/1803.06773)
* [Soft Actor-Critic Algorithms and Applications](https://arxiv.org/abs/1812.05905)
* [Soft Actor-Critic for Discrete Action Settings](https://arxiv.org/abs/1910.07207) (No peer review, preprint only)

Reference resources:

* :material-github: [haarnoja/sac](https://github.com/haarnoja/sac)
* :material-github: [openai/spinningup](https://github.com/openai/spinningup/tree/master/spinup/algos/tf1/sac)
* :material-github: [pranz24/pytorch-soft-actor-critic](https://github.com/pranz24/pytorch-soft-actor-critic)
* :material-github: [DLR-RM/stable-baselines3](https://github.com/DLR-RM/stable-baselines3/tree/master/stable_baselines3/sac)
* :material-github: [denisyarats/pytorch_sac](https://github.com/denisyarats/pytorch_sac)
* :material-github: [haarnoja/softqlearning](https://github.com/haarnoja/softqlearning)
* :material-github: [rail-berkeley/softlearning](https://github.com/rail-berkeley/softlearning)
* :material-github: [p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch](https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch)
* :material-github: [toshikwa/sac-discrete.pytorch](https://github.com/toshikwa/sac-discrete.pytorch)

| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`sac_continuous_actions.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py), :material-file-document: [docs](/rl-algorithms/sac/#sac_continuous_actionpy) | For continuous action spaces |
| :material-github: [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py), :material-file-document: [docs](/rl-algorithms/sac/#sac_ataripy) | For discrete action spaces |

Below are our single-file implementations of SAC:

## `sac_continuous_action.py`

The [sac_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py) has the following features:

* For continuous action spaces.
* Works with the `Box` observation space of low-level features.
* Works with the `Box` (continuous) action space.
* Numerically stable stochastic policy based on :material-github: [openai/spinningup](https://github.com/openai/spinningup/tree/master/spinup/algos/tf1/sac) and [pranz24/pytorch-soft-actor-critic](https://github.com/pranz24/pytorch-soft-actor-critic) implementations.
* Supports automatic entropy coefficient $\alpha$ tuning, enabled by default.

### Usage for continuous action spaces

=== "poetry"

    ```bash
    poetry install
    uv pip install ".[mujoco]"
    uv run python cleanrl/sac_continuous_action.py --help
    uv run python cleanrl/sac_continuous_action.py --env-id Hopper-v4
    uv run python cleanrl/sac_continuous_action.py --env-id Hopper-v4 --autotune False --alpha 0.2 ## Without Automatic entropy coef. tuning
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-mujoco.txt
    python cleanrl/sac_continuous_action.py --help
    python cleanrl/sac_continuous_action.py --env-id Mujoco-v4
    python cleanrl/sac_continuous_action.py --env-id Hopper-v4 --autotune False --alpha 0.2 ## Without Automatic entropy coef. tuning
    ```

### Explanation of the logged metrics

Running python cleanrl/sac_continuous_action.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

* `charts/episodic_return`: the episodic return of the game during training

* `charts/SPS`: number of steps per second

* `losses/qf1_loss`, `losses/qf2_loss`: for each Soft Q-value network $Q_{\theta_i}$, $i \in \{1,2\}$, this metric holds the mean squared error (MSE) between the soft Q-value estimate $Q_{\theta_i}(s, a)$ and the *entropy regularized* Bellman update target estimated as $r_t + \gamma \, Q_{\theta_{i}^{'}}(s', a') + \alpha \, \mathcal{H} \big[ \pi(a' \vert s') \big]$.

More formally, the Soft Q-value loss for the $i$-th network is obtained by:

$$
    J(\theta^{Q}_{i}) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \big[ (Q_{\theta_i}(s, a) - y)^2 \big]
$$

with the *entropy regularized*, *Soft Bellman update target*:
$$
    y = r(s, a) + \gamma ({\color{orange} \min_{\theta_{1,2}}Q_{\theta_i^{'}}(s',a')} - \alpha \, \text{log} \pi( \cdot \vert s'))
$$ where $a' \sim \pi( \cdot \vert s')$, $\text{log} \pi( \cdot \vert s')$ approximates the entropy of the policy, and $\mathcal{D}$ is the replay buffer storing samples of the agent during training.

Here, $\min_{\theta_{1,2}}Q_{\theta_i^{'}}(s',a')$ takes the minimum *Soft Q-value network* estimate between the two target Q-value networks $Q_{\theta_1^{'}}$ and $Q_{\theta_2^{'}}$ for the next state and action pair, so as to reduce over-estimation bias.

* `losses/qf_loss`: averages `losses/qf1_loss` and `losses/qf2_loss` for comparison with algorithms using a single Q-value network.

* `losses/actor_loss`: Given the stochastic nature of the policy in SAC, the actor (or policy) objective is formulated so as to maximize the likelihood of actions $a \sim \pi( \cdot \vert s)$ that would result in high Q-value estimate $Q(s, a)$. Additionally, the policy objective encourages the policy to maintain its entropy high enough to help explore, discover, and capture multi-modal optimal policies.

The policy's objective function can thus be defined as:

$$
    \text{max}_{\phi} \, J_{\pi}(\phi) = \mathbb{E}_{s \sim \mathcal{D}} \Big[ \text{min}_{i=1,2} Q_{\theta_i}(s, a) - \alpha \, \text{log}\pi_{\phi}(a \vert s) \Big]
$$

where the action is sampled using the reparameterization trick[^1]: $a = \mu_{\phi}(s) + \epsilon \, \sigma_{\phi}(s)$ with $\epsilon \sim \mathcal{N}(0, 1)$, $\text{log} \pi_{\phi}( \cdot \vert s')$ approximates the entropy of the policy, and $\mathcal{D}$ is the replay buffer storing samples of the agent during training.

* `losses/alpha`: $\alpha$ coefficient for *entropy regularization* of the policy.

* `losses/alpha_loss`: In the policy's objective defined above, the coefficient of the *entropy bonus* $\alpha$ is kept fixed all across the training.
As suggested by the authors in Section 5 of the [*Soft Actor-Critic And Applications*](https://arxiv.org/abs/1812.05905) paper, the original purpose of augmenting the standard reward with the entropy of the policy is to *encourage exploration* of not well enough explored states (thus high entropy).
Conversely, for states where the policy has already learned a near-optimal policy, it would be preferable to reduce the entropy bonus of the policy, so that it does not *become sub-optimal due to the entropy maximization incentive*.

Therefore, having a fixed value for $\alpha$ does not fit this desideratum of matching the entropy bonus with the knowledge of the policy at an arbitrary state during its training.

To mitigate this, the authors proposed a method to dynamically adjust $\alpha$ as the policy is trained, which is as follows:

$$
    \alpha^{*}_t = \text{argmin}_{\alpha_t} \mathbb{E}_{a_t \sim \pi^{*}_t} \big[ -\alpha_t \, \text{log}\pi^{*}_t(a_t \vert s_t; \alpha_t) - \alpha_t \mathcal{H} \big],
$$

where $\mathcal{H}$ represents the *target entropy*, the desired lower bound for the expected entropy of the policy over the trajectory distribution induced by the latter.
As a heuristic for the *target entropy*, the authors use the dimension of the action space of the task.

### Implementation details

CleanRL's [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py) implementation is based on :material-github: [openai/spinningup](https://github.com/openai/spinningup/tree/master/spinup/algos/pytorch/sac).

1. [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py) uses a *numerically stable* estimation method for the standard deviation $\sigma$ of the policy, which squashes it into a range of reasonable values for a standard deviation:

    ```
/* Detailed source-code truncated for AI context efficiency. */
```

    Note that unlike :material-github: [openai/spinningup](https://github.com/openai/spinningup/tree/master/spinup/algos/tf1/sac)'s implementation which uses `LOG_STD_MIN = -20`, CleanRL's uses `LOG_STD_MIN = -5` instead.

2. [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py) uses different learning rates for the policy and the Soft Q-value networks optimization.

    ```python
        parser.add_argument("--policy-lr", type=float, default=3e-4,
            help="the learning rate of the policy network optimizer")
        parser.add_argument("--q-lr", type=float, default=1e-3,
            help="the learning rate of the Q network network optimizer")
    ```

    while [openai/spinningup](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/tf1/sac/sac.py#L44)'s uses a single learning rate of `lr=1e-3` for both components.

    Note that in case it is used, the *automatic entropy coefficient* $\alpha$'s tuning shares the `q-lr` learning rate:

    ```python hl_lines="6"
        # Automatic entropy tuning
        if args.autotune:
            target_entropy = -torch.prod(torch.Tensor(envs.single_action_space.shape).to(device)).item()
            log_alpha = torch.zeros(1, requires_grad=True, device=device)
            alpha = log_alpha.exp().item()
            a_optimizer = optim.Adam([log_alpha], lr=args.q_lr)
        else:
            alpha = args.alpha
    ```

3. [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py) uses `--batch-size=256` while :material-github: [openai/spinningup](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/tf1/sac/sac.py#L44)'s uses `--batch-size=100` by default.

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/sac.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/sac.sh). Specifically, execute the following command:

``` title="benchmark/sac.sh" linenums="1"
--8<-- "benchmark/sac.sh::7"
```

The table below compares the results of CleanRL's [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py) with the [latest published results](https://arxiv.org/abs/1812.05905) by the original authors of the SAC algorithm.

???+ info
    Note that the results table above references the *training episodic return* for [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py), the results of [Soft Actor-Critic Algorithms and Applications](https://arxiv.org/abs/1812.05905) reference *evaluation episodic return* obtained by running the policy in the deterministic mode.

| Environment      | [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py) |[SAC: Algorithms and Applications](https://arxiv.org/abs/1812.05905) @ 1M steps|
| --------------- | ------------------ | ---------------- |
| HalfCheetah-v2  | 9634.89 ± 1423.73       | ~11,250          |
| Walker2d-v2     | 3591.45 ± 911.33         | ~4,800           |
| Hopper-v2       | 2310.46 ± 342.82         | ~3,250           |
| InvertedPendulum-v4 | 909.37 ± 55.66                                                        | N/A |
| Humanoid-v4         | 4996.29 ± 686.40                                                      | ~4500
| Pusher-v4           | -22.45 ± 0.51                                                         | N/A |

Learning curves:

``` title="benchmark/sac_plot.sh" linenums="1"
--8<-- "benchmark/sac_plot.sh::9"
```


<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/sac.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/sac-time.png">

Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/MuJoCo-CleanRL-s-SAC--VmlldzoxNzI1NDM0" style="width:100%; height:1200px" title="MuJoCo: CleanRL's SAC"></iframe>

## `sac_atari.py`

The [sac_atari.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) has the following features:

* For discrete action spaces.
* Works with the `Box` observation space of low-level features.
* Works with the `Discrete` action space.
* Improved stability and wall-clock efficiency through updates only every n-th step.
* Supports automatic entropy coefficient $\alpha$ tuning, enabled by default.

### Usage for discrete action spaces

=== "poetry"

    ```bash
    uv pip install ".[atari]"
    uv run python cleanrl/sac_atari.py --env-id PongNoFrameskip-v4
    uv run python cleanrl/sac_atari.py --env-id PongNoFrameskip-v4 --autotune False --alpha 0.2
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-atari.txt
    python cleanrl/sac_atari.py --env-id PongNoFrameskip-v4
    python cleanrl/sac_atari.py --env-id PongNoFrameskip-v4 --autotune False --alpha 0.2
    ```

### Explanation of the logged metrics

The metrics logged by `python cleanrl/sac_atari.py` are the same as the ones logged by `python cleanrl/sac_continuous_action.py` (see the [related docs](/rl-algorithms/sac/#explanation-of-the-logged-metrics)). However, the computations for the objectives differ in some details highlighted below:

* `losses/qf1_loss`, `losses/qf2_loss`: for each Soft Q-value network $Q_{\theta_i}$, $i \in \{1,2\}$, this metric holds the mean squared error (MSE) between the soft Q-value estimate $Q_{\theta_i}(s, a)$ and the *entropy regularized* Bellman update target estimated as $r_t + \gamma \, Q_{\theta_{i}^{'}}(s', a') + \alpha \, \mathcal{H} \big[ \pi(a' \vert s') \big]$.

SAC-discrete is able to exploit the discrete action space by using the full action distribution to calculate the Soft Q-targets instead of relying on a Monte Carlo approximation from a single Q-value. The new Soft Q-target is stated below with differences to the [continuous SAC target](/rl-algorithms/sac/#explanation-of-the-logged-metrics) highlighted in orange:
$$
    y = r(s, a) + \gamma \, {\color{orange}\pi (a | s^\prime)^{\mathsf T}} \Big(\min_{\theta_{1,2}} {\color{orange}Q_{\theta_i^{'}}(s')} - \alpha \, \log \pi( \cdot \vert s')\Big)~,
$$

Note how in the discrete setting the Q-function $Q_{\theta_i^{'}}(s')$ is a mapping $Q:S \rightarrow \mathbb R^{|\mathcal A|}$ that only takes states as inputs and outputs Q-values for all actions. Using all this available information and additionally weighing the target by the corresponding action selection probability reduces variance of the gradient.

* `losses/actor_loss`: Given the stochastic nature of the policy in SAC, the actor (or policy) objective is formulated so as to maximize the likelihood of actions $a \sim \pi( \cdot \vert s)$ that would result in high Q-value estimate $Q(s, a)$. Additionally, the policy objective encourages the policy to maintain its entropy high enough to help explore, discover, and capture multi-modal optimal policies.

SAC-discrete uses an action probability-weighted (highlighted in orange) objective given as:

$$
    \text{max}_{\phi} \, J_{\pi}(\phi) = \mathbb{E}_{s \sim \mathcal{D}} \bigg[ {\color{orange}\pi (a | s)^{\mathsf T}} \Big( \text{min}_{i=1,2} {\color{orange} Q_{\theta_i}(s)} - \alpha \, \log\pi_{\phi}(a \vert s) \Big) \bigg]
$$

Unlike for continuous action spaces, there is *no need for the reparameterization trick* due to using a Categorical policy. Similar to the critic objective, the Q-function $Q_{\theta_i}(s)$ is a function from states to real numbers and does not require actions as inputs.

* `losses/alpha_loss`: In the policy's objective defined above, the coefficient of the *entropy bonus* $\alpha$ is kept fixed all across the training.
As suggested by the authors in Section 5 of the [*Soft Actor-Critic And Applications*](https://arxiv.org/abs/1812.05905) paper, the original purpose of augmenting the standard reward with the entropy of the policy is to *encourage exploration* of not well enough explored states (thus high entropy).
Conversely, for states where the policy has already learned a near-optimal policy, it would be preferable to reduce the entropy bonus of the policy, so that it does not *become sub-optimal due to the entropy maximization incentive*.

In SAC-discrete, it is possible to weigh the target for the entropy coefficient by the policy's action selection probabilities to reduce gradient variance. This is the same trick that was already used in the critic and actor objectives, with differences to the regular SAC objective marked in orange:

$$
    \alpha^{*}_t = \text{argmin}_{\alpha_t} \mathbb{E}_{a_t \sim \pi^{*}_t} \, {\color{orange}\pi (a | s)^{\mathsf T}}\big[ -\alpha_t \, \log\pi^{*}_t(a_t \vert s_t; \alpha_t) - \alpha_t \tau \mathcal{H} \big],
$$

Since SAC-discrete uses a Categorical policy in a discrete action space, a different entropy target is required. The author uses the *maximum entropy Categorical distribution* (assigning uniform probability to all available actions) scaled by a factor of $\tau~.$

### Implementation details

[`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) uses the wrappers highlighted in the "9 Atari implementation details" in [The 37 Implementation Details of Proximal Policy Optimization](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/), which are as follows:

1. The Use of `NoopResetEnv` (:material-github: [common/atari_wrappers.py#L12](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L12))
2. The Use of `MaxAndSkipEnv` (:material-github: [common/atari_wrappers.py#L97](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L97))
3. The Use of `EpisodicLifeEnv` (:material-github: [common/atari_wrappers.py#L61](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L61))
4. The Use of `FireResetEnv` (:material-github: [common/atari_wrappers.py#L41](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L41))
5. The Use of `WarpFrame` (Image transformation) [common/atari_wrappers.py#L134](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L134)
6. The Use of `ClipRewardEnv` (:material-github: [common/atari_wrappers.py#L125](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L125))
7. The Use of `FrameStack` (:material-github: [common/atari_wrappers.py#L188](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/atari_wrappers.py#L188))
8. Shared Nature-CNN network for the policy and value functions (:material-github: [common/policies.py#L157](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/policies.py#L157), [common/models.py#L15-L26](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/models.py#L15-L26))
9. Scaling the Images to Range [0, 1] (:material-github: [common/models.py#L19](https://github.com/openai/baselines/blob/9b68103b737ac46bc201dfb3121cfa5df2127e53/baselines/common/models.py#L19))

Other noteworthy implementation details apart from the Atari wrapping are as follows:

1. [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) initializes the weights of its networks using He initialization (named after its author Kaiming He) from the paper ["Delving Deep into Rectifiers:
Surpassing Human-Level Performance on ImageNet Classification"](https://arxiv.org/pdf/1502.01852.pdf). The corresponding function in PyTorch is `kaiming_normal_`, its documentation can be found [here](https://pytorch.org/docs/stable/nn.init.html). In essence, it means the weights of each layer are initialized according to a Normal distribution with mean $\mu=0$ and

    $$
        \sigma = \frac{\text{gain}}{\sqrt{\text{fan}}}~,
    $$

    where $\text{fan}$ is the number of input neurons to the layer and $\text{gain}$ is a constant set to $\sqrt{2}$ for `ReLU` layers.

2. [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) uses the Adam[^2] optimizer with an increased $\epsilon$-parameter to improve its stability. This results in an increase in the denominator of the update rule:

    $$
        \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}
    $$

    Here $\hat{m}_t$ is the bias-corrected first moment and $\hat{v}_t$ the bias-corrected second raw moment.

3. [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) uses the action selection probabilities of the policy in multiple places to *reduce the variance of gradient estimates*. The target for the Soft Q-value estimate is weighted accordingly:

    ```python hl_lines="7"
    # CRITIC training
    with torch.no_grad():
        _, next_state_log_pi, next_state_action_probs = actor.get_action(data.next_observations)
        qf1_next_target = qf1_target(data.next_observations)
        qf2_next_target = qf2_target(data.next_observations)
        # we can use the action probabilities instead of MC sampling to estimate the expectation
        min_qf_next_target = next_state_action_probs * (
            torch.min(qf1_next_target, qf2_next_target) - alpha * next_state_log_pi
        )
    ```

    A similar action-probability weighting can be used for the actor gradient:

    ```python hl_lines="7"
    _, log_pi, action_probs = actor.get_action(data.observations)
    with torch.no_grad():
        qf1_values = qf1(data.observations)
        qf2_values = qf2(data.observations)
        min_qf_values = torch.min(qf1_values, qf2_values)
    # no need for reparameterization, the expectation can be calculated for discrete actions
    actor_loss = (action_probs * ((alpha * log_pi) - min_qf_values)).mean()
    ```

    Lastly, this variance reduction scheme is also used when automatic entropy tuning is enabled:

    ```python hl_lines="3"
    if args.autotune:
        # reuse action probabilities for temperature loss
        alpha_loss = (action_probs.detach() * (-log_alpha.exp() * (log_pi + target_entropy).detach())).mean()

        a_optimizer.zero_grad()
        alpha_loss.backward()
        a_optimizer.step()
        alpha = log_alpha.exp().item()
    ```

4. [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) uses `--target-entropy-scale=0.89` while the [SAC-discrete paper](https://arxiv.org/abs/1910.07207) uses `--target-entropy-scale=0.98` due to improved stability when training for more than 100k steps. Tuning this parameter to the environment at hand is advised and can lead to significant performance gains.

5. [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) performs learning updates only on every $n^{\text{th}}$ step. This leads to improved stability and prevents the agent's performance from degenerating during longer training runs.  
Note the difference to [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py): [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) updates every $n^{\text{th}}$ environment step and does a single update of actor and critic on every update step. [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py) updates the critic every step and the actor every $n^{\text{th}}$ step. It then compensates for the delayed actor updates by performing $n$ actor update steps.

6. [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) handles truncation and termination properly like (Mnih et al., 2015)[^3] by using SB3's replay buffer's `handle_timeout_termination=True`.

### Atari experiment results for SAC-discrete

Run benchmarks for :material-github: [benchmark/sac_atari.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/sac_atari.sh) by executing:

<script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Ftimoklein%2Fcleanrl%2Fblob%2Fsac-discrete%2Fbenchmark%2Fsac_atari.sh&style=github&type=code&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script>

The table below compares the results of CleanRL's [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) with the [original paper results](https://arxiv.org/abs/1910.07207).

???+ info
    Note that the results table above references the *training episodic return* for [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) without evaluation mode.

| Environment      | [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) 100k steps |[SAC for Discrete Action Settings](https://arxiv.org/abs/1910.07207) 100k steps| [`sac_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_atari.py) 5M steps | [`dqn_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py) 10M steps
| --------------- | ------------------ | ---------------- | ---------------- | ---------------- |
| PongNoFrameskip-v4  | ~ -20.21 ± 0.62       | -20.98 ± 0.0        |   ~19.24 ± 1.81  |     20.25 ± 0.41   |
| BreakoutNoFrameskip-v4     | ~2.33 ± 1.28         | -           | ~343.66 ± 93.34  |  366.928 ± 39.89  |
| BeamRiderNoFrameskip-v4       | ~396.15 ± 155.81         | 432.1 ± 44.0           |   ~8658.97 ± 1554.66 |    6673.24 ± 1434.37   |

Learning curves:

<div class="grid-container">
    <img src="../sac/PongNoFrameskip-v4.png">
    <img src="../sac/BreakoutNoFrameskip-v4.png">
    <img src="../sac/BeamRiderNoFrameskip-v4.png">
</div>

<div></div>


Tracked experiments:

<iframe src="https://wandb.ai/openrlbenchmark/cleanrl/reports/Atari-CleanRL-s-SAC-discrete--VmlldzoyNzgxMTI2" style="border:none;height:1024px;width:100%"></iframe>

[^1]:Diederik P Kingma, Max Welling (2016). Auto-Encoding Variational Bayes. ArXiv, abs/1312.6114. https://arxiv.org/abs/1312.6114

[^2]:Diederik P Kingma, Jimmy Lei Ba (2015). Adam: A Method for Stochastic Optimization. ArXiv, abs/1412.6980. https://arxiv.org/abs/1412.6980

[^3]:Mnih, V., Kavukcuoglu, K., Silver, D. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236

---

### Rl Algorithms Td3 (`docs/rl-algorithms/td3.md`)

# Twin Delayed Deep Deterministic Policy Gradient (TD3)


## Overview

TD3 is a popular DRL algorithm for continuous control. It extends DDPG with three techniques: 1) Clipped Double Q-Learning, 2) Delayed Policy Updates, and 3) Target Policy Smoothing Regularization. With these three techniques TD3 shows significantly better performance compared to DDPG.


Original paper: 

* [Addressing Function Approximation Error in Actor-Critic Methods](https://arxiv.org/abs/1802.09477)

Reference resources:

* :material-github: [sfujim/TD3](https://github.com/sfujim/TD3)
* [Twin Delayed DDPG | Spinning Up in Deep RL](https://spinningup.openai.com/en/latest/algorithms/td3.html)

## Implemented Variants


| Variants Implemented      | Description |
| ----------- | ----------- |
| :material-github: [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py), :material-file-document: [docs](/rl-algorithms/td3/#td3_continuous_actionpy) | For continuous action space |
| :material-github: [`td3_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py), :material-file-document: [docs](/rl-algorithms/td3/#td3_continuous_action_jaxpy) | For continuous action space |

Below are our single-file implementations of TD3:

## `td3_continuous_action.py`

The [td3_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py) has the following features:

* For continuous action space
* Works with the `Box` observation space of low-level features
* Works with the `Box` (continuous) action space

### Usage

=== "poetry"

    ```bash
    uv pip install ".[mujoco]"
    uv run python cleanrl/td3_continuous_action.py --help
    uv run python cleanrl/td3_continuous_action.py --env-id Hopper-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-mujoco.txt
    python cleanrl/td3_continuous_action.py --help
    python cleanrl/td3_continuous_action.py --env-id Hopper-v4
    ```

### Explanation of the logged metrics

Running `python cleanrl/td3_continuous_action.py` will automatically record various metrics such as various losses in Tensorboard. Below are the documentation for these metrics:

* `charts/episodic_return`: episodic return of the game
* `charts/SPS`: number of steps per second
* `losses/qf1_loss`: the MSE between the Q values at timestep $t$ and the target Q values at timestep $t+1$, which minimizes temporal difference. 
* `losses/actor_loss`: implemented as `-qf1(data.observations, actor(data.observations)).mean()`; it is the *negative* average Q values calculated based on the 1) observations and the 2) actions computed by the actor based on these observations. By minimizing `actor_loss`, the optimizer updates the actors parameter using the following gradient (Fujimoto et al., 2018, Algorithm 1)[^2]:

$$ \nabla_{\phi} J(\phi)=\left.N^{-1} \sum \nabla_{a} Q_{\theta_{1}}(s, a)\right|_{a=\pi_{\phi}(s)} \nabla_{\phi} \pi_{\phi}(s) $$

* `losses/qf1_values`: implemented as `qf1(data.observations, data.actions).view(-1); it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimations happen


### Implementation details

Our [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py) is based on the [`TD3.py`](https://github.com/sfujim/TD3/blob/master/TD3.py) from :material-github: [sfujim/TD3](https://github.com/sfujim/TD3). Our [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py) presents the following implementation differences.

1. [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py) uses a two separate objects `qf1` and `qf2` to represents the two Q functions in the Clipped Double Q-learning architecture, whereas  [`TD3.py`](https://github.com/sfujim/TD3/blob/master/TD3.py)  (Fujimoto et al., 2018)[^2] uses a single `Critic` class that contains both Q networks. That said, these two implementations are virtually the same.

2. [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py) also adds support for handling continuous environments where the lower and higher bounds of the action space are not $[-1,1]$, or are asymmetric.
The case where the bounds are not $[-1,1]$ is handled in [`TD3.py`](https://github.com/sfujim/TD3/blob/385b33ac7de4767bab17eb02ade4a268d3e4e24f/TD3.py#L28) (Fujimoto et al., 2018)[^2] as follows:
```python
class Actor(nn.Module):

    ...

	def forward(self, state):
		a = F.relu(self.l1(state))
		a = F.relu(self.l2(a))
		return self.max_action * torch.tanh(self.l3(a)) # Scale from [-1,1] to [-action_high, action_high]
```
 On the other hand, in [`CleanRL's td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py), the mean and the scale of the action space are computed as `action_bias` and `action_scale` respectively.
 Those scalars are in turn used to scale the output of a `tanh` activation function in the actor to the original action space range:
```python
class Actor(nn.Module):
    def __init__(self, env):
        ...
        # action rescaling
        self.register_buffer("action_scale", torch.FloatTensor((env.action_space.high - env.action_space.low) / 2.0))
        self.register_buffer("action_bias", torch.FloatTensor((env.action_space.high + env.action_space.low) / 2.0))

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = torch.tanh(self.fc_mu(x))
        return x * self.action_scale + self.action_bias # Scale from [-1,1] to [-action_low, action_high]
```

Additionally, when drawing exploration noise that is added to the actions produced by the actor, [`CleanRL's td3_continuous_action.py`](https://github.com/dosssman/cleanrl/blob/10b606e7bd9bd1b06e455e8ef542df2b7699a20c/cleanrl/td3_continuous_action.py#L180) centers the distribution the sampled from at `action_bias`, and the scale of the distribution is set to `action_scale * exploration_noise`.

???+ info

    Note that `Humanoid-v2`, `InvertedPendulum-v2`, `Pusher-v2` have action space bounds that are not the standard `[-1, 1]`. See below and :material-github: [PR #196](https://github.com/vwxyzjn/cleanrl/issues/196)

    ```
    Ant-v2 Observation space: Box(-inf, inf, (111,), float64) Action space: Box(-1.0, 1.0, (8,), float32)
    HalfCheetah-v2 Observation space: Box(-inf, inf, (17,), float64) Action space: Box(-1.0, 1.0, (6,), float32)
    Hopper-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (3,), float32)
    Humanoid-v2 Observation space: Box(-inf, inf, (376,), float64) Action space: Box(-0.4, 0.4, (17,), float32)
    InvertedDoublePendulum-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (1,), float32)
    InvertedPendulum-v2 Observation space: Box(-inf, inf, (4,), float64) Action space: Box(-3.0, 3.0, (1,), float32)
    Pusher-v2 Observation space: Box(-inf, inf, (23,), float64) Action space: Box(-2.0, 2.0, (7,), float32)
    Reacher-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (2,), float32)
    Swimmer-v2 Observation space: Box(-inf, inf, (8,), float64) Action space: Box(-1.0, 1.0, (2,), float32)
    Walker2d-v2 Observation space: Box(-inf, inf, (17,), float64) Action space: Box(-1.0, 1.0, (6,), float32)
    ```

### Experiment results

To run benchmark experiments, see :material-github: [benchmark/td3.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/td3.sh). Specifically, execute the following command:

``` title="benchmark/td3.sh" linenums="1"
--8<-- "benchmark/td3.sh::7"
```


Below are the average episodic returns for [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py) (3 random seeds). To ensure the quality of the implementation, we compared the results against (Fujimoto et al., 2018)[^2].

| Environment      | [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py) | [`TD3.py`](https://github.com/sfujim/TD3/blob/master/TD3.py) (Fujimoto et al., 2018, Table 1)[^2]  |
| ----------- | ----------- | ----------- | 
| HalfCheetah-v4      |   9583.22 ± 126.09      |9636.95 ± 859.065  |
| Walker2d-v4   |         4057.59 ± 658.78     |  4682.82 ± 539.64 | 
| Hopper-v4   |           3134.61 ± 360.18        |  3564.07 ± 114.74 | 
| InvertedPendulum-v4 |   968.99 ± 25.80    | 1000.00 ± 0.00  | 
| Humanoid-v4 |           5035.36 ± 21.67      |  not available | 
| Pusher-v4 |             -30.92 ± 1.05      |  not available | 



???+ info

    Note that [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py) uses gym MuJoCo v4 environments while [`TD3.py`](https://github.com/sfujim/TD3/blob/master/TD3.py) (Fujimoto et al., 2018)[^2] uses the gym MuJoCo v1 environments.

    Also note the performance of our `td3_continuous_action.py` seems to be worse than the reference implementation on Walker2d. This is likely due to :material-github: [openai/gym#938](https://github.com/openai/baselines/issues/938). We would have a hard time reproducing gym MuJoCo v1 environments because they have been long deprecated.

    One other thing could cause the performance difference: the original code reported the average episodic return using determinisitc evaluation (i.e., without exploration noise), see [`sfujim/TD3/main.py#L15-L32`](https://github.com/sfujim/TD3/blob/385b33ac7de4767bab17eb02ade4a268d3e4e24f/main.py#L15-L32), whereas we reported the episodic return during training and the policy gets updated between environments steps.

Learning curves:

``` title="benchmark/td3_plot.sh" linenums="1"
--8<-- "benchmark/td3_plot.sh::9"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/td3.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/td3-time.png">

Tracked experiments and game play videos:

<iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/MuJoCo-CleanRL-s-TD3--VmlldzoxNjk4Mzk5" style="width:100%; height:500px" title="MuJoCo: CleanRL's TD3"></iframe>




## `td3_continuous_action_jax.py`

The [td3_continuous_action_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py) has the following features:

* Uses [Jax](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Optax](https://github.com/deepmind/optax) instead of `torch`.  [td3_continuous_action_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py) is roughly 2.5-4x faster than  [td3_continuous_action.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py)
* For continuous action space
* Works with the `Box` observation space of low-level features
* Works with the `Box` (continuous) action space

### Usage

=== "poetry"

    ```bash
    uv pip install ".[mujoco, jax]"
    uv run python cleanrl/td3_continuous_action_jax.py --help
    uv run python cleanrl/td3_continuous_action_jax.py --env-id Hopper-v4
    ```

=== "pip"

    ```bash
    pip install -r requirements/requirements-mujoco.txt  
    pip install -r requirements/requirements-jax.txt
    python cleanrl/td3_continuous_action_jax.py --help
    python cleanrl/td3_continuous_action_jax.py --env-id Hopper-v4
    ```

### Explanation of the logged metrics

See [related docs](/rl-algorithms/td3/#explanation-of-the-logged-metrics) for `td3_continuous_action.py`.


### Implementation details

See [related docs](/rl-algorithms/td3/#implementation-details) for `td3_continuous_action.py`.


### Experiment results

To run benchmark experiments, see :material-github: [benchmark/td3.sh](https://github.com/vwxyzjn/cleanrl/blob/master/benchmark/td3.sh). Specifically, execute the following command:

``` title="benchmark/td3.sh" linenums="1"
--8<-- "benchmark/td3.sh:12:19"
```

Below are the average episodic returns for [`td3_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py) (3 random seeds).

{!benchmark/td3.md!}

Learning curves:


``` title="benchmark/td3_plot.sh" linenums="1"
--8<-- "benchmark/td3_plot.sh:11:20"
```

<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/td3.png">
<img loading="lazy" src="https://huggingface.co/datasets/cleanrl/benchmark/resolve/main/benchmark/pr-424/td3-time.png">

???+ info

    These are some previous experiments with TPUs. Note the results are very similar to the ones above, but the runtime can be different due to different hardware used.

    Note that the experiments were conducted on different hardwares, so your mileage might vary. This inconsistency is because 1) re-running expeirments on the same hardware is computationally expensive and 2) requiring the same hardware is not inclusive nor feasible to other contributors who might have different hardwares.

    That said, we roughly expect to see a 2-4x speed improvement from using [`td3_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py) under the same hardware. And if you disable the `--capture_video` overhead, the speed improvement will be even higher.

    Learning curves:


    <div class="grid-container">
    <img loading="lazy" src="../td3-jax/HalfCheetah-v2.png">
    <img loading="lazy" src="../td3-jax/HalfCheetah-v2-time.png">

    <img loading="lazy" src="../td3-jax/Walker2d-v2.png">
    <img loading="lazy" src="../td3-jax/Walker2d-v2-time.png">

    <img loading="lazy" src="../td3-jax/Hopper-v2.png">
    <img loading="lazy" src="../td3-jax/Hopper-v2-time.png">
    </div>



    Tracked experiments and game play videos:

    <iframe src="https://wandb.ai/openrlbenchmark/openrlbenchmark/reports/MuJoCo-CleanRL-s-TD3-JAX--VmlldzoyMzU1OTA4" style="width:100%; height:500px" title="MuJoCo: CleanRL's TD3 + JAX"></iframe>



[^1]:Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N.M., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2016). Continuous control with deep reinforcement learning. CoRR, abs/1509.02971. https://arxiv.org/abs/1509.02971

[^2]:Fujimoto, S., Hoof, H.V., & Meger, D. (2018). Addressing Function Approximation Error in Actor-Critic Methods. ArXiv, abs/1802.09477. https://arxiv.org/abs/1802.09477

---

### Blog Posts Cleanrl V1 (`docs/blog/posts/cleanrl-v1.md`)

---
date: 2022-10-05
authors: [costa]
description: >
  🎉 We are thrilled to announce the v1.0.0 CleanRL Release. Along with our CleanRL paper's recent publication in Journal of Machine Learning Research (https://www.jmlr.org/papers/v23/21-1342.html), our v1.0.0 release includes reworked documentation, new algorithm variants, support for google's new ML framework JAX, hyperparameter tuning utilities, and more.
categories:
  - Blog
---



# CleanRL v1 Release


🎉 We are thrilled to announce the v1.0.0 CleanRL Release. Along with our [CleanRL paper's recent publication in Journal of Machine Learning Research](https://www.jmlr.org/papers/v23/21-1342.html), our v1.0.0 release includes reworked documentation, new algorithm variants, support for google's new ML framework [JAX](https://github.com/google/jax), hyperparameter tuning utilities, and more. CleanRL has come a long way making high-quality deep reinforcement learning implementations easy to understand and reproducible. This release is a major milestone for the project and we are excited to share it with you. Over 90 PRs were merged to make this release possible. We would like to thank all the contributors who made this release possible.

More detailed release notes are available at [v1.0.0b1](https://github.com/vwxyzjn/cleanrl/releases/tag/v1.0.0b1), [v1.0.0b2](https://github.com/vwxyzjn/cleanrl/releases/tag/v1.0.0b2), and [v1.0.0](https://github.com/vwxyzjn/cleanrl/releases/tag/v1.0.0).




## Reworked documentation

One of the biggest change of the v1 release is the added documentation at [docs.cleanrl.dev](https://docs.cleanrl.dev). Having great documentation is important for building a reliable and reproducible project. We have reworked the documentation to make it easier to understand and use. For each implemented algorithm, we have documented as much as we can to promote transparency:

* [Short description of the algorithm and references](/rl-algorithms/ppo/#overview)
* [A list of implemented variant](/rl-algorithms/ppo/#implemented-variants)
* [The usage information](/rl-algorithms/ppo/#usage)
* [The explanation of the logged metrics](/rl-algorithms/ppo/#explanation-of-the-logged-metrics)
* [The documentation of implementation details](/rl-algorithms/ppo/#implementation-details)
* [Experimental results](/rl-algorithms/ppo/#experiment-results)

Here is a list of the algorithm variants and their documentation:


| Algorithm      | Variants Implemented |
| ----------- | ----------- |
| ✅ [Proximal Policy Optimization (PPO)](https://arxiv.org/pdf/1707.06347.pdf)  | :material-github: [`ppo.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppopy) |
| | :material-github: [`ppo_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_ataripy)
| | :material-github: [`ppo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_continuous_actionpy)
| | :material-github: [`ppo_atari_lstm.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_lstm.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_lstmpy)
| | :material-github: [`ppo_atari_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_envpoolpy)
| | :material-github: [`ppo_atari_envpool_xla_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_envpool_xla_jaxpy)
| | :material-github: [`ppo_procgen.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_procgen.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_procgenpy)
| | :material-github: [`ppo_atari_multigpu.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_multigpu.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_atari_multigpupy)
| | :material-github: [`ppo_pettingzoo_ma_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_pettingzoo_ma_atari.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_pettingzoo_ma_ataripy)
| | :material-github: [`ppo_continuous_action_isaacgym.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py),  :material-file-document: [docs](/rl-algorithms/ppo/#ppo_continuous_action_isaacgympy)
| ✅ [Deep Q-Learning (DQN)](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf) | :material-github: [`dqn.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqnpy) |
| | :material-github: [`dqn_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_ataripy) |
| | :material-github: [`dqn_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_jax.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_jaxpy) |
| | :material-github: [`dqn_atari_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py), :material-file-document: [docs](/rl-algorithms/dqn/#dqn_atari_jaxpy) |
| ✅ [Categorical DQN (C51)](https://arxiv.org/pdf/1707.06887.pdf) | :material-github: [`c51.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51.py), :material-file-document: [docs](/rl-algorithms/c51/#c51py) |
| | :material-github: [`c51_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/c51_atari.py), :material-file-document: [docs](/rl-algorithms/c51/#c51_ataripy) |
| ✅ [Soft Actor-Critic (SAC)](https://arxiv.org/pdf/1812.05905.pdf) | :material-github: [`sac_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/sac_continuous_action.py), :material-file-document: [docs](/rl-algorithms/sac/#sac_continuous_actionpy) |
| ✅ [Deep Deterministic Policy Gradient (DDPG)](https://arxiv.org/pdf/1509.02971.pdf) | :material-github: [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py), :material-file-document: [docs](/rl-algorithms/ddpg/#ddpg_continuous_actionpy) |
| | :material-github: [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py),  :material-file-document: [docs](/rl-algorithms/ddpg/#ddpg_continuous_action_jaxpy)
| ✅ [Twin Delayed Deep Deterministic Policy Gradient (TD3)](https://arxiv.org/pdf/1802.09477.pdf) | :material-github: [`td3_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action.py), :material-file-document: [docs](/rl-algorithms/td3/#td3_continuous_actionpy) |
|  | :material-github: [`td3_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/td3_continuous_action_jax.py), :material-file-document: [docs](/rl-algorithms/td3/#td3_continuous_action_jaxpy) |
| ✅ [Phasic Policy Gradient (PPG)](https://arxiv.org/abs/2009.04416) | :material-github: [`ppg_procgen.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppg_procgen.py), :material-file-document: [docs](/rl-algorithms/ppg/#ppg_procgenpy) |
| ✅ [Random Network Distillation (RND)](https://arxiv.org/abs/1810.12894) | :material-github: [`ppo_rnd_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_rnd_envpool.py), :material-file-document: [docs](/rl-algorithms/ppo-rnd/#ppo_rnd_envpoolpy) |


We also improved the [contribution guide](https://github.com/vwxyzjn/cleanrl/blob/master/CONTRIBUTING.md) to make it easier for new contributors to get started. We are still working on improving the documentation. If you have any suggestions, please let us know in the [GitHub Issues](https://github.com/vwxyzjn/cleanrl/issues).


## New algorithm variants, support for JAX

We now support JAX-based learning algorithm variants, which are usually faster than the `torch` equivalent! Here are the docs of the new JAX-based DQN, TD3, and DDPG implementations:


* [`dqn_atari_jax.py`](https://docs.cleanrl.dev/rl-algorithms/dqn/#dqn_atari_jaxpy) [@kinalmehta](https://github.com/kinalmehta) in [:material-github: vwxyzjn/cleanrl#222](https://github.com/vwxyzjn/cleanrl/pull/222)
    * about 25% faster than `dqn_atari.py`.
* [`td3_continuous_action_jax.py`](https://docs.cleanrl.dev/rl-algorithms/td3/#td3_continuous_action_jaxpy) by [@joaogui1](https://github.com/joaogui1) in [:material-github: vwxyzjn/cleanrl#225](https://github.com/vwxyzjn/cleanrl/pull/225)
    * about 2.5-4x faster than `td3_continuous_action.py`.
* [`ddpg_continuous_action_jax.py`](https://docs.cleanrl.dev/rl-algorithms/ddpg/#ddpg_continuous_action_jaxpy) by [@vwxyzjn](https://github.com/vwxyzjn) in [:material-github: vwxyzjn/cleanrl#187](https://github.com/vwxyzjn/cleanrl/pull/187)
    * about 2.5-4x faster than `ddpg_continuous_action.py`.
* [`ppo_atari_envpool_xla_jax.py`](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_envpool_xla_jaxpy) by [@vwxyzjn](https://github.com/vwxyzjn) in [:material-github: vwxyzjn/cleanrl#227](https://github.com/vwxyzjn/cleanrl/pull/227)
    * about 3x faster than openai/baselines' PPO.

For example, below are the benchmark of DDPG + JAX (see docs [here](/rl-algorithms/ddpg/#ddpg_continuous_action_jaxpy) for further detail):


<div class="grid-container">
<img src="/rl-algorithms/ddpg-jax/HalfCheetah-v2.png">
<img src="/rl-algorithms/ddpg-jax/HalfCheetah-v2-time.png">
</div>

Other new algorithm variants include multi-GPU PPO, PPO prototype that works with [Isaac Gym](https://github.com/NVIDIA-Omniverse/IsaacGymEnvs), multi-agent Atari PPO, and refactored PPG and PPO-RND implementations:

* [`ppo_atari_multigpu.pu`](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_multigpupy) by [@vwxyzjn](https://github.com/vwxyzjn) in [:material-github: vwxyzjn/cleanrl#178]( https://github.com/vwxyzjn/cleanrl/pull/178)
    * about 34% faster than `ppo_atari.py` which uses `SyncVectorEnv`.
* [`ppo_continuous_action_isaacgym.py`](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_continuous_action_isaacgympy) by [@vwxyzjn](https://github.com/vwxyzjn) in [:material-github: vwxyzjn/cleanrl#233](https://github.com/vwxyzjn/cleanrl/pull/233)
    * achieves 4000+ score and 30M steps on IsaacGymEnvs' `Ant` in 4 mins. 
* [`ppo_pettingzoo_ma_atari.py`](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_pettingzoo_ma_ataripy) by [@vwxyzjn](https://github.com/vwxyzjn) in [:material-github: vwxyzjn/cleanrl#188](https://github.com/vwxyzjn/cleanrl/pull/188)
    * achieves ~4000 *episodic length* (not episodic return) in Pong, creating competitive self play agents.
* [`ppg_procgen.py`](https://docs.cleanrl.dev/rl-algorithms/ppg/#ppg_procgenpy) by [@Dipamc77](https://github.com/Dipamc77) in [:material-github: vwxyzjn/cleanrl#186](https://github.com/vwxyzjn/cleanrl/pull/186)
    * matches openai/baselines' PPO performance in StarPilot (easy), BossFight (easy), and BigFish (easy).
* [`ppo_rnd_envpoolpy.py`](https://docs.cleanrl.dev/rl-algorithms/ppo-rnd/#ppo_rnd_envpoolpy) by [@yooceii](https://github.com/yooceii) in [:material-github: vwxyzjn/cleanrl#151](https://github.com/vwxyzjn/cleanrl/pull/151)
    * achieves ~7100 in `MontezumaRevengeNoFrameSkip-v4`.



## Tooling improvements

We love tools! The v1.0.0 release comes with a series of DevOps improvements, including pre-commit utilities, CI integration with GitHub to run end-to-end test cases. We also make available a new hyperparameter tuning tool and a new tool for running benchmark experiments.

### DevOps

We added a pre-commit utility to help contributors to format their code, check for spelling, and removing unused variables and imports before submitting a pull request (see [Contribution guide](/contribution/#pre-commit-utilities) for more detail).

<img src="/static/pre-commit.png">


To ensure our single-file implementations can run without error, we also added CI/CD pipeline which now runs end-to-end test cases for all the algorithm variants. The pipeline also tests builds across different operating systems, such as Linux, macOS, and Windows (see [here](https://github.com/vwxyzjn/cleanrl/actions/runs/3401991711/usage) as an example). GitHub actions are free for open source projects, and we are very happy to have this tool to help us maintain the project. 

<img src="/static/blog/cleanrl-v1/github-action.png">

### Hyperparameter tuning utilities

We now have preliminary support for hyperparameter tuning via `optuna` (see [docs](https://docs.cleanrl.dev/advanced/hyperparameter-tuning/)), which is designed to help researchers to find **a single set** of hyperparameters that work well with a kind of games. The current API looks like below:

```python
import optuna
from cleanrl_utils.tuner import Tuner
tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": [0, 500],
        "Acrobot-v1": [-500, 0],
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)
```

### Benchmarking utilities

We also added a new tool for running benchmark experiments. The tool is designed to help researchers to quickly run benchmark experiments across different algorithms environments with some random seeds. The tool lives in the `cleanrl_utils.benchmark` module, and the users can run commands such as:

```bash
OMP_NUM_THREADS=1 xvfb-run -a python -m cleanrl_utils.benchmark \
    --env-ids CartPole-v1 Acrobot-v1 MountainCar-v0 \
    --command "uv run python cleanrl/ppo.py --no_cuda --track --capture_video" \
    --num-seeds 3 \
    --workers 5
```

which will run the `ppo.py` script with `--no_cuda --track --capture_video` arguments across 3 random seeds for 3 environments. It uses `multiprocessing` to create a pool of 5 workers run the experiments in parallel.

## What’s next?

It is an exciting time and new improvements are coming to CleanRL. We plan to add more JAX-based implementations, huggingface integration, some RLops prototypes, and support Gymnasium. CleanRL is a community-based project and we always welcome new contributors. If there is an algorithm or new feature you would like to contribute, feel free to chat with us on our [discord channel](https://discord.gg/D6RCjA6sVT) or raise a GitHub issue. 

### More JAX implementations

More JAX-based implementation are coming. [Antonin Raffin](https://github.com/araffin), the core maintainer of [Stable-baselines3](https://github.com/DLR-RM/stable-baselines3), [SBX](https://github.com/araffin/sbx), and [rl-baselines3-zoo](https://github.com/DLR-RM/rl-baselines3-zoo), is contributing an optimized Soft Actor Critic implementation in JAX ([:material-github: vwxyzjn/cleanrl#300](https://github.com/vwxyzjn/cleanrl/pull/300)) and TD3+TQC, and DroQ ([:material-github: vwxyzjn/cleanrl#272](https://github.com/vwxyzjn/cleanrl/pull/272). These are incredibly exciting new algorithms. For example, DroQ is extremely sample efficient and can obtain ~5000 return in `HalfCheetah-v3` in just 100k steps ([tracked sbx experiment](https://wandb.ai/openrlbenchmark/sbx/runs/1tyzq3tu)).

### Huggingface integration

[Huggingface Hub 🤗](https://huggingface.co/models) is a great platform for sharing and collaborating models. We are working on a new integration with Huggingface Hub to make it easier for researchers to share their RL models and benchmark them against other models ([:material-github: vwxyzjn/cleanrl#292](https://github.com/vwxyzjn/cleanrl/pull/292)). Stay tuned! In the future, we will have a simple snippet for loading models like below:

```python
import random
from typing import Callable

import gym
import numpy as np
import torch


def evaluate(
    model_path: str,
    make_env: Callable,
    env_id: str,
    eval_episodes: int,
    run_name: str,
    Model: torch.nn.Module,
    device: torch.device,
    epsilon: float = 0.05,
    capture_video: bool = True,
):
    envs = gym.vector.SyncVectorEnv([make_env(env_id, 0, 0, capture_video, run_name)])
    model = Model(envs).to(device)
    model.load_state_dict(torch.load(model_path))
    model.eval()

    obs = envs.reset()
    episodic_returns = []
    while len(episodic_returns) < eval_episodes:
        if random.random() < epsilon:
            actions = np.array([envs.single_action_space.sample() for _ in range(envs.num_envs)])
        else:
            q_values = model(torch.Tensor(obs).to(device))
            actions = torch.argmax(q_values, dim=1).cpu().numpy()
        next_obs, _, _, infos = envs.step(actions)
        for info in infos:
            if "episode" in info.keys():
                print(f"eval_episode={len(episodic_returns)}, episodic_return={info['episode']['r']}")
                episodic_returns += [info["episode"]["r"]]
        obs = next_obs

    return episodic_returns


if __name__ == "__main__":
    from huggingface_hub import hf_hub_download

    from cleanrl.dqn import QNetwork, make_env

    model_path = hf_hub_download(repo_id="cleanrl/CartPole-v1-dqn-seed1", filename="q_network.pth")
```


<img src="/static/blog/cleanrl-v1/hf.png">

### RLops

How do we know the effect of a new feature / bug fix? DRL is brittle and has a series of reproducibility issues — even bug fixes sometimes could introduce performance regression (e.g., see [how a bug fix of contact force in MuJoCo results in worse performance for PPO](https://github.com/openai/gym/pull/2762#discussion_r853488897)). Therefore, it is essential to understand how the proposed changes impact the performance of the algorithms. 

We are working a prototype tool that allows us to compare the performance of the library at different versions of the tracked experiment ([:material-github: vwxyzjn/cleanrl#307](https://github.com/vwxyzjn/cleanrl/pull/307)). With this tool, we can confidently merge new features / bug fixes without worrying about introducing catastrophic regression. The users can run commands such as:

```bash
python -m cleanrl_utils.rlops --exp-name ddpg_continuous_action \
    --wandb-project-name cleanrl \
    --wandb-entity openrlbenchmark \
    --tags 'pr-299' 'rlops-pilot' \
    --env-ids HalfCheetah-v2 Walker2d-v2 Hopper-v2 InvertedPendulum-v2 Humanoid-v2 Pusher-v2 \
    --output-filename compare.png \
    --scan-history \
    --metric-last-n-average-window 100 \
    --report
```

which generates the following image

<img max-width="500px" src="/static/blog/cleanrl-v1/rlops.png">


### Support for Gymnasium

[Farama-Foundation/Gymnasium](https://github.com/Farama-Foundation/Gymnasium) is the next generation of [`openai/gym`](https://github.com/openai/gym) that will continue to be maintained and introduce new features. Please see their [announcement](https://farama.org/Announcing-The-Farama-Foundation) for further detail. We are migrating to `gymnasium` and the progress can be tracked in [:material-github: vwxyzjn/cleanrl#277](https://github.com/vwxyzjn/cleanrl/pull/277).

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Today we&#39;re launching the Farama Foundation, a new nonprofit dedicated to open source reinforcement learning, and we&#39;re beginning by maintaining and standardizing all the major open source reinforcement learning environments. Read more here: <a href="https://t.co/kQqFMQdVqn">https://t.co/kQqFMQdVqn</a></p>&mdash; Farama Foundation (@FaramaFound) <a href="https://twitter.com/FaramaFound/status/1584936111461502977?ref_src=twsrc%5Etfw">October 25, 2022</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

Also, the Farama foundation is working a project called [Shimmy](https://github.com/Farama-Foundation/Shimmy) which offers conversion wrapper for [`deepmind/dm_env`](https://github.com/deepmind/dm_env) environments, such as [`dm_control`](https://github.com/deepmind/dm_control) and [`deepmind/lab`](https://github.com/deepmind/lab). This is an exciting project that will allow us to support `deepmind/dm_env` in the future.


## Contributions

CleanRL has benefited from the contributions of many awesome folks. I would like to cordially thank the core dev members [@dosssman](https://github.com/dosssman) [@yooceii](https://github.com/yooceii) [@Dipamc](https://github.com/Dipamc) [@kinalmehta](https://github.com/kinalmehta) [@bragajj](https://github.com/bragajj) for their efforts in helping maintain the CleanRL repository.  I would also like to give a shout-out to our new contributors [@cool](https://github.com/cool)-RR, [@Howuhh](https://github.com/Howuhh), [@jseppanen](https://github.com/jseppanen), [@joaogui1](https://github.com/joaogui1), [@ALPH2H](https://github.com/ALPH2H), [@ElliotMunro200](https://github.com/ElliotMunro200), [@WillDudley](https://github.com/WillDudley), and [@sdpkjc](https://github.com/sdpkjc).

We always welcome new contributors to the project. If you are interested in contributing to CleanRL (e.g., new features, bug fixes, new algorithms), please check out our reworked [contributing guide](https://docs.cleanrl.dev/contribution/).


## New CleanRL Supported Publications

* Md Masudur Rahman and Yexiang Xue. "Bootstrap Advantage Estimation for Policy Optimization in Reinforcement Learning." In Proceedings of the IEEE International Conference on Machine Learning and Applications (ICMLA), 2022. [https://arxiv.org/pdf/2210.07312.pdf](https://arxiv.org/pdf/2210.07312.pdf)
* Weng, Jiayi, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu et al. "Envpool: A highly parallel reinforcement learning environment execution engine." In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [https://openreview.net/forum?id=BubxnHpuMbG](https://openreview.net/forum?id=BubxnHpuMbG)
* Huang, Shengyi, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. "The 37 Implementation Details of Proximal Policy Optimization." International Conference on Learning Representations 2022 Blog Post Track, [https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)
* Huang, Shengyi, and Santiago Ontañón. "A closer look at invalid action masking in policy gradient algorithms." The International FLAIRS Conference Proceedings, 35. [https://journals.flvc.org/FLAIRS/article/view/130584](https://journals.flvc.org/FLAIRS/article/view/130584)
* Schmidt, Dominik, and Thomas Schmied. "Fast and Data-Efficient Training of Rainbow: an Experimental Study on Atari." Deep Reinforcement Learning Workshop at the 35th Conference on Neural Information Processing Systems, [https://arxiv.org/abs/2111.10247](https://arxiv.org/abs/2111.10247)

---

### Rl Algorithms Ppo Ppo Atari Envpool Xla Jax Atari Hns (`docs/rl-algorithms/ppo/ppo_atari_envpool_xla_jax/atari_hns.md`)

| Environment         |   CleanRL ppo_atari_envpool_xla_jax.py |   openai/baselines' PPO |
|:--------------------|---------------------------------------:|------------------------:|
| Alien-v5            |                            0.219853    |             0.191542    |
| Amidar-v5           |                            0.356735    |             0.315461    |
| Assault-v5          |                           10.6075      |             7.36794     |
| Asterix-v5          |                            0.377642    |             0.39187     |
| Asteroids-v5        |                            0.0203601   |             0.0160295   |
| Atlantis-v5         |                          242.083       |           190.667       |
| BankHeist-v5        |                            1.59491     |             1.59851     |
| BattleZone-v5       |                            0.648278    |             0.51552     |
| BeamRider-v5        |                            0.125822    |             0.143463    |
| Berzerk-v5          |                            0.382584    |             0.304512    |
| Bowling-v5          |                            0.152385    |             0.284618    |
| Boxing-v5           |                            7.66295     |             7.77164     |
| Breakout-v5         |                           14.9339      |            13.4441      |
| Centipede-v5        |                            0.0825898   |             0.160915    |
| ChopperCommand-v5   |                            0.721451    |             0.0186007   |
| CrazyClimber-v5     |                            4.2051      |             4.0279      |
| Defender-v5         |                            3.07092     |             2.98276     |
| DemonAttack-v5      |                           12.465       |             6.60937     |
| DoubleDunk-v5       |                            4.56009     |             4.36364     |
| Enduro-v5           |                            1.4675      |             1.23315     |
| FishingDerby-v5     |                            2.13815     |             2.1809      |
| Freeway-v5          |                            1.1185      |             1.11205     |
| Frostbite-v5        |                            0.196544    |             0.201265    |
| Gopher-v5           |                            5.1566      |             1.22603     |
| Gravitar-v5         |                            0.304855    |             0.21953     |
| Hero-v5             |                            0.792007    |             0.837516    |
| IceHockey-v5        |                            0.519077    |             0.535946    |
| Jamesbond-v5        |                            1.73523     |             1.78046     |
| Kangaroo-v5         |                            2.4236      |             1.2536      |
| Krull-v5            |                            7.29433     |             6.62758     |
| KungFuMaster-v5     |                            1.17164     |             1.28383     |
| MontezumaRevenge-v5 |                            5.05722e-05 |             0           |
| MsPacman-v5         |                            0.324235    |             0.271832    |
| NameThisGame-v5     |                            0.547252    |             0.594366    |
| Phoenix-v5          |                            2.04392     |             1.22382     |
| Pitfall-v5          |                            0.0342621   |             0.0340544   |
| Pong-v5             |                            1.16173     |             1.16443     |
| PrivateEye-v5       |                            0.00107307  |            -5.30276e-05 |
| Qbert-v5            |                            1.22388     |             1.06233     |
| Riverraid-v5        |                            0.440997    |             0.502486    |
| RoadRunner-v5       |                            2.46769     |             5.14629     |
| Robotank-v5         |                            1.36598     |             1.42268     |
| Seaquest-v5         |                            0.0276666   |             0.040157    |
| Skiing-v5           |                            0.189151    |             0.250475    |
| Solaris-v5          |                            0.100747    |             0.0768074   |
| SpaceInvaders-v5    |                            0.666901    |             0.571875    |
| StarGunner-v5       |                            5.51553     |             4.198       |
| Surround-v5         |                            0.427541    |             0.237518    |
| Tennis-v5           |                            0.564862    |             0.908756    |
| TimePilot-v5        |                            1.59937     |             1.32888     |
| Tutankham-v5        |                            1.45338     |             1.19416     |
| UpNDown-v5          |                           38.4992      |            11.5527      |
| Venture-v5          |                            0           |             0.097076    |
| VideoPinball-v5     |                           18.9358      |            11.7083      |
| WizardOfWor-v5      |                            1.35567     |             1.0638      |
| YarsRevenge-v5      |                            1.04117     |             0.0999445   |
| Zaxxon-v5           |                            0.654571    |             0.694391    |

---

### Rl Algorithms Ppo Ppo Atari Envpool Xla Jax Atari Returns (`docs/rl-algorithms/ppo/ppo_atari_envpool_xla_jax/atari_returns.md`)

| Environment         |   CleanRL ppo_atari_envpool_xla_jax.py |   openai/baselines' PPO |
|:--------------------|---------------------------------------:|------------------------:|
| Alien-v5            |                         1744.76        |          1549.42        |
| Amidar-v5           |                          617.137       |           546.406       |
| Assault-v5          |                         5734.04        |          4050.78        |
| Asterix-v5          |                         3341.9         |          3459.9         |
| Asteroids-v5        |                         1669.3         |          1467.19        |
| Atlantis-v5         |                            3.92929e+06 |             3.09748e+06 |
| BankHeist-v5        |                         1192.68        |          1195.34        |
| BattleZone-v5       |                        24937.9         |         20314.3         |
| BeamRider-v5        |                         2447.84        |          2740.02        |
| Berzerk-v5          |                         1082.72        |           887.019       |
| Bowling-v5          |                           44.0681      |            62.2634      |
| Boxing-v5           |                           92.0554      |            93.3596      |
| Breakout-v5         |                          431.795       |           388.891       |
| Centipede-v5        |                         2910.69        |          3688.16        |
| ChopperCommand-v5   |                         5555.84        |           933.333       |
| CrazyClimber-v5     |                       116114           |        111675           |
| Defender-v5         |                        51439.2         |         50045.1         |
| DemonAttack-v5      |                        22824.8         |         12173.9         |
| DoubleDunk-v5       |                           -8.56781     |            -9           |
| Enduro-v5           |                         1262.79        |          1061.12        |
| FishingDerby-v5     |                           21.6222      |            23.8876      |
| Freeway-v5          |                           33.1075      |            32.9167      |
| Frostbite-v5        |                          904.346       |           924.5         |
| Gopher-v5           |                        11369.6         |          2899.57        |
| Gravitar-v5         |                         1141.95        |           870.755       |
| Hero-v5             |                        24628.3         |         25984.5         |
| IceHockey-v5        |                           -4.91917     |            -4.71505     |
| Jamesbond-v5        |                          504.105       |           516.489       |
| Kangaroo-v5         |                         7281.59        |          3791.5         |
| Krull-v5            |                         9384.7         |          8672.95        |
| KungFuMaster-v5     |                        26594.5         |         29116.1         |
| MontezumaRevenge-v5 |                            0.240385    |             0           |
| MsPacman-v5         |                         2461.62        |          2113.44        |
| NameThisGame-v5     |                         5442.67        |          5713.89        |
| Phoenix-v5          |                        14008.5         |          8693.21        |
| Pitfall-v5          |                           -0.0801282   |            -1.47059     |
| Pong-v5             |                           20.309       |            20.4043      |
| PrivateEye-v5       |                           99.5283      |            21.2121      |
| Qbert-v5            |                        16430.7         |         14283.4         |
| Riverraid-v5        |                         8297.21        |          9267.48        |
| RoadRunner-v5       |                        19342.2         |         40325           |
| Robotank-v5         |                           15.45        |            16           |
| Seaquest-v5         |                         1230.02        |          1754.44        |
| Skiing-v5           |                       -14684.3         |        -13901.7         |
| Solaris-v5          |                         2353.62        |          2088.12        |
| SpaceInvaders-v5    |                         1162.16        |          1017.65        |
| StarGunner-v5       |                        53535.9         |         40906           |
| Surround-v5         |                           -2.94558     |            -6.08095     |
| Tennis-v5           |                          -15.0446      |            -9.71429     |
| TimePilot-v5        |                         6224.87        |          5775.53        |
| Tutankham-v5        |                          238.419       |           197.929       |
| UpNDown-v5          |                       430177           |        129459           |
| Venture-v5          |                            0           |           115.278       |
| VideoPinball-v5     |                        42975.3         |         32777.4         |
| WizardOfWor-v5      |                         6247.83        |          5024.03        |
| YarsRevenge-v5      |                        56696.7         |          8238.44        |
| Zaxxon-v5           |                         6015.8         |          6379.79        |

---

--- METRICS ---
- Files Extracted: 57
- Estimated Token Budget: ~105226 tokens
- Canonical Reference: https://codewiki.google/github.com/vwxyzjn/cleanrl