cleanrl Documentation

Technical Documentation: vwxyzjn/cleanrl

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

CleanRL (Clean Implementation of RL Algorithms)

CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:

• 📜 Single-file implementation
  • Every detail about an algorithm variant is put into a single standalone file.
  • For example, our ppo_atari.py only has 340 lines of code but contains all implementation details on how PPO works with Atari games, so it is a great reference implementation to read for folks who do not wish to read an entire modular library.
• 📊 Benchmarked Implementation (7+ algorithms and 34+ games at https://benchmark.cleanrl.dev)
• 📈 Tensorboard Logging
• 🪛 Local Reproducibility via Seeding
• 🎮 Videos of Gameplay Capturing
• 🧫 Experiment Management with Weights and Biases
• 💸 Cloud Integration with docker and AWS

You can read more about CleanRL in our JMLR paper and documentation.

Notable CleanRL-related projects:

• corl-team/CORL: Offline RL algorithm implemented in CleanRL style
• pytorch-labs/LeanRL: Fast optimized PyTorch implementation of CleanRL RL algorithms using CUDAGraphs.

ℹ️ Support for Gymnasium: Farama-Foundation/Gymnasium is the next generation of openai/gym that will continue to be maintained and introduce new features. Please see their announcement for further detail. We are migrating to gymnasium and the progress can be tracked in vwxyzjn/cleanrl#277.

⚠️ NOTE: CleanRL is not a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's variant or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).

Get started Copy Chapter

Prerequisites:

• Python >=3.7.1,<3.11
• uv 0.7.9+

To run experiments locally, give the following a try:

git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .

# alternatively, you could use `uv venv` and do
# `python run cleanrl/ppo.py`
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000

# open another terminal and enter `cd cleanrl/cleanrl`
tensorboard --logdir runs

To use experiment tracking with wandb, run

wandb login # only required for the first time
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000 \
    --track \
    --wandb-project-name cleanrltest

If you are not using uv, you can install CleanRL with requirements.txt:

# core dependencies
pip install -r requirements/requirements.txt

# optional dependencies
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-mujoco_py.txt
pip install -r requirements/requirements-procgen.txt
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-jax.txt
pip install -r requirements/requirements-docs.txt
pip install -r requirements/requirements-cloud.txt
pip install -r requirements/requirements-memory_gym.txt

To run training scripts in other games:

uv venv

# classic control
python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1

# atari
uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
uv pip install ".[envpool]"
python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
uv run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3

# procgen
uv pip install ".[procgen]"
python cleanrl/ppo_procgen.py --env-id starpilot
python cleanrl/ppg_procgen.py --env-id starpilot

# ppo + lstm
uv pip install ".[atari]"
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4

You may also use a prebuilt development environment hosted in Gitpod:

Algorithms Implemented Copy Chapter

Open RL Benchmark Copy Chapter

To make our experimental data transparent, CleanRL participates in a related project called Open RL Benchmark, which contains tracked experiments from popular DRL libraries such as ours, Stable-baselines3, openai/baselines, jaxrl, and others.

Check out https://benchmark.cleanrl.dev/ for a collection of Weights and Biases reports showcasing tracked DRL experiments. The reports are interactive, and researchers can easily query information such as GPU utilization and videos of an agent's gameplay that are normally hard to acquire in other RL benchmarks. In the future, Open RL Benchmark will likely provide an dataset API for researchers to easily access the data (see repo).

Support and get involved Copy Chapter

We have a Discord Community for support. Feel free to ask questions. Posting in Github Issues and PRs are also welcome. Also our past video recordings are available at YouTube

Citing CleanRL Copy Chapter

If you use CleanRL in your work, please cite our technical paper:

@article{huang2022cleanrl,
  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {http://jmlr.org/papers/v23/21-1342.html}
}

Acknowledgement Copy Chapter

CleanRL is a community-powered by project and our contributors run experiments on a variety of hardware.

• We thank many contributors for using their own computers to run experiments
• We thank Google's TPU research cloud for providing TPU resources.
• We thank Hugging Face's cluster for providing GPU resources.

Index (`docs/index.md`) Copy Section

CleanRL - Overview

CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:

• 📜 Single-file implementation
  • Every detail about an algorithm variant is put into a single standalone file.
  • For example, our ppo_atari.py only has 340 lines of code but contains all implementation details on how PPO works with Atari games, so it is a great reference implementation to read for folks who do not wish to read an entire modular library.
• 📊 Benchmarked Implementation (7+ algorithms and 34+ games at https://benchmark.cleanrl.dev)
• 📈 Tensorboard Logging
• 🪛 Local Reproducibility via Seeding
• 🎮 Videos of Gameplay Capturing
• 🧫 Experiment Management with Weights and Biases
• 💸 Cloud Integration with docker and AWS

You can read more about CleanRL in our technical paper and documentation.

CleanRL only contains implementations of online deep reinforcement learning algorithms. If you are looking for offline algorithms, please check out corl-team/CORL, which shares a similar design philosophy as CleanRL.

!!! info

**Support for Gymnasium**: [Farama-Foundation/Gymnasium](https://github.com/Farama-Foundation/Gymnasium) is the next generation of [`openai/gym`](https://github.com/openai/gym) that will continue to be maintained and introduce new features. Please see their [announcement](https://farama.org/Announcing-The-Farama-Foundation) for further detail. We are migrating to `gymnasium` and the progress can be tracked in [vwxyzjn/cleanrl#277](https://github.com/vwxyzjn/cleanrl/pull/277).

!!! warning

CleanRL is *not* a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's variant or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).

Citing CleanRL Copy Chapter

If you use CleanRL in your work, please cite our technical paper:

@article{huang2022cleanrl,
  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {http://jmlr.org/papers/v23/21-1342.html}
}

Rl Algorithms Overview (`docs/rl-algorithms/overview.md`) Copy Section

Overview

Cleanrl Supported Papers Projects (`docs/cleanrl-supported-papers-projects.md`) Copy Section

CleanRL-supported Papers / Projects

CleanRL has become an increasingly popular deep reinforcement learning library, especially among practitioners who prefer more customizable code. Since its debut in July 2019, CleanRL has supported many open source projects and publications. Below are some CleanRL-supported projects and publications.

Feel free to edit this list if your project or paper has used CleanRL.

Publications Copy Chapter

• Md Masudur Rahman and Yexiang Xue. "Bootstrap Advantage Estimation for Policy Optimization in Reinforcement Learning." In Proceedings of the IEEE International Conference on Machine Learning and Applications (ICMLA), 2022. https://arxiv.org/pdf/2210.07312.pdf

• Centa, Matheus, and Philippe Preux. "Soft Action Priors: Towards Robust Policy Transfer." arXiv preprint arXiv:2209.09882 (2022). https://arxiv.org/pdf/2209.09882.pdf

• Weng, Jiayi, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu et al. "Envpool: A highly parallel reinforcement learning environment execution engine." In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=BubxnHpuMbG

• Huang, Shengyi, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. "The 37 Implementation Details of Proximal Policy Optimization." International Conference on Learning Representations 2022 Blog Post Track, https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/

• Huang, Shengyi, and Santiago Ontañón. "A closer look at invalid action masking in policy gradient algorithms." The International FLAIRS Conference Proceedings, 35. https://journals.flvc.org/FLAIRS/article/view/130584

• Schmidt, Dominik, and Thomas Schmied. "Fast and Data-Efficient Training of Rainbow: an Experimental Study on Atari." Deep Reinforcement Learning Workshop at the 35th Conference on Neural Information Processing Systems, https://arxiv.org/abs/2111.10247

• Dossa, Rousslan Fernand Julien, Shengyi Huang, Santiago Ontañón, and Takashi Matsubara. "An Empirical Investigation of Early Stopping Optimizations in Proximal Policy Optimization." IEEE Access 9 (2021): 117981-117992. https://ieeexplore.ieee.org/abstract/document/9520424

• Huang, Shengyi, Santiago Ontañón, Chris Bamford, and Lukasz Grela. "Gym-µRTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning." In 2021 IEEE Conference on Games (CoG), pp. 1-8. IEEE, 2021. https://ieeexplore.ieee.org/abstract/document/9619076

• Huang, Shengyi, and Santiago Ontañón. "Measuring Generalization of Deep Reinforcement Learning Applied to Real-time Strategy Games", AAAI 2021 Reinforcement Learning in Games Workshop, http://aaai-rlg.mlanctot.info/papers/AAAI21-RLG_paper_33.pdf

• Bamford, Chris, Huang, Shengyi, and Lucas, Simon, "Griddly: A platform for AI research in games", AAAI 2021 Reinforcement Learning in Games Workshop, https://arxiv.org/abs/2011.06363

• Huang, Shengyi, and Santiago Ontañón. "Action guidance: Getting the best of sparse rewards and shaped rewards for real-time strategy games." AIIDE Workshop on Artificial Intelligence for Strategy Games, https://arxiv.org/abs/2010.03956

• Huang, Shengyi, and Santiago Ontañón. "Comparing Observation and Action Representations for Deep Reinforcement Learning in $\mu $ RTS." AIIDE Workshop on Artificial Intelligence for Strategy Gamee, October 2019 https://arxiv.org/abs/1910.12134

Contribution (`docs/contribution.md`) Copy Section

👍🎉 Thank you for taking the time to contribute! 🎉👍

Feel free to open an issue or a Pull Request if you have any questions or suggestions. You can also join our Discord and ask questions there. If you plan to work on an issue, let us know in the issue thread to avoid duplicate work.

Good luck and have fun!

Development Environment Setup Copy Chapter

To setup the development environment, please clone the repository and follow the installation docs and usage docs. They should help you set a working uv environment, so you have the same set-up as other contributors. Additionally, you may want to run the following command to install dev-dependencies for documentation:

uv pip install ".[docs]"

To build the documentation, you can run the following command:

uv run mkdocs serve

For testing, we generally recommend making a PR and let GitHub run the tests in .github/workflows/tests.yaml automatically. These tests are cross-platform and run on Linux, macOS, and Windows. However, if you want to run the tests locally, you can run the following command:

uv run install --all-extras
uv run pytest tests/.

Also, we use pre-commit to helps us automate a sequence of short tasks (called pre-commit "hooks") such as code formatting. In particular, we always use the following hooks when submitting code to the main repository.

• pyupgrade: pyupgrade upgrades syntax for newer versions of the language.
• isort: isort sorts imported dependencies according to their type (e.g, standard library vs third-party library) and name.
• black: black enforces an uniform code style across the codebase.
• codespell: codespell helps avoid common incorrect spelling.

You can run the following command to run the following hooks:

uv run pre-commit run --all-files

which in most cases should automatically fix things as shown below:

Contribution Process Copy Chapter

!!! warning

**Before opening a pull request**, please open an issue first to discuss with us since this is likely a sizable effort. Once we agree on the plan, feel free to make a PR to include the new algorithm.

To ensure the validity of new features or bug fixes and prevent regressions, we use the "RLops" process. Deep Reinforcement Learning (DRL) is brittle and suffers from various reproducibility issues. Even bug fixes can sometimes lead to performance regressions (e.g., see how a bug fix of contact force in MuJoCo results in worse performance for PPO).

Therefore, it is essential to understand how the proposed changes impact the performance of the algorithms. Broadly, we categorize contributions into two types: 1) non-performance-impacting changes and 2) performance-impacting changes.

• non-performance-impacting changes: this type of change does not impact the performance of the algorithm, such as documentation fixes (:material-github: #282), renaming variables (:material-github: #257), and removing unused code (:material-github: #287). For this type of change, we can easily merge them without worrying too much about the consequences.
• performance-impacting changes: this type of change influences the algorithm's performance. Examples include making a slight modification to the gamma parameter in PPO (:material-github: #209), properly handling action bounds in DDPG (:material-github: #211), and fixing bugs (:material-github: #281)

Non-performance-impacting changes are relatively straightforward — we just need to make sure the tests pass and the code is formatted correctly. However, performance-impacting changes are more complicated. We need to make sure that the proposed changes do not lead to performance regressions. To do so, we use the "RLops" process detailed below, which is a set of procedures to help us run experiments and understand the impacts of the proposed changes.

RLops for Performance-impacting Changes: Copy Chapter

Importantly, regardless of the slight difference in performance-impacting changes, we need to re-run the benchmark to ensure there is no regression. To do so, we are going to leverage the openrlbenchmark.rlops CLI utility. This utility is a command-line interface that helps us run experiments and compare the performance of different versions of the codebase.

(Step 1) Run the benchmark Copy Section

Given a new feature, we create a PR and then run the benchmark experiments through benchmark.py, such as the following:

uv pip install ".[docs, mujoco]"
xvfb-run -a python -m cleanrl_utils.benchmark \
    --env-ids HalfCheetah-v4 Walker2d-v4 Hopper-v4 \
    --command "uv run python cleanrl/ddpg_continuous_action.py --track --capture_video" \
    --num-seeds 3 \
    --workers 1

under the hood, this script by default invokes an --autotag feature that tries to tag the the experiments with version control information, such as the git tag (e.g., v1.0.0b2-8-g6081d30) and the github PR number (e.g., pr-299). This is useful for us to compare the performance of the same algorithm across different versions.

(Step 2) Regression check Copy Section

Let's say our latest experiments is tagged with pr-299. We can then run the following command to compare its performance with our pilot experiments rlops-pilot. Note that the pilot experiments include all experiments before we started using RLops (i.e., rlops-pilot is the baseline).

python -m openrlbenchmark.rlops \
    --filters '?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return' \
        'ddpg_continuous_action?tag=pr-299' \
        'ddpg_continuous_action?tag=rlops-pilot' \
    --output-filename compare \
    --scan-history \
    --report

Here, we created a filter '?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return', which is a query string that specifies the following:

• we: the W&B entity name
• wpn: the W&B project name
• ceik: the custom key for the environment id
• cen: the custom key for the experiment name
• metric: the metric we are interested in

So we are fetching metrics from https://wandb.ai/openrlbenchmark/cleanrl. The environment id is stored in the env key, and the experiment name is stored in the exp_name key. The metric we are interested in is charts/episodic_return.

It generates a table comparing the runtime (in minutes):

It also generates a table comparing the episodic return in the last 100 episodes:

!!! info

For more documentation on `openrlbenchmark.rlops`, please refer to its [documentation](https://github.com/openrlbenchmark/openrlbenchmark).

!!! tip

To make the script run faster, we can choose not to use `--scan-history` which allows wandb to sample 500 data points from the training data. This is the default behavior and is much faster.

It also generates the following image and a wandb report.

(Step 3) Update the documentation Copy Section

Once we confirm there is no regression in the performance, we can update the documentation to display the new benchmark results. Run the previous command without comparing previous tags:

python -m cleanrl_utils.rlops --exp-name ddpg_continuous_action \
    --wandb-project-name cleanrl \
    --wandb-entity openrlbenchmark \
    --tags 'pr-299' \
    --env-ids HalfCheetah-v2 Walker2d-v2 Hopper-v2 \
    --output-filename compare.png \
    --scan-history \
    --metric-last-n-average-window 100

which gives us a table like below and a compare.png as the learning curve.

CleanRL's ddpg_continuous_action (pr-299)
HalfCheetah-v2                         10210.57 ± 196.22
Walker2d-v2                             1661.14 ± 250.01
Hopper-v2                               1007.44 ± 148.29

We will use them to update the experimental result section in the docs and replace the learning curves with the new ones.

(Step 4) Update test cases Copy Section

In CleanRL, we have end-to-end test cases that at least ensure the code can run without crashing. When applicable, you should update the test cases in tests and CI setting at .github/workflows/tests.yaml.

(Step 5) Merge the PR Copy Section

Finally, we can merge the PR.

Checklist Copy Chapter

Here is a checklist of the contribution process. See :material-github: #331 as an example for the list of deliverables.

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have updated the tests accordingly (if applicable).
• I have updated the documentation and previewed the changes via mkdocs serve.
  • I have explained note-worthy implementation details.
  • I have explained the logged metrics.
  • I have added links to the original paper and related papers.

If you need to run benchmark experiments for a performance-impacting changes:

• I have contacted @vwxyzjn to obtain access to the openrlbenchmark W&B team.
• I have used the benchmark utility to submit the tracked experiments to the openrlbenchmark/cleanrl W&B project, optionally with --capture_video.
• I have performed RLops with python -m openrlbenchmark.rlops.
  • For new feature or bug fix:
    • I have used the RLops utility to understand the performance impact of the changes and confirmed there is no regression.
  • For new algorithm:
    • I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • I have added the learning curves generated by the python -m openrlbenchmark.rlops utility to the documentation.
  • I have added links to the tracked experiments in W&B, generated by python -m openrlbenchmark.rlops ....your_args... --report, to the documentation.

Advanced Hyperparameter Tuning (`docs/advanced/hyperparameter-tuning.md`) Copy Section

Hyperparameter Tuning

CleanRL comes with a lightweight hyperparameter tuning utility Tuner, with a primary purpose of helping researchers find a single set of hyperparameters that works well with multiple tasks of similar type (e.g., one set of parameters for MuJoCo, another set for Atari). We try to avoid tuning hyperparameters per task, which is usually adhoc and do not generalize.

Get started Copy Chapter

Consider the :material-github: tuner_example.py in the CleanRL's root directory, which has the following content:

import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": [0, 500],
        "Acrobot-v1": [-500, 0],
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)

You can run the tuner with the following command:

uv pip install ".[optuna]"
python tuner_example.py

Often it is useful to find a single set of hyper parameters that work well with multiple environments, but this is challenging because each environment may have different reward scales, so we need to normalize the reward scales.

This is where target_scores comes in. We can use it to specify an upper and lower threshold of rewards for each environment (they don't have to be exact boundaries and can be just ballpark estimates). For example, if we want to find a set of hyper parameters that work well with CartPole-v1 and Acrobot-v1, we can set the target_scores and aggregation_type as follows:

tuner = Tuner(
    ...,
    target_scores={
        "CartPole-v1": [0, 500],
        "Acrobot-v1": [-500, 0],
    }
    aggregation_type="average",
)

Here is what happened when running python tuner_example.py:

1. The tuner_example.py launches num_trials=100 trials to find the best single set of hyperparameters for CartPole-v1 and Acrobot-v1 in script="cleanrl/ppo.py".
2. Each trial samples a set of hyperparameters from the params_fn to run num_seeds=3 experiments with different random seeds, mitigating the impact of randomness on the results.
   • In each experiment, tuner_example.py averages the last metric_last_n_average_window=50 reported metric="charts/episodic_return" to a number $x_i$ and calculate a normalized score $z_i$ according to the target_scores. In this case, $z_{i_0} = (x_{i_0} - 0) / (500 - 0)$ and $z_{i_1} = (x_{i_1} - -500) / (0 - -500)$. Then the tuner will aggregation_type="average" the two scores to get the normalized score $z_i = (z_{i_0} + z_{i_1}) / 2$.
3. Each trial then averages the normalized scores $z_i$ of the num_seeds=3 experiments to a number $z$ and the tuner optimizes $z$ according direction="maximize".

Note that we are using 3 random seeds for each environment in ["CartPole-v1","Acrobot-v1"], totalling 2*3=6 experiments per trial.

???+ info

When optimizing Atari games, you can put the human normalized scores in `target_scores` (Mnih et al., 2015, Extended Data Table 2)[^1], as done in the following example. The first number for each environment is the score obtained by random play and the second number is the score obtained by professional game testers. Note here we using `aggregation_type="median"`, this means we will optimize for the **median** of the human normalized scores **averaged** over the `num_seeds=3` experiments for each trial (pay close attention to the phrasing).

```python
tuner = Tuner(
    script="cleanrl/ppo_atari_envpool.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="median",
    target_scores={
        "Alien-v5": [227.8, 6875],
        "Amidar-v5": [5.8, 1676],
        'Assault-v5': (222.4, 1496),
        'Asterix-v5': (210.0, 8503),
        'Asteroids-v5': (719.1, 13157),
        ...
    },
    num_seeds=3,
    ...
)
```

Visualization Copy Chapter

Running python tuner_example.py will create a sqlite database containing all of the hyperparameter trials in ./cleanrl_hpopt.db We can use optuna-dashboard to visualize the process.

uv run optuna-dashboard sqlite:///cleanrl_hpopt.db

In the panel above, the y-axis shows the normalized score of the two environments (CartPole-v1 and Acrobot-v1) and the x-axis shows the number of trials.

The panel above shows the parallel coordinates plot of the normalized score — for example, we can see having a lower learning rate and a higher number of minibatches is a good choice.

???+ info

You can use a different database by passing `Tuner(..., storage="mysql://root@localhost/example")` to use a more persistent storage.

Work w/o knowing the reward scales Copy Chapter

What if we don't know the reward scales for the environments? We can optionally set the value target_scores to None, but this will only work with one environment at a time because it's difficult to optimize without knowing the scale of the rewards in each environment.

import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": None,
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)

In the example above, we will set the normalized score $z_i$ to be the average of the last metric_last_n_average_window=50 reported metric="charts/episodic_return" for CartPole-v1.

Work w/ pruners and samplers Copy Chapter

You can use Tuner with any pruner from optuna to prune less promising experiments or samplers to sample new hyperparameters. If you don't specify them explicitly, the script will use the default ones.

import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": None,
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)

Track experiments w/ Weights and Biases Copy Chapter

The Tuner can track all the experiments into Weights and Biases to help you visualize the progress of the tuning.

import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": None,
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
    wandb_kwargs={"project": "cleanrl"},
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)

[^1]:Mnih, V., Kavukcuoglu, K., Silver, D. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236

Advanced Resume Training (`docs/advanced/resume-training.md`) Copy Section

Resume Training

A common question we get asked is how to set up model checkpoints to continue training. In this document, we take this PPO example to explain that question.

Save model checkpoints Copy Chapter

The first step is to save models periodically. By default, we save the model to wandb.

num_updates = args.total_timesteps // args.batch_size

CHECKPOINT_FREQUENCY = 50
starting_update = 1

for update in range(starting_update, num_updates + 1):
    # ... do rollouts and train models

    if args.track:
        # make sure to tune `CHECKPOINT_FREQUENCY` 
        # so models are not saved too frequently
        if update % CHECKPOINT_FREQUENCY == 0:
            torch.save(agent.state_dict(), f"{wandb.run.dir}/agent.pt")
            wandb.save(f"{wandb.run.dir}/agent.pt", policy="now")

Then we could run the following to train our agents

python ppo_gridnet.py --prod-mode --capture_video

If the training was terminated early, we can still see the last updated model agent.pt in W&B like in this URL https://wandb.ai/costa-huang/cleanRL/runs/21421tda/files or as follows

Resume training Copy Chapter

The second step is to automatically download the agent.pt from the URL above and resume training as follows:

num_updates = args.total_timesteps // args.batch_size

CHECKPOINT_FREQUENCY = 50
starting_update = 1

if args.track and wandb.run.resumed:
    starting_update = run.summary.get("charts/update") + 1
    global_step = starting_update * args.batch_size
    api = wandb.Api()
    run = api.run(f"{run.entity}/{run.project}/{run.id}")
    model = run.file("agent.pt")
    model.download(f"models/{experiment_name}/")
    agent.load_state_dict(torch.load(
        f"models/{experiment_name}/agent.pt", map_location=device))
    agent.eval()
    print(f"resumed at update {starting_update}")

for update in range(starting_update, num_updates + 1):
    # ... do rollouts and train models

    if args.track:
        # make sure to tune `CHECKPOINT_FREQUENCY` 
        # so models are not saved too frequently
        if update % CHECKPOINT_FREQUENCY == 0:
            torch.save(agent.state_dict(), f"{wandb.run.dir}/agent.pt")
            wandb.save(f"{wandb.run.dir}/agent.pt", policy="now")

To resume training, note the ID of the experiment is 21421tda as in the URL https://wandb.ai/costa-huang/cleanRL/runs/21421tda, so we need to pass in the ID via environment variable to trigger the resume mode of W&B:

WANDB_RUN_ID=21421tda WANDB_RESUME=must python ppo_gridnet.py --prod-mode --capture_video

Benchmark Ddpg (`docs/benchmark/ddpg.md`) Copy Section

Benchmark Ppo (`docs/benchmark/ppo.md`) Copy Section

Benchmark Ppo Atari (`docs/benchmark/ppo_atari.md`) Copy Section

Benchmark Ppo Atari Envpool (`docs/benchmark/ppo_atari_envpool.md`) Copy Section

Benchmark Ppo Atari Envpool Runtimes (`docs/benchmark/ppo_atari_envpool_runtimes.md`) Copy Section

Benchmark Ppo Atari Envpool Xla Jax (`docs/benchmark/ppo_atari_envpool_xla_jax.md`) Copy Section

Benchmark Ppo Atari Envpool Xla Jax Runtimes (`docs/benchmark/ppo_atari_envpool_xla_jax_runtimes.md`) Copy Section

Benchmark Ppo Atari Envpool Xla Jax Scan (`docs/benchmark/ppo_atari_envpool_xla_jax_scan.md`) Copy Section

Benchmark Ppo Atari Envpool Xla Jax Scan Runtimes (`docs/benchmark/ppo_atari_envpool_xla_jax_scan_runtimes.md`) Copy Section

Benchmark Ppo Atari Lstm (`docs/benchmark/ppo_atari_lstm.md`) Copy Section

Benchmark Ppo Atari Lstm Runtimes (`docs/benchmark/ppo_atari_lstm_runtimes.md`) Copy Section

Benchmark Ppo Atari Multigpu (`docs/benchmark/ppo_atari_multigpu.md`) Copy Section

Benchmark Ppo Atari Multigpu Runtimes (`docs/benchmark/ppo_atari_multigpu_runtimes.md`) Copy Section

Benchmark Ppo Atari Runtimes (`docs/benchmark/ppo_atari_runtimes.md`) Copy Section

Benchmark Ppo Continuous Action (`docs/benchmark/ppo_continuous_action.md`) Copy Section

Benchmark Ppo Continuous Action Runtimes (`docs/benchmark/ppo_continuous_action_runtimes.md`) Copy Section

Benchmark Ppo Envpool (`docs/benchmark/ppo_envpool.md`) Copy Section

Benchmark Ppo Envpool Runtimes (`docs/benchmark/ppo_envpool_runtimes.md`) Copy Section

Benchmark Ppo Procgen (`docs/benchmark/ppo_procgen.md`) Copy Section

Benchmark Ppo Procgen Runtimes (`docs/benchmark/ppo_procgen_runtimes.md`) Copy Section

Benchmark Ppo Runtimes (`docs/benchmark/ppo_runtimes.md`) Copy Section

Benchmark Sac (`docs/benchmark/sac.md`) Copy Section

Benchmark Sac Runtimes (`docs/benchmark/sac_runtimes.md`) Copy Section

Benchmark Td3 (`docs/benchmark/td3.md`) Copy Section

Benchmark Td3 Runtimes (`docs/benchmark/td3_runtimes.md`) Copy Section

Cloud Installation (`docs/cloud/installation.md`) Copy Section

Installation

The rough idea behind the cloud integration is to package our code into a docker container and use AWS Batch to
run thousands of experiments concurrently.

Prerequisites Copy Chapter

• Terraform (see installation tutorial here)

We use Terraform to define our infrastructure with AWS Batch, which you can spin up as follows

# assuming you are at the root of the CleanRL project
uv pip install ".[cloud]"
cd cloud
python -m awscli configure
terraform init
export AWS_DEFAULT_REGION=$(aws configure get region --profile default)
terraform apply

!!! note
Don't worry about the cost of spinning up these AWS Batch compute environments and job queues. They are completely free and you are only charged when you submit experiments.

Then your AWS Batch console should look like

Clean Up Copy Section

Uninstalling/Deleting the infrastructure is pretty straightforward:

export AWS_DEFAULT_REGION=$(aws configure get region --profile default)
terraform destroy

Cloud Submit Experiments (`docs/cloud/submit-experiments.md`) Copy Section

Submit Experiments

Inspection Copy Section

Dry run to inspect the generated docker command

uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --num-seed 1

The generated docker command should look like

docker run -d --cpuset-cpus="0" -e WANDB_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx vwxyzjn/cleanrl:latest /bin/bash -c "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video --seed 1"

Run on AWS Copy Section

Submit a job using AWS's compute-optimized spot instances

uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --job-queue c5a-large-spot \
    --num-seed 1 \
    --num-vcpu 1 \
    --num-memory 2000 \
    --num-hours 48.0 \
    --provider aws

Submit a job using AWS's accelerated-computing spot instances

uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4 --track --capture_video" \
    --job-queue g4dn-xlarge-spot \
    --num-seed 1 \
    --num-vcpu 1 \
    --num-gpu 1 \
    --num-memory 4000 \
    --num-hours 48.0 \
    --provider aws

Submit a job using AWS's compute-optimized on-demand instances

uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --job-queue c5a-large \
    --num-seed 1 \
    --num-vcpu 1 \
    --num-memory 2000 \
    --num-hours 48.0 \
    --provider aws

Submit a job using AWS's accelerated-computing on-demand instances

uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4 --track --capture_video" \
    --job-queue g4dn-xlarge \
    --num-seed 1 \
    --num-vcpu 1 \
    --num-gpu 1 \
    --num-memory 4000 \
    --num-hours 48.0 \
    --provider aws

Then you should see:

Customize the Docker Container Copy Chapter

Set up docker's buildx and login in to your preferred registry.

docker buildx create --use
docker login

Then you could build a container using the --build flag based on the Dockerfile in the current directory. Also, --push will auto-push to the docker registry.

uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --build --push

To build a multi-arch image using --archs linux/arm64,linux/amd64:

uv run python -m cleanrl_utils.submit_exp \
    --docker-tag vwxyzjn/cleanrl:latest \
    --command "uv run python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 100000 --track --capture_video" \
    --archs linux/arm64,linux/amd64
    --build --push

!!! note
Building an multi-arch image is quite slow but will allow you to use ARM instances such as m6gd.medium that is 20-70% cheaper than X86 instances.
However, note there is no cloud providers that give ARM instances with Nvidia's GPU (to my knowledge), so this effort might not be worth it.

If you still wants to pursue multi-arch, you can speed things up by using a native ARM server and connect it to your `buildx` instance:

```
docker -H ssh://costa@gpu info
docker buildx create --name remote --use
docker buildx create --name remote --append ssh://costa@gpu
docker buildx inspect --bootstrap
python -m cleanrl_utils.submit_exp -b --archs linux/arm64,linux/amd64
```

Get Started Basic Usage (`docs/get-started/basic-usage.md`) Copy Section

Basic Usage

Two Ways to Run Copy Chapter

After the dependencies have been installed, there are two ways to run
the CleanRL script under the uv virtual environments.

1. Using uv run:

   bash Copy

   uv run python cleanrl/ppo.py \
       --seed 1 \
       --env-id CartPole-v0 \
       --total-timesteps 50000

2. Using uv venv:

   1. We first activate the virtual environment by using
      uv venv
   2. Then, run any desired CleanRL script

   Attention: Each step must be executed separately!

   bash Copy

   uv venv

   bash Copy

   python cleanrl/ppo.py \
       --seed 1 \
       --env-id CartPole-v0 \
       --total-timesteps 50000

!!! note

We recommend `uv venv` workflow for development. When the shell is activated, you should
be seeing a prefix like `(cleanrl-iXg02GqF-py3.9)` in your shell's prompt, which is the name
of the poetry's virtual environment.
**We will assume to run other commands (e.g. `tensorboard`) in the documentation within the poetry's shell.**

!!! warning

If you are using NVIDIA ampere GPUs (e.g., 3060 TI), you might meet the following error

```bash
NVIDIA GeForce RTX 3060 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3060 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Traceback (most recent call last):
File "ppo_atari_envpool.py", line 240, in <module>
    action, logprob, _, value = agent.get_action_and_value(next_obs)
File "ppo_atari_envpool.py", line 156, in get_action_and_value
    hidden = self.network(x / 255.0)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

This is because the `torch` wheel on PyPi is built with cuda 10.2. You would need to manually install the cuda 11.3 wheel like this:
```bash
uv pip install torch==1.12.1 --upgrade --extra-index-url https://download.pytorch.org/whl/cu113
```
Then, you can run the script again.

Visualize Training Metrics Copy Chapter

By default, the CleanRL scripts record all the training metrics via Tensorboard
into the runs folder. So, after running the training script above, feel free to run

tensorboard --logdir runs

Visualize the Agent's Gameplay Videos Copy Chapter

CleanRL helps record the agent's gameplay videos with a --capture_video flag,
which will save the videos in the videos/{$run_name} folder.

python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000 \
    --capture_video

Get Documentation Copy Chapter

You can directly obtained the documentation by using the --help flag.

/* Detailed source-code truncated for AI context efficiency. */

Get Started Benchmark Utility (`docs/get-started/benchmark-utility.md`) Copy Section

Benchmark Utility

CleanRL comes with a utility module cleanrl_utils.benchmark to help schedule and run benchmark experiments on your local machine.

Usage Copy Chapter

Try running python -m cleanrl_utils.benchmark --help to get the help text.

/* Detailed source-code truncated for AI context efficiency. */

Examples Copy Chapter

The following example demonstrates how to run classic control benchmark experiments.

OMP_NUM_THREADS=1 xvfb-run -a python -m cleanrl_utils.benchmark \
    --env-ids CartPole-v1 Acrobot-v1 MountainCar-v0 \
    --command "uv run python cleanrl/ppo.py --no_cuda --track --capture_video" \
    --num-seeds 3 \
    --workers 5

What just happened here? In principle the helps run the following commands in 5 subprocesses:

uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id CartPole-v1 --seed 1
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id Acrobot-v1 --seed 1
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id MountainCar-v0 --seed 1
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id CartPole-v1 --seed 2
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id Acrobot-v1 --seed 2
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id MountainCar-v0 --seed 2
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id CartPole-v1 --seed 3
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id Acrobot-v1 --seed 3
uv run python cleanrl/ppo.py --no_cuda --track --capture_video --env-id MountainCar-v0 --seed 3

More specifically:

1. --env-ids CartPole-v1 Acrobot-v1 MountainCar-v0 specifies that running experiments against these three environments
2. --command "uv run python cleanrl/ppo.py --no_cuda --track --capture_video" suggests running ppo.py with these settings:
   • turn off GPU usage via --no_cuda: because ppo.py has such as small neural network it often runs faster on CPU only
   • track the experiments via --track
   • render the agent gameplay videos via --capture_video; these videos algo get saved to the tracked experiments
     • xvfb-run -a virtualizes a display for video recording, enabling these commands on a headless linux system
3. --num-seeds 3 suggests running the the command with 3 random seeds for each env-id
4. --workers 5 suggests at maximum using 5 subprocesses to run the experiments
   • OMP_NUM_THREADS=1 suggests torch to use only 1 thread for each subprocesses; this way we don't have processes fighting each other.
5. --autotag tries to tag the the experiments with version control information, such as the git tag (e.g., v1.0.0b2-8-g6081d30) and the github PR number (e.g., pr-299). This is useful for us to compare the performance of the same algorithm across different versions.

Note that when you run with high-throughput environments such as envpool or procgen, it's recommended to set --workers 1 to maximuize SPS (steps per second), such as

xvfb-run -a python -m cleanrl_utils.benchmark \
    --env-ids Pong-v5 BeamRider-v5 Breakout-v5 \
    --command "uv run python cleanrl/ppo_atari_envpool.py --track --capture_video" \
    --num-seeds 3 \
    --workers 1

For more example usage, see https://github.com/vwxyzjn/cleanrl/blob/master/benchmark

Slurm integration Copy Chapter

If you have access to a slurm cluster, you can use cleanrl_utils.benchmark to schedule jobs on the cluster. The following example demonstrates how to run classic control benchmark experiments on a slurm cluster.

--8<-- "benchmark/ppo.sh:3:12"

poetry install
OMP_NUM_THREADS=1 xvfb-run -a uv run python -m cleanrl_utils.benchmark \
    --env-ids CartPole-v1 Acrobot-v1 MountainCar-v0 \
    --command "uv run python cleanrl/ppo.py --no_cuda --track --capture_video" \
    --num-seeds 3 \
    --workers 9 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 10 \
    --slurm-template-path benchmark/cleanrl_1gpu.slurm_template

Here, we have

• --slurm-gpus-per-task 1 suggests that each slurm job should use 1 GPU
• --slurm-ntasks 1 suggests that each slurm job should use 1 CPU
• --slurm-total-cpus 10 suggests that each slurm job should use 10 CPUs in total
• --slurm-template-path benchmark/cleanrl_1gpu.slurm_template suggests that we should use the template file benchmark/cleanrl_1gpu.slurm_template to generate the slurm job scripts. The template file looks like this:

--8<-- "benchmark/cleanrl_1gpu.slurm_template"

The utility will generate a slurm script based on the template file and submit the job to the cluster. The generated slurm script looks like this:

#!/bin/bash
#SBATCH --job-name=low-priority
#SBATCH --partition=production-cluster
#SBATCH --gpus-per-task=1
#SBATCH --cpus-per-gpu=10
#SBATCH --ntasks=1
#SBATCH --output=slurm/logs/%x_%j.out
#SBATCH --array=0-8%9
#SBATCH --mem-per-cpu=12G
#SBATCH --exclude=ip-26-0-147-[245,247],ip-26-0-156-239
##SBATCH --nodelist=ip-26-0-156-13


env_ids=(CartPole-v1 Acrobot-v1 MountainCar-v0)
seeds=(1 2 3)
env_id=${env_ids[$SLURM_ARRAY_TASK_ID / 3]}
seed=${seeds[$SLURM_ARRAY_TASK_ID % 3]}

echo "Running task $SLURM_ARRAY_TASK_ID with env_id: $env_id and seed: $seed"

srun uv run python cleanrl/ppo.py --no_cuda --track --env-id $env_id --seed $seed #

Get Started Examples (`docs/get-started/examples.md`) Copy Section

Examples

Atari Copy Chapter

uv venv

uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
uv pip install ".[envpool]"
python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
uv run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3

Demo Copy Section

You can also run training scripts in other games, such as:

Classic Control Copy Chapter

uv venv

python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1

Procgen Copy Chapter

uv venv

uv pip install ".[procgen]"
python cleanrl/ppo_procgen.py --env-id starpilot
python cleanrl/ppg_procgen.py --env-id starpilot

PPO + LSTM Copy Chapter

uv venv

uv pip install ".[atari]"
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4

Get Started Experiment Tracking (`docs/get-started/experiment-tracking.md`) Copy Section

Experiment tracking

To use experiment tracking with wandb, run with the --track flag, which will also
upload the videos recorded by the --capture_video flag.

uv venv
wandb login # only required for the first time
python cleanrl/ppo.py --track --capture_video

The console will output the url for the tracked experiment like the following

wandb:  View project at https://wandb.ai/costa-huang/cleanRL                            
wandb:  View run at https://wandb.ai/costa-huang/cleanRL/runs/10dwbgeh

When you open the URL, it's going to look like the following page:

Get Started Installation (`docs/get-started/installation.md`) Copy Section

Installation

Prerequisites Copy Chapter

• Python >=3.7.1,<3.11
• uv 0.7.19+

Simply run the following command for a quick start

git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .

!!! note "Working with different CUDA versions for torch"

By default, the `torch` wheel is built with CUDA 10.2. If you are using newer NVIDIA GPUs (e.g., 3060 TI), you may need to specifically install CUDA 11.3 wheels by overriding the `torch` dependency with `pip`:

```bash
uv pip install "torch==1.12.1" --upgrade --extra-index-url https://download.pytorch.org/whl/cu113
```

Install via `pip` Copy Chapter

While we recommend using uv to manage environments and dependencies, the traditional requirements.txt are available:

# core dependencies
pip install -r requirements/requirements.txt

# optional dependencies
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-mujoco_py.txt
pip install -r requirements/requirements-procgen.txt
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-jax.txt
pip install -r requirements/requirements-docs.txt
pip install -r requirements/requirements-cloud.txt

Optional Dependencies Copy Chapter

CleanRL makes it easy to install optional dependencies for common RL environments
and various development utilities. These optional dependencies are defined at the
pyproject.toml as optional dependencies

You can install them using the following command

uv pip install ".[atari]"
uv pip install ".[mujoco]"
uv pip install ".[dm_control]"
uv pip install ".[procgen]"
uv pip install ".[envpool]"
uv pip install ".[pettingzoo]"
uv pip install ".[jax]"
uv pip install ".[optuna]"
uv pip install ".[docs]"
uv pip install ".[cloud]"

Get Started Zoo (`docs/get-started/zoo.md`) Copy Section

🤗 Model Zoo

CleanRL now has 🧪 experimental support for saving and loading models from 🤗 HuggingFace's Model Hub. We are rolling out this feature in phases, and currently only support saving and loading models from the following algorithm variants:

Load models from the Model Hub Copy Chapter

We have a simple utility enjoy.py to load models from the hub and run them in an environment. We currently support the following commands:

uv pip install ".[dqn]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn --env-id CartPole-v1
uv pip install ".[dqn, jax]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_jax --env-id CartPole-v1

uv pip install ".[atari]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_atari --env-id BreakoutNoFrameskip-v4
uv pip install ".[atari, jax]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_atari_jax --env-id BreakoutNoFrameskip-v4

To see a list of supported models, please visit 🤗 https://huggingface.co/cleanrl.

???+ info "What happens under the hood?"

The `cleanrl_utils.enjoy` is a simple wrapper to load the models from the hub and run them in an environment. A minimal version of the script can be found at [cleanrl_utils/evals/dqn_eval.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/evals/dqn_eval.py), which may give you a more fine-grained control and access to the model.

<script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Fvwxyzjn%2Fcleanrl%2Fblob%2Fmaster%2Fcleanrl_utils%2Fevals%2Fdqn_eval.py&style=github&type=code&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script>

Save model to Model Hub Copy Chapter

In the supported algorithm variants, you can run the script with the --save-model flag, which saves a model to the runs folder, and the --upload-model flag, which upload the model to huggingface under your default entity (username). Optionally, you may override the default entity with --hf-entity flag.

uv run python cleanrl/dqn_jax.py --env-id CartPole-v1 --save-model --upload-model # --hf-entity cleanrl
uv run python cleanrl/dqn_atari_jax.py --env-id SeaquestNoFrameskip-v4  --save-model --upload-model # --hf-entity cleanrl

Rl Algorithms C51 (`docs/rl-algorithms/c51.md`) Copy Section

Categorical DQN (C51)

Overview Copy Chapter

C51 introduces a distributional perspective for DQN: instead of learning a single value for an action, C51 learns to predict a distribution of values for the action. Empirically, C51 demonstrates impressive performance in ALE.

Original papers:

• A Distributional Perspective on Reinforcement Learning

Implemented Variants Copy Chapter

Below are our single-file implementations of C51:

`c51_atari.py` Copy Chapter

The c51_atari.py has the following features:

• For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[atari]"
uv run python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/c51_atari.py --env-id PongNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id PongNoFrameskip-v4
```

Explanation of the logged metrics Copy Section

Running python cleanrl/c51_atari.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

• charts/episodic_return: episodic return of the game
• charts/SPS: number of steps per second
• losses/loss: the cross entropy loss between the $t$ step state value distribution and the projected $t+1$ step state value distribution
• losses/q_values: implemented as (old_pmfs * q_network.atoms).sum(1), which is the sum of the probability of getting returns $x$ (old_pmfs) multiplied by $x$ (q_network.atoms), averaged over the sample obtained from the replay buffer; useful when gauging if under or over estimation happens

Implementation details Copy Section

c51_atari.py is based on (Bellemare et al., 2017)[^1] but presents a few implementation differences:

1. (Bellemare et al., 2017)[^1] injects stochaticity by doing "on each frame the environment rejects the agent’s selected action with probability $p = 0.25$", but c51_atari.py does not do this
2. c51_atari.py use a self-contained evaluation scheme: c51_atari.py reports the episodic returns obtained throughout training, whereas (Bellemare et al., 2017)[^1] is trained with --end-e=0.01 but reported episodic returns using a separate evaluation process with --end-e=0.001 (See "5.2. State-of-the-Art Results" on page 7).

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/c51.sh. Specifically, execute the following command:

Below are the average episodic returns for c51_atari.py.

Note that we save computational time by reducing timesteps from 50M to 10M, but our c51_atari.py scores the same or higher than (Mnih et al., 2015)[^1] in 10M steps.

Learning curves:

Tracked experiments and game play videos:

`c51.py` Copy Chapter

The c51.py has the following features:

• Works with the Box observation space of low-level features
• Works with the Discrete action space
• Works with envs like CartPole-v1

Usage Copy Section

=== "poetry"

```bash
uv run python cleanrl/c51.py --env-id CartPole-v1
```

=== "pip"

```bash
python cleanrl/c51.py --env-id CartPole-v1
```

Explanation of the logged metrics Copy Section

See related docs for c51_atari.py.

Implementation details Copy Section

The c51.py shares the same implementation details as c51_atari.py except the c51.py runs with different hyperparameters and neural network architecture. Specifically,

1. c51.py uses a simpler neural network as follows:
   python self.network = nn.Sequential( nn.Linear(np.array(env.single_observation_space.shape).prod(), 120), nn.ReLU(), nn.Linear(120, 84), nn.ReLU(), nn.Linear(84, env.single_action_space.n), )

2. c51.py runs with different hyperparameters:

   bash Copy

   python c51.py --total-timesteps 500000 \
       --learning-rate 2.5e-4 \
       --buffer-size 10000 \
       --gamma 0.99 \
       --target-network-frequency 500 \
       --max-grad-norm 0.5 \
       --batch-size 128 \
       --start-e 1 \
       --end-e 0.05 \
       --exploration-fraction 0.5 \
       --learning-starts 10000 \
       --train-frequency 10

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/c51.sh. Specifically, execute the following command:

Below are the average episodic returns for c51.py.

Note that the C51 has no official benchmark on classic control environments, so we did not include a comparison. That said, our c51.py was able to achieve near perfect scores in CartPole-v1 and Acrobot-v1; further, it can obtain successful runs in the sparse environment MountainCar-v0.

Learning curves:

Tracked experiments and game play videos:

`c51_atari_jax.py` Copy Chapter

The c51_atari_jax.py has the following features:

• Uses Jax, Flax, and Optax instead of torch. c51_atari_jax.py is roughly 25% faster than c51_atari.py
• For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[atari, jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/c51_atari_jax.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/c51_atari_jax.py --env-id PongNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/c51_atari_jax.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari_jax.py --env-id PongNoFrameskip-v4
```

Explanation of the logged metrics Copy Section

See related docs for c51_atari.py.

Implementation details Copy Section

See related docs for c51_atari.py.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/c51.sh. Specifically, execute the following command:

Below are the average episodic returns for c51_atari_jax.py.

Learning curves:

Tracked experiments and game play videos:

`c51_jax.py` Copy Chapter

The c51_jax.py has the following features:

• Uses Jax, Flax, and Optax instead of torch. c51_jax.py is roughly 55% faster than c51.py
• Works with the Box observation space of low-level features
• Works with the Discrete action space
• Works with envs like CartPole-v1

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/c51_jax.py --env-id CartPole-v1
```

=== "pip"

```bash
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/c51_jax.py --env-id CartPole-v1
```

Explanation of the logged metrics Copy Section

See related docs for c51_atari.py.

Implementation details Copy Section

See related docs for c51.py.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/c51.sh. Specifically, execute the following command:

Below are the average episodic returns for c51_jax.py.

Learning curves:

Tracked experiments and game play videos:

[^1]:Bellemare, M.G., Dabney, W., & Munos, R. (2017). A Distributional Perspective on Reinforcement Learning. ICML.
[^2]:[Proposal] Formal API handling of truncation vs termination. https://github.com/openai/gym/issues/2510
[^3]: Hessel, M., Modayil, J., Hasselt, H.V., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.G., & Silver, D. (2018). Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI.

Rl Algorithms Ddpg (`docs/rl-algorithms/ddpg.md`) Copy Section

Deep Deterministic Policy Gradient (DDPG)

Overview Copy Chapter

DDPG is a popular DRL algorithm for continuous control. It extends DQN to work with the continuous action space by introducing a deterministic actor that directly outputs continuous actions. DDPG also combines techniques from DQN, such as the replay buffer and target network.

Original paper:

• Continuous control with deep reinforcement learning

Reference resources:

• :material-github: sfujim/TD3
• Deep Deterministic Policy Gradient | Spinning Up in Deep RL
• :material-github: ikostrikov/jaxrl (helpful reference when implemented ddpg_continuous_action_jax.py)

Implemented Variants Copy Chapter

Below is our single-file implementation of DDPG:

`ddpg_continuous_action.py` Copy Chapter

The ddpg_continuous_action.py has the following features:

• For continuous action space
• Works with the Box observation space of low-level features
• Works with the Box (continuous) action space

Usage Copy Section

=== "poetry"

```bash
uv pip install .
uv run python cleanrl/ddpg_continuous_action.py --help
uv pip install ".[mujoco]"
uv run python cleanrl/ddpg_continuous_action.py --env-id Hopper-v4
```

=== "pip"

```bash
python cleanrl/ddpg_continuous_action.py --help
pip install -r requirements/requirements-mujoco.txt
python cleanrl/ddpg_continuous_actions.py --env-id Hopper-v4
```

Explanation of the logged metrics Copy Section

Running python cleanrl/ddpg_continuous_action.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

• charts/episodic_return: episodic return of the game

• charts/SPS: number of steps per second

• losses/qf1_loss: the mean squared error (MSE) between the Q values at timestep $t$ and the Bellman update target estimated using the reward $r_t$ and the Q values at timestep $t+1$, thus minimizing the one-step temporal difference. Formally, it can be expressed by the equation below.
  $$
  J(\theta^{Q}) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \big[ (Q(s, a) - y)^2 \big],
  $$
  with the Bellman update target $y = r + \gamma , Q^{'}(s', a')$, where $a' \sim \mu^{'}(s')$, and the replay buffer $\mathcal{D}$.

• losses/actor_loss: implemented as -qf1(data.observations, actor(data.observations)).mean(); it is the negative average Q values calculated based on the 1) observations and the 2) actions computed by the actor based on these observations. By minimizing actor_loss, the optimizer updates the actors parameter using the following gradient (Lillicrap et al., 2016, Algorithm 1)[^1]:

$$ \nabla_{\theta^{\mu}} J \approx \frac{1}{N}\sum_i\left.\left.\nabla_{a} Q\left(s, a \mid \theta^{Q}\right)\right|{s=s{i}, a=\mu\left(s_{i}\right)} \nabla_{\theta^{\mu}} \mu\left(s \mid \theta^{\mu}\right)\right|{s{i}} $$

• losses/qf1_values: implemented as qf1(data.observations, data.actions).view(-1), it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimation happens.

Implementation details Copy Section

Our ddpg_continuous_action.py is based on the OurDDPG.py from :material-github: sfujim/TD3, which presents the the following implementation difference from (Lillicrap et al., 2016)[^1]:

1. ddpg_continuous_action.py uses a gaussian exploration noise $\mathcal{N}(0, 0.1)$, while (Lillicrap et al., 2016)[^1] uses Ornstein-Uhlenbeck process with $\theta=0.15$ and $\sigma=0.2$.

2. ddpg_continuous_action.py runs the experiments using the openai/gym MuJoCo environments, while (Lillicrap et al., 2016)[^1] uses their proprietary MuJoCo environments.

3. ddpg_continuous_action.py uses the following architecture:

   python Copy

   class QNetwork(nn.Module):
       def __init__(self, env):
           super().__init__()
           self.fc1 = nn.Linear(np.array(env.single_observation_space.shape).prod() + np.prod(env.single_action_space.shape), 256)
           self.fc2 = nn.Linear(256, 256)
           self.fc3 = nn.Linear(256, 1)
   
       def forward(self, x, a):
           x = torch.cat([x, a], 1)
           x = F.relu(self.fc1(x))
           x = F.relu(self.fc2(x))
           x = self.fc3(x)
           return x
   
   
   class Actor(nn.Module):
       def __init__(self, env):
           super().__init__()
           self.fc1 = nn.Linear(np.array(env.single_observation_space.shape).prod(), 256)
           self.fc2 = nn.Linear(256, 256)
           self.fc_mu = nn.Linear(256, np.prod(env.single_action_space.shape))
           # action rescaling
           self.register_buffer(
               "action_scale", torch.tensor((env.action_space.high - env.action_space.low) / 2.0, dtype=torch.float32)
           )
           self.register_buffer(
               "action_bias", torch.tensor((env.action_space.high + env.action_space.low) / 2.0, dtype=torch.float32)
           )
   
       def forward(self, x):
           x = F.relu(self.fc1(x))
           x = F.relu(self.fc2(x))
           x = torch.tanh(self.fc_mu(x))
           return x * self.action_scale + self.action_bias

   while (Lillicrap et al., 2016, see Appendix 7 EXPERIMENT DETAILS)[^1] uses the following architecture (difference highlighted):

   python hl_lines="4-6 9-11 19-21" Copy

   class QNetwork(nn.Module):
       def __init__(self, env):
           super(QNetwork, self).__init__()
           self.fc1 = nn.Linear(p.array(env.single_observation_space.shape).prod() + np.prod(env.single_action_space.shape), 400)
           self.fc2 = nn.Linear(400 + np.prod(env.single_action_space.shape), 300)
           self.fc3 = nn.Linear(300, 1)
   
       def forward(self, x, a):
           x = F.relu(self.fc1(x))
           x = torch.cat([x, a], 1)
           x = F.relu(self.fc2(x))
           x = self.fc3(x)
           return x
   
   
   class Actor(nn.Module):
       def __init__(self, env):
           super(Actor, self).__init__()
           self.fc1 = nn.Linear(np.array(env.single_observation_space.shape).prod(), 400)
           self.fc2 = nn.Linear(400, 300)
           self.fc_mu = nn.Linear(300, np.prod(env.single_action_space.shape))
           # action rescaling
           self.register_buffer(
               "action_scale", torch.tensor((env.action_space.high - env.action_space.low) / 2.0, dtype=torch.float32)
           )
           self.register_buffer(
               "action_bias", torch.tensor((env.action_space.high + env.action_space.low) / 2.0, dtype=torch.float32)
           )
   
       def forward(self, x):
           x = F.relu(self.fc1(x))
           x = F.relu(self.fc2(x))
           x = torch.tanh(self.fc_mu(x))
           return x * self.action_scale + self.action_bias

4. ddpg_continuous_action.py uses the following learning rates:

   python Copy

   q_optimizer = optim.Adam(list(qf1.parameters()), lr=3e-4)
   actor_optimizer = optim.Adam(list(actor.parameters()), lr=3e-4)

   while (Lillicrap et al., 2016, see Appendix 7 EXPERIMENT DETAILS)[^1] uses the following learning rates:

   python Copy

   q_optimizer = optim.Adam(list(qf1.parameters()), lr=1e-4)
   actor_optimizer = optim.Adam(list(actor.parameters()), lr=1e-3)

5. ddpg_continuous_action.py uses --batch-size=256 --tau=0.005, while (Lillicrap et al., 2016, see Appendix 7 EXPERIMENT DETAILS)[^1] uses --batch-size=64 --tau=0.001

6. ddpg_continuous_action.py also adds support for handling continuous environments where the lower and higher bounds of the action space are not $[-1,1]$, or are asymmetric.
   The case where the bounds are not $[-1,1]$ is handled in DDPG.py (Fujimoto et al., 2018)[^2] as follows:

class Actor(nn.Module):

    ...

	def forward(self, state):
		a = F.relu(self.l1(state))
		a = F.relu(self.l2(a))
		return self.max_action * torch.tanh(self.l3(a)) # Scale from [-1,1] to [-action_high, action_high]

On the other hand, in CleanRL's ddpg_continuous_action.py, the mean and the scale of the the action space are computed as action_bias and action_scale respectively.
Those scalars are in turn used to scale the output of a tanh activation function in the actor to the original action space range:

class Actor(nn.Module):
    def __init__(self, env):
        ...
        # action rescaling
        self.register_buffer(
            "action_scale", torch.tensor((env.action_space.high - env.action_space.low) / 2.0, dtype=torch.float32)
        )
        self.register_buffer(
            "action_bias", torch.tensor((env.action_space.high + env.action_space.low) / 2.0, dtype=torch.float32)
        )

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = torch.tanh(self.fc_mu(x))
        return x * self.action_scale + self.action_bias # Scale from [-1,1] to [-action_low, action_high]

Additionally, when drawing exploration noise that is added to the actions produced by the actor, CleanRL's ddpg_continuous_action.py centers the distribution the sampled from at action_bias, and the scale of the distribution is set to action_scale * exploration_noise.

???+ info

Note that `Humanoid-v2`, `InvertedPendulum-v2`, `Pusher-v2` have action space bounds that are not the standard `[-1, 1]`. See below.

```
Ant-v2 Observation space: Box(-inf, inf, (111,), float64) Action space: Box(-1.0, 1.0, (8,), float32)
HalfCheetah-v2 Observation space: Box(-inf, inf, (17,), float64) Action space: Box(-1.0, 1.0, (6,), float32)
Hopper-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (3,), float32)
Humanoid-v2 Observation space: Box(-inf, inf, (376,), float64) Action space: Box(-0.4, 0.4, (17,), float32)
InvertedDoublePendulum-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (1,), float32)
InvertedPendulum-v2 Observation space: Box(-inf, inf, (4,), float64) Action space: Box(-3.0, 3.0, (1,), float32)
Pusher-v2 Observation space: Box(-inf, inf, (23,), float64) Action space: Box(-2.0, 2.0, (7,), float32)
Reacher-v2 Observation space: Box(-inf, inf, (11,), float64) Action space: Box(-1.0, 1.0, (2,), float32)
Swimmer-v2 Observation space: Box(-inf, inf, (8,), float64) Action space: Box(-1.0, 1.0, (2,), float32)
Walker2d-v2 Observation space: Box(-inf, inf, (17,), float64) Action space: Box(-1.0, 1.0, (6,), float32)
```

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ddpg.sh. Specifically, execute the following command:

--8<-- "benchmark/ddpg.sh::7"

Below are the average episodic returns for ddpg_continuous_action.py (3 random seeds). To ensure the quality of the implementation, we compared the results against (Fujimoto et al., 2018)[^2].

???+ info

Note that [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) uses gym MuJoCo v4 environments while [`OurDDPG.py`](https://github.com/sfujim/TD3/blob/master/OurDDPG.py) (Fujimoto et al., 2018)[^2] uses the gym MuJoCo v1 environments.

Also note the performance of our `ddpg_continuous_action.py` seems to be worse than the reference implementation on Walker2d and Hopper. This is likely due to :material-github: [openai/gym#938](https://github.com/openai/baselines/issues/938). We would have a hard time reproducing gym MuJoCo v1 environments because they have been long deprecated.

One other thing could cause the performance difference: the original code reported the average episodic return using determinisitc evaluation (i.e., without exploration noise), see [`sfujim/TD3/main.py#L15-L32`](https://github.com/sfujim/TD3/blob/385b33ac7de4767bab17eb02ade4a268d3e4e24f/main.py#L15-L32), whereas we reported the episodic return during training and the policy gets updated between environments steps.

Learning curves:

--8<-- "benchmark/ddpg_plot.sh::9"

`ddpg_continuous_action_jax.py` Copy Chapter

The ddpg_continuous_action_jax.py has the following features:

• Uses Jax, Flax, and Optax instead of torch. ddpg_continuous_action_jax.py is roughly 2.5-4x faster than ddpg_continuous_action.py
• For continuous action space
• Works with the Box observation space of low-level features
• Works with the Box (continuous) action space

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[mujoco, jax]"
uv run python cleanrl/ddpg_continuous_action_jax.py --help
uv run python cleanrl/ddpg_continuous_action_jax.py --env-id Hopper-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-jax.txt
python cleanrl/ddpg_continuous_action_jax.py --help
python cleanrl/ddpg_continuous_action_jax.py --env-id Hopper-v4
```

???+ warning

Note that JAX does not work in Windows :fontawesome-brands-windows:. The official [docs](https://github.com/google/jax#installation) recommends using Windows Subsystem for Linux (WSL) to install JAX.

Explanation of the logged metrics Copy Section

See related docs for ddpg_continuous_action.py.

Implementation details Copy Section

See related docs for ddpg_continuous_action.py.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ddpg.sh. Specifically, execute the following command:

--8<-- "benchmark/ddpg.sh:12:19"

Below are the average episodic returns for ddpg_continuous_action_jax.py (3 random seeds).

{!benchmark/ddpg.md!}

Learning curves:

--8<-- "benchmark/ddpg_plot.sh:11:20"

???+ info

These are some previous experiments with TPUs. Note the results are very similar to the ones above, but the runtime can be different due to different hardware used.


Note that the experiments were conducted on different hardwares, so your mileage might vary. This inconsistency is because 1) re-running expeirments on the same hardware is computationally expensive and 2) requiring the same hardware is not inclusive nor feasible to other contributors who might have different hardwares.

That said, we roughly expect to see a 2-4x speed improvement from using [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) under the same hardware. And if you disable the `--capture_video` overhead, the speed improvement will be even higher.


| Environment      | [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) (RTX 3060 TI) | [`ddpg_continuous_action_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action_jax.py) (VM w/ TPU) | [`ddpg_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ddpg_continuous_action.py) (RTX 3060 TI) | [`OurDDPG.py`](https://github.com/sfujim/TD3/blob/master/OurDDPG.py) (Fujimoto et al., 2018, Table 1)[^2]    |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| HalfCheetah |  9592.25 ± 135.10 | 9125.06 ± 1477.58  | 10210.57 ± 196.22      |8577.29  |
| Walker2d |  1083.15 ± 567.65  | 1303.82 ± 448.41 | 1661.14 ± 250.01     |  3098.11 | 
| Hopper |  1275.28 ± 209.60  | 1145.05 ± 41.95 | 1007.44 ± 148.29         |  1860.02 |    

Learning curves:










 

Tracked experiments and game play videos:

[^1]:Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N.M., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2016). Continuous control with deep reinforcement learning. CoRR, abs/1509.02971. https://arxiv.org/abs/1509.02971

[^2]:Fujimoto, S., Hoof, H.V., & Meger, D. (2018). Addressing Function Approximation Error in Actor-Critic Methods. ArXiv, abs/1802.09477. https://arxiv.org/abs/1802.09477

Rl Algorithms Dqn (`docs/rl-algorithms/dqn.md`) Copy Section

Deep Q-Learning (DQN)

Overview Copy Chapter

As an extension of the Q-learning, DQN's main technical contribution is the use of replay buffer and target network, both of which would help improve the stability of the algorithm.

Original papers:

• Human-level control through deep reinforcement learning

Implemented Variants Copy Chapter

Below are our single-file implementations of DQN:

`dqn_atari.py` Copy Chapter

The dqn_atari.py has the following features:

• For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4

=== "poetry"

```bash
uv pip install ".[atari]"
uv run python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4
```

Explanation of the logged metrics Copy Section

Running python cleanrl/dqn_atari.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

• charts/episodic_return: episodic return of the game
• charts/SPS: number of steps per second
• losses/td_loss: the mean squared error (MSE) between the Q values at timestep $t$ and the Bellman update target estimated using the reward $r_t$ and the Q values at timestep $t+1$, thus minimizing the one-step temporal difference. Formally, it can be expressed by the equation below.
  $$
  J(\theta^{Q}) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \big[ (Q(s, a) - y)^2 \big],
  $$
  with the Bellman update target is $y = r + \gamma , Q^{'}(s', a')$ and the replay buffer is $\mathcal{D}$.
• losses/q_values: implemented as qf1(data.observations, data.actions).view(-1), it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimation happens.

Implementation details Copy Section

dqn_atari.py is based on (Mnih et al., 2015)[^1] but presents a few implementation differences:

1. dqn_atari.py use slightly different hyperparameters. Specifically,
   • dqn_atari.py uses the more popular Adam Optimizer with the --learning-rate=1e-4 as follows:
     python Copy

     optim.Adam(q_network.parameters(), lr=1e-4)

     whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses the RMSProp optimizer with --learning-rate=2.5e-4, gradient momentum 0.95, squared gradient momentum 0.95, and min squared gradient 0.01 as follows:
     python Copy

     optim.RMSprop(
         q_network.parameters(),
         lr=2.5e-4,
         momentum=0.95,
         # ... PyTorch's RMSprop does not directly support
         # squared gradient momentum and min squared gradient
         # so we are not sure what to put here.
     )

   • dqn_atari.py uses --learning-starts=80000 whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses --learning-starts=50000.
   • dqn_atari.py uses --target-network-frequency=1000 whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses --target-network-frequency=10000.
   • dqn_atari.py uses --total-timesteps=10000000 (i.e., 10M timesteps = 40M frames because of frame-skipping) whereas (Mnih et al., 2015)[^1] uses --total-timesteps=50000000 (i.e., 50M timesteps = 200M frames) (See "Training details" under "METHODS" on page 6 and the related source code run_gpu#L32, dqn/train_agent.lua#L81-L82, and dqn/train_agent.lua#L165-L169).
   • dqn_atari.py uses --end-e=0.01 (the final exploration epsilon) whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses --end-e=0.1.
   • dqn_atari.py uses --exploration-fraction=0.1 whereas (Mnih et al., 2015)[^1] (Extended Data Table 1) uses --exploration-fraction=0.02 (all corresponds to 250000 steps or 1M frames being the frame that epsilon is annealed to --end-e=0.1 ).
   • dqn_atari.py handles truncation and termination properly like (Mnih et al., 2015)[^1] by using SB3's replay buffer's handle_timeout_termination=True.
2. dqn_atari.py use a self-contained evaluation scheme: dqn_atari.py reports the episodic returns obtained throughout training, whereas (Mnih et al., 2015)[^1] is trained with --end-e=0.1 but reported episodic returns using a separate evaluation process with --end-e=0.01 (See "Evaluation procedure" under "METHODS" on page 6).
3. dqn_atari.py implements target network updates as Polyak updates. Compared to the original implementation in (Mnih et al., 2015)[^1], this version allows soft updates of the target network weights with --tau (update coefficient) values of less than 1 (i.e. --tau=0.9). Note that by default --tau=1.0 is used to be consistent with (Mnih et al., 2015)[^1].
4. dqn_atari.py uses the standard MSE loss function, whereas (Mnih et al., 2015)[^1] "...found it helpful to clip the error term from the update $r + \gamma \max_{a'}Q(s', a'; \theta^{-}{i}) - Q(s, a; \theta{i})$ to be between -1 and 1" (See "Training algorithm for deep Q-networks" under "METHODS" on page 7).

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/dqn.sh. Specifically, execute the following command:

Below are the average episodic returns for dqn_atari.py.

Note that we save computational time by reducing timesteps from 50M to 10M, but our dqn_atari.py scores the same or higher than (Mnih et al., 2015)[^1] in 10M steps.

Learning curves:

Tracked experiments and game play videos:

`dqn.py` Copy Chapter

The dqn.py has the following features:

• Works with the Box observation space of low-level features
• Works with the Discrete action space
• Works with envs like CartPole-v1

Usage Copy Section

=== "poetry"

```bash
uv run python cleanrl/dqn.py --env-id CartPole-v1
```

=== "pip"

```bash
python cleanrl/dqn.py --env-id CartPole-v1
```

Explanation of the logged metrics Copy Section

See related docs for dqn_atari.py.

Implementation details Copy Section

The dqn.py shares the same implementation details as dqn_atari.py except the dqn.py runs with different hyperparameters and neural network architecture. Specifically,

1. dqn.py uses a simpler neural network as follows:
   python self.network = nn.Sequential( nn.Linear(np.array(env.single_observation_space.shape).prod(), 120), nn.ReLU(), nn.Linear(120, 84), nn.ReLU(), nn.Linear(84, env.single_action_space.n), )

2. dqn.py runs with different hyperparameters:

   bash Copy

   python dqn.py --total-timesteps 500000 \
       --learning-rate 2.5e-4 \
       --buffer-size 10000 \
       --gamma 0.99 \
       --target-network-frequency 500 \
       --max-grad-norm 0.5 \
       --batch-size 128 \
       --start-e 1 \
       --end-e 0.05 \
       --exploration-fraction 0.5 \
       --learning-starts 10000 \
       --train-frequency 10

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/dqn.sh. Specifically, execute the following command:

Below are the average episodic returns for dqn.py.

Note that the DQN has no official benchmark on classic control environments, so we did not include a comparison. That said, our dqn.py was able to achieve near perfect scores in CartPole-v1 and Acrobot-v1; further, it can obtain successful runs in the sparse environment MountainCar-v0.

Learning curves:

Tracked experiments and game play videos:

`dqn_atari_jax.py` Copy Chapter

The dqn_atari_jax.py has the following features:

• Uses Jax, Flax, and Optax instead of torch. dqn_atari_jax.py is roughly 25%-50% faster than dqn_atari.py
• For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[atari, jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/dqn_atari_jax.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/dqn_atari_jax.py --env-id PongNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/dqn_atari_jax.py --env-id BreakoutNoFrameskip-v4
python cleanrl/dqn_atari_jax.py --env-id PongNoFrameskip-v4
```

???+ warning

Note that JAX does not work in Windows :fontawesome-brands-windows:. The official [docs](https://github.com/google/jax#installation) recommends using Windows Subsystem for Linux (WSL) to install JAX.

Explanation of the logged metrics Copy Section

See related docs for dqn_atari.py.

Implementation details Copy Section

See related docs for dqn_atari.py.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/dqn.sh. Specifically, execute the following command:

Below are the average episodic returns for dqn_atari_jax.py (3 random seeds).

???+ info

We observe a speedup of `~25%` in ['dqn_atari_jax.py'](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari_jax.py) compared to ['dqn_atari.py'](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py). This could be because the training loop is tightly integrated with the experience collection loop. We run a training loop every `4` environment steps by default. So more time is utilised in collecting experience than training the network. We observe much more speed-ups in algorithms which run a training step for each environment step. E.g., [DDPG](/rl-algorithms/ddpg/#experiment-results_1)

Learning curves:

Tracked experiments and game play videos:

`dqn_jax.py` Copy Chapter

• Uses Jax, Flax, and Optax instead of torch. dqn_jax.py is roughly 50% faster than dqn.py
• Works with the Box observation space of low-level features
• Works with the Discrete action space
• Works with envs like CartPole-v1

Usage Copy Section

python cleanrl/dqn_jax.py --env-id CartPole-v1

=== "poetry"

```bash
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/dqn_jax.py --env-id CartPole-v1
```

=== "pip"

```bash
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/dqn_jax.py --env-id CartPole-v1
```

Explanation of the logged metrics Copy Section

See related docs for dqn_atari.py.

Implementation details Copy Section

See related docs for dqn.py.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/dqn.sh. Specifically, execute the following command:

Below are the average episodic returns for dqn_jax.py (3 random seeds).

Tracked experiments and game play videos:

[^1]:Mnih, V., Kavukcuoglu, K., Silver, D. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236
[^2]:[Proposal] Formal API handling of truncation vs termination. https://github.com/openai/gym/issues/2510
[^3]: Hessel, M., Modayil, J., Hasselt, H.V., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.G., & Silver, D. (2018). Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI.

Rl Algorithms Ppg (`docs/rl-algorithms/ppg.md`) Copy Section

Phasic Policy Gradient (PPG)

Overview Copy Chapter

PPG is a DRL algorithm that separates policy and value function training by introducing an auxiliary phase. The training proceeds by running PPO during the policy phase, saving all the experience in a replay buffer. Then the replay buffer is used to train the value function. This makes the algorithm considerably slower than PPO, but improves sample efficiency on Procgen benchmark.

Original paper:

• Phasic Policy Gradient

Reference resources:

• Code for the paper "Phasic Policy Gradient" - by original authors from OpenAI

The original code has multiple code level details that are not mentioned in the paper. We found these changes to be important for reproducing the results claimed by the paper.

Implemented Variants Copy Chapter

Below are our single-file implementations of PPG:

`ppg_procgen.py` Copy Chapter

ppg_procgen.py works with the Procgen benchmark, which uses 64x64 RGB image observations, and discrete actions

Usage Copy Section

=== "uv"

```bash
uv pip install ".[procgen]"
uv run python cleanrl/ppg_procgen.py --help
uv run python cleanrl/ppg_procgen.py --env-id starpilot
```

=== "pip"

```bash
pip install -r requirements/requirements-procgen.txt
python cleanrl/ppg_procgen.py --help
python cleanrl/ppg_procgen.py --env-id starpilot
```

Explanation of the logged metrics Copy Section

Running python cleanrl/ppg_procgen.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

Same as PPO:

• charts/episodic_return: episodic return of the game
• charts/episodic_length: episodic length of the game
• charts/SPS: number of steps per second (this is initially high but drops off after the auxiliary phase)
• charts/learning_rate: the current learning rate (annealing is not done by default)
• losses/value_loss: the mean value loss across all data points
• losses/policy_loss: the mean policy loss across all data points
• losses/entropy: the mean entropy value across all data points
• losses/old_approx_kl: the approximate Kullback–Leibler divergence, measured by (-logratio).mean(), which corresponds to the k1 estimator in John Schulman’s blog post on approximating KL
• losses/approx_kl: better alternative to old_approx_kl measured by (logratio.exp() - 1) - logratio, which corresponds to the k3 estimator in approximating KL
• losses/clipfrac: the fraction of the training data that triggered the clipped objective
• losses/explained_variance: the explained variance for the value function

PPG specific:

• losses/aux/kl_loss: the mean value of the KL divergence when distilling the latest policy during the auxiliary phase.
• losses/aux/aux_value_loss: the mean value loss on the auxiliary value head
• losses/aux/real_value_loss: the mean value loss on the detached value head used to calculate the GAE returns during policy phase

Implementation details Copy Section

ppg_procgen.py includes the level implementation details that are different from PPO:

1. Full rollout sampling during auxiliary phase - (:material-github: phasic_policy_gradient/ppg.py#L173) - Instead of randomly sampling observations over the entire auxiliary buffer, PPG samples full rullouts from the buffer (Sets of 256 steps). This full rollout sampling is only done during the auxiliary phase. Note that the rollouts will still be at random starting points because PPO truncates the rollouts per env. This change gives a decent performance boost.

2. Batch level advantage normalization - PPG normalizes the full batch of advantage values before PPO updates instead of advantage normalization on each minibatch. (:material-github: phasic_policy_gradient/ppo.py#L70)

3. Normalized network initialization - (:material-github: phasic_policy_gradient/impala_cnn.py#L64) - PPG uses normalized initialization for all layers, with different scales.

   • Original PPO used orthogonal initialization of only the Policy head and Value heads with scale of 0.01 and 1. respectively.
   • For PPG
     • All weights are initialized with the default torch initialization (Kaiming Uniform)
     • Each layer’s weights are divided by the L2 norm of the weights such that the weights of input_channels axis are individually normalized (axis 1 for linear layers and 1,2,3 for convolutional layers). Then the weights are multiplied by a scale factor.
     • Scale factors for different layers
       • Value head, Policy head, Auxiliary value head - 0.1
       • Fully connected layer after last conv later - 1.4
       • Convolutional layers - Approximately 0.638

4. The Adam Optimizer's Epsilon Parameter -(:material-github: phasic_policy_gradient/ppg.py#L239) - Set to torch default of 1e-8 instead of 1e-5 which is used in PPO.

5. Use the same gamma parameter in the NormalizeReward wrapper. Note that the original implementation from openai/train-procgen uses the default gamma=0.99 in the VecNormalize wrapper but gamma=0.999 as PPO's parameter. The mismatch between the gammas is technically incorrect. See #209

Here are some additional notes:

• All the default hyperparameters from the original PPG implementation are used. Except setting 64 for the number of environments.
• The original PPG paper does not report results on easy environments, hence more hyperparameter tuning can give better results.
• Skipping every alternate auxiliary phase gives similar performance on easy environments while saving compute.
• Normalized network initialization scheme seems to matter a lot, but using layernorm with orthogonal initialization also works.
• Using mixed precision for auxiliary phase also works well to save compute, but using on policy phase makes training unstable.

Also, ppg_procgen.py differs from the original openai/phasic-policy-gradient implementation in the following ways.

• The original PPG code supports LSTM whereas the CleanRL code does not.
• The original PPG code uses separate optimizers for policy and auxiliary phase, but we do not implement this as we found it to not make too much difference.
• The original PPG code utilizes multiple GPUs but our implementation does not

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppg.sh. Specifically, execute the following command:

Below are the average episodic returns for ppg_procgen.py, and comparison with ppg_procgen.py on 25M timesteps.

???+ warning

Note that we have run the procgen experiments using the `easy` distribution for reducing the computational cost. However, the original paper's results were condcuted with the `hard` distribution mode. For convenience, in the learning curves below, we compared the performance of the original code base (`openai/phasic-policy-gradient` the purple curve) in the `easy` distribution.

Learning curves:

???+ info

Also note that our `ppo_procgen.py` which closely matches implementation details of `openai/baselines`' PPO which might not be the same as `openai/phasic-policy-gradient`'s PPO. We take the reported results from (Cobbe et al., 2020)[^1] and (Cobbe et al., 2021)[^2] and compared them in a [google sheet](https://docs.google.com/spreadsheets/d/1ZC_D2WPL6-PzhecM4ZFQWQ6nY6dkXeQDOIgRHVp1BNU/edit?usp=sharing) (screenshot shown below). As shown, the performance seems to diverge a bit. We also note that (Cobbe et al., 2020)[^1] used [`procgen==0.9.2`](https://github.com/openai/train-procgen/blob/1a2ae2194a61f76a733a39339530401c024c3ad8/environment.yml#L10) and (Cobbe et al., 2021)[^2] used [`procgen==0.10.4`](https://github.com/openai/phasic-policy-gradient/blob/7295473f0185c82f9eb9c1e17a373135edd8aacc/environment.yml#L10), which also could cause performance difference. It is for this reason, we ran our own `openai/phasic-policy-gradient` experiments on the `easy` distribution for comparison, but this does mean it's challenging to compare our results against those in the original PPG paper (Cobbe et al., 2021)[^2].

Tracked experiments and game play videos:

[^1]: Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2020, November). Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning (pp. 2048-2056). PMLR.
[^2]: Cobbe, K. W., Hilton, J., Klimov, O., & Schulman, J. (2021, July). Phasic policy gradient. In International Conference on Machine Learning (pp. 2020-2027). PMLR.

Rl Algorithms Ppo Isaacgymenvs (`docs/rl-algorithms/ppo-isaacgymenvs.md`) Copy Section

`ppo_continuous_action_isaacgym.py` Copy Chapter

???+ warning

`ppo_continuous_action_isaacgym.py` is temporarily deprecated. Please checkout the code in [https://github.com/vwxyzjn/cleanrl/releases/tag/v1.0.0](https://github.com/vwxyzjn/cleanrl/releases/tag/v1.0.0)

The ppo_continuous_action_isaacgym.py has the following features:

• Works with IsaacGymEnvs.
• Works with the Box observation space of low-level features
• Works with the Box (continuous) action space

IsaacGymEnvs is a hardware-accelerated (or GPU-accelerated) robotics simulation environment based on torch, which allows us to run thousands of simulation environments at the same time, empowering RL agents to learn many MuJoCo-style robotics tasks in minutes instead of hours. When creating an environment with IsaacGymEnvs via isaacgymenvs.make("Ant"), it creates a vectorized environment which produces GPU tensors as observations and take GPU tensors as actions to execute.

???+ info

Note that **Isaac Gym** is the underlying core physics engine, and **IssacGymEnvs** is a collection of environments built on Isaac Gym.

???+ info

`ppo_continuous_action_isaacgym.py` works with most environments in IsaacGymEnvs but it does not work with the following environments yet:

* AnymalTerrain
* FrankaCabinet
* ShadowHandOpenAI_FF
* ShadowHandOpenAI_LSTM
* Trifinger
* Ingenuity Quadcopter

🔥 we need contributors to work on supporting and tuning our PPO implementation in these envs. If you are interested, please read our [contribution guide](https://github.com/vwxyzjn/cleanrl/blob/master/CONTRIBUTING.md) and reach out!

Usage Copy Section

The installation of isaacgym requires a bit of work since it's not a standard Python package.

Please go to https://developer.nvidia.com/isaac-gym to download and install the latest version of Issac Gym which should look like IsaacGym_Preview_4_Package.tar.gz. Put this IsaacGym_Preview_4_Package.tar.gz into the ~/Downloads/ folder. Make sure your python version is either 3.7, or 3.8 (3.9 not supported yet).

# extract and move the content in `python` folder in the IsaacGym_Preview_4_Package.tar.gz
# into the `cleanrl/ppo_continuous_action_isaacgym/isaacgym/` folder
cp ~/Downloads/IsaacGym_Preview_4_Package.tar.gz IsaacGym_Preview_4_Package.tar.gz 
stat IsaacGym_Preview_4_Package.tar.gz
mkdir temp_isaacgym
tar -xf IsaacGym_Preview_4_Package.tar.gz -C temp_isaacgym
mv temp_isaacgym/isaacgym/python/* cleanrl/ppo_continuous_action_isaacgym/isaacgym
rm -rf temp_isaacgym

# if your global python version is not either 3.7 nor 3.8, you need to tell poetry specifically to use a 3.7 or 3.8 python
# e.g., `poetry env use /home/costa/.pyenv/versions/3.7.8/bin/python`
poetry install --with isaacgym
# if you are using NVIDIA's 30xx GPU, you need to specifically install cuda 11.3 wheels
# `uv pip install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu113`
uv run python cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py --help
uv run python cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py --env-id Ant

???+ warning

If you encounter the following installation error

```bash
Python.h: No such file or directory
#include <Python.h>
```

or 

```bash
libpython3.8.so.1.0: cannot open shared object file: No such file or directory
```

It usually means your python distribution does not include the shared library files. If you are ubuntu, you can install the following packages:

```bash
sudo apt-get install libpython3.8-dev # or sudo apt-get install libpython3.7-dev
```

If you are using [pyenv](https://github.com/pyenv/pyenv), you may try the following:

```bash
env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.7.8
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py.

Additionally, charts/consecutive_successes means the number of consecutive episodes that the agent has successfully manipulating the rubix cube to the desired state.

Implementation details Copy Section

ppo_continuous_action_isaacgym.py is based on ppo_continuous_action.py (see related docs), with a few modifications:

1. Different set of hyperparameters: ppo_continuous_action_isaacgym.py uses hyperparameters primarily derived from rl-games' configuration (see example). The basic spirit is to run more total_timesteps, with larger num_envs and smaller num_steps.

1. Slightly larger NN: ppo_continuous_action.py uses the following NN:
   python Copy

   self.critic = nn.Sequential(
       layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
       nn.Tanh(),
       layer_init(nn.Linear(64, 64)),
       nn.Tanh(),
       layer_init(nn.Linear(64, 1), std=1.0),
   )
   self.actor_mean = nn.Sequential(
       layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
       nn.Tanh(),
       layer_init(nn.Linear(64, 64)),
       nn.Tanh(),
       layer_init(nn.Linear(64, np.prod(envs.single_action_space.shape)), std=0.01),
   )

   while ppo_continuous_action_isaacgym.py uses the following NN:
   python Copy

   self.critic = nn.Sequential(
       layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 256)),
       nn.Tanh(),
       layer_init(nn.Linear(256, 256)),
       nn.Tanh(),
       layer_init(nn.Linear(256, 1), std=1.0),
   )
   self.actor_mean = nn.Sequential(
       layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 256)),
       nn.Tanh(),
       layer_init(nn.Linear(256, 256)),
       nn.Tanh(),
       layer_init(nn.Linear(256, np.prod(envs.single_action_space.shape)), std=0.01),
   )

2. No normalization and clipping: ppo_continuous_action_isaacgym.py does not do observation and reward normalization and clipping for simplicity. It does however optionally offer an option to scale the rewards via --reward-scaler x, which multiplies all the rewards obtained by x as an example.
3. Remove all CPU-related code: ppo_continuous_action_isaacgym.py needs to remove all CPU-related code (e.g. action.cpu().numpy()). This is because almost everything in IsaacGymEnvs happens in GPU. To do this, the major modifications include the following:
   1. Create a custom RecordEpisodeStatisticsTorch wrapper that records statstics using GPU tensors instead of numpy arrays.
   2. Avoid transferring the tensors to CPU. The related code in ppo_continuous_action.py looks like
   python Copy

   next_obs, reward, done, info = envs.step(action.cpu().numpy())
   rewards[step] = torch.tensor(reward).to(device).view(-1)
   next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(done).to(device)

   and the related code in ppo_continuous_action_isaacgym.py looks like

   python Copy

   next_obs, rewards[step], next_done, info = envs.step(action)

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

Below are the average episodic returns for ppo_continuous_action_isaacgym.py. To ensure the quality of the implementation, we compared the results against Denys88/rl_games' PPO and present the training time (units being s (seconds), m (minutes)). The hardware used is a NVIDIA RTX A6000 in a 24 core machine.

Learning curves:

???+ info

Note `ppo_continuous_action_isaacgym.py`'s performance seems poor compared to [Denys88/rl_games](https://github.com/Denys88/rl_games)' PPO. This is likely due to a few reasons.

1. [Denys88/rl_games](https://github.com/Denys88/rl_games)' PPO uses different sets of tuned hyperparameters  and neural network architecture configuration for different tasks, whereas `ppo_continuous_action_isaacgym.py` only uses one neural network architecture and 2 set of hyperparameters (ignoring `--total-timesteps`).
1. `ppo_continuous_action_isaacgym.py` does not use observation normalization (because in my preliminary testing for some reasons it did not help).

While it should be possible to obtain higher scores with more tuning, the purpose of `ppo_continuous_action_isaacgym.py` is to hit a balance between simplicity and performance. I think `ppo_continuous_action_isaacgym.py` has relatively good performance with a concise codebase, which should be easy to modify and extend for practitioners.

Tracked experiments and game play videos:

Old Learning curves w/ Isaac Gym Preview 3 (no longer available in Nvidia's website for download):

???+ info

Note the `AllegroHand` and `ShadowHand` experiments used the following command `ppo_continuous_action_isaacgym.py --track --capture_video --num-envs 16384 --num-steps 8 --update-epochs 5 --reward-scaler 0.01 --total-timesteps 600000000 --record-video-step-frequency 3660`. Costa: I was able to run this during my internship at NVIDIA, but in my home setup, the computer has less GPU memory which makes it hard to replicate the results w/ `--num-envs 16384`.

Rl Algorithms Ppo Rnd (`docs/rl-algorithms/ppo-rnd.md`) Copy Section

Random Network Distillation (RND)

Overview Copy Chapter

RND is an exploration bonus for RL methods that's easy to implement and enables significant progress in some hard exploration Atari games such as Montezuma's Revenge. We use Proximal Policy Optimization as our RL method as used by original paper's implementation

Original paper:

• Exploration by Random Network Distillation

Implemented Variants Copy Chapter

Below are our single-file implementations of RND:

`ppo_rnd_envpool.py` Copy Chapter

The ppo_rnd_envpool.py has the following features:

• Uses the blazing fast Envpool vectorized environment.
• For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discerete action space

???+ warning

Note that `ppo_rnd_envpool.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PPO implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

* Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
* Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

This causes the $s_{last}$ to be off by one. 
See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[envpool]"
uv run python cleanrl/ppo_rnd_envpool.py --help
uv run python cleanrl/ppo_rnd_envpool.py --env-id MontezumaRevenge-v5
```

=== "pip"

```bash
pip install -r requirements/requirements-envpool.txt
python cleanrl/ppo_rnd_envpool.py --help
python cleanrl/ppo_rnd_envpool.py --env-id MontezumaRevenge-v5
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py.
Below is the additional metric for RND:

• charts/episode_curiosity_reward: episodic intrinsic rewards.
• losses/fwd_loss: the prediction error between predict network and target network, can also be viewed as a proxy of the curiosity reward in that batch.

Implementation details Copy Section

ppo_rnd_envpool.py uses a customized RecordEpisodeStatistics to work with envpool but has the same other implementation details as ppo_atari.py (see related docs). Additionally, it has the following additional details:

1. We initialize the normalization parameters by stepping a random agent in the environment by args.num_steps * args.num_iterations_obs_norm_init. args.num_iterations_obs_norm_init=50 comes from the original implementation.
2. We uses sticky action from envpool to facilitate the exploration like done in the original implementation.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/rnd.sh. Specifically, execute the following command:

Below are the average episodic returns for ppo_rnd_envpool.py. To ensure the quality of the implementation, we compared the results against openai/random-network-distillation' PPO.

Note the MontezumaRevengeNoFrameSkip-v4 has same setting to MontezumaRevenge-v5.
Our benchmark has one seed due to limited compute resource and extreme long run time (~250 hours).

Learning curves:

Tracked experiments and game play videos:

[^1]:Burda, Yuri, et al. "Exploration by random network distillation." Seventh International Conference on Learning Representations. 2019.

Rl Algorithms Ppo Trxl (`docs/rl-algorithms/ppo-trxl.md`) Copy Section

Tranformer-XL (PPO-TrXL)

Overview Copy Chapter

Real-world tasks may expose imperfect information (e.g. partial observability). Such tasks require an agent to leverage memory capabilities. One way to do this is to use recurrent neural networks (e.g. LSTM) as seen in :material-github: ppo_atari_lstm.py, :material-file-document: docs. Here, Transformer-XL is used as episodic memory in Proximal Policy Optimization (PPO).

Original Paper and Implementation

• :material-file-document: Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents
• :material-github: neroRL
• :material-github: Episodic Transformer Memory PPO
• :material-github: Endless Memory Gym
• :material-play-circle: Interactive Visualizations of Trained Agents

Related Publications and Repositories

• :material-file-document: Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
• :material-file-document: Stabilizing Transformers for Reinforcement Learning
• :material-file-document: Towards mental time travel: a hierarchical memory for reinforcement learning agents
• :material-file-document: Grounded Language Learning Fast and Slow
• :material-github: transformerXL_PPO_JAX

Implemented Variants Copy Chapter

Below is our single-file implementation of PPO-TrXL:

`ppo_trxl.py` Copy Chapter

ppo_trxl.py has the following features:

• Works with Memory Gym's environments (84x84 RGB image observation).
• Works with Minigrid Memory (84x84 RGB image observation).
• Works also with environments exposing only game state vector observations (e.g. Proof of Memory Environment).
• Works with just single or multi-discrete action spaces.

Usage Copy Section

As the recommended way, the requirements default to PyTorch's CUDA packages.

=== "poetry"

```bash
cd cleanrl/ppo_trxl
poetry install
uv run python ppo_trxl.py --help
uv run python ppo_trxl.py --env-id Endless-MortarMayhem-v0
```

=== "pip"

```bash
pip install -r requirements/requirements-memory_gym.txt
python cleanrl/ppo_trxl/ppo_trxl.py --help
python cleanrl/ppo_trxl/ppo_trxl.py --env-id Endless-MortarMayhem-v0
```

Explanation of the logged metrics Copy Section

• episode/r_mean: mean of the episodic return of the game
• episode/l_mean: mean of the episode length of the game in steps
• episode/t_mean: mean of the episode duration of the game in seconds
• episode/advantage_mean: mean of all computed advantages
• episode/value_mean: mean of all approximated values
• charts/SPS: number of steps per second
• charts/learning_rate: the current learning rate
• charts/entropy_coefficient: the current entropy coefficient
• losses/value_loss: the mean value loss across all data points
• losses/policy_loss: the mean policy loss across all data points
• losses/entropy: the mean entropy value across all data points
• losses/reconstruction_loss: the mean observation reconstruction loss value across all data points
• losses/loss: the mean of all summed losses across all data points
• losses/old_approx_kl: the approximate Kullback–Leibler divergence, measured by (-logratio).mean(), which corresponds to the k1 estimator in John Schulman’s blog post on approximating KL
• losses/approx_kl: better alternative to old_approx_kl measured by (logratio.exp() - 1) - logratio, which corresponds to the k3 estimator in approximating KL
• losses/clipfrac: the fraction of the training data that triggered the clipped objective
• losses/explained_variance: the explained variance for the value function

Implementation details Copy Section

Most details are derived from ppo.py. These are additional or differing details:

1. The policy and value function share parameters.
2. Multi-head attention is implemented so that all heads share parameters.
3. Absolute positional encoding is used as default. Learned positional encodings are supported.
4. Previously computed hidden states of the TrXL layers are cached and reused for up to trxl_memory_length. Only 1 hidden state is computed anew.
5. TrXL layers adhere to pre-layer normalization.
6. Support for multi-discrete action spaces.
7. Support for an auxiliary observation reconstruction loss, which reconstructs TrXL's output to the fed visual observation.
8. The learning rate and the entropy bonus coefficient linearly decay until reaching a lower threshold.

Experiment results Copy Section

Note: When training on potentially endless episodes, the cached hidden states demand a large GPU memory. To reproduce the following experiments a minimum of 40GB is required. One workaround is to cache the hidden states in the buffer with lower precision as bfloat16. This is under examination for future updates.

Learning curves:

Tracked experiments:

Hyperparameters Copy Section

Memory Gym Environments

Please refer to the defaults in ppo_trxl.py and the single modifications as found in benchmark/ppo_trxl.sh

ProofofMemory-v0

uv run python ppo_trxl.py \
  --env_id ProofofMemory-v0 \
  --total_timesteps 25000 \
  --num_envs 16 \
  --num_steps 128 \
  --num_minibatches 8 \
  --update_epochs 4 \
  --trxl_num_layers 4 \
  --trxl_num_heads 1 \
  --trxl_dim 64 \
  --trxl_memory_length 16 \
  --trxl_positional_encoding none \
  --vf_coef 0.1 \
  --max_grad_norm 0.5 \
  --init_lr 3.0e-4 \
  --init_ent_coef 0.001 \
  --clip_coef 0.2

MiniGrid-MemoryS9-v0

uv run python ppo_trxl.py \
  --env_id MiniGrid-MemoryS9-v0 \
  --total_timesteps 2048000 \
  --num_envs 16 \
  --num_steps 256 \
  --trxl_num_layers 2 \
  --trxl_num_heads 4 \
  --trxl_dim 256 \
  --trxl_memory_length 64 \
  --max_grad_norm 0.25 \
  --anneal_steps 4096000
  --clip_coef 0.2

Enjoy pre-trained models Copy Section

Use cleanrl/ppo_trxl/enjoy.py to watch pre-trained agents.
You can retrieve pre-trained models from huggingface.
Note that Memory Gym environments are usually rendered using the debug_rgb_array render mode, which shows ground truth information about the current task that the agent cannot observe.

Run models from the hub:

python cleanrl/ppo_trxl/enjoy.py --hub --name Endless-MortarMayhem-v0_12.nn
python cleanrl/ppo_trxl/enjoy.py --hub --name Endless-MysterPath-v0_11.nn
python cleanrl/ppo_trxl/enjoy.py --hub --name Endless-SearingSpotlights-v0_30.nn
python cleanrl/ppo_trxl/enjoy.py --hub --name MiniGrid-MemoryS9-v0_10.nn
python cleanrl/ppo_trxl/enjoy.py --hub --name ProofofMemory-v0_1.nn

Run local models (or download them from the hub manually):

python cleanrl/ppo_trxl/enjoy.py --name Your.cleanrl_model

Rl Algorithms Ppo (`docs/rl-algorithms/ppo.md`) Copy Section

Proximal Policy Optimization (PPO)

Overview Copy Chapter

PPO is one of the most popular DRL algorithms. It runs reasonably fast by leveraging vector (parallel) environments and naturally works well with different action spaces, therefore supporting a variety of games. It also has good sample efficiency compared to algorithms such as DQN.

Original paper:

• Proximal Policy Optimization Algorithms

Reference resources:

• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• ⭐ The 37 Implementation Details of Proximal Policy Optimization

All our PPO implementations below are augmented with the same code-level optimizations presented in openai/baselines's PPO. To achieve this, see how we matched the implementation details in our blog post The 37 Implementation Details of Proximal Policy Optimization.

Implemented Variants Copy Chapter

Below are our single-file implementations of PPO:

`ppo.py` Copy Chapter

The ppo.py has the following features:

• Works with the Box observation space of low-level features
• Works with the Discrete action space
• Works with envs like CartPole-v1

Usage Copy Section

=== "uv"

```bash
uv pip install .
uv run python cleanrl/ppo.py --help
uv run python cleanrl/ppo.py --env-id CartPole-v1
```

=== "pip"

```bash
python cleanrl/ppo.py --help
python cleanrl/ppo.py --env-id CartPole-v1
```

Explanation of the logged metrics Copy Section

Running python cleanrl/ppo.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

• charts/episodic_return: episodic return of the game
• charts/episodic_length: episodic length of the game
• charts/SPS: number of steps per second
• charts/learning_rate: the current learning rate
• losses/value_loss: the mean value loss across all data points
• losses/policy_loss: the mean policy loss across all data points
• losses/entropy: the mean entropy value across all data points
• losses/old_approx_kl: the approximate Kullback–Leibler divergence, measured by (-logratio).mean(), which corresponds to the k1 estimator in John Schulman’s blog post on approximating KL
• losses/approx_kl: better alternative to old_approx_kl measured by (logratio.exp() - 1) - logratio, which corresponds to the k3 estimator in approximating KL
• losses/clipfrac: the fraction of the training data that triggered the clipped objective
• losses/explained_variance: the explained variance for the value function

Implementation details Copy Section

ppo.py is based on the "13 core implementation details" in The 37 Implementation Details of Proximal Policy Optimization, which are as follows:

1. Vectorized architecture (:material-github: common/cmd_util.py#L22)
2. Orthogonal Initialization of Weights and Constant Initialization of biases (:material-github: a2c/utils.py#L58))
3. The Adam Optimizer's Epsilon Parameter (:material-github: ppo2/model.py#L100)
4. Adam Learning Rate Annealing (:material-github: ppo2/ppo2.py#L133-L135)
5. Generalized Advantage Estimation (:material-github: ppo2/runner.py#L56-L65)
6. Mini-batch Updates (:material-github: ppo2/ppo2.py#L157-L166)
7. Normalization of Advantages (:material-github: ppo2/model.py#L139)
8. Clipped surrogate objective (:material-github: ppo2/model.py#L81-L86)
9. Value Function Loss Clipping (:material-github: ppo2/model.py#L68-L75)
10. Overall Loss and Entropy Bonus (:material-github: ppo2/model.py#L91)
11. Global Gradient Clipping (:material-github: ppo2/model.py#L102-L108)
12. Debug variables (:material-github: ppo2/model.py#L115-L116)
13. Separate MLP networks for policy and value functions (:material-github: common/policies.py#L156-L160, baselines/common/models.py#L75-L103)

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

--8<-- "benchmark/ppo.sh:3:8"

Below are the average episodic returns for ppo.py. To ensure the quality of the implementation, we compared the results against openai/baselines' PPO.

Learning curves:

--8<-- "benchmark/ppo_plot.sh::9"

Tracked experiments and game play videos:

Video tutorial Copy Section

If you'd like to learn ppo.py in-depth, consider checking out the following video tutorial:

`ppo_atari.py` Copy Chapter

The ppo_atari.py has the following features:

• For Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[atari]"
uv run python cleanrl/ppo_atari.py --help
uv run python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/ppo_atari.py --help
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py.

Implementation details Copy Section

ppo_atari.py is based on the "9 Atari implementation details" in The 37 Implementation Details of Proximal Policy Optimization, which are as follows:

1. The Use of NoopResetEnv (:material-github: common/atari_wrappers.py#L12)
2. The Use of MaxAndSkipEnv (:material-github: common/atari_wrappers.py#L97)
3. The Use of EpisodicLifeEnv (:material-github: common/atari_wrappers.py#L61)
4. The Use of FireResetEnv (:material-github: common/atari_wrappers.py#L41)
5. The Use of WarpFrame (Image transformation) common/atari_wrappers.py#L134
6. The Use of ClipRewardEnv (:material-github: common/atari_wrappers.py#L125)
7. The Use of FrameStack (:material-github: common/atari_wrappers.py#L188)
8. Shared Nature-CNN network for the policy and value functions (:material-github: common/policies.py#L157, common/models.py#L15-L26)
9. Scaling the Images to Range [0, 1] (:material-github: common/models.py#L19)

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

--8<-- "benchmark/ppo.sh:14:19"

Below are the average episodic returns for ppo_atari.py. To ensure the quality of the implementation, we compared the results against openai/baselines' PPO.

Learning curves:

--8<-- "benchmark/ppo_plot.sh:11:19"

Tracked experiments and game play videos:

Video tutorial Copy Section

If you'd like to learn ppo_atari.py in-depth, consider checking out the following video tutorial:

`ppo_continuous_action.py` Copy Chapter

The ppo_continuous_action.py has the following features:

• For continuous action space. Also implemented Mujoco-specific code-level optimizations
• Works with the Box observation space of low-level features
• Works with the Box (continuous) action space
• adding experimental support for Gymnasium
• 🧪 support dm_control environments via Shimmy

???+ warning

We are now recommending users to use [`rpo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/rpo_continuous_action.py) instead of [`ppo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py) because `rpo_continuous_action.py` empirically performs better than `ppo_continuous_action.py` in 93% of the environments we tested. Please see [experiment results](/rl-algorithms/rpo/#experiment-results) for detailed analysis.

Usage Copy Section

=== "poetry"

```bash
# mujoco v4 environments
uv pip install ".[mujoco]"
python cleanrl/ppo_continuous_action.py --help
python cleanrl/ppo_continuous_action.py --env-id Hopper-v4
# dm_control environments
uv pip install ".[mujoco, dm_control]"
python cleanrl/ppo_continuous_action.py --env-id dm_control/cartpole-balance-v0
```

=== "pip"

```bash
pip install -r requirements/requirements-mujoco.txt
python cleanrl/ppo_continuous_action.py --help
python cleanrl/ppo_continuous_action.py --env-id Hopper-v4
pip install -r requirements/requirements-dm_control.txt
python cleanrl/ppo_continuous_action.py --env-id dm_control/cartpole-balance-v0
```

???+ warning "dm_control installation issue"

If you run into error like `AttributeError: 'GLFWContext' object has no attribute '_context'` in Linux, it's because the rendering dependencies are not installed properly. To fix it, try running

```
sudo apt-get update && sudo apt-get -y install libgl1-mesa-glx libosmesa6 libglfw3 
```

See [https://github.com/deepmind/dm_control#rendering](https://github.com/deepmind/dm_control#rendering) for more detail.

Explanation of the logged metrics Copy Section

See related docs for ppo.py.

Implementation details Copy Section

ppo_continuous_action.py is based on the "9 details for continuous action domains (e.g. Mujoco)" in The 37 Implementation Details of Proximal Policy Optimization, which are as follows:

1. Continuous actions via normal distributions (:material-github: common/distributions.py#L103-L104)
2. State-independent log standard deviation (:material-github: common/distributions.py#L104)
3. Independent action components (:material-github: common/distributions.py#L238-L246)
4. Separate MLP networks for policy and value functions (:material-github: common/policies.py#L160, baselines/common/models.py#L75-L103
5. Handling of action clipping to valid range and storage (:material-github: common/cmd_util.py#L99-L100)
6. Normalization of Observation (:material-github: common/vec_env/vec_normalize.py#L4)
7. Observation Clipping (:material-github: common/vec_env/vec_normalize.py#L39)
8. Reward Scaling (:material-github: common/vec_env/vec_normalize.py#L28)
9. Reward Clipping (:material-github: common/vec_env/vec_normalize.py#L32)

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

MuJoCo v4

--8<-- "benchmark/ppo.sh:25:30"

{!benchmark/ppo_continuous_action.md!}

Learning curves:

--8<-- "benchmark/ppo_plot.sh:11:19"

Tracked experiments and game play videos:

--8<-- "benchmark/ppo.sh:36:41"

Below are the average episodic returns for ppo_continuous_action.py in dm_control environments.

Note that the dm_control/lqr-lqr_2_1-v0 dm_control/lqr-lqr_6_2-v0 environments are never terminated or truncated. See https://wandb.ai/openrlbenchmark/cleanrl/runs/3tm00923 and https://wandb.ai/openrlbenchmark/cleanrl/runs/1z9us07j as an example.

Learning curves:

Tracked experiments and game play videos:

???+ info

In the gymnasium environments, we use the v4 mujoco environments, which roughly results in the same performance as the v2 mujoco environments.

Video tutorial Copy Section

If you'd like to learn ppo_continuous_action.py in-depth, consider checking out the following video tutorial:

`ppo_atari_lstm.py` Copy Chapter

The ppo_atari_lstm.py has the following features:

• For Atari games using LSTM without stacked frames. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[atari]"
uv run python cleanrl/ppo_atari_lstm.py --help
uv run python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/ppo_atari_lstm.py --help
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py.

Implementation details Copy Section

ppo_atari_lstm.py is based on the "5 LSTM implementation details" in The 37 Implementation Details of Proximal Policy Optimization, which are as follows:

1. Layer initialization for LSTM layers (:material-github: a2c/utils.py#L84-L86)
2. Initialize the LSTM states to be zeros (:material-github: common/models.py#L179)
3. Reset LSTM states at the end of the episode (:material-github: common/models.py#L141)
4. Prepare sequential rollouts in mini-batches (:material-github: a2c/utils.py#L81)
5. Reconstruct LSTM states during training (:material-github: a2c/utils.py#L81)

To help test out the memory, we remove the 4 stacked frames from the observation (i.e., using env = gym.wrappers.FrameStack(env, 1) instead of env = gym.wrappers.FrameStack(env, 4) like in ppo_atari.py )

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

--8<-- "benchmark/ppo.sh:47:52"

Below are the average episodic returns for ppo_atari_lstm.py. To ensure the quality of the implementation, we compared the results against openai/baselines' PPO.

Learning curves:

--8<-- "benchmark/ppo_plot.sh:11:19"

Tracked experiments and game play videos:

`ppo_atari_envpool.py` Copy Chapter

The ppo_atari_envpool.py has the following features:

• Uses the blazing fast Envpool vectorized environment.
• For Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

???+ warning

Note that `ppo_atari_envpool.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PPO implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

* Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
* Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

This causes the $s_{last}$ to be off by one. 
See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[envpool]"
uv run python cleanrl/ppo_atari_envpool.py --help
uv run python cleanrl/ppo_atari_envpool.py --env-id Breakout-v5
```

=== "pip"

```bash
pip install -r requirements/requirements-envpool.txt
python cleanrl/ppo_atari_envpool.py --help
python cleanrl/ppo_atari_envpool.py --env-id Breakout-v5
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py.

Implementation details Copy Section

ppo_atari_envpool.py uses a customized RecordEpisodeStatistics to work with envpool but has the same other implementation details as ppo_atari.py (see related docs).

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

--8<-- "benchmark/ppo.sh:58:63"

{!benchmark/ppo_atari_envpool.md!}

Learning curves:

--8<-- "benchmark/ppo_plot.sh:51:62"

Tracked experiments and game play videos:

`ppo_atari_envpool_xla_jax.py` Copy Chapter

The ppo_atari_envpool_xla_jax.py has the following features:

• Uses the blazing fast Envpool vectorized environment.
  • Uses EnvPool's experimental XLA interface.
• Uses Jax, Flax, and Optax instead of torch.
• For Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

???+ warning

Note that `ppo_atari_envpool_xla_jax.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PPO implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

* Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
* Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

This causes the $s_{last}$ to be off by one. 
See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[envpool, jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/ppo_atari_envpool_xla_jax.py --help
uv run python cleanrl/ppo_atari_envpool_xla_jax.py --env-id Breakout-v5
```

=== "pip"

```bash
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/ppo_atari_envpool_xla_jax.py --help
python cleanrl/ppo_atari_envpool_xla_jax.py --env-id Breakout-v5
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py. In ppo_atari_envpool_xla_jax.py we omit logging losses/old_approx_kl and losses/clipfrac for brevity.

Additionally, we record the following metric:

• charts/avg_episodic_return: the average value of the latest episodic returns of args.num_envs=8 envs
• charts/avg_episodic_length: the average value of the latest episodic lengths of args.num_envs=8 envs

???+ info

Note that we use `charts/avg_episodic_return` and `charts/avg_episodic_length` in place of `charts/episodic_return` and `charts/episodic_length` because under the EnvPool's XLA interface, we can only record fixed-shape metrics where as there could be a variable number of raw episodic returns / lengths. To resolve this challenge, we create variables (e.g., `returned_episode_returns`, `returned_episode_lengths`) to keep track of the *latest* episodic returns / lengths of each environment and average them for reporting purposes.

Implementation details Copy Section

ppo_atari_envpool_xla_jax.py uses the same other implementation details as ppo_atari.py (see related docs), with two differences

1. ppo_atari_envpool_xla_jax.py does not use the value function clipping by default, because there is no sufficient evidence that value function clipping actually improves performance.
2. ppo_atari_envpool_xla_jax.py uses a customized EpisodeStatistics to record episode statistics instead of the RecordEpisodeStatistics used in other variants. RecordEpisodeStatistics is a stateful python wrapper which is incompatible with EnvPool's stateless XLA interface. To address this issue, we used a EpisodeStatistics dataclass and simply implement the logic of RecordEpisodeStatistics. However, EpisodeStatistics comes with a major limitation: its storage has a fixed shape and can only record the latest episodic return of the sub-environments. Furthermore, the default episodic return values in EpisodeStatistics are set to zeros, which does not necessarily correspond to the episodic return obtained by a random policy. For example, we would report charts/avg_episodic_return=0 for Pong-v5, even if they should have been charts/avg_episodic_return=-21. That said, this issue goes away as soon as the sub-environments finished their first episodes, therefore not impacting the reported results.

???+ info

We benchmarked the PPO implementation w/ and w/o value function clipping, finding no significant difference in performance, which is consistent with the findings in Andrychowicz et al.[^2]. See the related report [part 1](https://wandb.ai/costa-huang/cleanRL/reports/CleanRL-PPO-JAX-EnvPool-s-XLA-w-and-w-o-value-loss-clipping-vs-openai-baselins-PPO-part-1---VmlldzoyNzQ3MzQ1) and [part 2](https://wandb.ai/costa-huang/cleanRL/reports/CleanRL-PPO-JAX-EnvPool-s-XLA-w-and-w-o-value-loss-clipping-vs-openai-baselins-PPO-part-2---VmlldzoyNzQ3MzUw).

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

--8<-- "benchmark/ppo.sh:69:74"

{!benchmark/ppo_atari_envpool_xla_jax.md!}

Learning curves:

--8<-- "benchmark/ppo_plot.sh:64:85"

???+ info

Note the original openai/baselines uses `atari-py==0.2.6` which hangs on `gym.make("DefenderNoFrameskip-v4")` and does not support SurroundNoFrameskip-v4 (see issue [:material-github: openai/atari-py#73](https://github.com/openai/atari-py/issues/73)). To get results on these environments, we use `gym==0.23.1 ale-py==0.7.4 "AutoROM[accept-rom-license]==0.4.2` and [manually register `SurroundNoFrameskip-v4` in our fork](https://github.com/vwxyzjn/baselines/blob/e2cb1c938a62fa8d7fe98187246cde08dfd57bd1/baselines/common/register_all_atari_envs.py#L2).

Median Human Normalized Score (HNS) compared to SEEDRL's R2D2 (data available here).

???+ info

Note the SEEDRL's R2D2's median HNS data does not include learning curves for `Defender` and `Surround` (see [google-research/seed_rl#78](https://github.com/google-research/seed_rl/issues/78)). Also note the SEEDRL's R2D2 uses slightly different Atari preprocessing than our `ppo_atari_envpool_xla_jax.py`, so we may be comparing apples and oranges; however, the results are still informative at the scale of 57 Atari games — we would be at least comparing similar apples.

Tracked experiments and game play videos:

`ppo_atari_envpool_xla_jax_scan.py` Copy Chapter

The ppo_atari_envpool_xla_jax_scan.py has the following features:

• Replaces python loops in compute_gae, update_ppo, and rollout functions of ppo_atari_envpool_xla_jax.py with native jax.scan
• Warnings and caveats from ppo_atari_envpool_xla_jax.py also apply here

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[envpool, jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/ppo_atari_envpool_xla_jax_scan.py --help
uv run python cleanrl/ppo_atari_envpool_xla_jax_scan.py --env-id Breakout-v5
```

=== "pip"

```bash
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/ppo_atari_envpool_xla_jax_scan.py --help
python cleanrl/ppo_atari_envpool_xla_jax_scan.py --env-id Breakout-v5
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py. The metrics are the same as those in ppo_atari_envpool_xla_jax.py.

Implementation details Copy Section

ppo_atari_envpool_xla_jax_scan.py is a clone of ppo_atari_envpool_xla_jax.py that replaces the python loops with native jax.scan.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

--8<-- "benchmark/ppo.sh:80:85"

{!benchmark/ppo_atari_envpool_xla_jax_scan.md!}

Learning curves:

--8<-- "benchmark/ppo_plot.sh:87:96"

Learning curves:

???+ info

The training time of this variant and that of [ppo_atari_envpool_xla_jax.py](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py) are very similar but the compilation time is reduced significantly (see [vwxyzjn/cleanrl#328](https://github.com/vwxyzjn/cleanrl/pull/328#issuecomment-1340474894)). Note that the hardware also affects the speed in the learning curve below. Runs from [`costa-huang`](https://github.com/vwxyzjn/) (red) are slower from those of [`51616`](https://github.com/51616/) (blue and orange) because of hardware differences.

Tracked experiments:

`ppo_procgen.py` Copy Chapter

The ppo_procgen.py has the following features:

• For the procgen environments
• Uses IMPALA-style neural network
• Works with the Discrete action space

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[procgen]"
uv run python cleanrl/ppo_procgen.py --help
uv run python cleanrl/ppo_procgen.py --env-id starpilot
```

=== "pip"

```bash
pip install -r requirements/requirements-procgen.txt
python cleanrl/ppo_procgen.py --help
python cleanrl/ppo_procgen.py --env-id starpilot
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py.

Implementation details Copy Section

ppo_procgen.py is based on the details in "Appendix" in The 37 Implementation Details of Proximal Policy Optimization, which are as follows:

1. IMPALA-style Neural Network (:material-github: common/models.py#L28)
2. Use the same gamma parameter in the NormalizeReward wrapper. Note that the original implementation from openai/train-procgen uses the default gamma=0.99 in the VecNormalize wrapper but gamma=0.999 as PPO's parameter. The mismatch between the gammas is technically incorrect. See #209

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

--8<-- "benchmark/ppo.sh:91:100"

We try to match the default setting in openai/train-procgen except that we use the easy distribution mode and total_timesteps=25e6 to save compute. Notice openai/train-procgen has the following settings:

1. Learning rate annealing is turned off by default
2. Reward scaling and reward clipping is used

Below are the average episodic returns for ppo_procgen.py. To ensure the quality of the implementation, we compared the results against openai/baselines' PPO.

Learning curves:

--8<-- "benchmark/ppo_plot.sh:98:106"

???+ info

Note that we have run the procgen experiments using the `easy` distribution for reducing the computational cost.

Tracked experiments and game play videos:

`ppo_atari_multigpu.py` Copy Chapter

The ppo_atari_multigpu.py leverages data parallelism to speed up training time at no cost of sample efficiency.

ppo_atari_multigpu.py has the following features:

• Allows the users to use do training leveraging data parallelism
• For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

???+ warning

Note that `ppo_atari_multigpu.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. It will error out with `NOTE: Redirects are currently not supported in Windows or MacOs.` See [pytorch/pytorch#20380](https://github.com/pytorch/pytorch/issues/20380)

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[atari]"
uv run python cleanrl/ppo_atari_multigpu.py --help

# `--nproc_per_node=2` specifies how many subprocesses we spawn for training with data parallelism
# note it is possible to run this with a *single GPU*: each process will simply share the same GPU
uv run torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4

# by default we use the `gloo` backend, but you can use the `nccl` backend for better multi-GPU performance
uv run torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --backend nccl

# it is possible to spawn more processes than the amount of GPUs you have via `--device-ids`
# e.g., the command below spawns two processes using GPU 0 and two processes using GPU 1
uv run torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --device-ids 0 0 1 1
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/ppo_atari_multigpu.py --help

# `--nproc_per_node=2` specifies how many subprocesses we spawn for training with data parallelism
# note it is possible to run this with a *single GPU*: each process will simply share the same GPU
torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4

# by default we use the `gloo` backend, but you can use the `nccl` backend for better multi-GPU performance
torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --backend nccl

# it is possible to spawn more processes than the amount of GPUs you have via `--device-ids`
# e.g., the command below spawns two processes using GPU 0 and two processes using GPU 1
torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --device-ids 0 0 1 1
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py.

Implementation details Copy Section

ppo_atari_multigpu.py is based on ppo_atari.py (see its related docs).

We use Pytorch's distributed API to implement the data parallelism paradigm. The basic idea is that the user can spawn $N$ processes each running a copy of ppo_atari.py, holding a copy of the model, stepping the environments, and averaging their gradients together for the backward pass. Here are a few note-worthy implementation details.

1. Local versus global parameters: All of the parameters in ppo_atari.py are global (such as batch size), but in ppo_atari_multigpu.py we have local parameters as well. Say we run torchrun --standalone --nnodes=1 --nproc_per_node=2 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --local-num-envs=4; here are how all multi-gpu related parameters are adjusted:

   • number of environments: num_envs = local_num_envs * world_size = 4 * 2 = 8
   • batch size: local_batch_size = local_num_envs * num_steps = 4 * 128 = 512, batch_size = num_envs * num_steps) = 8 * 128 = 1024
   • minibatch size: local_minibatch_size = int(args.local_batch_size // args.num_minibatches) = 512 // 4 = 128, minibatch_size = int(args.batch_size // args.num_minibatches) = 1024 // 4 = 256
   • number of updates: num_iterations = args.total_timesteps // args.batch_size = 10000000 // 1024 = 9765

2. Adjust seed per process: we need be very careful with seeding: we could have used the exact same seed for each subprocess. To ensure this does not happen, we do the following

   python hl_lines="2 5 16" Copy

   # CRUCIAL: note that we needed to pass a different seed for each data parallelism worker
   args.seed += local_rank
   random.seed(args.seed)
   np.random.seed(args.seed)
   torch.manual_seed(args.seed - local_rank)
   torch.backends.cudnn.deterministic = args.torch_deterministic
   
   # ...
   
   envs = gym.vector.SyncVectorEnv(
       [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)]
   )
   assert isinstance(envs.single_action_space, gym.spaces.Discrete), "only discrete action space is supported"
   
   agent = Agent(envs).to(device)
   torch.manual_seed(args.seed)
   optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5)

   Notice that we adjust the seed with args.seed += local_rank (line 2), where local_rank is the index of the subprocesses. This ensures we seed packages and envs with uncorrealted seeds. However, we do need to use the same torch seed for all process to initialize same weights for the agent (line 5), after which we can use a different seed for torch (line 16).

3. Efficient gradient averaging: PyTorch recommends to average the gradient across the whole world via the following (see docs)

   python Copy

   for param in agent.parameters():
       dist.all_reduce(param.grad.data, op=dist.ReduceOp.SUM)
       param.grad.data /= world_size

   However, @cswinter introduces a more efficient gradient averaging scheme with proper batching (see :material-github: entity-neural-network/incubator#220), which looks like:

   python Copy

   all_grads_list = []
   for param in agent.parameters():
       if param.grad is not None:
           all_grads_list.append(param.grad.view(-1))
   all_grads = torch.cat(all_grads_list)
   dist.all_reduce(all_grads, op=dist.ReduceOp.SUM)
   offset = 0
   for param in agent.parameters():
       if param.grad is not None:
           param.grad.data.copy_(
               all_grads[offset : offset + param.numel()].view_as(param.grad.data) / world_size
           )
           offset += param.numel()

   In our previous empirical testing (see :material-github: vwxyzjn/cleanrl#162), we have found @cswinter's implementation to be faster, hence we adopt it in our implementation.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

--8<-- "benchmark/ppo.sh:102:107"

Below are the average episodic returns for ppo_atari_multigpu.py. To ensure no loss of sample efficiency, we compared the results against ppo_atari.py.

{!benchmark/ppo_atari_multigpu.md!}

Learning curves:

--8<-- "benchmark/ppo_plot.sh:108:117"

Under the same hardware, we see that ppo_atari_multigpu.py is about 30% faster than ppo_atari.py with no loss of sample efficiency.

???+ info

The experiments above is to show correctness -- we show that by aligning the same hyperparameters of `ppo_atari.py` and `ppo_atari_multigpu.py`, we can achieve the same sample efficiency. However, we can train even faster by simply running a much larger batch size. For example, we can run `torchrun --standalone --nnodes=1 --nproc_per_node=8 cleanrl/ppo_atari_multigpu.py --env-id BreakoutNoFrameskip-v4 --local-num-envs=8`, which will run 8 x 8 = 64 environments in parallel and achieve a batch size of 64 x 128 = 8192. This will likely result in a sample efficiency but should increase the wall time efficiency.

???+ info

Although `ppo_atari_multigpu.py` is 30% faster than `ppo_atari.py`, `ppo_atari_multigpu.py` is still slower than `ppo_atari_envpool.py`, as shown below.  This comparison really highlights the different kinds of optimization possible.



    
    


The purpose of `ppo_atari_multigpu.py` is not (yet) to achieve the fastest PPO + Atari example. Rather, its purpose is to *rigorously validate data parallelism does provide performance benefits*. We could do something like `ppo_atari_multigpu_envpool.py` to possibly obtain the fastest PPO + Atari possible, but that is for another day. Note we may need `numba` to pin the threads `envpool` is using in each subprocess to avoid threads fighting each other and lowering the throughput.

Tracked experiments and game play videos:

`ppo_pettingzoo_ma_atari.py` Copy Chapter

ppo_pettingzoo_ma_atari.py trains an agent to learn playing Atari games via selfplay. The selfplay environment is implemented as a vectorized environment from PettingZoo.ml. The basic idea is to create vectorized environment $E$ with num_envs = N, where $N$ is the number of players in the game. Say $N = 2$, then the 0-th sub environment of $E$ will return the observation for player 0 and 1-th sub environment will return the observation of player 1. Then the two environments takes a batch of 2 actions and execute them for player 0 and player 1, respectively. See "Vectorized architecture" in The 37 Implementation Details of Proximal Policy Optimization for more detail.

ppo_pettingzoo_ma_atari.py has the following features:

• For playing the pettingzoo's multi-agent Atari game.
• Works with the pixel-based observation space
• Works with the Box action space

???+ warning

Note that `ppo_pettingzoo_ma_atari.py` does not work in Windows :fontawesome-brands-windows:. See [https://pypi.org/project/multi-agent-ale-py/#files](https://pypi.org/project/multi-agent-ale-py/#files)

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[pettingzoo, atari]"
uv run AutoROM --accept-license
uv run  cleanrl/ppo_pettingzoo_ma_atari.py --help
uv run  cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3
uv run  cleanrl/ppo_pettingzoo_ma_atari.py --env-id surround_v2
```

=== "pip"

```bash
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-atari.txt
AutoROM --accept-license
python cleanrl/ppo_pettingzoo_ma_atari.py --help
python cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3
python cleanrl/ppo_pettingzoo_ma_atari.py --env-id surround_v2
```

See https://www.pettingzoo.ml/atari for a full-list of supported environments such as basketball_pong_v3. Notice pettingzoo sometimes introduces breaking changes, so make sure to install the pinned dependencies via poetry.

Explanation of the logged metrics Copy Section

Additionally, it logs the following metrics

• charts/episodic_return-player0: episodic return of the game for player 0
• charts/episodic_return-player1: episodic return of the game for player 1
• charts/episodic_length-player0: episodic length of the game for player 0
• charts/episodic_length-player1: episodic length of the game for player 1

See other logged metrics in the related docs for ppo.py.

Implementation details Copy Section

ppo_pettingzoo_ma_atari.py is based on ppo_atari.py (see its related docs).

ppo_pettingzoo_ma_atari.py additionally has the following implementation details:

1. supersuit wrappers: uses preprocessing wrappers from supersuit instead of from stable_baselines3, which looks like the following. In particular note that the supersuit does not offer a wrapper similar to NoopResetEnv, and that it uses the agent_indicator_v0 to add two channels indicating the which player the agent controls.

   diff Copy

   -env = gym.make(env_id)
   -env = NoopResetEnv(env, noop_max=30)
   -env = MaxAndSkipEnv(env, skip=4)
   -env = EpisodicLifeEnv(env)
   -if "FIRE" in env.unwrapped.get_action_meanings():
   -    env = FireResetEnv(env)
   -env = ClipRewardEnv(env)
   -env = gym.wrappers.ResizeObservation(env, (84, 84))
   -env = gym.wrappers.GrayScaleObservation(env)
   -env = gym.wrappers.FrameStack(env, 4)
   +env = importlib.import_module(f"pettingzoo.atari.{args.env_id}").parallel_env()
   +env = ss.max_observation_v0(env, 2)
   +env = ss.frame_skip_v0(env, 4)
   +env = ss.clip_reward_v0(env, lower_bound=-1, upper_bound=1)
   +env = ss.color_reduction_v0(env, mode="B")
   +env = ss.resize_v1(env, x_size=84, y_size=84)
   +env = ss.frame_stack_v1(env, 4)
   +env = ss.agent_indicator_v0(env, type_only=False)
   +env = ss.pettingzoo_env_to_vec_env_v1(env)
   +envs = ss.concat_vec_envs_v1(env, args.num_envs // 2, num_cpus=0, base_class="gym")

2. A more detailed note on the agent_indicator_v0 wrapper: let's dig deeper into how agent_indicator_v0 works. We do print(envs.reset(), envs.reset().shape)

   python Copy

   [  0.,   0.,   0., 236.,   1,   0.]],
   
   [[  0.,   0.,   0., 236.,   0.,   1.],
   [  0.,   0.,   0., 236.,   0.,   1.],
   [  0.,   0.,   0., 236.,   0.,   1.],
   ...,
   [  0.,   0.,   0., 236.,   0.,   1.],
   [  0.,   0.,   0., 236.,   0.,   1.],
   [  0.,   0.,   0., 236.,   0.,   1.]]]]) torch.Size([16, 84, 84, 6])

   So the agent_indicator_v0 adds the last two columns, where [ 0., 0., 0., 236., 1, 0.]] means this observation is for player 0, and [ 0., 0., 0., 236., 0., 1.] is for player 1. Notice the observation still has the range of $[0, 255]$ but the agent indicator channel has the range of $[0,1]$, so we need to be careful when dividing the observation by 255. In particular, we would only divide the first four channels by 255 and leave the agent indicator channels untouched as follows:

   py Copy

   def get_action_and_value(self, x, action=None):
       x = x.clone()
       x[:, :, :, [0, 1, 2, 3]] /= 255.0
       hidden = self.network(x.permute((0, 3, 1, 2)))

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/ppo.sh. Specifically, execute the following command:

???+ info

Note that evaluation is usually tricker in in selfplay environments. The usual episodic return is not a good indicator of the agent's performance in zero-sum games because the episodic return converges to zero. To evaluate the agent's ability, an intuitive approach is to take a look at the videos of the agents playing the game (included below), visually inspect the agent's behavior. The best scheme, however, is rating systems like [Trueskill](https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/) or [ELO scores](https://en.wikipedia.org/wiki/Elo_rating_system). However, they are more difficult to implement and are outside the scode of `ppo_pettingzoo_ma_atari.py`. 


For simplicity, we measure the **episodic length** instead, which in a sense measures how many "back and forth" the agent can create. In other words, the longer the agent can play the game, the better the agent can play. Empirically, we have found episodic length to be a good indicator of the agent's skill, especially in `pong_v3` and `surround_v2`. However, it is not the case for `tennis_v3` and we'd need to visually inspect the agents' game play videos.

Below are the average episodic length for ppo_pettingzoo_ma_atari.py. To ensure no loss of sample efficiency, we compared the results against ppo_atari.py.

Learning curves:

Tracked experiments and game play videos:

{!rl-algorithms/ppo-isaacgymenvs.md!}

[^1]: Huang, Shengyi; Dossa, Rousslan Fernand Julien; Raffin, Antonin; Kanervisto, Anssi; Wang, Weixun (2022). The 37 Implementation Details of Proximal Policy Optimization. ICLR 2022 Blog Track https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/

[^2]: Andrychowicz, Marcin, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot et al. "What matters in on-policy reinforcement learning? a large-scale empirical study." International Conference on Learning Representations 2021, https://openreview.net/forum?id=nIAxjsniDzg

Rl Algorithms Pqn (`docs/rl-algorithms/pqn.md`) Copy Section

Parallel Q Network (PQN)

Overview Copy Chapter

PQN is a parallelized version of the Deep Q-learning algorithm. It is designed to be more efficient than DQN by using multiple agents to interact with the environment in parallel. PQN can be thought of as DQN (1) without replay buffer and target networks, and (2) with layer normalizations and parallel environments.

Original paper:

• Simplifying Deep Temporal Difference Learning

Reference resources:

• :material-github: purejaxql

Implemented Variants Copy Chapter

Below are our single-file implementations of PQN:

`pqn.py` Copy Chapter

The pqn.py has the following features:

• Works with the Box observation space of low-level features
• Works with the Discrete action space
• Works with envs like CartPole-v1

Usage Copy Section

=== "poetry"

```bash
poetry install
uv run python cleanrl/pqn.py --help
uv run python cleanrl/pqn.py --env-id CartPole-v1
```

=== "pip"

```bash
python cleanrl/pqn.py --help
python cleanrl/pqn.py --env-id CartPole-v1
```

Explanation of the logged metrics Copy Section

Running python cleanrl/pqn.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

• charts/episodic_return: episodic return of the game
• charts/episodic_length: episodic length of the game
• charts/SPS: number of steps per second
• charts/learning_rate: the current learning rate
• losses/td_loss: the mean squared error (MSE) between the Q values at timestep $t$ and the Bellman update target estimated using the $Q(\lambda)$ returns.
• losses/q_values: it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimation happens.

Implementation details Copy Section

1. Vectorized architecture (:material-github: common/cmd_util.py#L22)
2. Orthogonal Initialization of Weights and Constant Initialization of biases (:material-github: a2c/utils.py#L58))
3. Normalized Q Network (:material-github: purejaxql/pqn_atari.py#L200)
4. Uses the RAdam Optimizer with the default epsilon parameter(:material-github: purejaxql/pqn_atari.py#L362)
5. Adam Learning Rate Annealing (:material-github: pqn2/pqn2.py#L133-L135)
6. Q Lambda Returns (:material-github: purejaxql/pqn_atari.py#L446)
7. Mini-batch Updates (:material-github: pqn2/pqn2.py#L157-L166)
8. Global Gradient Clipping (:material-github: purejaxql/pqn_atari.py#L360)

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/pqn.sh. Specifically, execute the following command:

--8<-- "benchmark/pqn.sh:0:6"

Episode Rewards:

Runtime:

Learning curves:

--8<-- "benchmark/pqn_plot.sh:1:9"

Tracked experiments:

`pqn_atari_envpool.py` Copy Chapter

The pqn_atari_envpool.py has the following features:

• Uses the blazing fast Envpool vectorized environment.
• For Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

???+ warning

Note that `pqn_atari_envpool.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PQN implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

* Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
* Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

This causes the $s_{last}$ to be off by one. 
See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[envpool]"
uv run python cleanrl/pqn_atari_envpool.py --help
uv run python cleanrl/pqn_atari_envpool.py --env-id Breakout-v5
```

=== "pip"

```bash
pip install -r requirements/requirements-envpool.txt
python cleanrl/pqn_atari_envpool.py --help
python cleanrl/pqn_atari_envpool.py --env-id Breakout-v5
```

Explanation of the logged metrics Copy Section

See related docs for pqn.py.

Implementation details Copy Section

pqn_atari_envpool.py uses a customized RecordEpisodeStatistics to work with envpool but has the same other implementation details as ppo_atari.py.

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/pqn.sh. Specifically, execute the following command:

--8<-- "benchmark/pqn.sh:12:17"

Episode Rewards:

Runtime:

Learning curves:

--8<-- "benchmark/pqn_plot.sh:11:29"

Tracked experiments:

`pqn_atari_envpool_lstm.py` Copy Chapter

The pqn_atari_envpool_lstm.py has the following features:

• Uses the blazing fast Envpool vectorized environment.
• For Atari games using LSTM without stacked frames. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

???+ warning

Note that `pqn_atari_envpool.py` does not work in Windows :fontawesome-brands-windows: and MacOs :fontawesome-brands-apple:. See envpool's built wheels here: [https://pypi.org/project/envpool/#files](https://pypi.org/project/envpool/#files)

???+ bug

EnvPool's vectorized environment **does not behave the same** as gym's vectorized environment, which causes a compatibility bug in our PQN implementation. When an action $a$ results in an episode termination or truncation, the environment generates $s_{last}$ as the terminated or truncated state; we then use $s_{new}$ to denote the initial state of the new episodes. Here is how the bahviors differ:

* Under the vectorized environment of `envpool<=0.6.4`, the `obs` in `obs, reward, done, info = env.step(action)` is the truncated state $s_{last}$
* Under the vectorized environment of `gym==0.23.1`, the `obs` in `obs, reward, done, info = env.step(action)` is the initial state $s_{new}$.

This causes the $s_{last}$ to be off by one. 
See [:material-github: sail-sg/envpool#194](https://github.com/sail-sg/envpool/issues/194) for more detail. However, it does not seem to impact performance, so we take a note here and await for the upstream fix.

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[atari]"
uv run python cleanrl/pqn_atari_envpool_lstm.py --help
uv run python cleanrl/pqn_atari_envpool_lstm.py --env-id Breakout-v5
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/pqn_atari_envpool_lstm.py --help
python cleanrl/pqn_atari_envpool_lstm.py --env-id Breakout-v5
```

Explanation of the logged metrics Copy Section

See related docs for pqn.py.

Implementation details Copy Section

pqn_atari_envpool_lstm.py is based on the "5 LSTM implementation details" in The 37 Implementation Details of Proximal Policy Optimization, which are as follows:

1. Layer initialization for LSTM layers (:material-github: a2c/utils.py#L84-L86)
2. Initialize the LSTM states to be zeros (:material-github: common/models.py#L179)
3. Reset LSTM states at the end of the episode (:material-github: common/models.py#L141)
4. Prepare sequential rollouts in mini-batches (:material-github: a2c/utils.py#L81)
5. Reconstruct LSTM states during training (:material-github: a2c/utils.py#L81)

To help test out the memory, we remove the 4 stacked frames from the observation (i.e., using env = gym.wrappers.FrameStack(env, 1) instead of env = gym.wrappers.FrameStack(env, 4) like in ppo_atari.py )

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/pqn.sh. Specifically, execute the following command:

--8<-- "benchmark/pqn.sh:23:28"

Episode Rewards:

Runtime:

--8<-- "benchmark/pqn_plot.sh:32:50"

Tracked experiments:

Rl Algorithms Qdagger (`docs/rl-algorithms/qdagger.md`) Copy Section

QDagger

Overview Copy Chapter

QDagger is an extension of the DQN algorithm that uses previously computed results, like teacher policy and teacher replay buffer, to help train student policy. This method eliminates the need for learning from scratch, improving sample efficiency and reducing computational effort in training new policy.

Original paper:

• Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress

Reference resources:

• :material-github: google-research/reincarnating_rl
• Original Paper's Website

Implemented Variants Copy Chapter

Below are our single-file implementations of QDagger:

`qdagger_dqn_atari_impalacnn.py` Copy Chapter

The qdagger_dqn_atari_impalacnn.py has the following features:

• For playing Atari games. It uses Impala-CNN from RainbowDQN and common atari-based pre-processing techniques.
• Its teacher policy uses CleanRL's dqn_atari policy from the huggingface/cleanrl repository.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

uv pip install ".[atari]"
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4

=== "poetry"

```bash
uv pip install ".[atari]"
uv run python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4
```

Explanation of the logged metrics Copy Section

Running python cleanrl/qdagger_dqn_atari_impalacnn.py will automatically record various metrics such as value or distillation losses in Tensorboard. Below is the documentation for these metrics:

• charts/episodic_return: episodic return of the game
• charts/SPS: number of steps per second
• losses/td_loss: the mean squared error (MSE) between the Q values at timestep $t$ and the Bellman update target estimated using the reward $r_t$ and the Q values at timestep $t+1$, thus minimizing the one-step temporal difference. Formally, it can be expressed by the equation below.
  $$
  J(\theta^{Q}) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \big[ (Q(s, a) - y)^2 \big],
  $$
  with the Bellman update target is $y = r + \gamma , Q^{'}(s', a')$ and the replay buffer is $\mathcal{D}$.
• losses/q_values: implemented as qf1(data.observations, data.actions).view(-1), it is the average Q values of the sampled data in the replay buffer; useful when gauging if under or over estimation happens.
• losses/distill_loss: the distillation loss, which is the KL divergence between the teacher policy $\pi_T$ and the student policy $\pi$. Formally, it can be expressed by the equation below.
  $$
  L_{\text{distill}} = \lambda_t \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[ \sum_a \pi_T(a|s)\log\pi(a|s)\right]
  $$
• Charts/distill_coeff: the coefficient $\lambda_t$ for the distillation loss, which is a function of the ratio between the teacher policy $\pi_T$ and the student policy $\pi$. Formally, it can be expressed by the equation below.
  $$
  \lambda_t = 1_{t<t_0}\max(1 - G^\pi/G^{\pi_T}, 0)
  $$
• losses/loss: the total loss, which is the sum of the TD loss and the distillation loss.
  $$
  L_{\text{qdagger}} = J(\theta^{Q}) + L_{\text{distill}}
  $$
• charts/teacher/avg_episodic_return: average episodic return of teacher policy evaluation
• charts/offline/avg_episodic_return: average episodic return of policy evaluation in offline training phase

Implementation details Copy Section

qdagger_dqn_atari_impalacnn.py is based on (Agarwal et al., 2022)[^1] but presents a few implementation differences:

• (Agarwal et al., 2022)[^1] uses the teacher replay buffer data saved during the training of the teacher policy, but our teacher policy, which uses CleanRL's dqn_atari from the huggingface/cleanrl, does not contain replay buffer data, so we populate the teacher replay buffer with the teacher policy before training. For more details, see section A.5 "Additional ablations for QDagger" in the original paper.
• (Agarwal et al., 2022)[^1] uses DQN (Adam) @ 400M frames, but we use CleanRL's dqn_atari, which is DQN (Adam) @ 10M steps(40M frames).
• We have used an old set of Atari preprocessing techniques that doesn't use sticky action, but the original paper does.

Experiment results Copy Section

Below are the average episodic returns for qdagger_dqn_atari_impalacnn.py.

Learning curves:

Learning curve comparison with dqn_atari:

Tracked experiments and game play videos:

`qdagger_dqn_atari_jax_impalacnn.py` Copy Chapter

The qdagger_dqn_atari_jax_impalacnn.py has the following features:

• Uses Jax, Flax, and Optax instead of torch. qdagger_dqn_atari_jax_impalacnn.py is roughly 25%-50% faster than qdagger_dqn_atari_impalacnn.py
• For playing Atari games. It uses Impala-CNN from RainbowDQN and common atari-based pre-processing techniques.
• Its teacher policy uses CleanRL's dqn_atari_jax policy from the huggingface/cleanrl repository.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

=== "poetry"

```bash
uv pip install ".[atari, jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/qdagger_dqn_atari_jax_impalacnn.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/qdagger_dqn_atari_jax_impalacnn.py --env-id PongNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/qdagger_dqn_atari_jax_impalacnn.py --env-id BreakoutNoFrameskip-v4
python cleanrl/qdagger_dqn_atari_jax_impalacnn.py --env-id PongNoFrameskip-v4
```

???+ warning

Note that JAX does not work in Windows :fontawesome-brands-windows:. The official [docs](https://github.com/google/jax#installation) recommends using Windows Subsystem for Linux (WSL) to install JAX.

Explanation of the logged metrics Copy Section

See related docs for qdagger_dqn_atari_impalacnn.py.

Implementation details Copy Section

See related docs for qdagger_dqn_atari_impalacnn.py.

Experiment results Copy Section

Below are the average episodic returns for qdagger_dqn_atari_jax_impalacnn.py.

Learning curves:

Learning curve comparison with dqn_atari_jax:

[^1]:Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. “Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress.” arXiv, October 4, 2022. http://arxiv.org/abs/2206.01626.

Rl Algorithms Rainbow (`docs/rl-algorithms/rainbow.md`) Copy Section

Rainbow

Overview Copy Chapter

The Rainbow algorithm is an extension of DQN that combines multiple improvements:

• Prioritized Experience Replay
• Dueling Network Architecture
• Noisy Networks
• Distributional Q-Learning
• N-step Learning
• Double Q-Learning

Original papers:

• Rainbow: Combining Improvements in Deep Reinforcement Learning

Reference resources:

• :material-github: Dopamine

• :material-github: Kaixhin

Implemented Variants Copy Chapter

`rainbow_atari.py` Copy Chapter

The rainbow_atari.py has the following features:

• For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discrete action space

Usage Copy Section

poetry install -E atari
python cleanrl/rainbow_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/rainbow_atari.py --env-id PongNoFrameskip-v4

=== "poetry"

```bash
poetry install -E atari
poetry run python cleanrl/rainbow_atari.py --env-id BreakoutNoFrameskip-v4
poetry run python cleanrl/rainbow_atari.py --env-id PongNoFrameskip-v4
```

=== "pip"

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/rainbow_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/rainbow_atari.py --env-id PongNoFrameskip-v4
```

Explanation of the logged metrics Copy Section

Running python cleanrl/rainbow_atari.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

• charts/episodic_return: episodic return of the game
• charts/SPS: number of steps per second
• losses/td_loss: the n-step distributional TD loss
• losses/q_values: the mean Q values of the sampled data in the replay buffer
• charts/beta: the beta value of the prioritized experience replay

Implementation details Copy Section

rainbow_atari.py is based on (Hessel et al., 2018)[^1], and uses the same hyperparameters as (Hessel et al., 2018)[^1]. See Table 1 in (Hessel et al., 2018)[^1] for the hyperparameters. However, there are a few implementation differences:

1. rainbow_atari.py uses the more popular Adam Optimizer with the --learning-rate=0.0000625 as follows:
   python Copy

   optim.Adam(q_network.parameters(), lr=0.0000625)

   whereas (Hessel et al., 2018)[^1] uses the RMSProp optimizer with --learning-rate=0.0000625, gradient momentum 0.95, squared gradient momentum 0.95, and min squared gradient 0.01 as follows:
   python Copy

   optim.RMSprop(
       q_network.parameters(),
       lr=2.5e-4,
       momentum=0.95,
       # ... PyTorch's RMSprop does not directly support
       # squared gradient momentum and min squared gradient
       # so we are not sure what to put here.
   )

Experiment results Copy Section

To run benchmark experiments, see :material-github: benchmark/rainbow.sh. Specifically, execute the following command:

--8<-- "benchmark/rainbow.sh:0:6"

Below are the average episodic returns for rainbow_atari.py.

Learning curves:

Rainbow shows better performance than C51 and DQN.

Rainbow is also more sample efficient than C51 and DQN.

Rainbow obtains better aggregated performance than C51 and DQN.

[^1]: Hessel, M., Modayil, J., Hasselt, H.V., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.G., & Silver, D. (2018). Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI.

Rl Algorithms Rpo (`docs/rl-algorithms/rpo.md`) Copy Section

Robust Policy Optimization (RPO)

Overview Copy Chapter

RPO leverages a method of perturbing the distribution representing actions. The goal is to encourage high-entropy actions and provide a better representation of the action space. The method consists of a simple modification on top of the objective of the PPO algorithm. In the RPO algorithm, the mean of the action distribution is perturbed using a random number drawn from a Uniform distribution.

Original paper:

• Robust Policy Optimization in Deep Reinforcement Learning

Implemented Variants Copy Chapter

Below are our single-file implementations of RPO:

`rpo_continuous_action.py` Copy Chapter

rpo_continuous_action.py works with Gym (Gymnasium), dm_control, Mujoco environments with continuous action and vector observations.

The rpo_continuous_action.py has the following features (similar to ppo_continuous_action.py):

• For continuous action space. Also implemented Mujoco-specific code-level optimizations
• Works with the Box observation space of low-level features
• Works with the Box (continuous) action space
• adding experimental support for Gymnasium
• 🧪 support dm_control environments via Shimmy

Usage Copy Section

# mujoco v4 environments
uv pip install ".[mujoco]"
python cleanrl/rpo_continuous_action.py --help
python cleanrl/rpo_continuous_action.py --env-id Walker2d-v4
# NOTE: we recommend using --rpo-alpha 0.01 for Ant Hopper InvertedDoublePendulum Reacher Pusher
python cleanrl/rpo_continuous_action.py --env-id Ant-v4 --rpo-alpha 0.01
# dm_control v4 environments
uv pip install ".[mujoco, dm_control]"
python cleanrl/rpo_continuous_action.py --env-id dm_control/cartpole-balance-v0
# BipedalWalker-v3 experiment (hack)
uv pip install .
uv pip install box2d-py==2.3.5
python cleanrl/rpo_continuous_action.py --env-id BipedalWalker-v3

=== "poetry"

```bash
# mujoco v4 environments
uv pip install ".[mujoco]"
python cleanrl/rpo_continuous_action.py --help
python cleanrl/rpo_continuous_action.py --env-id Hopper-v4
# NOTE: we recommend using --rpo-alpha 0.01 for Ant Hopper InvertedDoublePendulum Reacher Pusher
python cleanrl/rpo_continuous_action.py --env-id Ant-v4 --rpo-alpha 0.01
# dm_control environments
uv pip install ".[mujoco, dm_control]"
python cleanrl/rpo_continuous_action.py --env-id dm_control/cartpole-balance-v0
# BipedalWalker-v3 experiment (hack)
uv pip install box2d-py==2.3.5
uv run python cleanrl/rpo_continuous_action.py --env-id BipedalWalker-v3
```

=== "pip"

```bash
pip install -r requirements/requirements-mujoco.txt
python cleanrl/rpo_continuous_action.py --help
python cleanrl/rpo_continuous_action.py --env-id Hopper-v4
# NOTE: we recommend using --rpo-alpha 0.01 for Ant Hopper InvertedDoublePendulum Reacher Pusher
python cleanrl/rpo_continuous_action.py --env-id Ant-v4 --rpo-alpha 0.01
pip install -r requirements/requirements-dm_control.txt
python cleanrl/rpo_continuous_action.py --env-id dm_control/cartpole-balance-v0
pip install box2d-py==2.3.5
python cleanrl/rpo_continuous_action.py --env-id BipedalWalker-v3
```

Explanation of the logged metrics Copy Section

See related docs for ppo.py.

Implementation details Copy Section

rpo_continuous_action.py has the same implementation details as ppo_continuous_action.py (see related docs) but with a few lines of code differences.

class Agent(nn.Module):
    def __init__(self, envs):
        super().__init__()
        self.critic = nn.Sequential(
            layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
            nn.Tanh(),
            layer_init(nn.Linear(64, 64)),
            nn.Tanh(),
            layer_init(nn.Linear(64, 1), std=1.0),
        )
        self.actor_mean = nn.Sequential(
            layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
            nn.Tanh(),
            layer_init(nn.Linear(64, 64)),
            nn.Tanh(),
            layer_init(nn.Linear(64, np.prod(envs.single_action_space.shape)), std=0.01),
        )
        self.actor_logstd = nn.Parameter(torch.zeros(1, np.prod(envs.single_action_space.shape)))

    def get_value(self, x):
        return self.critic(x)

    def get_action_and_value(self, x, action=None):
        action_mean = self.actor_mean(x)
        action_logstd = self.actor_logstd.expand_as(action_mean)
        action_std = torch.exp(action_logstd)
        probs = Normal(action_mean, action_std)
        if action is None:
            action = probs.sample()
        else: # new to RPO
            # sample again to add stochasticity, for the policy update
            z = torch.FloatTensor(action_mean.shape).uniform_(-self.rpo_alpha, self.rpo_alpha)
            action_mean = action_mean + z
            probs = Normal(action_mean, action_std)

???+ note

RPO usages the same PPO-specific hyperparameters. In benchmarking results, we run both algorithms for 8M timesteps.
RPO has one additional hyperparameter, `rpo_alpha`, which determines the amount of random perturbation on the action mean. 
We set a default value of `rpo_alpha=0.5` at which RPO is strictly equal to or better than the default PPO in 93% of environments tested (all 48/48 dm_control, 2/2 Gym, 7/11 mujoco_v4. Total 57 out of 61 environments tested.).
With finetuning `rpo_alpha=0.01` on four mujoco environments, namely, Ant, InvertedDoublePendulum, Reacher, and Pusher, RPO is strictly equal to or better than the default PPO in all tested environments.

Experiment results Copy Section

To run benchmark experiments, see benchmark/rpo.sh. Specifically, execute the following command:

???+ note "Result tables, learning curves"

=== "dm_control"

    Results on all dm_control environments. The PPO and RPO run for 8M timesteps, and results are computed over 10 random seeds.

    |                                       | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
    |:--------------------------------------|:-------------------------------------------------------------|:----------------------------------------------|
    | dm_control/acrobot-swingup-v0         | 26.87 ± 7.93                                                 | 42.97 ± 2.71                                  |
    | dm_control/acrobot-swingup_sparse-v0  | 1.70 ± 0.88                                                  | 3.31 ± 0.84                                   |
    | dm_control/ball_in_cup-catch-v0       | 935.26 ± 12.57                                               | 939.75 ± 10.18                                |
    | dm_control/cartpole-balance-v0        | 790.36 ± 17.64                                               | 795.12 ± 10.49                                |
    | dm_control/cartpole-balance_sparse-v0 | 986.04 ± 9.93                                                | 988.56 ± 4.28                                 |
    | dm_control/cartpole-swingup-v0        | 590.21 ± 16.72                                               | 613.46 ± 10.91                                |
    | dm_control/cartpole-swingup_sparse-v0 | 240.14 ± 299.93                                              | 525.49 ± 185.96                               |
    | dm_control/cartpole-two_poles-v0      | 216.31 ± 4.03                                                | 218.31 ± 7.30                                 |
    | dm_control/cartpole-three_poles-v0    | 160.03 ± 2.52                                                | 159.97 ± 2.28                                 |
    | dm_control/cheetah-run-v0             | 472.14 ± 99.62                                               | 565.51 ± 58.03                                |
    | dm_control/dog-stand-v0               | 332.06 ± 23.66                                               | 501.22 ± 131.98                               |
    | dm_control/dog-walk-v0                | 124.92 ± 23.13                                               | 166.39 ± 44.65                                |
    | dm_control/dog-trot-v0                | 79.89 ± 12.30                                                | 115.39 ± 29.68                                |
    | dm_control/dog-run-v0                 | 69.07 ± 8.17                                                 | 104.27 ± 24.44                                |
    | dm_control/dog-fetch-v0               | 28.34 ± 4.87                                                 | 43.58 ± 6.88                                  |
    | dm_control/finger-spin-v0             | 630.06 ± 252.99                                              | 848.67 ± 25.21                                |
    | dm_control/finger-turn_easy-v0        | 237.76 ± 78.10                                               | 450.88 ± 133.54                               |
    | dm_control/finger-turn_hard-v0        | 83.76 ± 28.96                                                | 259.99 ± 144.83                               |
    | dm_control/fish-upright-v0            | 559.72 ± 65.79                                               | 803.21 ± 28.36                                |
    | dm_control/fish-swim-v0               | 80.42 ± 9.18                                                 | 140.33 ± 49.10                                |
    | dm_control/hopper-stand-v0            | 3.26 ± 1.65                                                  | 404.39 ± 198.17                               |
    | dm_control/hopper-hop-v0              | 6.48 ± 18.92                                                 | 62.60 ± 87.29                                 |
    | dm_control/humanoid-stand-v0          | 20.76 ± 29.35                                                | 140.43 ± 57.27                                |
    | dm_control/humanoid-walk-v0           | 8.92 ± 18.01                                                 | 77.00 ± 53.00                                 |
    | dm_control/humanoid-run-v0            | 5.44 ± 9.16                                                  | 24.00 ± 19.54                                 |
    | dm_control/humanoid-run_pure_state-v0 | 1.13 ± 0.11                                                  | 3.24 ± 2.41                                   |
    | dm_control/humanoid_CMU-stand-v0      | 4.64 ± 0.37                                                  | 4.32 ± 0.33                                   |
    | dm_control/humanoid_CMU-run-v0        | 0.88 ± 0.09                                                  | 0.80 ± 0.09                                   |
    | dm_control/manipulator-bring_ball-v0  | 0.42 ± 0.14                                                  | 0.44 ± 0.23                                   |
    | dm_control/manipulator-bring_peg-v0   | 0.95 ± 0.43                                                  | 1.07 ± 1.01                                   |
    | dm_control/manipulator-insert_ball-v0 | 41.24 ± 27.27                                                | 43.63 ± 12.77                                 |
    | dm_control/manipulator-insert_peg-v0  | 40.72 ± 15.95                                                | 44.87 ± 26.55                                 |
    | dm_control/pendulum-swingup-v0        | 472.19 ± 385.47                                              | 774.30 ± 21.03                                |
    | dm_control/point_mass-easy-v0         | 534.23 ± 264.35                                              | 653.73 ± 23.14                                |
    | dm_control/point_mass-hard-v0         | 129.75 ± 61.18                                               | 185.81 ± 36.25                                |
    | dm_control/quadruped-walk-v0          | 247.29 ± 90.48                                               | 602.64 ± 223.23                               |
    | dm_control/quadruped-run-v0           | 171.50 ± 37.90                                               | 367.98 ± 117.18                               |
    | dm_control/quadruped-escape-v0        | 23.11 ± 10.48                                                | 68.50 ± 27.81                                 |
    | dm_control/quadruped-fetch-v0         | 183.71 ± 25.14                                               | 216.32 ± 17.44                                |
    | dm_control/reacher-easy-v0            | 773.01 ± 56.70                                               | 716.89 ± 50.07                                |
    | dm_control/reacher-hard-v0            | 637.84 ± 81.15                                               | 576.81 ± 48.25                                |
    | dm_control/stacker-stack_2-v0         | 58.02 ± 11.04                                                | 70.95 ± 16.84                                 |
    | dm_control/stacker-stack_4-v0         | 73.84 ± 14.48                                                | 65.54 ± 18.06                                 |
    | dm_control/swimmer-swimmer6-v0        | 164.22 ± 18.44                                               | 159.60 ± 39.52                                |
    | dm_control/swimmer-swimmer15-v0       | 161.02 ± 24.56                                               | 153.91 ± 28.08                                |
    | dm_control/walker-stand-v0            | 439.24 ± 210.22                                              | 734.74 ± 142.52                               |
    | dm_control/walker-walk-v0             | 305.74 ± 92.15                                               | 787.11 ± 125.97                               |
    | dm_control/walker-run-v0              | 128.18 ± 91.52                                               | 391.56 ± 119.75                               |
            
    Learning curves:
    

    Tracked experiments: 
    
    
    




=== "MuJoCo v4"

    |                     | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
    |:--------------------|:-------------------------------------------------------------|:----------------------------------------------|
    | HumanoidStandup-v4  | 109325.87 ± 16161.71                                         | 150972.11 ± 6926.19                           |
    | Humanoid-v4         | 583.17 ± 27.88                                               | 799.44 ± 170.85                               |
    | InvertedPendulum-v4 | 888.83 ± 34.66                                               | 879.81 ± 35.52                                |
    | Walker2d-v4         | 2872.92 ± 690.53                                             | 3665.48 ± 278.61                              |

    Learning curves:
    


    Tracked experiments:     

    

    
    The following environments require tuning of `alpha` (Algorithm 1, line 13, paper: https://arxiv.org/pdf/2212.07536.pdf). As described in the paper, this variable should be tuned for environments tested. A larger value means more randomness, whereas a smaller value indicates less randomness. Some mujoco environments require a smaller `alpha=0.01` value to achieve a reasonable performance compared to `alpha=0.5` for the rest of the environments. This version (`alpha=0.01`) of runs is indicated as `rpo_continuous_action_alpha_0_01` in the table and learning curves.

    |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action_alpha_0_01 ({'tag': ['pr-331']})   |
    |:--------------------------|:-------------------------------------------------------------|:---------------------------------------------------------|
    | Ant-v4                    | 1824.17 ± 905.78                                             | 2702.91 ± 683.53                                         |
    | HalfCheetah-v4            | 2637.19 ± 1068.49                                            | 2716.51 ± 1314.93                                        |
    | Hopper-v4                 | 2741.42 ± 269.11                                             | 2334.22 ± 441.89                                         |
    | InvertedDoublePendulum-v4 | 5626.22 ± 289.23                                             | 5409.03 ± 318.68                                         |
    | Reacher-v4                | -4.65 ± 0.96                                                 | -3.93 ± 0.19                                             |
    | Swimmer-v4                | 124.88 ± 22.24                                               | 129.97 ± 12.02                                           |
    | Pusher-v4                 | -30.35 ± 6.43                                                | -31.48 ± 9.83                                            |

    Learning curves:
    

    Tracked experiments:     

    
    

    Results with `rpo_alpha=0.5` (not tuned) on the tuned environments:

    |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
    |:--------------------------|:-------------------------------------------------------------|:----------------------------------------------|
    | Ant-v4                    | 1774.42 ± 819.08                                             | -7.99 ± 2.47                                  |
    | HalfCheetah-v4            | 2667.34 ± 1109.99                                            | 2163.57 ± 790.16                              |
    | Hopper-v4                 | 2761.77 ± 286.88                                             | 1557.18 ± 206.74                              |
    | InvertedDoublePendulum-v4 | 5644.00 ± 353.46                                             | 296.97 ± 15.95                                |
    | Reacher-v4                | -4.67 ± 0.88                                                 | -66.35 ± 0.66                                 |
    | Swimmer-v4                | 124.52 ± 22.10                                               | 117.82 ± 10.07                                |
    | Pusher-v4                 | -30.62 ± 6.80                                                | -276.32 ± 26.99                               |

    Learning curves:
    


    Tracked experiments:     

    



=== "MuJoCo v2"

    |                     | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
    |:--------------------|:-------------------------------------------------------------|:----------------------------------------------|
    | HumanoidStandup-v2  | 109118.07 ± 19422.20                                         | 156848.90 ± 11414.50                          |
    | Humanoid-v2         | 588.22 ± 43.80                                               | 717.37 ± 97.18                                |
    | InvertedPendulum-v2 | 867.64 ± 19.97                                               | 866.60 ± 27.06                                |
    | Walker2d-v2         | 3220.99 ± 923.84                                             | 4150.51 ± 348.03                              |

    Learning curves:
    

    Tracked experiments:     

    

    The following environments require tuning of `alpha` (Algorithm 1, line 13, paper: https://arxiv.org/pdf/2212.07536.pdf). As described in the paper, this variable should be tuned for environments tested. A larger value means more randomness, whereas a smaller value indicates less randomness. Some mujoco environments require a smaller `alpha=0.01` value to achieve a reasonable performance compared to `alpha=0.5` for the rest of the environments. This version (`alpha=0.01`) of runs is indicated as `rpo_continuous_action_alpha_0_01` in the table and learning curves.

    |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action_alpha_0_01 ({'tag': ['pr-331']})   |
    |:--------------------------|:-------------------------------------------------------------|:---------------------------------------------------------|
    | Ant-v2                    | 2412.35 ± 949.44                                             | 3084.95 ± 759.51                                         |
    | HalfCheetah-v2            | 2717.27 ± 1269.85                                            | 2707.91 ± 1215.21                                        |
    | Hopper-v2                 | 2387.39 ± 645.41                                             | 2272.78 ± 588.66                                         |
    | InvertedDoublePendulum-v2 | 5630.91 ± 377.93                                             | 5661.29 ± 316.04                                         |
    | Reacher-v2                | -4.61 ± 0.53                                                 | -4.24 ± 0.25                                             |
    | Swimmer-v2                | 132.07 ± 9.92                                                | 141.37 ± 8.70                                            |
    | Pusher-v2                 | -33.93 ± 8.55                                                | -26.22 ± 2.52                                            |
            
    
    Learning curves:
    

    Tracked experiments:     

    

    Results with `rpo_alpha=0.5` (not tuned) on the tuned environments:

    |                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
    |:--------------------------|:-------------------------------------------------------------|:----------------------------------------------|
    | Ant-v2                    | 2495.65 ± 991.65                                             | -7.81 ± 3.57                                  |
    | HalfCheetah-v2            | 2722.03 ± 1231.28                                            | 2605.06 ± 1183.30                             |
    | Hopper-v2                 | 2356.83 ± 650.91                                             | 1609.79 ± 164.16                              |
    | InvertedDoublePendulum-v2 | 5675.31 ± 244.34                                             | 274.78 ± 16.40                                |
    | Reacher-v2                | -4.67 ± 0.48                                                 | -66.55 ± 0.20                                 |
    | Swimmer-v2                | 131.53 ± 9.94                                                | 114.34 ± 3.95                                 |
    | Pusher-v2                 | -33.46 ± 8.41                                                | -275.09 ± 15.65                               |

    Learning curves:
    


    Tracked experiments:     

    

    


=== "Gym(Gymnasium)"

    Results on two continuous gym environments. The PPO and RPO run for 8M timesteps, and results are computed over 10 random seeds.


    |                  | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
    |:-----------------|:-------------------------------------------------------------|:----------------------------------------------|
    | Pendulum-v1      | -1141.98 ± 135.55                                            | -151.08 ± 3.73                                |
    | BipedalWalker-v3 | 172.12 ± 96.05                                               | 227.11 ± 18.23                                |

    Learning curves:
    

    Tracked experiments:

???+ failure

Failure case of `rpo_alpha=0.5`:

Overall, we observed that `rpo_alpha=0.5` is strictly equal to or better than the default PPO in 93% of environments tested (all 48/48 dm_control, 2/2 Gym, 7/11 mujoco_v4. Total 57 out of 61 environments tested). 

Here are the failure cases:
`Mujoco v4 and v2: Ant InvertedDoublePendulum  Reacher Pusher`

|                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
|:--------------------------|:-------------------------------------------------------------|:----------------------------------------------|
| Ant-v4                    | 1831.63 ± 867.71                                             | -10.43 ± 8.16                                 |
| InvertedDoublePendulum-v4 | 5490.71 ± 261.50                                             | 303.36 ± 13.39                                |
| Reacher-v4                | -4.58 ± 0.73                                                 | -66.62 ± 0.56                                 |
| Pusher-v4                 | -30.63 ± 6.42                                                | -276.11 ± 26.52                               |

Learning curves:


Tracked experiments:     





|                           | ppo_continuous_action_8M ({'tag': ['v1.0.0-13-gcbd83f6']})   | rpo_continuous_action ({'tag': ['pr-331']})   |
|:--------------------------|:-------------------------------------------------------------|:----------------------------------------------|
| Ant-v2                    | 2493.50 ± 993.24                                             | -7.26 ± 2.28                                  |
| InvertedDoublePendulum-v2 | 5568.37 ± 401.65                                             | 278.94 ± 15.34                                |
| Reacher-v2                | -4.62 ± 0.47                                                 | -66.61 ± 0.23                                 |
| Pusher-v2                 | -33.51 ± 8.47                                                | -276.01 ± 15.93                               |


Learning curves:



Tracked experiments:     




However, tuning of `rpo_alpha` (`rpo_alpha=0.01` on failed cases) helps RPO to overcome the failure, and it performs strictly equal to or better than the default PPO in all (100%) of tested environments.

Rl Algorithms Sac (`docs/rl-algorithms/sac.md`) Copy Section

Soft Actor-Critic (SAC)

Overview Copy Chapter

The Soft Actor-Critic (SAC) algorithm extends the DDPG algorithm by 1) using a stochastic policy, which in theory can express multi-modal optimal policies.
This also enables the use of 2) entropy regularization based on the stochastic policy's entropy. It serves as a built-in, state-dependent exploration heuristic for the agent, instead of relying on non-correlated noise processes as in DDPG, or TD3
Additionally, it incorporates the 3) usage of two Soft Q-network to reduce the overestimation bias issue in Q-network-based methods.

Original papers:
The SAC algorithm's initial proposal, and later updates and improvements can be chronologically traced through the following publications:

• Reinforcement Learning with Deep Energy-Based Policies
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Composable Deep Reinforcement Learning for Robotic Manipulation
• Soft Actor-Critic Algorithms and Applications
• Soft Actor-Critic for Discrete Action Settings (No peer review, preprint only)

Reference resources:

• :material-github: haarnoja/sac
• :material-github: openai/spinningup
• :material-github: pranz24/pytorch-soft-actor-critic
• :material-github: DLR-RM/stable-baselines3
• :material-github: denisyarats/pytorch_sac
• :material-github: haarnoja/softqlearning
• :material-github: rail-berkeley/softlearning
• :material-github: p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch
• :material-github: toshikwa/sac-discrete.pytorch

Below are our single-file implementations of SAC:

`sac_continuous_action.py` Copy Chapter

The sac_continuous_action.py has the following features:

• For continuous action spaces.
• Works with the Box observation space of low-level features.
• Works with the Box (continuous) action space.
• Numerically stable stochastic policy based on :material-github: openai/spinningup and pranz24/pytorch-soft-actor-critic implementations.
• Supports automatic entropy coefficient $\alpha$ tuning, enabled by default.

Usage for continuous action spaces Copy Section

=== "poetry"

```bash
poetry install
uv pip install ".[mujoco]"
uv run python cleanrl/sac_continuous_action.py --help
uv run python cleanrl/sac_continuous_action.py --env-id Hopper-v4
uv run python cleanrl/sac_continuous_action.py --env-id Hopper-v4 --autotune False --alpha 0.2 ## Without Automatic entropy coef. tuning
```

=== "pip"

```bash
pip install -r requirements/requirements-mujoco.txt
python cleanrl/sac_continuous_action.py --help
python cleanrl/sac_continuous_action.py --env-id Mujoco-v4
python cleanrl/sac_continuous_action.py --env-id Hopper-v4 --autotune False --alpha 0.2 ## Without Automatic entropy coef. tuning
```

Explanation of the logged metrics Copy Section

Running python cleanrl/sac_continuous_action.py will automatically record various metrics such as actor or value losses in Tensorboard. Below is the documentation for these metrics:

• charts/episodic_return: the episodic return of the game during training

• charts/SPS: number of steps per second

• losses/qf1_loss, losses/qf2_loss: for each Soft Q-value network $Q_{\theta_i}$, $i \in {1,2}$, this metric holds the mean squared error (MSE) between the soft Q-value estimate $Q_{\theta_i}(s, a)$ and the entropy regularized Bellman update target estimated as $r_t + \gamma , Q_{\theta_{i}^{'}}(s', a') + \alpha , \mathcal{H} \big[ \pi(a' \vert s') \big]$.

More formally, the Soft Q-value loss for the $i$-th network is obtained by:

$$
J(\theta^{Q}{i}) = \mathbb{E}{(s,a,r,s') \sim \mathcal{D}} \big[ (Q_{\theta_i}(s, a) - y)^2 \big]
$$

with the entropy regularized, Soft Bellman update target:
$$
y = r(s, a) + \gamma ({\color{orange} \min_{\theta_{1,2}}Q_{\theta_i^{'}}(s',a')} - \alpha , \text{log} \pi( \cdot \vert s'))
$$ where $a' \sim \pi( \cdot \vert s')$, $\text{log} \pi( \cdot \vert s')$ approximates the entropy of the policy, and $\mathcal{D}$ is the replay buffer storing samples of the agent during training.

Here, $\min_{\theta_{1,2}}Q_{\theta_i^{'}}(s',a')$ takes the minimum Soft Q-value network estimate between the two target Q-value networks $Q_{\theta_1^{'}}$ and $Q_{\theta_2^{'}}$ for the next state and action pair, so as to reduce over-estimation bias.

• losses/qf_loss: averages losses/qf1_loss and losses/qf2_loss for comparison with algorithms using a single Q-value network.

• losses/actor_loss: Given the stochastic nature of the policy in SAC, the actor (or policy) objective is formulated so as to maximize the likelihood of actions $a \sim \pi( \cdot \vert s)$ that would result in high Q-value estimate $Q(s, a)$. Additionally, the policy objective encourages the policy to maintain its entropy high enough to help explore, discover, and capture multi-modal optimal policies.

The policy's objective function can thus be defined as:

$$
\text{max}{\phi} , J{\pi}(\phi) = \mathbb{E}{s \sim \mathcal{D}} \Big[ \text{min}{i=1,2} Q_{\theta_i}(s, a) - \alpha , \text{log}\pi_{\phi}(a \vert s) \Big]
$$

where the action is sampled using the reparameterization trick[^1]: $a = \mu_{\phi}(s) + \epsilon , \sigma_{\phi}(s)$ with $\epsilon \sim \mathcal{N}(0, 1)$, $\text{log} \pi_{\phi}( \cdot \vert s')$ approximates the entropy of the policy, and $\mathcal{D}$ is the replay buffer storing samples of the agent during training.

• losses/alpha: $\alpha$ coefficient for entropy regularization of the policy.

• losses/alpha_loss: In the policy's objective defined above, the coefficient of the entropy bonus $\alpha$ is kept fixed all across the training.
  As suggested by the authors in Section 5 of the Soft Actor-Critic And Applications paper, the original purpose of augmenting the standard reward with the entropy of the policy is to encourage exploration of not well enough explored states (thus high entropy).
  Conversely, for states where the policy has already learned a near-optimal policy, it would be preferable to reduce the entropy bonus of the policy, so that it does not become sub-optimal due to the entropy maximization incentive.

Therefore, having a fixed value for $\alpha$ does not fit this desideratum of matching the entropy bonus with the knowledge of the policy at an arbitrary state during its training.

To mitigate this, the authors proposed a method to dynamically adjust $\alpha$ as the policy is trained, which is as follows:

$$
\alpha^{}t = \text{argmin}{\alpha_t} \mathbb{E}_{a_t \sim \pi^{}_t} \big[ -\alpha_t , \text{log}\pi^{*}_t(a_t \vert s_t; \alpha_t) - \alpha_t \mathcal{H} \big],
$$

where $\mathcal{H}$ represents the target entropy, the desired lower bound for the expected entropy of the policy over the trajectory distribution induced by the latter.
As a heuristic for the target entropy, the authors use the dimension of the action space of the task.