# Technical Documentation: DLR-RM/stable-baselines3
> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/DLR-RM/stable-baselines3)
### Overview (`README.md`)
[](https://github.com/DLR-RM/stable-baselines3/actions/workflows/ci.yml)
[](https://stable-baselines3.readthedocs.io/en/master/?badge=master) [](https://github.com/DLR-RM/stable-baselines3/actions/workflows/ci.yml)
[](https://github.com/psf/black)
# Stable Baselines3
Stable Baselines3 (SB3) is a set of reliable implementations of reinforcement learning algorithms in PyTorch. It is the next major version of [Stable Baselines](https://github.com/hill-a/stable-baselines).
You can read a detailed presentation of Stable Baselines3 in the [v1.0 blog post](https://araffin.github.io/post/sb3/) or our [JMLR paper](https://jmlr.org/papers/volume22/20-1364/20-1364.pdf).
These algorithms will make it easier for the research community and industry to replicate, refine, and identify new ideas, and will create good baselines to build projects on top of. We expect these tools will be used as a base around which new ideas can be added, and as a tool for comparing a new approach against existing ones. We also hope that the simplicity of these tools will allow beginners to experiment with a more advanced toolset, without being buried in implementation details.
**Note: Despite its simplicity of use, Stable Baselines3 (SB3) assumes you have some knowledge about Reinforcement Learning (RL).** You should not utilize this library without some practice. To that extent, we provide good resources in the [documentation](https://stable-baselines3.readthedocs.io/en/master/guide/rl.html) to get started with RL.
## Main Features
**The performance of each algorithm was tested** (see *Results* section in their respective page).
You can take a look at the issues [#48](https://github.com/DLR-RM/stable-baselines3/issues/48) and [#49](https://github.com/DLR-RM/stable-baselines3/issues/49) for more details.
We also provide detailed logs and reports on the [OpenRL Benchmark](https://wandb.ai/openrlbenchmark/sb3) platform.
| **Features** | **Stable-Baselines3** |
| --------------------------- | ----------------------|
| State of the art RL methods | :heavy_check_mark: |
| Documentation | :heavy_check_mark: |
| Custom environments | :heavy_check_mark: |
| Custom policies | :heavy_check_mark: |
| Common interface | :heavy_check_mark: |
| `Dict` observation space support | :heavy_check_mark: |
| Ipython / Notebook friendly | :heavy_check_mark: |
| Tensorboard support | :heavy_check_mark: |
| PEP8 code style | :heavy_check_mark: |
| Custom callback | :heavy_check_mark: |
| High code coverage | :heavy_check_mark: |
| Type hints | :heavy_check_mark: |
### Planned features
Since most of the features from the [original roadmap](https://github.com/DLR-RM/stable-baselines3/issues/1) have been implemented, there are no major changes planned for SB3; it is now *stable*.
If you want to contribute, you can search in the issues for the ones where [help is welcomed](https://github.com/DLR-RM/stable-baselines3/labels/help%20wanted) and the other [proposed enhancements](https://github.com/DLR-RM/stable-baselines3/labels/enhancement).
While SB3 development is now focused on bug fixes and maintenance (doc update, user experience, ...), there is more active development going on in the associated repositories:
- newer algorithms are regularly added to the [SB3 Contrib](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib) repository
- faster variants are developed in the [SBX (SB3 + Jax)](https://github.com/araffin/sbx) repository
- the training framework for SB3, the RL Zoo, has an active [roadmap](https://github.com/DLR-RM/rl-baselines3-zoo/issues/299)
## Migration guide: from Stable-Baselines (SB2) to Stable-Baselines3 (SB3)
A migration guide from SB2 to SB3 can be found in the [documentation](https://stable-baselines3.readthedocs.io/en/master/guide/migration.html).
## Documentation
Documentation is available online: [https://stable-baselines3.readthedocs.io/](https://stable-baselines3.readthedocs.io/)
## Integrations
Stable-Baselines3 has some integration with other libraries/services like Weights & Biases for experiment tracking or Hugging Face for storing/sharing trained models. You can find out more in the [dedicated section](https://stable-baselines3.readthedocs.io/en/master/guide/integrations.html) of the documentation.
## RL Baselines3 Zoo: A Training Framework for Stable Baselines3 Reinforcement Learning Agents
[RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) is a training framework for Reinforcement Learning (RL).
It provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.
In addition, it includes a collection of tuned hyperparameters for common environments and RL algorithms, and agents trained with those settings.
Goals of this repository:
1. Provide a simple interface to train and enjoy RL agents
2. Benchmark the different Reinforcement Learning algorithms
3. Provide tuned hyperparameters for each environment and RL algorithm
4. Have fun with the trained agents!
Github repo: https://github.com/DLR-RM/rl-baselines3-zoo
Documentation: https://rl-baselines3-zoo.readthedocs.io/en/master/
## SB3-Contrib: Experimental RL Features
We implement experimental features in a separate contrib repository: [SB3-Contrib](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib)
This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Recurrent PPO (PPO LSTM), CrossQ, Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO).
Documentation is available online: [https://sb3-contrib.readthedocs.io/](https://sb3-contrib.readthedocs.io/)
## Stable-Baselines Jax (SBX)
[Stable Baselines Jax (SBX)](https://github.com/araffin/sbx) is a proof of concept version of Stable-Baselines3 in Jax, with recent algorithms like DroQ or CrossQ.
It provides a minimal number of features compared to SB3 but can be much faster (up to 20x times!): https://twitter.com/araffin2/status/1590714558628253698
## Installation
**Note:** Stable-Baselines3 supports PyTorch >= 2.8
### Prerequisites
Stable Baselines3 requires Python 3.10+.
#### Windows
To install stable-baselines on Windows, please look at the [documentation](https://stable-baselines3.readthedocs.io/en/master/guide/install.html#prerequisites).
### Install using pip
Install the Stable Baselines3 package:
```sh
pip install 'stable-baselines3[extra]'
```
This includes optional dependencies like Tensorboard, OpenCV, `ale-py` to train on atari games, as well as `pandas` and `matplotlib` for plotting and analyzing results. If you do not need those, you can use:
```sh
pip install stable-baselines3
```
Please read the [documentation](https://stable-baselines3.readthedocs.io/) for more details and alternatives (from source, using docker).
## Example
Most of the code in the library tries to follow a sklearn-like syntax for the Reinforcement Learning algorithms.
Here is a quick example of how to train and run PPO on a cartpole environment:
```python
import gymnasium as gym
from stable_baselines3 import PPO
env = gym.make("CartPole-v1", render_mode="human")
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10_000)
vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
actions, _states = model.predict(obs, deterministic=True)
obs, rewards, dones, info = vec_env.step(actions)
vec_env.render()
# VecEnv resets automatically
# if done:
# obs = env.reset()
env.close()
```
Or just train a model with a one liner if [the environment is registered in Gymnasium](https://gymnasium.farama.org/tutorials/gymnasium_basics/environment_creation/#registering-envs) and if [the policy is registered](https://stable-baselines3.readthedocs.io/en/master/guide/custom_policy.html):
```python
from stable_baselines3 import PPO
model = PPO("MlpPolicy", "CartPole-v1").learn(10_000)
```
Please read the [documentation](https://stable-baselines3.readthedocs.io/) for more examples.
## Try it online with Colab Notebooks!
All the following examples can be executed online using Google Colab notebooks:
- [Full Tutorial](https://github.com/araffin/rl-tutorial-jnrr19)
- [All Notebooks](https://github.com/Stable-Baselines-Team/rl-colab-notebooks/tree/sb3)
- [Getting Started](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/stable_baselines_getting_started.ipynb)
- [Training, Saving, Loading](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/saving_loading_dqn.ipynb)
- [Multiprocessing](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb)
- [Monitor Training and Plotting](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/monitor_training.ipynb)
- [Atari Games](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/atari_games.ipynb)
- [RL Baselines Zoo](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/rl-baselines-zoo.ipynb)
- [PyBullet](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/pybullet.ipynb)
## Implemented Algorithms
| **Name** | **Recurrent** | `Box` | `Discrete` | `MultiDiscrete` | `MultiBinary` | **Multi Processing** |
| ------------------- | ------------------ | ------------------ | ------------------ | ------------------- | ------------------ | --------------------------------- |
| A2C | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| ARS[1](#f1) | :x: | :heavy_check_mark: | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: |
| CrossQ[1](#f1)[2](#f2) | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
| DDPG | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
| DQN | :x: | :x: | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: |
| DroQ[2](#f2) | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
| HER | :x: | :heavy_check_mark: | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: |
| Maskable PPO[1](#f1) | :x: | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| PPO | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| QR-DQN[1](#f1) | :x: | :x: | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: |
| RecurrentPPO[1](#f1) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| SAC | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
| SAC-N[2](#f2) | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
| SimBa[2](#f2) | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
| TD3 | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
| TQC[1](#f1) | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
| TRPO[1](#f1) | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
1: Implemented in [SB3 Contrib](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib) GitHub repository.
2: Implemented in [SBX (SB3 + Jax)](https://github.com/araffin/sbx)
Actions `gymnasium.spaces`:
* `Box`: A N-dimensional box that contains every point in the action space.
* `Discrete`: A list of possible actions, where only one action can be used per timestep.
* `MultiDiscrete`: A list of possible actions, where each timestep only one action of each discrete set can be used.
* `MultiBinary`: A list of possible actions, where each timestep any of the actions can be used in any combination.
## Testing the installation
### Install dependencies
```sh
pip install -e '.[docs,tests,extra]'
```
### Run tests
All unit tests in stable baselines3 can be run using `pytest` runner:
```sh
make pytest
```
To run a single test file:
```sh
python3 -m pytest -v tests/test_env_checker.py
```
To run a single test:
```sh
python3 -m pytest -v -k 'test_check_env_dict_action'
```
You can also do a static type check using `mypy`:
```sh
pip install mypy
make type
```
Codestyle check with `ruff`:
```sh
pip install ruff
make lint
```
## Projects Using Stable-Baselines3
We try to maintain a list of projects using stable-baselines3 in the [documentation](https://stable-baselines3.readthedocs.io/en/master/misc/projects.html),
please tell us if you want your project to appear on this page ;)
## Citing the Project
To cite this repository in publications:
```bibtex
@article{stable-baselines3,
author = {Antonin Raffin and Ashley Hill and Adam Gleave and Anssi Kanervisto and Maximilian Ernestus and Noah Dormann},
title = {Stable-Baselines3: Reliable Reinforcement Learning Implementations},
journal = {Journal of Machine Learning Research},
year = {2021},
volume = {22},
number = {268},
pages = {1-8},
url = {https://jmlr.org/papers/v22/20-1364.html}
}
```
Note: If you need to refer to a specific version of SB3, you can also use the [Zenodo DOI](https://doi.org/10.5281/zenodo.8123988).
## Maintainers
Stable-Baselines3 is currently maintained by [Ashley Hill](https://github.com/hill-a) (aka @hill-a), [Antonin Raffin](https://araffin.github.io/) (aka [@araffin](https://github.com/araffin)), [Maximilian Ernestus](https://github.com/ernestum) (aka @ernestum), [Adam Gleave](https://github.com/adamgleave) (@AdamGleave), [Anssi Kanervisto](https://github.com/Miffyli) (@Miffyli) and [Quentin Gallouédec](https://github.com/qgallouedec) (@qgallouedec).
**Important Note: We do not provide technical support, or consulting** and do not answer personal questions via email.
Please post your question on the [RL Discord](https://discord.com/invite/xhfNqQv), [Reddit](https://www.reddit.com/r/reinforcementlearning/), or [Stack Overflow](https://stackoverflow.com/) in that case.
## How To Contribute
For anyone interested in making the baselines better, there is still some documentation that needs to be done.
If you want to contribute, please read [**CONTRIBUTING.md**](./CONTRIBUTING.md) guide first.
## Acknowledgments
The initial work to develop Stable Baselines3 was partially funded by the project *Reduced Complexity Models* from the *Helmholtz-Gemeinschaft Deutscher Forschungszentren*, and by the EU Horizon 2020 Research and Innovation Programme under grant number 951992 ([VeriDream](https://www.veridream.eu/)).
The original version, Stable Baselines, was created in the [robotics lab U2IS](http://u2is.ensta-paris.fr/?lang=en) ([INRIA Flowers](https://flowers.inria.fr/) team) at [ENSTA ParisTech](https://www.ensta.fr/en).
Logo credits: [L.M. Tenkes](https://www.instagram.com/lucillehue/)
---
### Overview (`docs/README.md`)
# Stable Baselines3 Documentation
This folder contains documentation for the RL baselines.
### Build the Documentation
#### Install Sphinx and Theme
Execute this command in the project root:
```
pip install -e ".[docs]"
```
#### Building the Docs
In the `docs/` folder:
```
make html
```
for automatic rebuilding each time a file is changed:
```
sphinx-autobuild . _build/html
```
---
### Guide Quickstart (`docs/guide/quickstart.md`)
(quickstart)=
# Getting Started
:::{note}
Stable-Baselines3 (SB3) uses [vectorized environments (VecEnv)](vec_envs.md) internally.
Please read the associated section to learn more about its features and differences compared to a single Gym environment.
:::
Most of the library follows a sklearn-like syntax for the reinforcement learning algorithms.
Here is a quick example of how to train and run A2C on a CartPole environment:
```python
import gymnasium as gym
from stable_baselines3 import A2C
env = gym.make("CartPole-v1", render_mode="rgb_array")
model = A2C("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10_000)
vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
actions, _states = model.predict(obs, deterministic=True)
obs, rewards, dones, info = vec_env.step(actions)
vec_env.render("human")
# VecEnv resets automatically
# if done:
# obs = vec_env.reset()
```
:::{note}
You can find explanations about the logger output and names in the {ref}`Logger ` section.
:::
Or just train a model with a one-liner if
[the environment is registered in Gymnasium](https://gymnasium.farama.org/tutorials/gymnasium_basics/environment_creation/#registering-envs) and the
policy is registered:
```python
from stable_baselines3 import A2C
model = A2C("MlpPolicy", "CartPole-v1").learn(10_000)
```
---
### Conda Env.Yml (`docs/conda_env.yml`)
name: root
channels:
- conda-forge
dependencies:
- pip=26.1.2
- python=3.12
- pytorch-cpu=2.9.1
- pip:
- gymnasium>=0.29.1,<2.0
- cloudpickle
- opencv-python-headless
- pandas
- numpy>=1.20,<3.0
- matplotlib
- sphinx>=5,<10
- sphinx_rtd_theme>=3.0
- sphinx_copybutton
- myst-parser>=4,<6
---
### Index (`docs/index.rst`)
.. Stable Baselines3 documentation master file, created by
sphinx-quickstart on Thu Sep 26 11:06:54 2019.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Stable-Baselines3 Docs - Reliable Reinforcement Learning Implementations
========================================================================
`Stable Baselines3 (SB3) `_ is a set of reliable implementations of reinforcement learning algorithms in PyTorch.
It is the next major version of `Stable Baselines `_.
GitHub repository: https://github.com/DLR-RM/stable-baselines3
Paper: https://jmlr.org/papers/volume22/20-1364/20-1364.pdf
RL Baselines3 Zoo (training framework for SB3): https://github.com/DLR-RM/rl-baselines3-zoo
RL Baselines3 Zoo provides a collection of pre-trained agents, scripts for training, evaluating agents, tuning hyperparameters, plotting results, and recording videos.
SB3 Contrib (experimental RL code, latest algorithms): https://github.com/Stable-Baselines-Team/stable-baselines3-contrib
SBX (SB3 + Jax): https://github.com/araffin/sbx
Main Features
--------------
- Unified structure for all algorithms
- PEP 8 compliant (unified code style)
- Documented functions and classes
- Tests, high code coverage, and type hints
- Clean code
- TensorBoard support
- **The performance of each algorithm was tested** (see *Results* section in their respective page)
.. toctree::
:maxdepth: 2
:caption: User Guide
guide/install
guide/quickstart
guide/rl_tips
guide/rl
guide/algos
guide/examples
guide/vec_envs
guide/custom_policy
guide/custom_env
guide/callbacks
guide/tensorboard
guide/integrations
guide/rl_zoo
guide/sb3_contrib
guide/sbx
guide/plotting
guide/imitation
guide/migration
guide/checking_nan
guide/developer
guide/save_format
guide/export
.. toctree::
:maxdepth: 1
:caption: RL Algorithms
modules/base
modules/a2c
modules/ddpg
modules/dqn
modules/her
modules/ppo
modules/sac
modules/td3
.. toctree::
:maxdepth: 1
:caption: Common
common/atari_wrappers
common/env_util
common/envs
common/distributions
common/evaluation
common/env_checker
common/monitor
common/logger
common/noise
common/utils
.. toctree::
:maxdepth: 1
:caption: Misc
misc/changelog
misc/projects
Citing Stable Baselines3
------------------------
To cite this project in publications:
.. code-block:: bibtex
@article{stable-baselines3,
author = {Antonin Raffin and Ashley Hill and Adam Gleave and Anssi Kanervisto and Maximilian Ernestus and Noah Dormann},
title = {Stable-Baselines3: Reliable Reinforcement Learning Implementations},
journal = {Journal of Machine Learning Research},
year = {2021},
volume = {22},
number = {268},
pages = {1-8},
url = {https://jmlr.org/papers/v22/20-1364.html}
}
Note: If you need to refer to a specific version of SB3, you can also use the `Zenodo DOI `_.
Contributing
------------
For anyone interested in making the RL baselines better, there are still some improvements
that need to be done.
You can check issues in the `repository `_.
If you want to contribute, please read `CONTRIBUTING.md `_ first.
Indices and tables
-------------------
* :ref:`genindex`
* :ref:`search`
* :ref:`modindex`
---
### Spelling Wordlist (`docs/spelling_wordlist.txt`)
py
env
atari
argparse
Argparse
TensorFlow
feedforward
envs
VecEnv
pretrain
petrained
tf
th
nn
np
str
mujoco
cpu
ndarray
ndarrays
timestep
timesteps
stepsize
dataset
adam
fn
normalisation
Kullback
Leibler
boolean
deserialized
pretrained
minibatch
subprocesses
ArgumentParser
Tensorflow
Gaussian
approximator
minibatches
hyperparameters
hyperparameter
vectorized
rl
colab
dataloader
npz
datasets
vf
logits
num
Utils
backpropagate
prepend
NaN
preprocessing
Cloudpickle
async
multiprocess
tensorflow
mlp
cnn
neglogp
tanh
coef
repo
Huber
params
ppo
arxiv
Arxiv
func
DQN
Uhlenbeck
Ornstein
multithread
cancelled
Tensorboard
parallelize
customising
serializable
Multiprocessed
cartpole
toolset
lstm
rescale
ffmpeg
avconv
unnormalized
Github
pre
preprocess
backend
attr
preprocess
Antonin
Raffin
araffin
Homebrew
Numpy
Theano
rollout
kfac
Piecewise
csv
nvidia
visdom
tensorboard
preprocessed
namespace
sklearn
GoalEnv
Torchy
pytorch
dicts
optimizers
Deprecations
forkserver
cuda
Polyak
gSDE
rollouts
Pyro
softmax
stdout
Contrib
Quantile
---
### Common Atari Wrappers (`docs/common/atari_wrappers.md`)
(atari-wrapper)=
# Atari Wrappers
```{eval-rst}
.. automodule:: stable_baselines3.common.atari_wrappers
:members:
```
---
### Common Distributions (`docs/common/distributions.md`)
(distributions)=
# Probability Distributions
Probability distributions used for the different action spaces:
- `CategoricalDistribution` -> Discrete
- `DiagGaussianDistribution` -> Box (continuous actions)
- `StateDependentNoiseDistribution` -> Box (continuous actions) when `use_sde=True`
% - ``MultiCategoricalDistribution`` -> MultiDiscrete
% - ``BernoulliDistribution`` -> MultiBinary
The policy networks output parameters for the distributions (named `flat` in the methods).
Actions are then sampled from those distributions.
For instance, in the case of discrete actions. The policy network outputs probability
of taking each action. The `CategoricalDistribution` allows sampling from it,
computes the entropy, the log probability (`log_prob`) and backpropagate the gradient.
In the case of continuous actions, a Gaussian distribution is used. The policy network outputs
mean and (log) std of the distribution (assumed to be a `DiagGaussianDistribution`).
```{eval-rst}
.. automodule:: stable_baselines3.common.distributions
:members:
```
---
### Common Env Checker (`docs/common/env_checker.md`)
(env-checker)=
# Gym Environment Checker
```{eval-rst}
.. automodule:: stable_baselines3.common.env_checker
:members:
```
---
### Common Env Util (`docs/common/env_util.md`)
(env-util)=
# Environments Utils
```{eval-rst}
.. automodule:: stable_baselines3.common.env_util
:members:
```
---
### Common Envs (`docs/common/envs.md`)
(envs)=
```{eval-rst}
.. automodule:: stable_baselines3.common.envs
```
# Custom Environments
Those environments were created for testing purposes.
## BitFlippingEnv
```{eval-rst}
.. autoclass:: BitFlippingEnv
:members:
```
## SimpleMultiObsEnv
```{eval-rst}
.. autoclass:: SimpleMultiObsEnv
:members:
```
---
### Common Evaluation (`docs/common/evaluation.md`)
(eval)=
# Evaluation Helper
```{eval-rst}
.. automodule:: stable_baselines3.common.evaluation
:members:
```
---
### Common Logger (`docs/common/logger.md`)
(logger)=
# Logger
To overwrite the default logger, you can pass one to the algorithm.
Available formats are `["stdout", "csv", "log", "tensorboard", "json"]`.
:::{warning}
When passing a custom logger object,
this will overwrite `tensorboard_log` and `verbose` settings
passed to the constructor.
:::
```python
from stable_baselines3 import A2C
from stable_baselines3.common.logger import configure
tmp_path = "/tmp/sb3_log/"
# set up logger
new_logger = configure(tmp_path, ["stdout", "csv", "tensorboard"])
model = A2C("MlpPolicy", "CartPole-v1", verbose=1)
# Set new logger
model.set_logger(new_logger)
model.learn(10000)
```
## Explanation of logger output
You can find below short explanations of the values logged in Stable-Baselines3 (SB3).
Depending on the algorithm used and of the wrappers/callbacks applied, SB3 only logs a subset of those keys during training.
Below you can find an example of the logger output when training a PPO agent:
```bash
-----------------------------------------
| eval/ | |
| mean_ep_length | 200 |
| mean_reward | -157 |
| rollout/ | |
| ep_len_mean | 200 |
| ep_rew_mean | -227 |
| time/ | |
| fps | 972 |
| iterations | 19 |
| time_elapsed | 80 |
| total_timesteps | 77824 |
| train/ | |
| approx_kl | 0.037781604 |
| clip_fraction | 0.243 |
| clip_range | 0.2 |
| entropy_loss | -1.06 |
| explained_variance | 0.999 |
| learning_rate | 0.001 |
| loss | 0.245 |
| n_updates | 180 |
| policy_gradient_loss | -0.00398 |
| std | 0.205 |
| value_loss | 0.226 |
-----------------------------------------
```
### eval/
All `eval/` values are computed by the `EvalCallback`.
- `mean_ep_length`: Mean episode length
- `mean_reward`: Mean episodic reward (during evaluation)
- `success_rate`: Mean success rate during evaluation (1.0 means 100% success), the environment info dict must contain an `is_success` key to compute that value
### rollout/
- `ep_len_mean`: Mean episode length (averaged over `stats_window_size` episodes, 100 by default)
- `ep_rew_mean`: Mean episodic training reward (averaged over `stats_window_size` episodes, 100 by default), a `Monitor` wrapper is required to compute that value (automatically added by `make_vec_env`).
- `exploration_rate`: Current value of the exploration rate when using DQN, it corresponds to the fraction of actions taken randomly (epsilon of the "epsilon-greedy" exploration)
- `success_rate`: Mean success rate during training (averaged over `stats_window_size` episodes, 100 by default), you must pass an extra argument to the `Monitor` wrapper to log that value (`info_keywords=("is_success",)`) and provide `info["is_success"]=True/False` on the final step of the episode
### time/
- `episodes`: Total number of episodes
- `fps`: Number of frames per seconds (includes time taken by gradient update)
- `iterations`: Number of iterations (data collection + policy update for A2C/PPO)
- `time_elapsed`: Time in seconds since the beginning of training
- `total_timesteps`: Total number of timesteps (steps in the environments)
### train/
- `actor_loss`: Current value for the actor loss for off-policy algorithms
- `approx_kl`: approximate mean KL divergence between old and new policy (for PPO), it is an estimation of how much changes happened in the update
- `clip_fraction`: mean fraction of surrogate loss that was clipped (above `clip_range` threshold) for PPO.
- `clip_range`: Current value of the clipping factor for the surrogate loss of PPO
- `critic_loss`: Current value for the critic function loss for off-policy algorithms, usually error between value function output and TD(0), temporal difference estimate
- `ent_coef`: Current value of the entropy coefficient (when using SAC)
- `ent_coef_loss`: Current value of the entropy coefficient loss (when using SAC)
- `entropy_loss`: Mean value of the entropy loss (negative of the average policy entropy)
- `explained_variance`: Fraction of the return variance explained by the value function, see
(ev=0 => might as well have predicted zero, ev=1 => perfect prediction, ev\<0 => worse than just predicting zero)
- `learning_rate`: Current learning rate value
- `loss`: Current total loss value
- `n_updates`: Number of gradient updates applied so far
- `policy_gradient_loss`: Current value of the policy gradient loss (its value does not have much meaning)
- `value_loss`: Current value for the value function loss for on-policy algorithms, usually error between value function output and Monte-Carlo estimate (or TD(lambda) estimate)
- `std`: Current standard deviation of the noise when using generalized State-Dependent Exploration (gSDE)
```{eval-rst}
.. automodule:: stable_baselines3.common.logger
:members:
```
---
### Common Monitor (`docs/common/monitor.md`)
(monitor)=
# Monitor Wrapper
```{eval-rst}
.. automodule:: stable_baselines3.common.monitor
:members:
```
---
### Common Noise (`docs/common/noise.md`)
(noise)=
# Action Noise
```{eval-rst}
.. automodule:: stable_baselines3.common.noise
:members:
```
---
### Common Utils (`docs/common/utils.md`)
(utils)=
# Utils
```{eval-rst}
.. automodule:: stable_baselines3.common.utils
:members:
```
---
### Guide Algos (`docs/guide/algos.md`)
# RL Algorithms
This table displays the RL algorithms that are implemented in the Stable Baselines3 project,
along with some useful characteristics: support for discrete/continuous actions, multiprocessing.
:::{note}
More algorithms (like QR-DQN or TQC) are implemented in our [contrib repo](sb3_contrib.md)
and in our {ref}`SBX (SB3 + Jax) repo ` (DroQ, CrossQ, SimBa, ...).
:::
| Name | `Box` | `Discrete` | `MultiDiscrete` | `MultiBinary` | Multi Processing |
| ------------------ | ----- | ---------- | --------------- | ------------- | ---------------- |
| A2C | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| ARS [^f1] | ✔️ | ✔️ | ❌ | ❌ | ✔️ |
| CrossQ [^f1],[^f2] | ✔️ | ❌ | ❌ | ❌ | ✔️ |
| DDPG | ✔️ | ❌ | ❌ | ❌ | ✔️ |
| DQN | ❌ | ✔️ | ❌ | ❌ | ✔️ |
| DroQ [^f2] | ✔️ | ❌ | ❌ | ❌ | ✔️ |
| HER | ✔️ | ✔️ | ❌ | ❌ | ✔️ |
| Maskable PPO [^f1] | ❌ | ✔️ | ✔️ | ✔️ | ✔️ |
| PPO | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| QR-DQN [^f1] | ❌ | ️✔️ | ❌ | ❌ | ✔️ |
| RecurrentPPO [^f1] | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| SAC | ✔️ | ❌ | ❌ | ❌ | ✔️ |
| SAC-N [^f2] | ✔️ | ❌ | ❌ | ❌ | ✔️ |
| SimBa [^f2] | ✔️ | ❌ | ❌ | ❌ | ✔️ |
| TD3 | ✔️ | ❌ | ❌ | ❌ | ✔️ |
| TQC [^f1] | ✔️ | ❌ | ❌ | ❌ | ✔️ |
| TRPO [^f1] | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
[^f1]: Implemented in [SB3 Contrib](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib)
[^f2]: Implemented in [SBX (SB3 + Jax)](https://github.com/araffin/sbx)
:::{note}
`Tuple` observation spaces are not supported by any environment,
however, single-level `Dict` spaces are (cf. {ref}`Examples `).
:::
Actions `gym.spaces`:
- `Box`: A N-dimensional box that contains every point in the action
space.
- `Discrete`: A list of possible actions, where each timestep only
one of the actions can be used.
- `MultiDiscrete`: A list of possible actions, where each timestep only one action of each discrete set can be used.
- `MultiBinary`: A list of possible actions, where each timestep any of the actions can be used in any combination.
:::{note}
Some logging values (like `ep_rew_mean`, `ep_len_mean`) are only available when using a `Monitor` wrapper
See [Issue #339](https://github.com/hill-a/stable-baselines/issues/339) for more info.
:::
:::{note}
When using off-policy algorithms, [Time Limits](https://arxiv.org/abs/1712.00378) (aka timeouts) are handled
properly (cf. [issue #284](https://github.com/DLR-RM/stable-baselines3/issues/284)).
You can revert to SB3 < 2.1.0 behavior by passing `handle_timeout_termination=False`
via the `replay_buffer_kwargs` argument.
:::
## Reproducibility
Completely reproducible results are not guaranteed across PyTorch releases or different platforms.
Furthermore, results need not be reproducible between CPU and GPU executions, even when using identical seeds.
In order to make computations deterministic for your specific problem on one specific platform,
you need to pass a `seed` argument at the creation of a model.
If you pass an environment to the model using `set_env()`, then you also need to seed the environment first.
Credit: part of the *Reproducibility* section comes from [PyTorch Documentation](https://pytorch.org/docs/stable/notes/randomness.html)
## Training exceeds `total_timesteps`
When you train an agent using SB3, you pass a `total_timesteps` parameter to the `learn()` method which defines the training budget for the agent (how many interactions with the environment are allowed).
For example:
```python
from stable_baselines3 import PPO
model = PPO("MlpPolicy", "CartPole-v1").learn(total_timesteps=1_000)
```
Because of the way the algorithms work, `total_timesteps` is a lower bound (see [issue #1150](https://github.com/DLR-RM/stable-baselines3/issues/1150)).
In the example above, PPO will effectively collect `n_steps * n_envs = 2048 * 1` steps despite `total_timesteps=1_000`
In more details:
- PPO/A2C and derivates collect `n_steps * n_envs` of experience
before performing an update, so if you want to have exactly
`total_timesteps`, you will need to adjust those values
- SAC/DQN/TD3 and other off-policy algorithms collect
`train_freq * n_envs` steps before doing an update (when `train_freq` is in steps and not episodes), so if you want to have exactly `total_timesteps`
you have to adjust these values (`train_freq=4` by default for DQN)
- ARS and other population-based algorithms evaluate the policy for
`n_episodes` with `n_envs`, so unless the number of steps per
episode is fixed, it is not possible to exactly achieve
`total_timesteps`
- when using multiple envs, each call to `env.step()` corresponds to
`n_envs` timesteps, so it is no longer possible to use the
`EvaluationCallback` at an exact timestep
---
### Guide Callbacks (`docs/guide/callbacks.md`)
(callbacks)=
# Callbacks
A callback is a set of functions that will be called at given stages of the training procedure.
You can use callbacks to access internal state of the RL model during training.
It allows one to do monitoring, auto saving, model manipulation, progress bars, ...
## Custom Callback
To build a custom callback, you need to create a class that derives from `BaseCallback`.
This will give you access to events (`_on_training_start`, `_on_step`) and useful variables (like `self.model` for the RL model).
You can find two examples of custom callbacks in the documentation: one for saving the best model according to the training reward (see {ref}`Examples `), and one for logging additional values with Tensorboard (see {ref}`Tensorboard section `).
```
/* Detailed source-code truncated for AI context efficiency. */
```
:::{note}
`self.num_timesteps` corresponds to the total number of steps taken in the environment, i.e., it is the number of environments multiplied by the number of time `env.step()` was called
For the other algorithms, `self.num_timesteps` is incremented by `n_envs` (number of environments) after each call to `env.step()`
:::
:::{note}
For off-policy algorithms like SAC, DDPG, TD3 or DQN, the notion of `rollout` corresponds to the steps taken in the environment between two updates.
:::
(eventcallback)=
## Event Callback
Compared to Keras, Stable Baselines provides a second type of `BaseCallback`, named `EventCallback` that is meant to trigger events. When an event is triggered, then a child callback is called.
As an example, {ref}`EvalCallback` is an `EventCallback` that will trigger its child callback when there is a new best model.
A child callback is for instance {ref}`StopTrainingOnRewardThreshold ` that stops the training if the mean reward achieved by the RL model is above a threshold.
:::{note}
We recommend taking a look at the source code of {ref}`EvalCallback` and {ref}`StopTrainingOnRewardThreshold ` to have a better overview of what can be achieved with this kind of callbacks.
:::
```python
class EventCallback(BaseCallback):
"""
Base class for triggering callback on event.
:param callback: Callback that will be called when an event is triggered.
:param verbose: Verbosity level: 0 for no output, 1 for info messages, 2 for debug messages
"""
def __init__(self, callback: BaseCallback, verbose: int = 0):
super().__init__(verbose=verbose)
self.callback = callback
# Give access to the parent
self.callback.parent = self
...
def _on_event(self) -> bool:
return self.callback()
```
## Callback Collection
Stable Baselines provides you with a set of common callbacks for:
- saving the model periodically ({ref}`CheckpointCallback`)
- evaluating the model periodically and saving the best one ({ref}`EvalCallback`)
- chaining callbacks ({ref}`CallbackList`)
- triggering callback on events ({ref}`EventCallback`, {ref}`EveryNTimesteps`)
- logging data every N timesteps ({ref}`LogEveryNTimesteps`)
- stopping the training early based on a reward threshold ({ref}`StopTrainingOnRewardThreshold `)
(checkpointcallback)=
### CheckpointCallback
Callback for saving a model every `save_freq` calls to `env.step()`, you must specify a log folder (`save_path`)
and optionally a prefix for the checkpoints (`rl_model` by default).
If you are using this callback to stop and resume training, you may want to optionally save the replay buffer if the
model has one (`save_replay_buffer`, `False` by default).
Additionally, if your environment uses a [VecNormalize](vec_envs.md#vecnormalize) wrapper, you can save the
corresponding statistics using `save_vecnormalize` (`False` by default).
:::{warning}
When using multiple environments, each call to `env.step()` will effectively correspond to `n_envs` steps.
If you want the `save_freq` to be similar when using a different number of environments,
you need to account for it using `save_freq = max(save_freq // n_envs, 1)`.
The same goes for the other callbacks.
:::
```python
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import CheckpointCallback
# Save a checkpoint every 1000 steps
checkpoint_callback = CheckpointCallback(
save_freq=1000,
save_path="./logs/",
name_prefix="rl_model",
save_replay_buffer=True,
save_vecnormalize=True,
)
model = SAC("MlpPolicy", "Pendulum-v1")
model.learn(2000, callback=checkpoint_callback)
```
(evalcallback)=
### EvalCallback
Evaluate periodically the performance of an agent, using a separate test environment.
It will save the best model if `best_model_save_path` folder is specified and save the evaluations results in a NumPy archive (`evaluations.npz`) if `log_path` folder is specified.
:::{note}
You can pass child callbacks via `callback_after_eval` and `callback_on_new_best` arguments. `callback_after_eval` will be triggered after every evaluation, and `callback_on_new_best` will be triggered each time there is a new best model.
:::
:::{warning}
You need to make sure that `eval_env` is wrapped the same way as the training environment, for instance using the `VecTransposeImage` wrapper if you have a channel-last image as input.
The `EvalCallback` class outputs a warning if it is not the case.
:::
```python
import gymnasium as gym
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback
# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
# Use deterministic actions for evaluation
eval_callback = EvalCallback(eval_env, best_model_save_path="./logs/",
log_path="./logs/", eval_freq=500,
deterministic=True, render=False)
model = SAC("MlpPolicy", "Pendulum-v1")
model.learn(5000, callback=eval_callback)
```
(progressbarcallback)=
### ProgressBarCallback
Display a progress bar with the current progress, elapsed time and estimated remaining time.
This callback is integrated inside SB3 via the `progress_bar` argument of the `learn()` method.
:::{note}
`ProgressBarCallback` callback requires `tqdm` and `rich` packages to be installed. This is done automatically when using `pip install stable-baselines3[extra]`
:::
```python
from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import ProgressBarCallback
model = PPO("MlpPolicy", "Pendulum-v1")
# Display progress bar using the progress bar callback
# this is equivalent to model.learn(100_000, callback=ProgressBarCallback())
model.learn(100_000, progress_bar=True)
```
(callbacklist)=
### CallbackList
Class for chaining callbacks, they will be called sequentially.
Alternatively, you can pass directly a list of callbacks to the `learn()` method, it will be converted automatically to a `CallbackList`.
```python
import gymnasium as gym
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import CallbackList, CheckpointCallback, EvalCallback
checkpoint_callback = CheckpointCallback(save_freq=1000, save_path="./logs/")
# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
eval_callback = EvalCallback(eval_env, best_model_save_path="./logs/best_model",
log_path="./logs/results", eval_freq=500)
# Create the callback list
callback = CallbackList([checkpoint_callback, eval_callback])
model = SAC("MlpPolicy", "Pendulum-v1")
# Equivalent to:
# model.learn(5000, callback=[checkpoint_callback, eval_callback])
model.learn(5000, callback=callback)
```
(stoptrainingcallback)=
### StopTrainingOnRewardThreshold
Stop the training once a threshold in episodic reward (mean episode reward over the evaluations) has been reached (i.e., when the model is good enough).
It must be used with the {ref}`EvalCallback` and use the event triggered by a new best model.
```python
import gymnasium as gym
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback, StopTrainingOnRewardThreshold
# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
# Stop training when the model reaches the reward threshold
callback_on_best = StopTrainingOnRewardThreshold(reward_threshold=-200, verbose=1)
eval_callback = EvalCallback(eval_env, callback_on_new_best=callback_on_best, verbose=1)
model = SAC("MlpPolicy", "Pendulum-v1", verbose=1)
# Almost infinite number of timesteps, but the training will stop
# early as soon as the reward threshold is reached
model.learn(int(1e10), callback=eval_callback)
```
(everyntimesteps)=
### EveryNTimesteps
An {ref}`EventCallback` that will trigger its child callback every `n_steps` timesteps.
:::{note}
Because of the way `VecEnv` work, `n_steps` is a lower bound between two events when using multiple environments.
:::
```python
import gymnasium as gym
from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import CheckpointCallback, EveryNTimesteps
# this is equivalent to defining CheckpointCallback(save_freq=500)
# checkpoint_callback will be triggered every 500 steps
checkpoint_on_event = CheckpointCallback(save_freq=1, save_path="./logs/")
event_callback = EveryNTimesteps(n_steps=500, callback=checkpoint_on_event)
model = PPO("MlpPolicy", "Pendulum-v1", verbose=1)
model.learn(20_000, callback=event_callback)
```
(logeveryntimesteps)=
### LogEveryNTimesteps
A callback derived from {ref}`EveryNTimesteps` that will dump the logged data every `n_steps` timesteps.
```python
import gymnasium as gym
from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import LogEveryNTimesteps
event_callback = LogEveryNTimesteps(n_steps=1_000)
model = PPO("MlpPolicy", "Pendulum-v1", verbose=1)
# Disable auto-logging by passing `log_interval=None`
model.learn(10_000, callback=event_callback, log_interval=None)
```
(stoptrainingonmaxepisodes)=
### StopTrainingOnMaxEpisodes
Stop the training upon reaching the maximum number of episodes, regardless of the model's `total_timesteps` value.
Also, presumes that, for multiple environments, the desired behavior is that the agent trains on each env for `max_episodes`
and in total for `max_episodes * n_envs` episodes.
:::{note}
For multiple environments, the agent will train for a total of `max_episodes * n_envs` episodes.
However, it can't be guaranteed that this training will occur for an exact number of `max_episodes` per environment.
Thus, there is an assumption that, on average, each environment ran for `max_episodes`.
:::
```python
from stable_baselines3 import A2C
from stable_baselines3.common.callbacks import StopTrainingOnMaxEpisodes
# Stops training when the model reaches the maximum number of episodes
callback_max_episodes = StopTrainingOnMaxEpisodes(max_episodes=5, verbose=1)
model = A2C("MlpPolicy", "Pendulum-v1", verbose=1)
# Almost infinite number of timesteps, but the training will stop
# early as soon as the max number of episodes is reached
model.learn(int(1e10), callback=callback_max_episodes)
```
(stoptrainingonnomodelimprovement)=
### StopTrainingOnNoModelImprovement
Stop the training if there is no new best model (no new best mean reward) after more than a specific number of consecutive evaluations.
The idea is to save time in experiments when you know that the learning curves are somehow well-behaved and, therefore,
after many evaluations without improvement the learning has probably stabilized.
It must be used with the {ref}`EvalCallback` and use the event triggered after every evaluation.
```python
import gymnasium as gym
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback, StopTrainingOnNoModelImprovement
# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
# Stop training if there is no improvement after more than 3 evaluations
stop_train_callback = StopTrainingOnNoModelImprovement(max_no_improvement_evals=3, min_evals=5, verbose=1)
eval_callback = EvalCallback(eval_env, eval_freq=1000, callback_after_eval=stop_train_callback, verbose=1)
model = SAC("MlpPolicy", "Pendulum-v1", learning_rate=1e-3, verbose=1)
# Almost infinite number of timesteps, but the training will stop early
# as soon as the number of consecutive evaluations without model
# improvement is greater than 3
model.learn(int(1e10), callback=eval_callback)
```
```{eval-rst}
.. automodule:: stable_baselines3.common.callbacks
:members:
```
---
### Guide Checking Nan (`docs/guide/checking_nan.md`)
# Dealing with NaNs and infs
During the training of a model on a given environment, it is possible that the RL model becomes completely
corrupted when a NaN or an inf is given or returned from the RL model.
## How and why?
The issue arises when NaNs or infs do not crash, but simply get propagated through the training,
until all the floating point number converge to NaN or inf. This is in line with the
[IEEE Standard for Floating-Point Arithmetic (IEEE 754)](https://ieeexplore.ieee.org/document/4610935) standard, as it says:
:::{note}
Five possible exceptions can occur:
: - Invalid operation ($\sqrt{-1}$, $\inf \times 1$, $\text{NaN}\ \mathrm{mod}\ 1$, ...) return NaN
- Division by zero:
: - if the operand is not zero ($1/0$, $-2/0$, ...) returns $\pm\inf$
- if the operand is zero ($0/0$) returns signaling NaN
- Overflow (exponent too high to represent) returns $\pm\inf$
- Underflow (exponent too low to represent) returns $0$
- Inexact (not representable exactly in base 2, eg: $1/5$) returns the rounded value (ex: {code}`assert (1/5) * 3 == 0.6000000000000001`)
:::
And of these, only `Division by zero` will signal an exception, the rest will propagate invalid values quietly.
In python, dividing by zero will indeed raise the exception: `ZeroDivisionError: float division by zero`,
but ignores the rest.
The default in numpy, will warn: `RuntimeWarning: invalid value encountered`
but will not halt the code.
## Anomaly detection with PyTorch
To enable NaN detection in PyTorch you can do
```python
import torch as th
th.autograd.set_detect_anomaly(True)
```
## Numpy parameters
Numpy has a convenient way of dealing with invalid value: [numpy.seterr](https://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html),
which defines for the python process, how it should handle floating point error.
```python
import numpy as np
np.seterr(all="raise") # define before your code.
print("numpy test:")
a = np.float64(1.0)
b = np.float64(0.0)
val = a / b # this will now raise an exception instead of a warning.
print(val)
```
but this will also avoid overflow issues on floating point numbers:
```python
import numpy as np
np.seterr(all="raise") # define before your code.
print("numpy overflow test:")
a = np.float64(10)
b = np.float64(1000)
val = a ** b # this will now raise an exception
print(val)
```
but will not avoid the propagation issues:
```python
import numpy as np
np.seterr(all="raise") # define before your code.
print("numpy propagation test:")
a = np.float64("NaN")
b = np.float64(1.0)
val = a + b # this will neither warn nor raise anything
print(val)
```
## VecCheckNan Wrapper
In order to find when and from where the invalid value originated from, stable-baselines3 comes with a `VecCheckNan` wrapper.
It will monitor the actions, observations, and rewards, indicating what action or observation caused it and from what.
```python
import gymnasium as gym
from gymnasium import spaces
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv, VecCheckNan
class NanAndInfEnv(gym.Env):
"""Custom Environment that raised NaNs and Infs"""
metadata = {"render.modes": ["human"]}
def __init__(self):
super(NanAndInfEnv, self).__init__()
self.action_space = spaces.Box(low=-np.inf, high=np.inf, shape=(1,), dtype=np.float64)
self.observation_space = spaces.Box(low=-np.inf, high=np.inf, shape=(1,), dtype=np.float64)
def step(self, _action):
randf = np.random.rand()
if randf > 0.99:
obs = float("NaN")
elif randf > 0.98:
obs = float("inf")
else:
obs = randf
return [obs], 0.0, False, {}
def reset(self):
return [0.0]
def render(self, close=False):
pass
# Create environment
env = DummyVecEnv([lambda: NanAndInfEnv()])
env = VecCheckNan(env, raise_exception=True)
# Instantiate the agent
model = PPO("MlpPolicy", env)
# Train the agent
model.learn(total_timesteps=int(2e5)) # this will crash explaining that the invalid value originated from the environment.
```
## RL Model hyperparameters
Depending on your hyperparameters, NaN can occur much more often.
A great example of this:
Be aware, the hyperparameters given by default seem to work in most cases,
however your environment might not play nice with them.
If this is the case, try to read up on the effect each hyperparameter has on the model,
so that you can try and tune them to get a stable model. Alternatively, you can try automatic hyperparameter tuning (included in the rl zoo).
## Missing values from datasets
If your environment is generated from an external dataset, do not forget to make sure your dataset does not contain NaNs.
As some datasets will sometimes fill missing values with NaNs as a surrogate value.
Here is some reading material about finding NaNs:
And filling the missing values with something else (imputation):
---
### Guide Custom Env (`docs/guide/custom_env.md`)
(custom-env)=
# Using Custom Environments
To use the RL baselines with custom environments, they just need to follow the *gymnasium* [interface](https://gymnasium.farama.org/tutorials/gymnasium_basics/environment_creation/#sphx-glr-tutorials-gymnasium-basics-environment-creation-py).
That is to say, your environment must implement the following methods (and inherits from Gym Class):
:::{note}
If you are using images as input, the observation must be of type `np.uint8` and be within a space `Box` bounded by [0, 255] (`Box(low=0, high=255, shape=())`).
By default, the observation is normalized by SB3 pre-processing (dividing by 255 to have values in [0, 1], i.e. `Box(low=0, high=1)`) when using CNN policies.
Images can be either channel-first or channel-last.
If you want to use `CnnPolicy` or `MultiInputPolicy` with image-like observation (3D tensor) that are already normalized, you must pass `normalize_images=False`
to the policy (using `policy_kwargs` parameter, `policy_kwargs=dict(normalize_images=False)`)
and make sure your image is in the **channel-first** format.
:::
:::{note}
Although SB3 supports both channel-last and channel-first images as input, we recommend using the channel-first convention when possible.
Under the hood, when a channel-last image is passed, SB3 uses a `VecTransposeImage` wrapper to re-order the channels.
:::
:::{note}
SB3 doesn't support `Discrete` and `MultiDiscrete` spaces with `start!=0`. However, you can update your environment or use a wrapper to make your env compatible with SB3:
```python
import gymnasium as gym
class ShiftWrapper(gym.Wrapper):
"""Allow to use Discrete() action spaces with start!=0"""
def __init__(self, env: gym.Env) -> None:
super().__init__(env)
assert isinstance(env.action_space, gym.spaces.Discrete)
self.action_space = gym.spaces.Discrete(env.action_space.n, start=0)
def step(self, action: int):
return self.env.step(action + self.env.action_space.start)
```
:::
:::{note}
SB3 doesn't support `MultiDiscrete` spaces with multi-dimensional arrays. However, you can update your environment or use a wrapper to make your env compatible with SB3:
```python
import numpy as np
import gymnasium as gym
class ReshapeWrapper(gym.Wrapper):
"""Allow to use MultiDiscrete() action spaces with len(nvec.shape) > 1:"""
def __init__(self, env: gym.Env) -> None:
super().__init__(env)
assert isinstance(env.action_space, gym.spaces.MultiDiscrete)
self.original_shape = env.action_space.nvec.shape
self.action_space = gym.spaces.MultiDiscrete(env.action_space.nvec.flatten())
def step(self, action: np.ndarray):
return self.env.step(action.reshape(self.original_shape))
```
:::
```python
import gymnasium as gym
import numpy as np
from gymnasium import spaces
class CustomEnv(gym.Env):
"""Custom Environment that follows gym interface."""
metadata = {"render_modes": ["human"], "render_fps": 30}
def __init__(self, arg1, arg2, ...):
super().__init__()
# Define action and observation space
# They must be gym.spaces objects
# Example when using discrete actions:
self.action_space = spaces.Discrete(N_DISCRETE_ACTIONS)
# Example for using image as input (channel-first; channel-last also works):
self.observation_space = spaces.Box(low=0, high=255,
shape=(N_CHANNELS, HEIGHT, WIDTH), dtype=np.uint8)
def step(self, action):
...
return observation, reward, terminated, truncated, info
def reset(self, seed=None, options=None):
...
return observation, info
def render(self):
...
def close(self):
...
```
Then you can define and train a RL agent with:
```python
# Instantiate the env
env = CustomEnv(arg1, ...)
# Define and Train the agent
model = A2C("CnnPolicy", env).learn(total_timesteps=1000)
```
To check that your environment follows the Gym interface that SB3 supports, please use:
```python
from stable_baselines3.common.env_checker import check_env
env = CustomEnv(arg1, ...)
# It will check your custom environment and output additional warnings if needed
check_env(env)
```
Gymnasium also has its own [env checker](https://gymnasium.farama.org/api/utils/#gymnasium.utils.env_checker.check_env) but it checks a superset of what SB3 supports (SB3 does not support all Gym features).
We have created a [colab notebook](https://colab.research.google.com/github/araffin/rl-tutorial-jnrr19/blob/sb3/5_custom_gym_env.ipynb) for a concrete example of creating a custom environment along with an example of using it with Stable-Baselines3 interface.
Alternatively, you may look at Gymnasium [built-in environments](https://gymnasium.farama.org).
Optionally, you can also register the environment with gym, that will allow you to create the RL agent in one line (and use `gym.make()` to instantiate the env):
```python
from gymnasium.envs.registration import register
# Example for the CartPole environment
register(
# unique identifier for the env `name-version`
id="CartPole-v1",
# path to the class for creating the env
# Note: entry_point also accept a class as input (and not only a string)
entry_point="gym.envs.classic_control:CartPoleEnv",
# Max number of steps per episode, using a `TimeLimitWrapper`
max_episode_steps=500,
)
```
In the project, for testing purposes, we use a custom environment named `IdentityEnv`
defined [in this file](https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/envs/identity_env.py).
An example of how to use it can be found [here](https://github.com/DLR-RM/stable-baselines3/blob/master/tests/test_identity.py).
---
### Guide Custom Policy (`docs/guide/custom_policy.md`)
(custom-policy)=
# Policy Networks
Stable Baselines3 provides policy networks for images (CnnPolicies),
other type of input features (MlpPolicies) and multiple different inputs (MultiInputPolicies).
:::{warning}
For A2C and PPO, continuous actions are clipped during training and testing
(to avoid out of bound error). SAC, DDPG and TD3 squash the action, using a `tanh()` transformation,
which handles bounds more correctly.
:::
## SB3 Policy
SB3 networks are separated into two main parts (see figure below):
- A features extractor (usually shared between actor and critic when applicable, to save computation)
whose role is to extract features (i.e. convert to a feature vector) from high-dimensional observations, for instance, a CNN that extracts features from images.
This is the `features_extractor_class` parameter. You can change the default parameters of that features extractor
by passing a `features_extractor_kwargs` parameter.
- A (fully-connected) network that maps the features to actions/value. Its architecture is controlled by the `net_arch` parameter.
:::{note}
All observations are first pre-processed (e.g. images are normalized, discrete obs are converted to one-hot vectors, ...) before being fed to the features extractor.
In the case of vector observations, the features extractor is just a `Flatten` layer.
:::
```{image} ../_static/img/net_arch.png
```
SB3 policies are usually composed of several networks (actor/critic networks + target networks when applicable) together
with the associated optimizers.
Each of these network have a features extractor followed by a fully-connected network.
:::{note}
When we refer to "policy" in Stable-Baselines3, this is usually an abuse of language compared to RL terminology.
In SB3, "policy" refers to the class that handles all the networks useful for training,
so not only the network used to predict actions (the "learned controller").
:::
```{image} ../_static/img/sb3_policy.png
```
## Default Network Architecture
The default network architecture used by SB3 depends on the algorithm and the observation space.
You can visualize the architecture by printing `model.policy` (see [issue #329](https://github.com/DLR-RM/stable-baselines3/issues/329)).
For 1D observation space, a 2 layers fully connected net is used with:
- 64 units (per layer) for PPO/A2C/DQN
- 256 units for SAC
- [400, 300] units for TD3/DDPG (values are taken from the original TD3 paper)
For image observation spaces, the "Nature CNN" (see code for more details) is used for feature extraction, and SAC/TD3 also keeps the same fully connected network after it.
The other algorithms only have a linear layer after the CNN.
The CNN is shared between actor and critic for A2C/PPO (on-policy algorithms) to reduce computation.
Off-policy algorithms (TD3, DDPG, SAC, ...) have separate feature extractors: one for the actor and one for the critic, since the best performance is obtained with this configuration.
For mixed observations (dictionary observations), the two architectures from above are used, i.e., CNN for images and then two layers fully-connected network
(with a smaller output size for the CNN).
## Custom Network Architecture
One way of customising the policy network architecture is to pass arguments when creating the model,
using `policy_kwargs` parameter:
:::{note}
An extra linear layer will be added on top of the layers specified in `net_arch`, in order to have the right output dimensions and activation functions (e.g. Softmax for discrete actions).
In the following example, as CartPole's action space has a dimension of 2, the final dimensions of the `net_arch`'s layers will be:
```none
obs
<4>
/ \
<32> <32>
| |
<32> <32>
| |
<2> <1>
action value
```
:::
```python
import gymnasium as gym
import torch as th
from stable_baselines3 import PPO
# Custom actor (pi) and value function (vf) networks
# of two layers of size 32 each with Relu activation function
# Note: an extra linear layer will be added on top of the pi and the vf nets, respectively
policy_kwargs = dict(activation_fn=th.nn.ReLU,
net_arch=dict(pi=[32, 32], vf=[32, 32]))
# Create the agent
model = PPO("MlpPolicy", "CartPole-v1", policy_kwargs=policy_kwargs, verbose=1)
# Retrieve the environment
env = model.get_env()
# Train the agent
model.learn(total_timesteps=20_000)
# Save the agent
model.save("ppo_cartpole")
del model
# the policy_kwargs are automatically loaded
model = PPO.load("ppo_cartpole", env=env)
```
## Custom Feature Extractor
If you want to have a custom features extractor (e.g. custom CNN when using images), you can define class
that derives from `BaseFeaturesExtractor` and then pass it to the model when training.
:::{note}
For on-policy algorithms, the features extractor is shared by default between the actor and the critic to save computation (when applicable).
However, this can be changed setting `share_features_extractor=False` in the
`policy_kwargs` (both for on-policy and off-policy algorithms).
:::
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Multiple Inputs and Dictionary Observations
Stable Baselines3 supports handling of multiple inputs by using `Dict` Gym space. This can be done using
`MultiInputPolicy`, which by default uses the `CombinedExtractor` features extractor to turn multiple
inputs into a single vector, handled by the `net_arch` network.
By default, `CombinedExtractor` processes multiple inputs as follows:
1. If input is an image (automatically detected, see `common.preprocessing.is_image_space`), process image with Nature Atari CNN network and
output a latent vector of size `256`.
2. If input is not an image, flatten it (no layers).
3. Concatenate all previous vectors into one long vector and pass it to policy.
Much like above, you can define custom features extractors. The following example assumes the environment has two keys in the
observation space dictionary: "image" is a (1,H,W) image (channel first), and "vector" is a (D,) dimensional vector. We process "image" with a simple
downsampling and "vector" with a single linear layer.
```
/* Detailed source-code truncated for AI context efficiency. */
```
## On-Policy Algorithms
### Custom Networks
If you need a network architecture that is different for the actor and the critic when using `PPO`, `A2C` or `TRPO`,
you can pass a dictionary of the following structure: `dict(pi=[], vf=[])`.
For example, if you want a different architecture for the actor (aka `pi`) and the critic (value-function aka `vf`) networks,
then you can specify `net_arch=dict(pi=[32, 32], vf=[64, 64])`.
Otherwise, to have actor and critic that share the same network architecture,
you only need to specify `net_arch=[128, 128]` (here, two hidden layers of 128 units each, this is equivalent to `net_arch=dict(pi=[128, 128], vf=[128, 128])`).
If shared layers are needed, you need to implement a custom policy network (see [advanced example below](#advanced-example)).
#### Examples
Same architecture for actor and critic with two layers of size 128: `net_arch=[128, 128]`
```none
obs
/ \
<128> <128>
| |
<128> <128>
| |
action value
```
Different architectures for actor and critic: `net_arch=dict(pi=[32, 32], vf=[64, 64])`
```none
obs
/ \
<32> <64>
| |
<32> <64>
| |
action value
```
#### Advanced Example
If your task requires even more granular control over the policy/value architecture, you can redefine the policy directly:
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Off-Policy Algorithms
If you need a network architecture that is different for the actor and the critic when using `SAC`, `DDPG`, `TQC` or `TD3`,
you can pass a dictionary of the following structure: `dict(pi=[], qf=[])`.
For example, if you want a different architecture for the actor (aka `pi`) and the critic (Q-function aka `qf`) networks,
then you can specify `net_arch=dict(pi=[64, 64], qf=[400, 300])`.
Otherwise, to have actor and critic that share the same network architecture,
you only need to specify `net_arch=[256, 256]` (here, two hidden layers of 256 units each).
:::{note}
For advanced customization of off-policy algorithms policies, please take a look at the code.
A good understanding of the algorithm used is required, see discussion in [issue #425](https://github.com/DLR-RM/stable-baselines3/issues/425)
:::
```python
from stable_baselines3 import SAC
# Custom actor architecture with two layers of 64 units each
# Custom critic architecture with two layers of 400 and 300 units
policy_kwargs = dict(net_arch=dict(pi=[64, 64], qf=[400, 300]))
# Create the agent
model = SAC("MlpPolicy", "Pendulum-v1", policy_kwargs=policy_kwargs, verbose=1)
model.learn(5000)
```
---
### Guide Developer (`docs/guide/developer.md`)
(developer)=
# Developer Guide
This guide is meant for those who want to understand the internals and the design choices of Stable-Baselines3.
At first, you should read the two issues where the design choices were discussed:
-
-
The library is not meant to be modular, although inheritance is used to reduce code duplication.
## Algorithms Structure
Each algorithm (on-policy and off-policy ones) follows a common structure.
Policy contains code for acting in the environment, and algorithm updates this policy.
There is one folder per algorithm, and in that folder there is the algorithm and the policy definition (`policies.py`).
Each algorithm has two main methods:
- `.collect_rollouts()` which defines how new samples are collected, usually inherited from the base class. Those samples are then stored in a `RolloutBuffer` (discarded after the gradient update) or `ReplayBuffer`
- `.train()` which updates the parameters using samples from the buffer
```{image} ../_static/img/sb3_loop.png
```
## Where to start?
The first thing you need to read and understand are the base classes in the `common/` folder:
- `BaseAlgorithm` in `base_class.py` which defines how an RL class should look like.
It contains also all the "glue code" for saving/loading and the common operations (wrapping environments)
- `BasePolicy` in `policies.py` which defines how a policy class should look like.
It contains also all the magic for the `.predict()` method, to handle as many spaces/cases as possible
- `OffPolicyAlgorithm` in `off_policy_algorithm.py` that contains the implementation of `collect_rollouts()` for the off-policy algorithms,
and similarly `OnPolicyAlgorithm` in `on_policy_algorithm.py`.
All the environments handled internally are assumed to be `VecEnv` (`gym.Env` are automatically wrapped).
## Pre-Processing
To handle different observation spaces, some pre-processing needs to be done (e.g. one-hot encoding for discrete observation).
Most of the code for pre-processing is in `common/preprocessing.py` and `common/policies.py`.
For images, environment is automatically wrapped with `VecTransposeImage` if observations are detected to be images with
channel-last convention to transform it to PyTorch's channel-first convention.
## Policy Structure
When we refer to "policy" in Stable-Baselines3, this is usually an abuse of language compared to RL terminology.
In SB3, "policy" refers to the class that handles all the networks useful for training,
so not only the network used to predict actions (the "learned controller").
For instance, the `TD3` policy contains the actor, the critic and the target networks.
To avoid the hassle of importing specific policy classes for specific algorithm (e.g. both A2C and PPO use `ActorCriticPolicy`),
SB3 uses names like "MlpPolicy" and "CnnPolicy" to refer policies using small feed-forward networks or convolutional networks,
respectively. Importing `[algorithm]/policies.py` registers an appropriate policy for that algorithm under those names.
## Probability distributions
When needed, the policies handle the different probability distributions.
All distributions are located in `common/distributions.py` and follow the same interface.
Each distribution corresponds to a type of action space (e.g. `Categorical` is the one used for discrete actions.
For continuous actions, we can use multiple distributions ("DiagGaussian", "SquashedGaussian" or "StateDependentDistribution")
## State-Dependent Exploration
State-Dependent Exploration (SDE) is a type of exploration that allows to use RL directly on real robots,
that was the starting point for the Stable-Baselines3 library.
I (@araffin) published a paper about a generalized version of SDE (the one implemented in SB3):
## Misc
The rest of the `common/` is composed of helpers (e.g. evaluation helpers) or basic components (like the callbacks).
The `type_aliases.py` file contains common type hint aliases like `GymStepReturn`.
Et voilà?
After reading this guide and the mentioned files, you should be now able to understand the design logic behind the library ;)
---
### Guide Examples (`docs/guide/examples.md`)
---
myst:
substitutions:
colab: |-
```{image} ../_static/img/colab.svg
```
---
(examples)=
# Examples
:::{note}
These examples are only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in the RL Zoo [repository](https://github.com/DLR-RM/rl-baselines3-zoo).
:::
## Try it online with Colab Notebooks!
All the following examples can be executed online using Google Colab
{{ colab }} notebooks:
- [Full Tutorial](https://github.com/araffin/rl-tutorial-jnrr19/tree/sb3)
- [All Notebooks](https://github.com/Stable-Baselines-Team/rl-colab-notebooks/tree/sb3)
- [Getting Started](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/stable_baselines_getting_started.ipynb)
- [Training, Saving, Loading](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/saving_loading_dqn.ipynb)
- [Multiprocessing]
- [Monitor Training and Plotting]
- [Atari Games]
- [RL Baselines zoo]
- [PyBullet]
- [Hindsight Experience Replay]
- [Advanced Saving and Loading]
## Basic Usage: Training, Saving, Loading
In the following example, we will train, save and load a DQN model on the Lunar Lander environment.
```{image} ../_static/img/colab-badge.svg
:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/saving_loading_dqn.ipynb
```
:::{figure} https://cdn-images-1.medium.com/max/960/1*f4VZPKOI0PYNWiwt0la0Rg.gif
Lunar Lander Environment
:::
:::{note}
LunarLander requires the Python package `box2d`.
You can install it using `apt install swig` and then `pip install box2d box2d-kengz`
:::
:::{warning}
The `load` method recreates the model from scratch and should be called on the algorithm without instantiating it first,
e.g. `model = DQN.load("dqn_lunar", env=env)` instead of `model = DQN(env=env)` followed by `model.load("dqn_lunar")`. The latter **will not work** as `load` is not an in-place operation.
If you want to load parameters without recreating the model, e.g. to evaluate the same model
with multiple different sets of parameters, consider using `set_parameters` instead.
:::
```python
import gymnasium as gym
from stable_baselines3 import DQN
from stable_baselines3.common.evaluation import evaluate_policy
# Create environment
env = gym.make("LunarLander-v3", render_mode="rgb_array")
# Instantiate the agent
# Note: you can enable multi-step replay buffer by passing `n_steps=3` for instance
model = DQN("MlpPolicy", env, verbose=1)
# Train the agent and display a progress bar
model.learn(total_timesteps=int(2e5), progress_bar=True)
# Save the agent
model.save("dqn_lunar")
del model # delete trained model to demonstrate loading
# Load the trained agent
# NOTE: if you have loading issue, you can pass `print_system_info=True`
# to compare the system on which the model was trained vs the current one
# model = DQN.load("dqn_lunar", env=env, print_system_info=True)
model = DQN.load("dqn_lunar", env=env)
# Evaluate the agent
# NOTE: If you use wrappers with your environment that modify rewards,
# this will be reflected here. To evaluate with original rewards,
# wrap the environment in a "Monitor" wrapper before other wrappers.
mean_reward, std_reward = evaluate_policy(model, model.get_env(), n_eval_episodes=10)
# Enjoy trained agent
vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
action, _states = model.predict(obs, deterministic=True)
obs, rewards, dones, info = vec_env.step(action)
vec_env.render("human")
```
## Multiprocessing: Unleashing the Power of Vectorized Environments
```{image} ../_static/img/colab-badge.svg
:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb
```
:::{figure} https://cdn-images-1.medium.com/max/960/1*h4WTQNVIsvMXJTCpXm_TAw.gif
CartPole Environment
:::
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Multiprocessing with off-policy algorithms
:::{warning}
When using multiple environments with off-policy algorithms, you should update the `gradient_steps`
parameter too. Set it to `gradient_steps=-1` to perform as many gradient steps as transitions collected.
There is usually a trade-off between wall-clock time and sample efficiency,
see this [example in PR #439](https://github.com/DLR-RM/stable-baselines3/pull/439#issuecomment-961796799)
:::
```python
import gymnasium as gym
from stable_baselines3 import SAC
from stable_baselines3.common.env_util import make_vec_env
vec_env = make_vec_env("Pendulum-v0", n_envs=4, seed=0)
# We collect 4 transitions per call to `env.step()`
# and performs 2 gradient steps per call to `env.step()`
# if gradient_steps=-1, then we would do 4 gradients steps per call to `env.step()`
model = SAC("MlpPolicy", vec_env, train_freq=1, gradient_steps=2, verbose=1)
model.learn(total_timesteps=10_000)
```
## Dict Observations
You can use environments with dictionary observation spaces. This is useful in the case where one can't directly
concatenate observations such as an image from a camera combined with a vector of servo sensor data (e.g., rotation angles).
Stable Baselines3 provides `SimpleMultiObsEnv` as an example of this kind of setting.
The environment is a simple grid world, but the observations for each cell come in the form of dictionaries.
These dictionaries are randomly initialized on the creation of the environment and contain a vector observation and an image observation.
```python
from stable_baselines3 import PPO
from stable_baselines3.common.envs import SimpleMultiObsEnv
# Stable Baselines provides SimpleMultiObsEnv as an example environment with Dict observations
env = SimpleMultiObsEnv(random_start=False)
model = PPO("MultiInputPolicy", env, verbose=1)
model.learn(total_timesteps=100_000)
```
## Callbacks: Monitoring Training
:::{note}
We recommend reading the [Callback section](callbacks.md)
:::
You can define a custom callback function that will be called inside the agent.
This could be useful when you want to monitor training, for instance, display live
learning curves in TensorBoard or save the best agent.
If your callback returns False, training is aborted early.
```{image} ../_static/img/colab-badge.svg
:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/monitor_training.ipynb
```
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Callbacks: Evaluate Agent Performance
To periodically evaluate an agent's performance on a separate test environment, use `EvalCallback`.
You can control the evaluation frequency with `eval_freq` to monitor your agent's progress during training.
```python
import os
import gymnasium as gym
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback
from stable_baselines3.common.env_util import make_vec_env
env_id = "Pendulum-v1"
n_training_envs = 1
n_eval_envs = 5
# Create log dir where evaluation results will be saved
eval_log_dir = "./eval_logs/"
os.makedirs(eval_log_dir, exist_ok=True)
# Initialize a vectorized training environment with default parameters
train_env = make_vec_env(env_id, n_envs=n_training_envs, seed=0)
# Separate evaluation env, with different parameters passed via env_kwargs
# Eval environments can be vectorized to speed up evaluation.
eval_env = make_vec_env(env_id, n_envs=n_eval_envs, seed=0,
env_kwargs={'g':0.7})
# Create a callback that evaluates an agent for 5 episodes every 500 training environment steps.
# When using multiple training environments, agent will be evaluated every
# eval_freq calls to train_env.step(), thus it will be evaluated every
# (eval_freq * n_envs) training steps. See EvalCallback doc for more information.
eval_callback = EvalCallback(eval_env, best_model_save_path=eval_log_dir,
log_path=eval_log_dir, eval_freq=max(500 // n_training_envs, 1),
n_eval_episodes=5, deterministic=True,
render=False)
model = SAC("MlpPolicy", train_env)
model.learn(5000, callback=eval_callback)
```
## Atari Games
:::{figure} ../_static/img/breakout.gif
Trained A2C agent on Breakout
:::
:::{figure} https://cdn-images-1.medium.com/max/960/1*UHYJE7lF8IDZS_U5SsAFUQ.gif
Pong Environment
:::
Training an RL agent on Atari games is straightforward thanks to the `make_atari_env` helper function.
It will do [all the preprocessing](https://danieltakeshi.github.io/2016/11/25/frame-skipping-and-preprocessing-for-deep-q-networks-on-atari-2600-games/)
and multiprocessing for you. To install the Atari environments, run the command `pip install gymnasium[atari,accept-rom-license]` to install the Atari environments and ROMs, or install Stable Baselines3 with `pip install stable-baselines3[extra]` to install this and other optional dependencies.
```{image} ../_static/img/colab-badge.svg
:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/atari_games.ipynb
```
:::{note}
When working with Atari environments, be aware that the default `terminal_on_life_loss=True` behavior
can cause `env.reset()` to perform a no-op step instead of truly resetting the environment when
the episode ends due to a life loss (not game over, see [issue #666](https://github.com/DLR-RM/stable-baselines3/issues/666)).
To ensure `reset()` always resets the environment, use:
```python
from stable_baselines3.common.env_util import make_atari_env
import ale_py
env = make_atari_env(
"BreakoutNoFrameskip-v4",
n_envs=1,
wrapper_kwargs=dict(terminal_on_life_loss=False)
)
```
:::
```python
from stable_baselines3.common.env_util import make_atari_env
from stable_baselines3.common.vec_env import VecFrameStack
from stable_baselines3 import A2C
import ale_py
# There already exists an environment generator
# that will make and wrap atari environments correctly.
# Here we are also multi-worker training (n_envs=4 => 4 environments)
vec_env = make_atari_env("PongNoFrameskip-v4", n_envs=4, seed=0)
# Frame-stacking with 4 frames
vec_env = VecFrameStack(vec_env, n_stack=4)
model = A2C("CnnPolicy", vec_env, verbose=1)
model.learn(total_timesteps=25_000)
obs = vec_env.reset()
while True:
action, _states = model.predict(obs, deterministic=False)
obs, rewards, dones, info = vec_env.step(action)
vec_env.render("human")
```
## PyBullet: Normalizing input features
Normalizing input features may be essential for successful training of an RL agent
(by default, images are scaled, but other types of input are not),
for instance, when training on [PyBullet](https://github.com/bulletphysics/bullet3/) environments.
For this, there is a wrapper `VecNormalize` that will compute a running average and standard deviation of the input features (it can do the same for rewards).
:::{note}
You need to install PyBullet environments with `pip install pybullet_envs_gymnasium`
:::
```{image} ../_static/img/colab-badge.svg
:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/pybullet.ipynb
```
```python
from pathlib import Path
import pybullet_envs_gymnasium
from stable_baselines3.common.vec_env import VecNormalize
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3 import PPO
# Alternatively, you can use the MuJoCo equivalent "HalfCheetah-v4"
vec_env = make_vec_env("HalfCheetahBulletEnv-v0", n_envs=1)
# Automatically normalize the input features and reward
vec_env = VecNormalize(vec_env, norm_obs=True, norm_reward=True, clip_obs=10.0)
model = PPO("MlpPolicy", vec_env)
model.learn(total_timesteps=2000)
# Don't forget to save the VecNormalize statistics when saving the agent
log_dir = Path("/tmp/")
model.save(log_dir / "ppo_halfcheetah")
stats_path = log_dir / "vec_normalize.pkl"
vec_env.save(stats_path)
# To demonstrate loading
del model, vec_env
# Load the saved statistics
vec_env = make_vec_env("HalfCheetahBulletEnv-v0", n_envs=1)
vec_env = VecNormalize.load(stats_path, vec_env)
# do not update them at test time
vec_env.training = False
# reward normalization is not needed at test time
vec_env.norm_reward = False
# Load the agent
model = PPO.load(log_dir / "ppo_halfcheetah", env=vec_env)
```
## Hindsight Experience Replay (HER)
For this example, we use [Highway-Env](https://github.com/Farama-Foundation/HighwayEnv) by [@eleurent](https://github.com/eleurent).
```{image} ../_static/img/colab-badge.svg
:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/stable_baselines_her.ipynb
```
:::{figure} https://raw.githubusercontent.com/eleurent/highway-env/gh-media/docs/media/parking-env.gif
The highway-parking-v0 environment.
:::
The parking env is a goal-conditioned continuous control task, in which the vehicle must park in a given space with the appropriate heading.
:::{note}
The hyperparameters in the following example were optimized for that environment.
:::
```python
import gymnasium as gym
import highway_env
import numpy as np
from stable_baselines3 import HerReplayBuffer, SAC, DDPG, TD3
from stable_baselines3.common.noise import NormalActionNoise
env = gym.make("parking-v0")
# Create 4 artificial transitions per real transition
n_sampled_goal = 4
# SAC hyperparams:
model = SAC(
"MultiInputPolicy",
env,
replay_buffer_class=HerReplayBuffer,
replay_buffer_kwargs=dict(
n_sampled_goal=n_sampled_goal,
goal_selection_strategy="future",
),
verbose=1,
buffer_size=int(1e6),
learning_rate=1e-3,
gamma=0.95,
batch_size=256,
policy_kwargs=dict(net_arch=[256, 256, 256]),
)
model.learn(int(2e5))
model.save("her_sac_highway")
# Load saved model
# Because it needs access to `env.compute_reward()`
# HER must be loaded with the env
env = gym.make("parking-v0", render_mode="human") # Change the render mode
model = SAC.load("her_sac_highway", env=env)
obs, info = env.reset()
# Evaluate the agent
episode_reward = 0
for _ in range(100):
action, _ = model.predict(obs, deterministic=True)
obs, reward, terminated, truncated, info = env.step(action)
episode_reward += reward
if terminated or truncated or info.get("is_success", False):
print("Reward:", episode_reward, "Success?", info.get("is_success", False))
episode_reward = 0.0
obs, info = env.reset()
```
## Learning Rate Schedule
All algorithms allow you to pass a learning rate schedule that takes as input the current progress remaining (from 1 to 0).
`PPO`'s `` clip_range` `` parameter also accepts such schedule.
The [RL Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) already includes
linear and constant schedules.
```python
from typing import Callable
from stable_baselines3 import PPO
def linear_schedule(initial_value: float) -> Callable[[float], float]:
"""
Linear learning rate schedule.
:param initial_value: Initial learning rate.
:return: schedule that computes
current learning rate depending on remaining progress
"""
def func(progress_remaining: float) -> float:
"""
Progress will decrease from 1 (beginning) to 0.
:param progress_remaining:
:return: current learning rate
"""
return progress_remaining * initial_value
return func
# Initial learning rate of 0.001
model = PPO("MlpPolicy", "CartPole-v1", learning_rate=linear_schedule(0.001), verbose=1)
model.learn(total_timesteps=20_000)
# By default, `reset_num_timesteps` is True, in which case the learning rate schedule resets.
# progress_remaining = 1.0 - (num_timesteps / total_timesteps)
model.learn(total_timesteps=10_000, reset_num_timesteps=True)
```
## Advanced Saving and Loading
In this example, we show how to use a policy independently from a model (and how to save it, load it) and save/load a replay buffer.
By default, the replay buffer is not saved when calling `model.save()`, in order to save space on the disk (a replay buffer can be up to several GB when using images).
However, SB3 provides a `save_replay_buffer()` and `load_replay_buffer()` method to save it separately.
:::{note}
For training model after loading it, we recommend loading the replay buffer to ensure stable learning (for off-policy algorithms).
You also need to pass `reset_num_timesteps=True` to `learn` function which initializes the environment
and agent for training if a new environment was created since saving the model.
:::
```{image} ../_static/img/colab-badge.svg
:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/advanced_saving_loading.ipynb
```
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Accessing and modifying model parameters
You can access model's parameters via `set_parameters` and `get_parameters` functions,
or via `model.policy.state_dict()` (and `load_state_dict()`),
which use dictionaries that map variable names to PyTorch tensors.
These functions are useful when you need to e.g. evaluate large set of models with same network structure,
visualize different layers of the network or modify parameters manually.
Policies also offers a simple way to save/load weights as a NumPy vector, using `parameters_to_vector()`
and `load_from_vector()` method.
Following example demonstrates reading parameters, modifying some of them and loading them to model
by implementing [evolution strategy (es)](https://blog.otoro.net/2017/10/29/visual-evolution-strategies/)
for solving the `CartPole-v1` environment. The initial guess for parameters is obtained by running
A2C policy gradient updates on the model.
```
/* Detailed source-code truncated for AI context efficiency. */
```
## SB3 with Isaac Lab, Brax, Procgen, EnvPool
Some massively parallel simulation environments such as [EnvPool](https://github.com/sail-sg/envpool), [Isaac Lab](https://github.com/isaac-sim/IsaacLab), [Brax](https://github.com/google/brax) or [ProcGen](https://github.com/Farama-Foundation/Procgen2) already produce a vectorized environment to speed up data collection (see discussion in [issue #314](https://github.com/DLR-RM/stable-baselines3/issues/314)).
To use SB3 with these tools, you need to wrap the environment with tool-specific `VecEnvWrapper` that preprocesses the data for SB3,
you can find links to some of these wrappers in [issue #772](https://github.com/DLR-RM/stable-baselines3/issues/772#issuecomment-1048657002).
- Isaac Lab wrapper: [link](https://github.com/isaac-sim/IsaacLab/blob/3e73d6dd79080fd7632488c061052a6edd52e230/source/isaaclab_rl/isaaclab_rl/sb3.py#L93)
- Brax: [link](https://gist.github.com/araffin/a7a576ec1453e74d9bb93120918ef7e7)
- EnvPool: [link](https://github.com/sail-sg/envpool/blob/main/examples/sb3_examples/ppo.py)
- Getting SAC to Work on a Massive Parallel Simulator:
## SB3 with DeepMind Control (dm_control)
If you want to use SB3 with [dm_control](https://github.com/google-deepmind/dm_control), you need to apply two wrappers (one from [shimmy](https://github.com/Farama-Foundation/Shimmy), one pre-built one) to convert it to a Gymnasium compatible environment:
```python
import shimmy
import stable_baselines3 as sb3
from dm_control import suite
from gymnasium.wrappers import FlattenObservation
# Available envs:
# suite._DOMAINS and suite.dog.SUITE
env = suite.load(domain_name="dog", task_name="run")
gym_env = FlattenObservation(shimmy.DmControlCompatibilityV0(env))
model = sb3.PPO("MlpPolicy", gym_env, verbose=1)
model.learn(10_000, progress_bar=True)
```
## Record a Video
Record an MP4 video (here using a random agent).
:::{note}
It requires `ffmpeg` or `avconv` to be installed on the machine.
:::
```python
import gymnasium as gym
from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv
env_id = "CartPole-v1"
video_folder = "logs/videos/"
video_length = 100
vec_env = DummyVecEnv([lambda: gym.make(env_id, render_mode="rgb_array")])
obs = vec_env.reset()
# Record the video starting at the first step
vec_env = VecVideoRecorder(vec_env, video_folder,
record_video_trigger=lambda x: x == 0, video_length=video_length,
name_prefix=f"random-agent-{env_id}")
vec_env.reset()
for _ in range(video_length + 1):
action = [vec_env.action_space.sample()]
obs, _, _, _ = vec_env.step(action)
# Save the video
vec_env.close()
```
## Compiling models
To make models faster, you can compile them using `torch.compile()`.
```python
import torch as th
from stable_baselines3 import PPO
env_id = "CartPole-v1"
model = PPO("MlpPolicy", env_id, verbose=1)
model.policy = th.compile(model.policy, backend="inductor")
model.learn(total_timesteps=10_000)
```
:::{note}
To list available backends, you can use `torch._dynamo.list_backends()`.
:::
## Bonus: Make a GIF of a Trained Agent
```python
import imageio
import numpy as np
from stable_baselines3 import A2C
model = A2C("MlpPolicy", "LunarLander-v3").learn(100_000)
images = []
obs = model.env.reset()
img = model.env.render(mode="rgb_array")
for i in range(350):
images.append(img)
action, _ = model.predict(obs)
obs, _, _, _ = model.env.step(action)
img = model.env.render(mode="rgb_array")
imageio.mimsave("lander_a2c.gif", [np.array(img) for i, img in enumerate(images) if i%2 == 0], fps=29)
```
[advanced saving and loading]: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/advanced_saving_loading.ipynb
[atari games]: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/atari_games.ipynb
[hindsight experience replay]: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/stable_baselines_her.ipynb
[monitor training and plotting]: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/monitor_training.ipynb
[multiprocessing]: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb
[pybullet]: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/pybullet.ipynb
[rl baselines zoo]: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/rl-baselines-zoo.ipynb
---
### Guide Export (`docs/guide/export.md`)
(export)=
# Exporting models
After training an agent, you may want to deploy/use it in another language
or framework, like [tensorflowjs](https://github.com/tensorflow/tfjs).
Stable Baselines3 does not include tools to export models to other frameworks, but
this document aims to cover parts that are required for exporting along with
more detailed stories from users of Stable Baselines3.
## Background
In Stable Baselines3, the controller is stored inside policies which convert
observations into actions. Each learning algorithm (e.g. DQN, A2C, SAC)
contains a policy object which represents the currently learned behavior,
accessible via `model.policy`.
Policies hold enough information to do the inference (i.e. predict actions),
so it is enough to export these policies (cf {ref}`examples `)
to do inference in another framework.
:::{warning}
When using CNN policies, the observation is normalized during preprocessing.
This pre-processing is done *inside* the policy (dividing by 255 to have values in [0, 1])
:::
## Export to ONNX
If you are using PyTorch 2.0+ and ONNX Opset 14+, you can easily export SB3 policies using the following code:
:::{warning}
The following returns normalized actions and doesn't include the [post-processing](https://github.com/DLR-RM/stable-baselines3/blob/a9273f968eaf8c6e04302a07d803eebfca6e7e86/stable_baselines3/common/policies.py#L370-L377) step that is done with continuous actions (clip or unscale the action to the correct space).
:::
```
/* Detailed source-code truncated for AI context efficiency. */
```
For exporting `MultiInputPolicy`, please have a look at [GH#1873](https://github.com/DLR-RM/stable-baselines3/issues/1873#issuecomment-2710776085).
For SAC the procedure is similar. The example shown only exports the actor network as the actor is sufficient to roll out the trained policies.
```
/* Detailed source-code truncated for AI context efficiency. */
```
For more discussion around the topic, please refer to [GH#383](https://github.com/DLR-RM/stable-baselines3/issues/383) and [GH#1349](https://github.com/DLR-RM/stable-baselines3/issues/1349).
## Trace/Export to C++
You can use PyTorch JIT to trace and save a trained model that can be reused in other applications
(for instance inference code written in C++).
There is a draft PR in the RL Zoo about C++ export:
```python
# See "ONNX export" for imports and OnnxablePolicy
jit_path = "sac_traced.pt"
# Trace and optimize the module
traced_module = th.jit.trace(onnxable_model.eval(), dummy_input)
frozen_module = th.jit.freeze(traced_module)
frozen_module = th.jit.optimize_for_inference(frozen_module)
th.jit.save(frozen_module, jit_path)
##### Load and test with torch
import torch as th
dummy_input = th.randn(1, *observation_size)
loaded_module = th.jit.load(jit_path)
action_jit = loaded_module(dummy_input)
```
## Export to ONNX-JS / ONNX Runtime Web
Official documentation:
Full example code:
Demo:
The code linked above is a complete example (using car dodging environment) that:
1. Creates/Trains a PPO model
2. Exports the model to ONNX along with normalization stats in JSON
3. Runs in the browser with normalization using onnxruntime-web to achieve similar results
Below is a simple example with converting to ONNX then inferencing without postprocess in ONNX-JS
```python
import torch as th
from stable_baselines3 import SAC
class OnnxablePolicy(th.nn.Module):
def __init__(self, actor: th.nn.Module):
super().__init__()
self.actor = actor
def forward(self, observation: th.Tensor) -> th.Tensor:
# NOTE: You may have to postprocess (unnormalize or renormalize)
return self.actor(observation, deterministic=True)
# Example: model = SAC("MlpPolicy", "Pendulum-v1")
SAC("MlpPolicy", "Pendulum-v1").save("PathToTrainedModel.zip")
model = SAC.load("PathToTrainedModel.zip", device="cpu")
onnxable_model = OnnxablePolicy(model.policy.actor)
observation_size = model.observation_space.shape
dummy_input = th.randn(1, *observation_size)
th.onnx.export(
onnxable_model,
dummy_input,
"my_sac_actor.onnx",
opset_version=17,
input_names=["input"],
)
```
```javascript
// Install using `npm install onnxruntime-web` (tested with version 1.19) or using cdn
import * as ort from 'onnxruntime-web';
async function runInference() {
const session = await ort.InferenceSession.create('my_sac_actor.onnx');
// The observation_size = 3 (for Pendulum-v1)
const inputData = Float32Array.from([0.1, -0.2, 0.3]);
const inputTensor = new ort.Tensor('float32', inputData, [1, 3]);
const results = await session.run({ input: inputTensor });
const outputName = session.outputNames[0];
const action = results[outputName].data;
console.log('Predicted action=', action);
}
runInference();
```
## Export to TensorFlow.js
:::{warning}
As of November 2025, [onnx2tf](https://github.com/PINTO0309/onnx2tf) does not support TensorFlow.js. Therefore, [tfjs-converter](https://github.com/tensorflow/tfjs-converter) is used instead. However, tfjs-converter is not currently maintained and requires older opsets and TensorFlow versions.
:::
In order for this to work, you have to do multiple conversions: SB3 => ONNX => TensorFlow => TensorFlow.js.
The opset version needs to be changed for the conversion (`opset_version=14` is currently required). Please refer to the code above for more stable usage with a higher opset.
The following is a simple example that showcases the full conversion + inference.
Please refer to the previous sections for the first step (SB3 => ONNX).
The main difference is that you need to specify `opset_version=14`.
```python
# Tested with python3.10
# Then install these dependencies in a fresh env
"""
pip install --use-deprecated=legacy-resolver tensorflow==2.13.0 keras==2.13.1 onnx==1.16.0 onnx-tf==1.9.0 tensorflow-probability==0.21.0 tensorflowjs==4.15.0 jax==0.4.26 jaxlib==0.4.26
"""
# Then run this codeblock
# If there are no errors (the folder is structure correctly) then
"""
# tensorflowjs_converter --input_format=tf_saved_model --output_format=tfjs_graph_model tf_model tfjs_model
"""
# If you get an error exporting using `tensorflowjs_converter` then upgrade tensorflow
"""
pip install --upgrade tensorflow tensorflow-decision-forests tensorflowjs
"""
# And retry with and it should work (do not rerun this codeblock)
"""
tensorflowjs_converter --input_format=tf_saved_model --output_format=tfjs_graph_model tf_model tfjs_model
"""
import onnx
import onnx_tf.backend
import tensorflow as tf
ONNX_FILE_PATH = "my_sac_actor.onnx"
MODEL_PATH = "tf_model"
onnx_model = onnx.load(ONNX_FILE_PATH)
onnx.checker.check_model(onnx_model)
print(onnx.helper.printable_graph(onnx_model.graph))
print('Converting ONNX to TF...')
tf_rep = onnx_tf.backend.prepare(onnx_model)
tf_rep.export_graph(MODEL_PATH)
# After this do not forget to use `tensorflowjs_converter`
```
```javascript
import * as tf from 'https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.15.0/+esm';
// Post processing not included
async function runInference() {
const MODEL_URL = './tfjs_model/model.json';
const model = await tf.loadGraphModel(MODEL_URL);
// Observation_size is 3 for Pendulum-v1
const inputData = [1.0, 0.0, 0.0];
const inputTensor = tf.tensor2d([inputData], [1, 3]);
const resultTensor = model.execute(inputTensor);
const action = await resultTensor.data();
console.log('Predicted action=', action);
inputTensor.dispose();
resultTensor.dispose();
}
runInference();
```
## Export to TFLite / Coral (Edge TPU)
Full example code:
Google created a chip called the "Coral" for deploying AI to the
edge. It's available in a variety of form factors, including USB (using
the Coral on a Raspberry Pi, with a SB3-developed model, was the original
motivation for the code example above).
The Coral chip is fast, with very low power consumption, but only has limited
on-device training abilities. More information is on the webpage here:
.
To deploy to a Coral, one must work via TFLite, and quantize the
network to reflect the Coral's capabilities. The full chain to go from
SB3 to Coral is: SB3 (Torch) => ONNX => TensorFlow => TFLite => Coral.
The code linked above is a complete, minimal, example that:
1. Creates a model using SB3
2. Follows the path of exports all the way to TFLite and Google Coral
3. Demonstrates the forward pass for most exported variants
There are a number of pitfalls along the way to the complete conversion
that this example covers, including:
- Making the Gym's observation work with ONNX properly
- Quantising the TFLite model appropriately to align with Gym
while still taking advantage of Coral
- Using OnnxablePolicy as described in the above example
## Manual export
You can also manually export required parameters (weights) and construct the
network in your desired framework.
You can access parameters of the model via agents'
{func}`get_parameters ` function.
As policies are also PyTorch modules, you can also access `model.policy.state_dict()` directly.
To find the architecture of the networks for each algorithm, best is to check the `policies.py` file located
in their respective folders.
:::{note}
In most cases, we recommend using PyTorch methods `state_dict()` and `load_state_dict()` from the policy,
unless you need to access the optimizers' state dict too. In that case, you need to call `get_parameters()`.
:::
## SBX (SB3 + Jax) Export
As an example of manual export, {ref}`Stable Baselines Jax (SBX) ` policies can be exported to ONNX
by using an intermediate PyTorch representation, as shown in the following example:
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
### Guide Imitation (`docs/guide/imitation.md`)
(imitation)=
# Imitation Learning
The [imitation](https://github.com/HumanCompatibleAI/imitation) library implements
imitation learning algorithms on top of Stable-Baselines3, including:
- Behavioral Cloning
- [DAgger](https://arxiv.org/abs/1011.0686) with synthetic examples
- [Adversarial Inverse Reinforcement Learning](https://arxiv.org/abs/1710.11248) (AIRL)
- [Generative Adversarial Imitation Learning](https://arxiv.org/abs/1606.03476) (GAIL)
- [Deep RL from Human Preferences](https://arxiv.org/abs/1706.03741) (DRLHP)
You can install imitation with `pip install imitation`. The [imitation
documentation](https://imitation.readthedocs.io/en/latest/) has more details
on how to use the library, including [a quick start guide](https://imitation.readthedocs.io/en/latest/getting-started/first_steps.html)
for the impatient.
---
### Guide Install (`docs/guide/install.md`)
(install)=
# Installation
## Prerequisites
Stable-Baselines3 requires python 3.10+ and PyTorch >= 2.8
### Windows
We recommend using [miniforge](https://github.com/conda-forge/miniforge#windows) for Windows users for easier installation of Python packages and required libraries. You need an environment with Python version 3.10 or above.
For a quick start you can move straight to installing Stable-Baselines3 in the next step.
:::{note}
Trying to create Atari environments may result in vague errors related to missing DLL files and modules. This is an
issue with atari-py package. [See this discussion for more information](https://github.com/openai/atari-py/issues/65).
:::
### Stable Release
To install Stable Baselines3 with pip, execute:
```bash
pip install 'stable-baselines3[extra]'
```
:::{note}
Some shells such as Zsh require quotation marks around brackets, i.e. `pip install 'stable-baselines3[extra]'` [More information](https://stackoverflow.com/a/30539963).
:::
This includes optional dependencies like Tensorboard, OpenCV or `ale-py` to train on Atari games. If you do not need those, you can use:
```bash
pip install stable-baselines3
```
:::{note}
If you need to work with OpenCV on a machine without a X-server (for instance inside a docker image),
you will need to install `opencv-python-headless`, see [issue #298](https://github.com/DLR-RM/stable-baselines3/issues/298).
:::
## Bleeding-edge version
```bash
pip install git+https://github.com/DLR-RM/stable-baselines3
```
with extras:
```bash
pip install "stable_baselines3[extra,tests,docs] @ git+https://github.com/DLR-RM/stable-baselines3"
```
## Development version
To contribute to Stable-Baselines3, with support for running tests and building the documentation.
```bash
git clone https://github.com/DLR-RM/stable-baselines3 && cd stable-baselines3
pip install -e '.[docs,tests,extra]'
```
## Using Docker Images
If you are looking for docker images with stable-baselines already installed in it,
we recommend using images from [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo).
Otherwise, the following images contained all the dependencies for stable-baselines3 but not the stable-baselines3 package itself.
They are made for development.
### Use Built Images
GPU image (requires [nvidia-docker]):
```bash
docker pull stablebaselines/stable-baselines3
```
CPU only:
```bash
docker pull stablebaselines/stable-baselines3-cpu
```
### Build the Docker Images
Build GPU image (with nvidia-docker):
```bash
make docker-gpu
```
Build CPU image:
```bash
make docker-cpu
```
Note: if you are using a proxy, you need to pass extra params during
build and do some [tweaks]:
```bash
--network=host --build-arg HTTPS_PROXY=https://your.proxy.fr:8080/ --build-arg https_proxy=https://your.proxy.fr:8080/
```
### Run the images (CPU/GPU)
Run the nvidia-docker GPU image
```bash
docker run -it --runtime=nvidia --rm --network host --ipc=host --name test --mount src="$(pwd)",target=/home/mamba/stable-baselines3,type=bind stablebaselines/stable-baselines3 bash -c 'cd /home/mamba/stable-baselines3/ && pytest tests/'
```
Or, with the shell file:
```bash
./scripts/run_docker_gpu.sh pytest tests/
```
Run the docker CPU image
```bash
docker run -it --rm --network host --ipc=host --name test --mount src="$(pwd)",target=/home/mamba/stable-baselines3,type=bind stablebaselines/stable-baselines3-cpu bash -c 'cd /home/mamba/stable-baselines3/ && pytest tests/'
```
Or, with the shell file:
```bash
./scripts/run_docker_cpu.sh pytest tests/
```
Explanation of the docker command:
- `docker run -it` create an instance of an image (=container), and
run it interactively (so ctrl+c will work)
- `--rm` option means to remove the container once it exits/stops
(otherwise, you will have to use `docker rm`)
- `--network host` don't use network isolation, this allows to use
tensorboard/visdom on host machine
- `--ipc=host` Use the host system’s IPC namespace. IPC (POSIX/SysV IPC) namespace provides
separation of named shared memory segments, semaphores and message
queues.
- `--name test` give explicitly the name `test` to the container,
otherwise it will be assigned a random name
- `--mount src=...` give access of the local directory (`pwd`
command) to the container (it will be map to `/home/mamba/stable-baselines`), so
all the logs created in the container in this folder will be kept
- `bash -c '...'` Run command inside the docker image, here run the tests
(`pytest tests/`)
[nvidia-docker]: https://github.com/NVIDIA/nvidia-docker
[tweaks]: https://stackoverflow.com/questions/23111631/cannot-download-docker-images-behind-a-proxy
---
### Guide Integrations (`docs/guide/integrations.md`)
(integrations)=
# Integrations
## Weights & Biases
Weights & Biases provides a callback for experiment tracking that allows to visualize and share results.
The full documentation is available here:
```python
import gymnasium as gym
import wandb
from wandb.integration.sb3 import WandbCallback
from stable_baselines3 import PPO
config = {
"policy_type": "MlpPolicy",
"total_timesteps": 25000,
"env_id": "CartPole-v1",
}
run = wandb.init(
project="sb3",
config=config,
sync_tensorboard=True, # auto-upload sb3's tensorboard metrics
# monitor_gym=True, # auto-upload the videos of agents playing the game
# save_code=True, # optional
)
model = PPO(config["policy_type"], config["env_id"], verbose=1, tensorboard_log=f"runs/{run.id}")
model.learn(
total_timesteps=config["total_timesteps"],
callback=WandbCallback(
model_save_path=f"models/{run.id}",
verbose=2,
),
)
run.finish()
```
## Hugging Face 🤗
The Hugging Face Hub 🤗 is a central place where anyone can share and explore models. It allows you to host your saved models 💾.
You can see the list of stable-baselines3 saved models here:
Most of them are available via the RL Zoo.
Official pre-trained models are saved in the SB3 organization on the hub:
We wrote a tutorial on how to use 🤗 Hub and Stable-Baselines3
[here](https://colab.research.google.com/github/huggingface/huggingface_sb3/blob/main/notebooks/sb3_huggingface.ipynb).
### Installation
```bash
pip install huggingface_sb3
```
:::{note}
If you use the [RL Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), pushing/loading models from the hub are already integrated:
```bash
# Download model and save it into the logs/ folder
# Only use TRUST_REMOTE_CODE=True with HF models that can be trusted (here the SB3 organization)
TRUST_REMOTE_CODE=True python -m rl_zoo3.load_from_hub --algo a2c --env LunarLander-v3 -orga sb3 -f logs/
# Test the agent
python -m rl_zoo3.enjoy --algo a2c --env LunarLander-v3 -f logs/
# Push model, config and hyperparameters to the hub
python -m rl_zoo3.push_to_hub --algo a2c --env LunarLander-v3 -f logs/ -orga sb3 -m "Initial commit"
```
:::
### Download a model from the Hub
You need to copy the repo-id that contains your saved model.
For instance `sb3/demo-hf-CartPole-v1`:
```python
import os
import gymnasium as gym
from huggingface_sb3 import load_from_hub
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy
# Allow the use of `pickle.load()` when downloading model from the hub
# Please make sure that the organization from which you download can be trusted
os.environ["TRUST_REMOTE_CODE"] = "True"
# Retrieve the model from the hub
## repo_id = id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name})
## filename = name of the model zip file from the repository
checkpoint = load_from_hub(
repo_id="sb3/demo-hf-CartPole-v1",
filename="ppo-CartPole-v1.zip",
)
model = PPO.load(checkpoint)
# Evaluate the agent and watch it
eval_env = gym.make("CartPole-v1")
mean_reward, std_reward = evaluate_policy(
model, eval_env, render=True, n_eval_episodes=5, deterministic=True, warn=False
)
print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")
```
You need to define two parameters:
- `repo-id`: the name of the Hugging Face repo you want to download.
- `filename`: the file you want to download.
### Upload a model to the Hub
You can easily upload your models using two different functions:
1. `package_to_hub()`: save the model, evaluate it, generate a model card and record a replay video of your agent before pushing the complete repo to the Hub.
2. `push_to_hub()`: simply push a file to the Hub.
First, you need to be logged in to Hugging Face to upload a model:
- If you're using Colab/Jupyter Notebooks:
```python
from huggingface_hub import notebook_login
notebook_login()
```
- Otherwise:
```bash
huggingface-cli login
```
Then, in this example, we train a PPO agent to play CartPole-v1 and push it to a new repo `sb3/demo-hf-CartPole-v1`
#### With `package_to_hub()`
```python
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from huggingface_sb3 import package_to_hub
# Create the environment
env_id = "CartPole-v1"
env = make_vec_env(env_id, n_envs=1)
# Create the evaluation environment
eval_env = make_vec_env(env_id, n_envs=1)
# Instantiate the agent
model = PPO("MlpPolicy", env, verbose=1)
# Train the agent
model.learn(total_timesteps=int(5000))
# This method saves, evaluates, generates a model card and records a replay video of your agent before pushing the repo to the hub
package_to_hub(model=model,
model_name="ppo-CartPole-v1",
model_architecture="PPO",
env_id=env_id,
eval_env=eval_env,
repo_id="sb3/demo-hf-CartPole-v1",
commit_message="Test commit")
```
You need to define seven parameters:
- `model`: your trained model.
- `model_architecture`: name of the architecture of your model (DQN, PPO, A2C, SAC…).
- `env_id`: name of the environment.
- `eval_env`: environment used to evaluate the agent.
- `repo-id`: the name of the Hugging Face repo you want to create or update. It’s \/\.
- `commit-message`.
- `filename`: the file you want to push to the Hub.
#### With `push_to_hub()`
```python
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from huggingface_sb3 import push_to_hub
# Create the environment
env_id = "CartPole-v1"
env = make_vec_env(env_id, n_envs=1)
# Instantiate the agent
model = PPO("MlpPolicy", env, verbose=1)
# Train the agent
model.learn(total_timesteps=int(5000))
# Save the model
model.save("ppo-CartPole-v1")
# Push this saved model .zip file to the hf repo
# If this repo does not exist it will be created
## repo_id = id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name})
## filename: the name of the file == "name" inside model.save("ppo-CartPole-v1")
push_to_hub(
repo_id="sb3/demo-hf-CartPole-v1",
filename="ppo-CartPole-v1.zip",
commit_message="Added CartPole-v1 model trained with PPO",
)
```
You need to define three parameters:
- `repo-id`: the name of the Hugging Face repo you want to create or update. It’s \/\.
- `filename`: the file you want to push to the Hub.
- `commit-message`.
## MLflow
If you want to use [MLflow](https://github.com/mlflow/mlflow) to track your SB3 experiments,
you can adapt the following code which defines a custom logger output:
```python
import sys
from typing import Any, Dict, Tuple, Union
import mlflow
import numpy as np
from stable_baselines3 import SAC
from stable_baselines3.common.logger import HumanOutputFormat, KVWriter, Logger
class MLflowOutputFormat(KVWriter):
"""
Dumps key/value pairs into MLflow's numeric format.
"""
def write(
self,
key_values: Dict[str, Any],
key_excluded: Dict[str, Union[str, Tuple[str, ...]]],
step: int = 0,
) -> None:
for (key, value), (_, excluded) in zip(
sorted(key_values.items()), sorted(key_excluded.items())
):
if excluded is not None and "mlflow" in excluded:
continue
if isinstance(value, np.ScalarType):
if not isinstance(value, str):
mlflow.log_metric(key, value, step)
loggers = Logger(
folder=None,
output_formats=[HumanOutputFormat(sys.stdout), MLflowOutputFormat()],
)
with mlflow.start_run():
model = SAC("MlpPolicy", "Pendulum-v1", verbose=2)
# Set custom logger
model.set_logger(loggers)
model.learn(total_timesteps=10000, log_interval=1)
```
---
### Guide Migration (`docs/guide/migration.md`)
(migration)=
# Migrating from Stable-Baselines
This is a guide to migrate from Stable-Baselines (SB2) to Stable-Baselines3 (SB3).
It also references the main changes.
## Overview
Overall Stable-Baselines3 (SB3) keeps the high-level API of Stable-Baselines (SB2).
Most of the changes are to ensure more consistency and are internal ones.
Because of the backend change, from Tensorflow to PyTorch, the internal code is much more readable and easy to debug
at the cost of some speed (dynamic graph vs static graph., see [Issue #90](https://github.com/DLR-RM/stable-baselines3/issues/90))
However, the algorithms were extensively benchmarked on Atari games and continuous control PyBullet envs
(see [Issue #48](https://github.com/DLR-RM/stable-baselines3/issues/48) and [Issue #49](https://github.com/DLR-RM/stable-baselines3/issues/49))
so you should not expect performance drop when switching from SB2 to SB3.
## How to migrate?
In most cases, replacing `from stable_baselines` with `from stable_baselines3` will be sufficient.
Some files were moved to the common folder (cf below) and could result to import errors.
Some algorithms were removed because of their complexity to improve the maintainability of the project.
We recommend reading this guide carefully to understand all the changes that were made.
You can also take a look at the [rl-zoo3](https://github.com/DLR-RM/rl-baselines3-zoo) and compare the imports
to the [rl-zoo](https://github.com/araffin/rl-baselines-zoo) of SB2 to have a concrete example of successful migration.
:::{note}
If you experience massive slow-down switching to PyTorch, you may need to play with the number of threads used,
using `torch.set_num_threads(1)` or `OMP_NUM_THREADS=1`, see [issue #122](https://github.com/DLR-RM/stable-baselines3/issues/122)
and [issue #90](https://github.com/DLR-RM/stable-baselines3/issues/90).
:::
## Breaking Changes
- SB3 requires python 3.7+ (instead of python 3.5+ for SB2)
- Dropped MPI support
- Dropped layer normalized policies (`MlpLnLstmPolicy`, `CnnLnLstmPolicy`)
- LSTM policies (`MlpLstmPolicy`, `CnnLstmPolicy`) are not supported for the time being
(see [PR #53](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/53) for a recurrent PPO implementation)
- Dropped parameter noise for DDPG and DQN
- PPO is now closer to the original implementation (no clipping of the value function by default), cf PPO section below
- Orthogonal initialization is only used by A2C/PPO
- The features extractor (CNN extractor) is shared between policy and q-networks for DDPG/SAC/TD3 and only the policy loss used to update it (much faster)
- Tensorboard legacy logging was dropped in favor of having one logger for the terminal and Tensorboard (cf {ref}`Tensorboard integration `)
- We dropped ACKTR/ACER support because of their complexity compared to simpler alternatives (PPO, SAC, TD3) performing as good.
- We dropped GAIL support as we are focusing on model-free RL only, you can however take a look at the {ref}`imitation project ` which implements
GAIL and other imitation learning algorithms on top of SB3.
- `action_probability` is currently not implemented in the base class
- `pretrain()` method for behavior cloning was removed (see [issue #27](https://github.com/DLR-RM/stable-baselines3/issues/27))
You can take a look at the [issue about SB3 implementation design](https://github.com/hill-a/stable-baselines/issues/576) for more details.
### Moved Files
- `bench/monitor.py` -> `common/monitor.py`
- `logger.py` -> `common/logger.py`
- `results_plotter.py` -> `common/results_plotter.py`
- `common/cmd_util.py` -> `common/env_util.py`
Utility functions are no longer exported from `common` module, you should import them with their absolute path, e.g.:
```python
from stable_baselines3.common.env_util import make_atari_env, make_vec_env
from stable_baselines3.common.utils import set_random_seed
```
instead of `from stable_baselines3.common import make_atari_env`
### Changes and renaming in parameters
#### Base-class (all algorithms)
- `load_parameters` -> `set_parameters`
- `get/set_parameters` return a dictionary mapping object names
to their respective PyTorch tensors and other objects representing
their parameters, instead of simpler mapping of parameter name to
a NumPy array. These functions also return PyTorch tensors rather
than NumPy arrays.
#### Policies
- `cnn_extractor` -> `features_extractor`, as `features_extractor` is now used with `MlpPolicy` too
#### A2C
- `epsilon` -> `rms_prop_eps`
- `lr_schedule` is part of `learning_rate` (it can be a callable).
- `alpha`, `momentum` are modifiable through `policy_kwargs` key `optimizer_kwargs`.
:::{warning}
PyTorch implementation of RMSprop [differs from Tensorflow's](https://github.com/pytorch/pytorch/issues/23796),
which leads to [different and potentially more unstable results](https://github.com/DLR-RM/stable-baselines3/pull/110#issuecomment-663255241).
Use `stable_baselines3.common.sb2_compat.rmsprop_tf_like.RMSpropTFLike` optimizer to match the results
with TensorFlow's implementation. This can be done through `policy_kwargs`: `A2C(policy_kwargs=dict(optimizer_class=RMSpropTFLike, optimizer_kwargs=dict(eps=1e-5)))`
:::
#### PPO
- `cliprange` -> `clip_range`
- `cliprange_vf` -> `clip_range_vf`
- `nminibatches` -> `batch_size`
:::{warning}
`nminibatches` gave different batch size depending on the number of environments: `batch_size = (n_steps * n_envs) // nminibatches`
:::
- `clip_range_vf` behavior for PPO is slightly different: Set it to `None` (default) to deactivate clipping (in SB2, you had to pass `-1`, `None` meant to use `clip_range` for the clipping)
- `lam` -> `gae_lambda`
- `noptepochs` -> `n_epochs`
PPO default hyperparameters are the one tuned for continuous control environment.
We recommend taking a look at the [RL Zoo](rl_zoo.md) for hyperparameters tuned for Atari games.
#### DQN
Only the vanilla DQN is implemented right now but extensions will follow.
Default hyperparameters are taken from the Nature paper, except for the optimizer and learning rate that were taken from Stable Baselines defaults.
#### DDPG
DDPG now follows the same interface as SAC/TD3.
For state/reward normalization, you should use `VecNormalize` as for all other algorithms.
#### SAC/TD3
SAC/TD3 now accept any number of critics, e.g. `policy_kwargs=dict(n_critics=3)`, instead of only two before.
:::{note}
SAC/TD3 default hyperparameters (including network architecture) now match the ones from the original papers.
DDPG is using TD3 defaults.
:::
#### SAC
SAC implementation matches the latest version of the original implementation: it uses two Q function networks and two target Q function networks
instead of two Q function networks and one Value function network (SB2 implementation, first version of the original implementation).
Despite this change, no change in performance should be expected.
:::{note}
SAC `predict()` method has now `deterministic=False` by default for consistency.
To match SB2 behavior, you need to explicitly pass `deterministic=True`
:::
#### HER
The `HER` implementation now only supports online sampling of the new goals. This is done in a vectorized version.
The goal selection strategy `RANDOM` is no longer supported.
### New logger API
- Methods were renamed in the logger:
- `logkv` -> `record`, `writekvs` -> `write`, `writeseq` -> `write_sequence`,
- `logkvs` -> `record_dict`, `dumpkvs` -> `dump`,
- `getkvs` -> `get_log_dict`, `logkv_mean` -> `record_mean`,
### Internal Changes
Please read the {ref}`Developer Guide ` section.
## New Features (SB3 vs SB2)
- Much cleaner and consistent base code (and no more warnings =D!) and static type checks
- Independent saving/loading/predict for policies
- A2C now supports Generalized Advantage Estimation (GAE) and advantage normalization (both are deactivated by default)
- Generalized State-Dependent Exploration (gSDE) exploration is available for A2C/PPO/SAC. It allows using RL directly on real robots (cf )
- Better saving/loading: optimizers are now included in the saved parameters and there are two new methods `save_replay_buffer` and `load_replay_buffer` for the replay buffer when using off-policy algorithms (DQN/DDPG/SAC/TD3)
- You can pass `optimizer_class` and `optimizer_kwargs` to `policy_kwargs` in order to easily
customize optimizers
- Seeding now works properly to have deterministic results
- Replay buffer does not grow, allocate everything at build time (faster)
- We added a memory efficient replay buffer variant (pass `optimize_memory_usage=True` to the constructor), it reduces drastically the memory used especially when using images
- You can specify an arbitrary number of critics for SAC/TD3 (e.g. `policy_kwargs=dict(n_critics=3)`)
---
### Guide Plotting (`docs/guide/plotting.md`)
(plotting)=
# Plotting
Stable Baselines3 provides utilities for plotting training results, allowing you to monitor and visualize your agent's learning progress.
The plotting functionality is provided by the `results_plotter` module, which can load monitor files created during training and generate various plots.
:::{note}
Plotting requires `pandas` and `matplotlib`. Install them with:
```bash
pip install pandas matplotlib
```
Or install the extra dependencies:
```bash
pip install 'stable-baselines3[extra]'
```
:::
:::{note}
We recommend using the
[RL Baselines3 Zoo plotting scripts](https://rl-baselines3-zoo.readthedocs.io/en/master/guide/plot.html)
which provide plotting capabilities with confidence intervals, and publication-ready visualizations.
:::
## Recommended Approach: RL Baselines3 Zoo Plotting
The [RL Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) provides scripts that allows to compare results across different environments and have publication-ready plots with confidence intervals.
The three main plotting scripts are:
- [plot_train.py](https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/rl_zoo3/plots/plot_train.py): For training plots
- [all_plots.py](https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/rl_zoo3/plots/all_plots.py): For evaluation plots, to post-process the result
- [plot_from_file.py](https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/rl_zoo3/plots/plot_from_file.py): For more advanced plotting from post-processed results
These scripts offer features that are not included in the basic SB3 plotting utilities.
### Installation
First, install RL Baselines3 Zoo:
```bash
pip install 'rl_zoo3[plots]'
```
### Basic Training Plot Examples
```bash
# Train an agent
python -m rl_zoo3.train --algo ppo --env CartPole-v1 -f logs/
# Plot training results for a single algorithm
python -m rl_zoo3.plots.plot_train --algo ppo --env CartPole-v1 --exp-folder logs/
```
### Evaluation and Comparison Plots
```bash
# Generate evaluation plots and save post-processed results
# in `logs/demo_plots.pkl` in order to use `plot_from_file`
python -m rl_zoo3.plots.all_plots --algo ppo sac -e Pendulum-v1 -f logs/ -o logs/demo_plots
# More advanced plotting from post-processed results (with confidence intervals)
python -m rl_zoo3.plots.plot_from_file -i logs/demo_plots.pkl --rliable --ci-size 0.95
```
For more examples, please read the
[RL Baselines3 Zoo plotting guide](https://rl-baselines3-zoo.readthedocs.io/en/master/guide/plot.html).
## Real-Time Monitoring
For real-time monitoring during training, consider using the plotting functions within callbacks
(see the [Callbacks guide](callbacks.md)) or integrating with tools like [Tensorboard](tensorboard.md) or Weights & Biases
(see the [Integrations guide](integrations.md)).
## Monitor File Format
The `Monitor` wrapper saves training data in CSV format with the following columns:
- `r`: Episode return (sum of rewards for one episode)
- `l`: Episode length (number of steps)
- `t`: Timestamp (wall-clock time when episode ended)
Additional columns may be present if you log custom metrics in the environment's info dict and pass their names via the `info_keywords` parameter.
:::{note}
The plotting functions automatically handle multiple monitor files from the same directory.
This occurs when using vectorized environments. Episodes are loaded and sorted by timestamp
to ensure they are in the correct chronological order.
:::
## Basic SB3 Plotting (Simple Use Cases)
### Basic Plotting: Single Training Run
The simplest way to plot training results is to use the `plot_results` function after training an agent.
This function reads the monitor files created by the `Monitor` wrapper and plots the episode rewards over time.
```python
import os
import gymnasium as gym
import matplotlib.pyplot as plt
from stable_baselines3 import PPO
from stable_baselines3.common.monitor import Monitor
from stable_baselines3.common.results_plotter import plot_results
from stable_baselines3.common import results_plotter
# Create log directory
log_dir = "tmp/"
os.makedirs(log_dir, exist_ok=True)
# Create and wrap the environment with Monitor
env = gym.make("CartPole-v1")
env = Monitor(env, log_dir)
# Train the agent
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=20_000)
# Plot the results
plot_results([log_dir], 20_000, results_plotter.X_TIMESTEPS, "PPO CartPole")
plt.show()
```
### Different Plotting Modes
The plotting functions support three different x-axis modes:
- `X_TIMESTEPS`: Plot rewards vs. timesteps (default)
- `X_EPISODES`: Plot rewards vs. episode number
- `X_WALLTIME`: Plot rewards vs. wall-clock time in hours
```python
import matplotlib.pyplot as plt
from stable_baselines3.common import results_plotter
# Plot by timesteps (shows sample efficiency)
# plot_results([log_dir], None, results_plotter.X_TIMESTEPS, "Rewards vs Timesteps")
# By Episodes
plot_results([log_dir], None, results_plotter.X_EPISODES, "Rewards vs Episodes")
# plot_results([log_dir], None, results_plotter.X_WALLTIME, "Rewards vs Time")
plt.tight_layout()
plt.show()
```
### Advanced Plotting with Manual Data Processing
For more control over the plotting, you can use the underlying functions to process the data manually:
```python
import numpy as np
import matplotlib.pyplot as plt
from stable_baselines3.common.monitor import load_results
from stable_baselines3.common.results_plotter import ts2xy, window_func
# Load the results
df = load_results(log_dir)
# Convert dataframe (x=timesteps, y=episodic return)
x, y = ts2xy(df, "timesteps")
# Plot raw data
plt.figure(figsize=(10, 6))
plt.subplot(2, 1, 1)
plt.scatter(x, y, s=2, alpha=0.6)
plt.xlabel("Timesteps")
plt.ylabel("Episode Reward")
plt.title("Raw Episode Rewards")
# Plot smoothed data with custom window
plt.subplot(2, 1, 2)
if len(x) >= 50: # Only smooth if we have enough data
x_smooth, y_smooth = window_func(x, y, 50, np.mean)
plt.plot(x_smooth, y_smooth, linewidth=2)
plt.xlabel("Timesteps")
plt.ylabel("Average Episode Reward (50-episode window)")
plt.title("Smoothed Episode Rewards")
plt.tight_layout()
plt.show()
```
---
### Guide Rl (`docs/guide/rl.md`)
(rl)=
# Reinforcement Learning Resources
Stable-Baselines3 assumes that you already understand the basic concepts of Reinforcement Learning (RL).
However, if you want to learn about RL, there are several good resources to get started:
- [OpenAI Spinning Up](https://spinningup.openai.com/en/latest/)
- [The Deep Reinforcement Learning Course](https://huggingface.co/learn/deep-rl-course/unit0/introduction)
- [David Silver's course](https://davidstarsilver.wordpress.com/teaching/)
- [RL102: From Tabular Q-Learning to Deep Q-Learning (DQN)](https://araffin.github.io/post/rl102/)
- [RL103: From Deep Q-Learning (DQN) to Soft Actor-Critic (SAC) and Beyond](https://araffin.github.io/post/rl103/)
- [Lilian Weng's blog](https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html)
- [Berkeley's Deep RL Bootcamp](https://sites.google.com/view/deep-rl-bootcamp/lectures)
- [Berkeley's Deep Reinforcement Learning course](https://rail.eecs.berkeley.edu/deeprlcourse/)
- [Decisions & Dragons - FAQ for RL foundations](https://www.decisionsanddragons.com)
- [More resources](https://github.com/dennybritz/reinforcement-learning)
---
### Guide Rl Tips (`docs/guide/rl_tips.md`)
(rl-tips)=
# Reinforcement Learning Tips and Tricks
The aim of this section is to help you run reinforcement learning experiments.
It covers general advice about RL (where to start, which algorithm to choose, how to evaluate an algorithm, ...),
as well as tips and tricks when using a custom environment or implementing an RL algorithm.
:::{note}
We have a [video on YouTube](https://www.youtube.com/watch?v=Ikngt0_DXJg) that covers
this section in more detail. You can also find the [slides here](https://araffin.github.io/slides/rlvs-tips-tricks/).
:::
:::{note}
We also have a [video on Designing and Running Real-World RL Experiments](https://youtu.be/eZ6ZEpCi6D8), slides [can be found online](https://araffin.github.io/slides/design-real-rl-experiments/), and a video on [Recent Advances in RL for Continuous Control (SOTA)](https://www.youtube.com/watch?v=Sb0dgmxbGZY) with [slides (early 2026 update)](https://araffin.github.io/slides/advances-rl-sota-2026/).
:::
## General advice when using Reinforcement Learning
### TL;DR
1. Read about RL and Stable-Baselines3 (SB3)
2. Do quantitative experiments and hyperparameter tuning if needed
3. Evaluate the performance using a separate test environment (remember to check wrappers!)
4. For better performance, increase the training budget
Like any other subject, if you want to work with RL, you should first read about it (we have a dedicated [resource page](rl.md) to get you started) to understand what you are using.
We also recommend that you read the Stable Baselines3 (SB3) documentation and do the [tutorial](https://github.com/araffin/rl-tutorial-jnrr19).
It covers basic usage and guides you towards more advanced concepts of the library (e.g. callbacks and wrappers).
Reinforcement Learning differs from other machine learning methods in several ways. The data used to train the agent is collected
through interactions with the environment by the agent itself (as opposed to, for example, supervised learning where you have a fixed dataset).
This dependency can lead to a vicious circle: if the agent collects poor quality data (e.g. trajectories with no rewards), it will not improve and will continue to collect bad trajectories.
This factor, among others, explains that results in RL may vary from one run to another (i.e., when only the seed of the pseudo-random generator changes).
For this reason, you should always do several runs to obtain quantitative results.
Good results in RL generally depend on finding appropriate hyperparameters.
Recent algorithms (PPO, SAC, TD3, DroQ) normally require little hyperparameter tuning, however, *don't expect the default ones to work* in every environment.
Therefore, we *highly recommend* that you take a look at the [RL zoo](https://github.com/DLR-RM/rl-baselines3-zoo) (or the original papers) for tuned hyperparameters.
A best practice when you apply RL to a new problem is to do automatic [hyperparameter optimization](https://araffin.github.io/post/hyperparam-tuning/).
Again, this is included in the [RL zoo](https://github.com/DLR-RM/rl-baselines3-zoo).
When applying RL to a custom problem, you should always normalize the input to the agent (e.g. using `VecNormalize` for PPO/A2C)
and look at common preprocessing done on other environments (e.g. for [Atari](https://danieltakeshi.github.io/2016/11/25/frame-skipping-and-preprocessing-for-deep-q-networks-on-atari-2600-games/), frame-stack, ...).
Please refer to *Tips and Tricks when creating a custom environment* paragraph below for more advice related to custom environments.
### Current Limitations of RL
You have to be aware of the current [limitations](https://www.alexirpan.com/2018/02/14/rl-hard.html) of reinforcement learning.
Model-free RL algorithms (i.e. all the algorithms implemented in SB3) are usually *sample inefficient*. They require a lot of samples (sometimes millions of interactions) to learn anything useful.
That's why most of the successes in RL were achieved on games or in simulation only.
For instance, in this [work](https://www.youtube.com/watch?v=aTDkYFZFWug) by ETH Zurich, the ANYmal robot was trained in simulation only, and then tested in the real world.
As general advice, to obtain better performance, you should increase the budget of the agent (number of training timesteps).
In order to achieve the desired behavior, expert knowledge is often required to design an adequate reward function.
This *reward engineering* (or *RewArt* as coined by [Freek Stulp](http://www.freekstulp.net/)), necessitates several iterations. As a good example of reward shaping,
you can take a look at [Deep Mimic paper](https://xbpeng.github.io/projects/DeepMimic/index.html) which combines imitation learning and reinforcement learning to do acrobatic moves.
A final limitation of RL is the instability of training. That is, you can observe a huge drop in performance during training.
This behavior is particularly present in `DDPG`, that's why its extension `TD3` tries to tackle that issue.
Other methods, such as `TRPO` or `PPO` use a *trust region* to minimize this problem by avoiding too large updates.
### How to evaluate an RL algorithm?
:::{note}
Pay attention to environment wrappers when evaluating your agent and comparing results to others' results. Modifications to episode rewards
or lengths may also affect evaluation results which may not be desirable. Check `evaluate_policy` helper function in {ref}`Evaluation Helper ` section.
:::
Because most algorithms use exploration noise during training, you need a separate test environment to evaluate the performance of your agent at a given time.
It is recommended to periodically evaluate your agent for `n` test episodes (`n` is usually between 5 and 20) and average the reward per episode to have a good estimate.
:::{note}
We provide an `EvalCallback` for doing such evaluation. You can read more about it in the {ref}`Callbacks ` section.
:::
As some policies are stochastic by default (e.g. A2C or PPO), you should also try to set `deterministic=True` when calling the `.predict()` method,
this frequently leads to better performance.
Looking at the training curve (episode reward as a function of timesteps) is a good proxy but underestimates the agent's true performance.
We highly recommend reading [Empirical Design in Reinforcement Learning](https://arxiv.org/abs/2304.01315), as it provides valuable insights for best practices when running RL experiments.
We also suggest reading [Deep Reinforcement Learning that Matters](https://arxiv.org/abs/1709.06560) for a good discussion about RL evaluation,
and [Rliable: Better Evaluation for Reinforcement Learning](https://araffin.github.io/post/rliable/) for comparing results.
You can also take a look at this [blog post](https://openlab-flowers.inria.fr/t/how-many-random-seeds-should-i-use-statistical-power-analysis-in-deep-reinforcement-learning-experiments/457)
and this [issue](https://github.com/hill-a/stable-baselines/issues/199) by Cédric Colas.
## Which algorithm should I use?
There is no silver bullet in RL, you can choose one or the other depending on your needs and problems.
The first distinction comes from your action space, i.e., do you have discrete (e.g. LEFT, RIGHT, ...)
or continuous actions (ex: go to a certain speed)?
Some algorithms are only tailored for one or the other domain: `DQN` supports only discrete actions, while `SAC` is restricted to continuous actions.
The second difference that will help you decide is whether you can parallelize your training or not.
If what matters is the wall clock training time, then you should lean towards `A2C` and its derivatives (PPO, ...).
Take a look at the [Vectorized Environments](vec_envs.md) to learn more about training with multiple workers.
To accelerate training, you can also take a look at [SBX], which is SB3 + Jax, it has less features than SB3 but can be up to 20x faster than SB3 PyTorch thanks to JIT compilation of the gradient update.
In sparse reward settings, we recommend using dedicated methods like HER (see below) or population-based algorithms like ARS (available in our [contrib repo](sb3_contrib.md).
To sum up:
### Discrete Actions
:::{note}
This covers `Discrete`, `MultiDiscrete`, `Binary` and `MultiBinary` spaces
:::
#### Discrete Actions - Single Process
`DQN` with extensions (double DQN, prioritized replay, ...) are the recommended algorithms.
We notably provide `QR-DQN` in our [contrib repo](sb3_contrib.md).
`DQN` is usually slower to train (regarding wall clock time) but is the most sample efficient (because of its replay buffer).
#### Discrete Actions - Multiprocessed
You should give a try to `PPO` or `A2C`.
### Continuous Actions
#### Continuous Actions - Single Process
:::{note}
We have a video on [Recent Advances in RL for Continuous Control (SOTA)](https://www.youtube.com/watch?v=Sb0dgmxbGZY) with [slides (early 2026 update)](https://araffin.github.io/slides/advances-rl-sota-2026/).
:::
Current State Of The Art (SOTA) algorithms are `SAC`, `TD3`, `CrossQ` and `TQC` (available in our [contrib repo](sb3_contrib.md) and [SBX (SB3 + Jax) repo](sbx.md)).
Please use the hyperparameters in the [RL zoo](https://github.com/DLR-RM/rl-baselines3-zoo) for best results.
If you want an extremely sample-efficient algorithm, we recommend using the [DroQ configuration](https://twitter.com/araffin2/status/1575439865222660098) in [SBX] (it does many gradient steps per step in the environment).
#### Continuous Actions - Multiprocessed
Take a look at `PPO`, `TRPO` (available in our [contrib repo](sb3_contrib.md)) or `A2C`. Again, don't forget to take the hyperparameters from the [RL zoo](https://github.com/DLR-RM/rl-baselines3-zoo) for continuous actions problems (cf *Bullet* envs).
:::{note}
Normalization is critical for those algorithms
:::
### Goal Environment
If your environment follows the `GoalEnv` interface (cf [HER](../modules/her.md)), then you should use
HER + (SAC/TD3/DDPG/DQN/QR-DQN/TQC) depending on the action space.
:::{note}
The `batch_size` is an important hyperparameter for experiments with [HER](../modules/her.md)
:::
## Tips and Tricks when creating a custom environment
If you want to learn about how to create a custom environment, we recommend you read this [page](custom_env.md).
We also provide a [colab notebook](https://colab.research.google.com/github/araffin/rl-tutorial-jnrr19/blob/master/5_custom_gym_env.ipynb) for a concrete example of creating a custom gym environment.
Some basic advice:
- always normalize your observation space if you can, i.e. if you know the boundaries
- normalize your action space and make it symmetric if it is continuous (see potential problem below) A good practice is to rescale your actions so that they lie in [-1, 1]. This does not limit you, as you can easily rescale the action within the environment.
- start with a shaped reward (i.e. informative reward) and a simplified version of your problem
- debug with random actions to check if your environment works and follows the gym interface (with `check_env`, see below)
Two important things to keep in mind when creating a custom environment are avoiding breaking the Markov assumption
and properly handle termination due to a timeout (maximum number of steps in an episode).
For example, if there is a time delay between action and observation (e.g. due to wifi communication), you should provide a history of observations as input.
Termination due to timeout (max number of steps per episode) needs to be handled separately.
You should return `truncated = True`.
If you are using the gym `TimeLimit` wrapper, this will be done automatically.
You can read [Time Limit in RL](https://arxiv.org/abs/1712.00378), take a look at the [Designing and Running Real-World RL Experiments video](https://youtu.be/eZ6ZEpCi6D8) or [RL Tips and Tricks video](https://www.youtube.com/watch?v=Ikngt0_DXJg) for more details.
We provide a helper to check that your environment runs without error:
```python
from stable_baselines3.common.env_checker import check_env
env = CustomEnv(arg1, ...)
# It will check your custom environment and output additional warnings if needed
check_env(env)
```
If you want to quickly try a random agent on your environment, you can also do:
```python
env = YourEnv()
obs, info = env.reset()
n_steps = 10
for _ in range(n_steps):
# Random action
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
obs, info = env.reset()
```
**Why should I normalize the action space?**
Most reinforcement learning algorithms rely on a [Gaussian distribution](https://araffin.github.io/post/sac-massive-sim/) (initially centered at 0 with std 1) for continuous actions.
So, if you forget to normalize the action space when using a custom environment, this can [harm learning](https://araffin.github.io/post/sac-massive-sim/) and can be difficult to debug (cf attached image and [issue #473](https://github.com/hill-a/stable-baselines/issues/473)).
:::{figure} ../_static/img/mistake.png
:::
Another consequence of using a Gaussian distribution is that the action range is not bounded.
That's why clipping is usually used as a bandage to stay in a valid interval.
A better solution would be to use a squashing function (cf `SAC`) or a Beta distribution (cf [issue #112](https://github.com/hill-a/stable-baselines/issues/112)).
:::{note}
This statement is not true for `DDPG` or `TD3` because they don't rely on any probability distribution.
:::
## Tips and Tricks when implementing an RL algorithm
:::{note}
We have a [video on YouTube about reliable RL](https://www.youtube.com/watch?v=7-PUg9EAa3Y) that covers
this section in more details. You can also find the [slides online](https://araffin.github.io/slides/tips-reliable-rl/).
:::
When you try to reproduce a RL paper by implementing the algorithm, the [nuts and bolts of RL research](http://joschu.net/docs/nuts-and-bolts.pdf)
by John Schulman are quite useful ([video](https://www.youtube.com/watch?v=8EcdaCk9KaQ)).
We *recommend following those steps to have a working RL algorithm*:
1. Read the original paper several times
2. Read existing implementations (if available)
3. Try to have some "signs of life" on toy problems
4. Validate the implementation by making it run on harder and harder envs (you can compare results against the RL zoo).
You usually need to run hyperparameter optimization for that step.
You need to be particularly careful on the shape of the different objects you are manipulating (a broadcast mistake will fail silently cf. [issue #75](https://github.com/hill-a/stable-baselines/pull/76))
and when to stop the gradient propagation.
Don't forget to handle termination due to timeout separately (see remark in the custom environment section above),
you can also take a look at [Issue #284](https://github.com/DLR-RM/stable-baselines3/issues/284) and [Issue #633](https://github.com/DLR-RM/stable-baselines3/issues/633).
A personal pick (by @araffin) for environments with gradual difficulty in RL with continuous actions:
1. Pendulum (easy to solve)
2. HalfCheetahBullet (medium difficulty with local minima and shaped reward)
3. BipedalWalkerHardcore (if it works on that one, then you can have a cookie)
For RL with discrete actions:
1. CartPole-v1 (easy to be better than random agent, harder to achieve maximal performance)
2. LunarLander
3. Pong (one of the easiest Atari games)
4. other Atari games (e.g. Breakout)
[sbx]: https://github.com/araffin/sbx
---
### Guide Rl Zoo (`docs/guide/rl_zoo.md`)
(rl-zoo)=
# RL Baselines3 Zoo
[RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) is a training framework for Reinforcement Learning (RL).
It provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.
In addition, it includes a collection of tuned hyperparameters for common environments and RL algorithms, and agents trained with those settings.
Goals of this repository:
1. Provide a simple interface to train and enjoy RL agents
2. Benchmark the different Reinforcement Learning algorithms
3. Provide tuned hyperparameters for each environment and RL algorithm
4. Have fun with the trained agents!
Documentation is available online:
## Installation
Option 1: install the python package `pip install rl_zoo3`
or:
1. Clone the repository:
```
git clone --recursive https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
```
:::{note}
You can remove the `--recursive` option if you don't want to download the trained agents
:::
:::{note}
If you only need the training/plotting scripts and additional callbacks/wrappers from the RL Zoo, you can also install it via pip: `pip install rl_zoo3`
:::
2\. Install dependencies
```
apt-get install swig cmake ffmpeg
# full dependencies
pip install -r requirements.txt
# minimal dependencies
pip install -e .
```
## Train an Agent
The hyperparameters for each environment are defined in
`hyperparameters/algo_name.yml`.
If the environment exists in this file, then you can train an agent
using:
```
python -m rl_zoo3.train --algo algo_name --env env_id
```
For example (with evaluation and checkpoints):
```
python -m rl_zoo3.train --algo ppo --env CartPole-v1 --eval-freq 10000 --save-freq 50000
```
Continue training (here, load pretrained agent for Breakout and continue
training for 5000 steps):
```
python -m rl_zoo3.train --algo a2c --env BreakoutNoFrameskip-v4 -i trained_agents/a2c/BreakoutNoFrameskip-v4_1/BreakoutNoFrameskip-v4.zip -n 5000
```
## Enjoy a Trained Agent
If the trained agent exists, then you can see it in action using:
```
python -m rl_zoo3.enjoy --algo algo_name --env env_id
```
For example, enjoy A2C on Breakout during 5000 timesteps:
```
python -m rl_zoo3.enjoy --algo a2c --env BreakoutNoFrameskip-v4 --folder rl-trained-agents/ -n 5000
```
## Hyperparameter Optimization
We use [Optuna](https://optuna.org/) for optimizing the hyperparameters.
Tune the hyperparameters for PPO, using a random sampler and median pruner, 2 parallels jobs,
with a budget of 1000 trials and a maximum of 50000 steps:
```
python -m rl_zoo3.train --algo ppo --env MountainCar-v0 -n 50000 -optimize --n-trials 1000 --n-jobs 2 \
--sampler random --pruner median
```
## Colab Notebook: Try it Online!
You can train agents online using Google [colab notebook](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/rl-baselines-zoo.ipynb).
:::{note}
You can find more information about the rl baselines3 zoo in the repo [README](https://github.com/DLR-RM/rl-baselines3-zoo). For instance, how to record a video of a trained agent.
:::
---
### Guide Save Format (`docs/guide/save_format.md`)
(save-format)=
# On saving and loading
Stable Baselines3 (SB3) stores both neural network parameters and algorithm-related parameters such as
exploration schedule, number of environments and observation/action space. This allows continual learning and easy
use of trained agents without training, but it is not without its issues. Following describes the format
used to save agents in SB3 along with its pros and shortcomings.
Terminology used in this page:
- *parameters* refer to neural network parameters (also called "weights"). This is a dictionary
mapping variable name to a PyTorch tensor.
- *data* refers to RL algorithm parameters, e.g. learning rate, exploration schedule, action/observation space.
These depend on the algorithm used. This is a dictionary mapping classes variable names to their values.
## Zip-archive
A zip-archived JSON dump, PyTorch state dictionaries and PyTorch variables. The data dictionary (class parameters)
is stored as a JSON file, model parameters and optimizers are serialized with `torch.save()` function and these files
are stored under a single .zip archive.
Any objects that are not JSON serializable are serialized with cloudpickle and stored as base64-encoded
string in the JSON file, along with some information that was stored in the serialization. This allows
inspecting stored objects without deserializing the object itself.
This format allows skipping elements in the file, i.e. we can skip deserializing objects that are
broken/non-serializable.
This can be done via `custom_objects` argument to load functions.
:::{note}
If you encounter loading issue, for instance pickle issues or error after loading
(see [#171](https://github.com/DLR-RM/stable-baselines3/issues/171) or [#573](https://github.com/DLR-RM/stable-baselines3/issues/573)),
you can pass `print_system_info=True`
to compare the system on which the model was trained vs the current one
`model = PPO.load("ppo_saved", print_system_info=True)`
:::
File structure:
```
saved_model.zip/
├── data JSON file of class-parameters (dictionary)
├── *.optimizer.pth PyTorch optimizers serialized
├── policy.pth PyTorch state dictionary of the policy saved
├── pytorch_variables.pth Additional PyTorch variables
├── _stable_baselines3_version contains the SB3 version with which the model was saved
├── system_info.txt contains system info (os, python version, ...) on which the model was saved
```
Pros:
- More robust to unserializable objects (one bad object does not break everything).
- Saved files can be inspected/extracted with zip-archive explorers and by other languages.
Cons:
- More complex implementation.
- Still relies partly on cloudpickle for complex objects (e.g. custom functions)
which can lead to [incompatibilities](https://github.com/DLR-RM/stable-baselines3/issues/172) between Python versions.
---
### Guide Sb3 Contrib (`docs/guide/sb3_contrib.md`)
(sb3-contrib)=
# SB3 Contrib
We implement experimental features in a separate contrib repository:
[SB3-Contrib]
This allows Stable-Baselines3 (SB3) to maintain a stable and compact core, while still
providing the latest features, like RecurrentPPO (PPO LSTM), Truncated Quantile Critics (TQC), Augmented Random Search (ARS), Trust Region Policy Optimization (TRPO) or
Quantile Regression DQN (QR-DQN).
## Why create this repository?
Over the span of stable-baselines and stable-baselines3, the community
has been eager to contribute in form of better logging utilities,
environment wrappers, extended support (e.g. different action spaces)
and learning algorithms.
However sometimes these utilities were too niche to be considered for
stable-baselines or proved to be too difficult to integrate well into
the existing code without creating a mess. sb3-contrib aims to fix this by not
requiring the neatest code integration with existing code and not
setting limits on what is too niche: almost everything remotely useful
goes!
We hope this allows us to provide reliable implementations
following stable-baselines usual standards (consistent style, documentation, etc)
beyond the relatively small scope of utilities in the main repository.
### Features
See documentation for the full list of included features.
**RL Algorithms**:
- [Augmented Random Search (ARS)](https://arxiv.org/abs/1803.07055)
- [Quantile Regression DQN (QR-DQN)]
- [PPO with invalid action masking (Maskable PPO)](https://arxiv.org/abs/2006.14171)
- [PPO with recurrent policy (RecurrentPPO aka PPO LSTM)](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)
- [Truncated Quantile Critics (TQC)]
- [Trust Region Policy Optimization (TRPO)](https://arxiv.org/abs/1502.05477)
- [Batch Normalization in Deep Reinforcement Learning (CrossQ)](https://openreview.net/forum?id=PczQtTsTIX)
**Gym Wrappers**:
- [Time Feature Wrapper]
### Documentation
Documentation is available online:
### Installation
To install Stable-Baselines3 contrib with pip, execute:
```
pip install sb3-contrib
```
We recommend to use the `master` version of Stable Baselines3 and SB3-Contrib.
To install Stable Baselines3 `master` version:
```
pip install git+https://github.com/DLR-RM/stable-baselines3
```
To install Stable Baselines3 contrib `master` version:
```
pip install git+https://github.com/Stable-Baselines-Team/stable-baselines3-contrib
```
### Example
SB3-Contrib follows the SB3 API and folder structure. So, if you are familiar with SB3,
using SB3-Contrib should be easy too.
Here is an example of training a Quantile Regression DQN (QR-DQN) agent on the CartPole environment.
```python
from sb3_contrib import QRDQN
policy_kwargs = dict(n_quantiles=50)
model = QRDQN("MlpPolicy", "CartPole-v1", policy_kwargs=policy_kwargs, verbose=1)
model.learn(total_timesteps=10000, log_interval=4)
model.save("qrdqn_cartpole")
```
[quantile regression dqn (qr-dqn)]: https://arxiv.org/abs/1710.10044
[sb3-contrib]: https://github.com/Stable-Baselines-Team/stable-baselines3-contrib
[time feature wrapper]: https://arxiv.org/abs/1712.00378
[truncated quantile critics (tqc)]: https://arxiv.org/abs/2005.04269
---
### Guide Sbx (`docs/guide/sbx.md`)
(sbx)=
# Stable Baselines Jax (SBX)
[Stable Baselines Jax (SBX)](https://github.com/araffin/sbx) is a proof of concept version of Stable-Baselines3 in Jax.
It provides a minimal number of features compared to SB3 but can be much faster (up to 20x times!):
Implemented algorithms:
- Soft Actor-Critic (SAC) and SAC-N
- Truncated Quantile Critics (TQC)
- Dropout Q-Functions for Doubly Efficient Reinforcement Learning (DroQ)
- Proximal Policy Optimization (PPO)
- Deep Q Network (DQN)
- Twin Delayed DDPG (TD3)
- Deep Deterministic Policy Gradient (DDPG)
- Batch Normalization in Deep Reinforcement Learning (CrossQ)
- Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning (SimBa)
As SBX follows SB3 API, it is also compatible with the [RL Zoo](https://github.com/DLR-RM/rl-baselines3-zoo).
For that you will need to create two files:
`train_sbx.py`:
```python
import rl_zoo3
import rl_zoo3.train
from rl_zoo3.train import train
from sbx import DDPG, DQN, PPO, SAC, TD3, TQC, CrossQ
rl_zoo3.ALGOS["ddpg"] = DDPG
rl_zoo3.ALGOS["dqn"] = DQN
# See SBX readme to use DroQ configuration
# rl_zoo3.ALGOS["droq"] = DroQ
rl_zoo3.ALGOS["sac"] = SAC
rl_zoo3.ALGOS["ppo"] = PPO
rl_zoo3.ALGOS["td3"] = TD3
rl_zoo3.ALGOS["tqc"] = TQC
rl_zoo3.ALGOS["crossq"] = CrossQ
rl_zoo3.train.ALGOS = rl_zoo3.ALGOS
rl_zoo3.exp_manager.ALGOS = rl_zoo3.ALGOS
if __name__ == "__main__":
train()
```
Then you can call `python train_sbx.py --algo sac --env Pendulum-v1` and use the RL Zoo CLI.
`enjoy_sbx.py`:
```python
import rl_zoo3
import rl_zoo3.enjoy
from rl_zoo3.enjoy import enjoy
from sbx import DDPG, DQN, PPO, SAC, TD3, TQC, CrossQ
rl_zoo3.ALGOS["ddpg"] = DDPG
rl_zoo3.ALGOS["dqn"] = DQN
# See SBX readme to use DroQ configuration
# rl_zoo3.ALGOS["droq"] = DroQ
rl_zoo3.ALGOS["sac"] = SAC
rl_zoo3.ALGOS["ppo"] = PPO
rl_zoo3.ALGOS["td3"] = TD3
rl_zoo3.ALGOS["tqc"] = TQC
rl_zoo3.ALGOS["crossq"] = CrossQ
rl_zoo3.enjoy.ALGOS = rl_zoo3.ALGOS
rl_zoo3.exp_manager.ALGOS = rl_zoo3.ALGOS
if __name__ == "__main__":
enjoy()
```
---
### Guide Tensorboard (`docs/guide/tensorboard.md`)
(tensorboard)=
# Tensorboard Integration
## Basic Usage
To use Tensorboard with stable baselines3, you simply need to pass the location of the log folder to the RL agent:
```python
from stable_baselines3 import A2C
model = A2C("MlpPolicy", "CartPole-v1", verbose=1, tensorboard_log="./a2c_cartpole_tensorboard/")
model.learn(total_timesteps=10_000)
```
You can also define custom logging name when training (by default it is the algorithm name)
```python
from stable_baselines3 import A2C
model = A2C("MlpPolicy", "CartPole-v1", verbose=1, tensorboard_log="./a2c_cartpole_tensorboard/")
model.learn(total_timesteps=10_000, tb_log_name="first_run")
# Pass reset_num_timesteps=False to continue the training curve in tensorboard
# By default, it will create a new curve
# Keep tb_log_name constant to have continuous curve (see note below)
model.learn(total_timesteps=10_000, tb_log_name="second_run", reset_num_timesteps=False)
model.learn(total_timesteps=10_000, tb_log_name="third_run", reset_num_timesteps=False)
```
:::{note}
If you specify different `tb_log_name` in subsequent runs, you will have split graphs, like in the figure below.
If you want them to be continuous, you must keep the same `tb_log_name` (see [issue #975](https://github.com/DLR-RM/stable-baselines3/issues/975#issuecomment-1198992211)).
And, if you still managed to get your graphs split by other means, just put tensorboard log files into the same folder.
```{image} ../_static/img/split_graph.png
:alt: split_graph
:width: 330
```
:::
Once the learn function is called, you can monitor the RL agent during or after the training, with the following bash command:
```bash
tensorboard --logdir ./a2c_cartpole_tensorboard/
```
:::{note}
You can find explanations about the logger output and names in the {ref}`Logger ` section.
:::
you can also add past logging folders:
```bash
tensorboard --logdir ./a2c_cartpole_tensorboard/,./ppo2_cartpole_tensorboard/
```
It will display information such as the episode reward (when using a `Monitor` wrapper), the model losses and other parameter unique to some models.
```{image} ../_static/img/Tensorboard_example.png
:alt: plotting
:width: 600
```
## Logging More Values
Using a callback, you can easily log more values with TensorBoard.
Here is a simple example on how to log both additional tensor or arbitrary scalar value:
```python
import numpy as np
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback
model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)
class TensorboardCallback(BaseCallback):
"""
Custom callback for plotting additional values in tensorboard.
"""
def __init__(self, verbose=0):
super().__init__(verbose)
def _on_step(self) -> bool:
# Log scalar value (here a random variable)
value = np.random.random()
self.logger.record("random_value", value)
return True
model.learn(50000, callback=TensorboardCallback())
```
:::{note}
If you want to log values more often than the default to tensorboard, you manually call `self.logger.dump(self.num_timesteps)` in a callback
(see [issue #506](https://github.com/DLR-RM/stable-baselines3/issues/506)).
:::
## Logging Images
TensorBoard supports periodic logging of image data, which helps evaluating agents at various stages during training.
:::{warning}
To support image logging [pillow](https://github.com/python-pillow/Pillow) must be installed otherwise, TensorBoard ignores the image and logs a warning.
:::
Here is an example of how to render an image to TensorBoard at regular intervals:
```python
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import Image
model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)
class ImageRecorderCallback(BaseCallback):
def __init__(self, verbose=0):
super().__init__(verbose)
def _on_step(self):
image = self.training_env.render(mode="rgb_array")
# "HWC" specify the dataformat of the image, here channel last
# (H for height, W for width, C for channel)
# See https://docs.pytorch.org/docs/stable/tensorboard.html
# for supported formats
self.logger.record("trajectory/image", Image(image, "HWC"), exclude=("stdout", "log", "json", "csv"))
return True
model.learn(50000, callback=ImageRecorderCallback())
```
## Logging Figures/Plots
TensorBoard supports periodic logging of figures/plots created with matplotlib, which helps evaluate agents at various stages during training.
:::{warning}
To support figure logging [matplotlib](https://matplotlib.org/) must be installed otherwise, TensorBoard ignores the figure and logs a warning.
:::
Here is an example of how to store a plot in TensorBoard at regular intervals:
```python
import numpy as np
import matplotlib.pyplot as plt
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import Figure
model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)
class FigureRecorderCallback(BaseCallback):
def __init__(self, verbose=0):
super().__init__(verbose)
def _on_step(self):
# Plot values (here a random variable)
figure = plt.figure()
figure.add_subplot().plot(np.random.random(3))
# Close the figure after logging it
self.logger.record("trajectory/figure", Figure(figure, close=True), exclude=("stdout", "log", "json", "csv"))
plt.close()
return True
model.learn(50000, callback=FigureRecorderCallback())
```
## Logging Videos
TensorBoard supports periodic logging of video data, which helps evaluate agents at various stages during training.
:::{warning}
To support video logging [moviepy](https://zulko.github.io/moviepy/) must be installed otherwise, TensorBoard ignores the video and logs a warning.
:::
Here is an example of how to render an episode and log the resulting video to TensorBoard at regular intervals:
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Logging Hyperparameters
TensorBoard supports logging of hyperparameters in its HPARAMS tab, which helps to compare agents trainings.
:::{warning}
To display hyperparameters in the HPARAMS section, a `metric_dict` must be given (as well as a `hparam_dict`).
:::
Here is an example of how to save hyperparameters in TensorBoard:
```python
from stable_baselines3 import A2C
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import HParam
class HParamCallback(BaseCallback):
"""
Saves the hyperparameters and metrics at the start of the training, and logs them to TensorBoard.
"""
def _on_training_start(self) -> None:
hparam_dict = {
"algorithm": self.model.__class__.__name__,
"learning rate": self.model.learning_rate,
"gamma": self.model.gamma,
}
# define the metrics that will appear in the `HPARAMS` Tensorboard tab by referencing their tag
# Tensorbaord will find & display metrics from the `SCALARS` tab
metric_dict = {
"rollout/ep_len_mean": 0,
"train/value_loss": 0.0,
}
self.logger.record(
"hparams",
HParam(hparam_dict, metric_dict),
exclude=("stdout", "log", "json", "csv"),
)
def _on_step(self) -> bool:
return True
model = A2C("MlpPolicy", "CartPole-v1", tensorboard_log="runs/", verbose=1)
model.learn(total_timesteps=int(5e4), callback=HParamCallback())
```
## Directly Accessing The Summary Writer
If you would like to log arbitrary data (in one of the formats supported by [PyTorch](https://docs.pytorch.org/docs/stable/tensorboard.html)), you
can get direct access to the underlying SummaryWriter in a callback:
:::{warning}
This method is not recommended and should only be used by advanced users.
:::
:::{note}
If you want a concrete example, you can watch [how to log lap time with donkeycar env](https://www.youtube.com/watch?v=v8j2bpcE4Rg&t=4619s),
or read the code in the [RL Zoo](https://github.com/DLR-RM/rl-baselines3-zoo/blob/eb5d9c7770abe9a60f5511193ebcb260dfdc2706/rl_zoo3/callbacks.py#L262).
You might also want to take a look at [issue #1160](https://github.com/DLR-RM/stable-baselines3/issues/1160) and [issue #1219](https://github.com/DLR-RM/stable-baselines3/issues/1219).
:::
```python
from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import TensorBoardOutputFormat
model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)
class SummaryWriterCallback(BaseCallback):
def _on_training_start(self):
self._log_freq = 1000 # log every 1000 calls
output_formats = self.logger.output_formats
# Save reference to tensorboard formatter object
# note: the failure case (not formatter found) is not handled here, should be done with try/except.
self.tb_formatter = next(formatter for formatter in output_formats if isinstance(formatter, TensorBoardOutputFormat))
def _on_step(self) -> bool:
if self.n_calls % self._log_freq == 0:
# You can have access to info from the env using self.locals.
# for instance, when using one env (index 0 of locals["infos"]):
# lap_count = self.locals["infos"][0]["lap_count"]
# self.tb_formatter.writer.add_scalar("train/lap_count", lap_count, self.num_timesteps)
self.tb_formatter.writer.add_text("direct_access", "this is a value", self.num_timesteps)
self.tb_formatter.writer.flush()
model.learn(50000, callback=SummaryWriterCallback())
```
---
### Guide Vec Envs (`docs/guide/vec_envs.md`)
(vec-env)=
```{eval-rst}
.. automodule:: stable_baselines3.common.vec_env
```
# Vectorized Environments
Vectorized Environments are a method for stacking multiple independent environments into a single environment.
Instead of training an RL agent on 1 environment per step, it allows us to train it on `n` environments per step.
Because of this, `actions` passed to the environment are now a vector (of dimension `n`).
It is the same for `observations`, `rewards` and end of episode signals (`dones`).
In the case of non-array observation spaces such as `Dict` or `Tuple`, where different sub-spaces
may have different shapes, the sub-observations are vectors (of dimension `n`).
| Name | `Box` | `Discrete` | `Dict` | `Tuple` | Multi Processing |
| ------------- | ----- | ---------- | ------ | ------- | ---------------- |
| DummyVecEnv | ✔️ | ✔️ | ✔️ | ✔️ | ❌️ |
| SubprocVecEnv | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
:::{note}
Vectorized environments are required when using wrappers for frame-stacking or normalization.
:::
:::{note}
When using vectorized environments, the environments are automatically reset at the end of each episode.
Thus, the observation returned for the i-th environment when `done[i]` is true will in fact be the first observation of the next episode, not the last observation of the episode that has just terminated.
You can access the "real" final observation of the terminated episode—that is, the one that accompanied the `done` event provided by the underlying environment—using the `terminal_observation` keys in the info dicts returned by the `VecEnv`.
:::
:::{warning}
When defining a custom `VecEnv` (for instance, using gym3 `ProcgenEnv`), you should provide `terminal_observation` keys in the info dicts returned by the `VecEnv`
(cf. note above).
:::
:::{warning}
When using `SubprocVecEnv`, users must wrap the code in an `if __name__ == "__main__":` if using the `forkserver` or `spawn` start method (default on Windows).
On Linux, the default start method is `fork` which is not thread safe and can create deadlocks.
For more information, see Python's [multiprocessing guidelines](https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods).
:::
## VecEnv API vs Gym API
For consistency across Stable-Baselines3 (SB3) versions and because of its special requirements and features,
SB3 VecEnv API is not the same as Gym API.
SB3 VecEnv API is actually close to Gym 0.21 API but differs to Gym 0.26+ API:
- the `reset()` method only returns the observation (`obs = vec_env.reset()`) and not a tuple, the info at reset are stored in `vec_env.reset_infos`.
- only the initial call to `vec_env.reset()` is required, environments are reset automatically afterward (and `reset_infos` is updated automatically).
- the `vec_env.step(actions)` method expects an array as input
(with a batch size corresponding to the number of environments) and returns a 4-tuple (and not a 5-tuple): `obs, rewards, dones, infos` instead of `obs, reward, terminated, truncated, info`
where `dones = terminated or truncated` (for each env).
`obs, rewards, dones` are NumPy arrays with shape `(n_envs, shape_for_single_env)` (so with a batch dimension).
Additional information is passed via the `infos` value which is a list of dictionaries.
- at the end of an episode, `infos[env_idx]["TimeLimit.truncated"] = truncated and not terminated`
tells the user if an episode was truncated or not:
you should bootstrap if `infos[env_idx]["TimeLimit.truncated"] is True` (episode over due to a timeout/truncation)
or `dones[env_idx] is False` (episode not finished).
Note: compared to Gym 0.26+ `infos[env_idx]["TimeLimit.truncated"]` and `terminated` [are mutually exclusive](https://github.com/openai/gym/issues/3102).
The conversion from SB3 to Gym API is
```python
# done is True at the end of an episode
# dones[env_idx] = terminated[env_idx] or truncated[env_idx]
# In SB3, truncated and terminated are mutually exclusive
# infos[env_idx]["TimeLimit.truncated"] = truncated and not terminated
# terminated[env_idx] tells you whether you should bootstrap or not:
# when the episode has not ended or when the termination was a timeout/truncation
terminated[env_idx] = dones[env_idx] and not infos[env_idx]["TimeLimit.truncated"]
should_bootstrap[env_idx] = not terminated[env_idx]
```
- at the end of an episode, because the environment resets automatically,
we provide `infos[env_idx]["terminal_observation"]` which contains the last observation
of an episode (and can be used when bootstrapping, see note in the previous section)
- to overcome the current Gymnasium limitation (only one render mode allowed per env instance, see [issue #100](https://github.com/Farama-Foundation/Gymnasium/issues/100)),
we recommend using `render_mode="rgb_array"` since we can both have the image as a numpy array and display it with OpenCV.
if no mode is passed or `mode="rgb_array"` is passed when calling `vec_env.render` then we use the default mode, otherwise, we use the OpenCV display.
Note that if `render_mode != "rgb_array"`, you can only call `vec_env.render()` (without argument or with `mode=env.render_mode`).
- the `reset()` method doesn't take any parameter. If you want to seed the pseudo-random generator or pass options,
you should call `vec_env.seed(seed=seed)`/`vec_env.set_options(options)` and `obs = vec_env.reset()` afterward (seed and options are discarded after each call to `reset()`).
- methods and attributes of the underlying Gym envs can be accessed, called and set using `vec_env.get_attr("attribute_name")`,
`vec_env.env_method("method_name", args1, args2, kwargs1=kwargs1)` and `vec_env.set_attr("attribute_name", new_value)`.
## Modifying Vectorized Environments Attributes
If you plan to [modify the attributes of an environment](https://github.com/DLR-RM/stable-baselines3/issues/1573) while it is used (e.g., modifying an attribute specifying the task carried out for a portion of training when doing multi-task learning, or
a parameter of the environment dynamics), you must expose a setter method.
In fact, directly accessing the environment attribute in the callback can lead to unexpected behavior because environments can be wrapped (using gym or VecEnv wrappers, the `Monitor` wrapper being one example).
Consider the following example for a custom env:
```
/* Detailed source-code truncated for AI context efficiency. */
```
In this example `env.mu` cannot be accessed/changed directly because it is wrapped in a `VecEnv` and because it could be wrapped with other wrappers (see [GH#1573](https://github.com/DLR-RM/stable-baselines3/issues/1573) for a longer explanation).
Instead, the callback should use the `set_mu` method via the `env_method` method for Vectorized Environments.
```python
from itertools import cycle
class ChangeMuCallback(BaseCallback):
"""
This callback changes the value of mu during training looping
through a list of values until training is aborted.
The environment is implemented so that the impact of changing
the value of mu mid-episode is visible only after the episode is over
and the reset method has been called.
"""
def __init__(self):
super().__init__()
# An iterator that contains the different of the friction coefficient
self.mus = cycle([0.1, 0.2, 0.5, 0.13, 0.9])
def _on_step(self):
# Note: in practice, you should not change this value at every step
# but rather depending on some events/metrics like agent performance/episode termination
# both accessible via the `self.logger` or `self.locals` variables
self.training_env.env_method("set_mu", next(self.mus))
```
This callback can then be used to safely modify environment attributes during training since
it calls the environment setter method.
## Vectorized Environments Wrappers
If you want to alter or augment a `VecEnv` without redefining it completely (e.g. stack multiple frames, monitor the `VecEnv`, normalize the observation, ...), you can use `VecEnvWrapper` for that.
They are the vectorized equivalents (i.e., they act on multiple environments at the same time) of `gym.Wrapper`.
You can find below an example for extracting one key from the observation:
```python
import numpy as np
from stable_baselines3.common.vec_env.base_vec_env import VecEnv, VecEnvStepReturn, VecEnvWrapper
class VecExtractDictObs(VecEnvWrapper):
"""
A vectorized wrapper for filtering a specific key from dictionary observations.
Similar to Gym's FilterObservation wrapper:
https://github.com/openai/gym/blob/master/gym/wrappers/filter_observation.py
:param venv: The vectorized environment
:param key: The key of the dictionary observation
"""
def __init__(self, venv: VecEnv, key: str):
self.key = key
super().__init__(venv=venv, observation_space=venv.observation_space.spaces[self.key])
def reset(self) -> np.ndarray:
obs = self.venv.reset()
return obs[self.key]
def step_async(self, actions: np.ndarray) -> None:
self.venv.step_async(actions)
def step_wait(self) -> VecEnvStepReturn:
obs, rewards, dones, info = self.venv.step_wait()
return obs[self.key], rewards, dones, info
env = DummyVecEnv([lambda: gym.make("FetchReach-v1")])
# Wrap the VecEnv
env = VecExtractDictObs(env, key="observation")
```
:::{note}
When creating a vectorized environment, you can also specify ordinary gymnasium
wrappers to wrap each of the sub-environments. See the
{func}`make_vec_env `
documentation for details.
Example:
```python
from gymnasium.wrappers import RescaleAction
from stable_baselines3.common.env_util import make_vec_env
# Use gym wrapper for each sub-env of the VecEnv
wrapper_kwargs = dict(min_action=-1.0, max_action=1.0)
vec_env = make_vec_env(
"Pendulum-v1", n_envs=2, wrapper_class=RescaleAction, wrapper_kwargs=wrapper_kwargs
)
```
:::
## VecEnv
```{eval-rst}
.. autoclass:: VecEnv
:members:
```
## DummyVecEnv
```{eval-rst}
.. autoclass:: DummyVecEnv
:members:
```
## SubprocVecEnv
```{eval-rst}
.. autoclass:: SubprocVecEnv
:members:
```
## Wrappers
### VecFrameStack
```{eval-rst}
.. autoclass:: VecFrameStack
:members:
```
### StackedObservations
```{eval-rst}
.. autoclass:: stable_baselines3.common.vec_env.stacked_observations.StackedObservations
:members:
```
### VecNormalize
```{eval-rst}
.. autoclass:: VecNormalize
:members:
```
### VecVideoRecorder
```{eval-rst}
.. autoclass:: VecVideoRecorder
:members:
```
### VecCheckNan
```{eval-rst}
.. autoclass:: VecCheckNan
:members:
```
### VecTransposeImage
```{eval-rst}
.. autoclass:: VecTransposeImage
:members:
```
### VecMonitor
```{eval-rst}
.. autoclass:: VecMonitor
:members:
```
### VecExtractDictObs
```{eval-rst}
.. autoclass:: VecExtractDictObs
:members:
```
---
### Misc Changelog (`docs/misc/changelog.md`)
(changelog)=
# Changelog
## Release 2.9.2 (2026-10-26)
:::{warning}
Stable-Baselines3 (SB3) v2.9.2 will be the last one supporting Python 3.10 (end of life in October 2026).
We highly recommended you to upgrade to Python >= 3.11.
:::
### Breaking Changes:
### New Features:
### Bug Fixes:
- Fixed file descriptor leak in `SubprocVecEnv.close()` by closing parent-side pipes to prevent "too many open files" errors in long-running loops
- Fixed Pandas deprecation warnings on `inplace` operations
### [SB3-Contrib]
- Upgraded to Stable-Baselines3 >= 2.9.2
### [RL Zoo]
- Set `learning_starts` to 500 for `parking-v0` in TQC hyperparameters to ensure the first episode is finished
- Fixed "too many open files" issue when doing hyperparameter optimization: the trial environments are now closed in a `finally` block and `gc_after_trial=True` is passed to Optuna to free them between trials (also needs latest SB3 master version to fully work)
- Set `use_expln=True` for pybullet envs when using gSDE to avoid NaNs
- Add a note regarding pybullet source installation error on macOS due to legacy code
- Migrate packaging setup from `setup.py` to `pyproject.toml` and integrate `requirements.txt` as a new extra
### [SBX] (SB3 + Jax)
- Updated Jax dependency version range
### Deprecations:
### Others:
- Fixed moviepy test compatibility with moviepy >= 2.0 (PyTorch doesn't support it yet)
- Added filter for numpy `newshape` deprecation warning from torch tensorboard
- Fixed mypy error in test_vec_envs.py by wrapping `itertools.product` with `list()`
- Improved tests coverage for RMSpropTFLike optimizer (invalid params, centered/momentum/weight_decay branches, pickle roundtrip)
- Improved test coverage for `save_util.py`: added tests for `BadZipFile` error handling in `load_from_zip_file` and `IsADirectoryError`/`FileNotFoundError` handling in `open_path`
- Fixed Docker build by adding the missing `--system` flag to `uv pip uninstall opencv-python` (required by recent `uv`)
- Added a warning when `n_steps > 1` is passed together with a custom `replay_buffer_class`, as `n_steps` is only applied when the replay buffer class is selected automatically (@Koustav-github)
- Updated tests dependencies
### Documentation:
- Clarified in the `n_steps` docstring that it only applies when `replay_buffer_class` is `None`, and that a custom replay buffer must be configured with `replay_buffer_kwargs={"n_steps": ..., "gamma": ...}` (@Koustav-github)
- Updated algorithm table (added the ones from SBX and sorted the table)
- Fixed typos, grammar, and Gymnasium API inconsistencies across documentation
## Release 2.9.0 (2026-06-15)
**Updated dependencies (pandas is now optional, gymnasium 1.3.0 support, torch>=2.8)**
### Breaking Changes:
- Relaxed Gymnasium version range (from `"gymnasium>=0.29.1,<1.3.0"` to `"gymnasium>=0.29.1,<2.0"`)
- `pandas` and `matplotlib` are no longer core dependencies; they are now optional and only required for loading results and plotting (moved to `stable-baselines3[extra]`).
- Moved `read_json` and `read_csv` helper functions to test files
- Raised `torch` minimum version from 2.3 to 2.8 to mitigate https://github.com/advisories/GHSA-887c-mr87-cxwp
### New Features:
### Bug Fixes:
- Fixed deprecated error Taxi-v3 from gymnasium v1.3.0 in tests
### [SB3-Contrib]
- Optimized tests (faster to run)
- Fixed dead link for `RecurrentPPO`.
### [RL Zoo]
### [SBX] (SB3 + Jax)
- Added support for `rollout_buffer_class` and `rollout_buffer_kwargs` arguments in `PPO` and `OnPolicyAlgorithmJax` constructors, as in Stable Baselines3. (@Trenza1ore)
- Updated Jax dependency
### Deprecations:
### Others:
- Optimized tests (faster to run)
- Add workflow to automatically publish to PyPi when a new tag is created
### Documentation:
- Added example for using torch.compile
- Fixed many broken links and updated links to https whenever possible
## Release 2.8.0 (2026-04-01)
**Dropped Python 3.9, added Python 3.13 support, MaskablePPO bug fix, default hyperparams for unlisted env in the RL Zoo, Markdown doc**
### Breaking Changes:
- Removed support for Python 3.9, please upgrade to Python >= 3.10
- Set `strict=True` for every call to `zip(...)`
- Switched to `pygame-ce` when installing extras
### New Features:
- Added official support for Python 3.13
### Bug Fixes:
- Fixed saving and loading of Torch compiled models (using `th.compile()`) by updating `get_parameters()`
- Added a warning to env-checker if a multidiscrete space has multi-dimensional array (@unexploredtest)
- Fixed `pandas.concat` futurewarnings occuring when dataframes are empty by removing empty frames from the list before concatenating
### [SB3-Contrib]
- Set `strict=True` for every call to `zip(...)`
- Fixed `MaskablePPO` and `RecurrentPPO` inaccurate `n_updates` counting when `target_kl` early exits the training loop
- Fixed `RecurrentPPO` and `MaskablePPO` `forward` and `predict` not reshaping the action before clipping it (@immortal-boy)
- Do not call `forward()` method directly in `RecurrentPPO` (@immortal-boy)
- Fixed `MaskableCategorical.apply_masking()` crashing with `ValueError: Simplex` when cached `probs` deviate from sum=1 in float32 with large action spaces (torch 2.9+) (@kirann-05)
- Switched to Markdown documentation (using MyST parser)
### [RL Zoo]
- Set ``strict=True`` for every call to ``zip(...)``
- Allow to specify `env_kwargs` in the hyperparam config
- Switched to Markdown documentation (using MyST parser)
- Allow to use default hyperparameters for any environment
- Save training command in Weights & Biases (Wandb)
- Save training command and default hyperparameters as study attributes
### [SBX] (SB3 + Jax)
- Increased Jax version range and use tf-nightly
### Deprecations:
- `zip_strict()` is not needed anymore since Python 3.10, please use `zip(..., strict=True)` instead
### Others:
- Updated to Python 3.10+ annotations
- Removed some unused variables (@unexploredtest)
- Improved type hints for distributions
- Simplified zip file loading by removing Python 3.6 workaround and enabling `weights_only=True` (PyTorch 2.x)
- Sped up saving/loading tests
- Updated black from v25 to v26
- Updated monitor test to check handling of empty monitor files
### Documentation:
- Added a note on MultiDiscrete spaces with multi-dimensional arrays and a wrapper to fix the issue (@unexploredtest)
- Added an example of manual export of SBX (SB3 + Jax) model to ONNX (@m-abr)
- Switched to Markdown documentation (using MyST parser)
## Release 2.7.1 (2025-12-05)
:::{warning}
Stable-Baselines3 (SB3) v2.7.1 will be the last one supporting Python 3.9 (end of life in October 2025).
We highly recommended you to upgrade to Python >= 3.10.
:::
### Breaking Changes:
### New Features:
- `RolloutBuffer` and `DictRolloutBuffer` now uses the actual observation / action space `dtype` (instead of float32), this should save memory (@Trenza1ore)
### Bug Fixes:
- Fixed env checker to properly handle `Sequence` observation spaces when nested inside composite spaces (`Dict`, `Tuple`, `OneOf`) (@copilot)
- Update env checker to warn users when using Graph space (@dhruvmalik007).
- Fixed memory leak in `VecVideoRecorder` where `recorded_frames` stayed in memory due to reference in the moviepy clip (@copilot)
- Remove double space in `StopTrainingOnRewardThreshold` callback message (@sea-bass)
### [SB3-Contrib]
- Fixed tensorboard log name for `MaskablePPO`
### [SBX] (SB3 + Jax)
- Added `CnnPolicy` to PPO
### Documentation:
- Added plotting documentation and examples
- Added documentation clarifying gSDE (Generalized State-Dependent Exploration) inference behavior for PPO, SAC, and A2C algorithms
- Documented Atari wrapper reset behavior where `env.reset()` may perform a no-op step instead of truly resetting when `terminal_on_life_loss=True` (default), and how to avoid this behavior by setting `terminal_on_life_loss=False`
- Clarified comment in `_sample_action()` method to better explain action scaling behavior for off-policy algorithms (@copilot)
- Added sb3-plus to projects page
- Added example usage of ONNX JS
- Updated link to paper of community project DeepNetSlice (@AlexPasqua)
- Added example usage of Tensorflow JS
- Included exact versions in ONNX JS and example project
- Made step 2 (`pip install`) of `CONTRIBUTING.md` more robust
## Release 2.7.0 (2025-07-25)
**n-step returns for all off-policy algorithms**
### Breaking Changes:
### New Features:
- Added support for n-step returns for off-policy algorithms via the `n_steps` parameter
- Added `NStepReplayBuffer` that allows to compute n-step returns without additional memory requirement (and without for loops)
- Added Gymnasium v1.2 support
### Bug Fixes:
- Fixed docker GPU image (PyTorch GPU was not installed)
- Fixed segmentation faults caused by non-portable schedules during model loading (@akanto)
### [SB3-Contrib]
- Added support for n-step returns for off-policy algorithms via the `n_steps` parameter
- Use the `FloatSchedule` and `LinearSchedule` classes instead of lambdas in the ARS, PPO, and QRDQN implementations to improve model portability across different operating systems
### [RL Zoo]
- `linear_schedule` now returns a `SimpleLinearSchedule` object for better portability
- Renamed `LunarLander-v2` to `LunarLander-v3` in hyperparameters
- Renamed `CarRacing-v2` to `CarRacing-v3` in hyperparameters
- Docker GPU images are now working again
- Use `ConstantSchedule`, and `SimpleLinearSchedule` instead of `constant_fn` and `linear_schedule`
- Fixed `CarRacing-v3` hyperparameters for newer Gymnasium version
### [SBX] (SB3 + Jax)
- Added support for n-step returns for off-policy algorithms via the `n_steps` parameter
- Added KL Adaptive LR for PPO and LR schedule for SAC/TQC
### Deprecations:
- `get_schedule_fn()`, `get_linear_fn()`, `constant_fn()` are deprecated, please use `FloatSchedule()`, `LinearSchedule()`, `ConstantSchedule()` instead
### Others:
### Documentation:
- Clarify `evaluate_policy` documentation
- Added doc about training exceeding the `total_timesteps` parameter
- Updated `LunarLander` and `LunarLanderContinuous` environment versions to v3 (@j0m0k0)
- Added sb3-extra-buffers to the project page (@Trenza1ore)
## Release 2.6.0 (2025-03-24)
**New \`\`LogEveryNTimesteps\`\` callback and \`\`has_attr\`\` method, refactored hyperparameter optimization**
### Breaking Changes:
### New Features:
- Added `has_attr` method for `VecEnv` to check if an attribute exists
- Added `LogEveryNTimesteps` callback to dump logs every N timesteps (note: you need to pass `log_interval=None` to avoid any interference)
- Added Gymnasium v1.1 support
### Bug Fixes:
- `SubProcVecEnv` will now exit gracefully (without big traceback) when using `KeyboardInterrupt`
### [SB3-Contrib]
- Renamed `_dump_logs()` to `dump_logs()`
- Fixed issues with `SubprocVecEnv` and `MaskablePPO` by using `vec_env.has_attr()` (pickling issues, mask function not present)
### [RL Zoo]
- Refactored hyperparameter optimization. The Optuna [Journal storage backend](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.storages.JournalStorage.html) is now supported (recommended default) and you can easily load tuned hyperparameter via the new `--trial-id` argument of `train.py`.
- Save the exact command line used to launch a training
- Added support for special vectorized env (e.g. Brax, IsaacSim) by allowing to override the `VecEnv` class use to instantiate the env in the `ExperimentManager`
- Allow to disable auto-logging by passing `--log-interval -2` (useful when logging things manually)
- Added Gymnasium v1.1 support
- Fixed use of old HF api in `get_hf_trained_models()`
### [SBX] (SB3 + Jax)
- Updated PPO to support `net_arch`, and additional fixes
- Fixed entropy coeff wrongly logged for SAC and derivatives.
- Fixed PPO `predict()` for env that were not normalized (action spaces with limits != [-1, 1])
- PPO now logs the standard deviation
### Deprecations:
- `algo._dump_logs()` is deprecated in favor of `algo.dump_logs()` and will be removed in SB3 v2.7.0
### Others:
- Updated black from v24 to v25
- Improved error messages when checking Box space equality (loading `VecNormalize`)
- Updated test to reflect how `set_wrapper_attr` should be used now
### Documentation:
- Clarify the use of Gym wrappers with `make_vec_env` in the section on Vectorized Environments (@pstahlhofen)
- Updated callback doc for `EveryNTimesteps`
- Added doc on how to set env attributes via `VecEnv` calls
- Added ONNX export example for `MultiInputPolicy` (@darkopetrovic)
## Release 2.5.0 (2025-01-27)
**New algorithm: SimBa in SBX, NumPy 2.0 support**
### Breaking Changes:
- Increased minimum required version of PyTorch to 2.3.0
- Removed support for Python 3.8
### New Features:
- Added support for NumPy v2.0: `VecNormalize` now cast normalized rewards to float32, updated bit flipping env to avoid overflow issues too
- Added official support for Python 3.12
### [SBX] (SB3 + Jax)
- Added SimBa Policy: Simplicity Bias for Scaling Up Parameters in DRL
- Added support for parameter resets
### Others:
- Updated Dockerfile
### Documentation:
- Added Decisions and Dragons to resources. (@jmacglashan)
- Updated PyBullet example, now compatible with Gymnasium
- Added link to policies for `policy_kwargs` parameter (@kplers)
- Add FootstepNet Envs to the project page (@cgaspard3333)
- Added FRASA to the project page (@MarcDcls)
- Fixed atari example (@chrisgao99)
- Add a note about `Discrete` action spaces with `start!=0`
- Update doc for massively parallel simulators (Isaac Lab, Brax, ...)
- Add dm_control example
## Release 2.4.1 (2024-12-20)
### Bug Fixes:
- Fixed a bug introduced in v2.4.0 where the `VecVideoRecorder` would override videos
## Release 2.4.0 (2024-11-18)
**New algorithm: CrossQ in SB3 Contrib, Gymnasium v1.0 support**
:::{note}
DQN (and QR-DQN) models saved with SB3 < 2.4.0 will show a warning about
truncation of optimizer state when loaded with SB3 >= 2.4.0.
To suppress the warning, simply save the model again.
You can find more info in [PR #1963](https://github.com/DLR-RM/stable-baselines3/pull/1963)
:::
:::{warning}
Stable-Baselines3 (SB3) v2.4.0 will be the last one supporting Python 3.8 (end of life in October 2024)
and PyTorch < 2.3.
We highly recommended you to upgrade to Python >= 3.9 and PyTorch >= 2.3 (compatible with NumPy v2).
:::
### Breaking Changes:
- Increased minimum required version of Gymnasium to 0.29.1
### New Features:
- Added support for `pre_linear_modules` and `post_linear_modules` in `create_mlp` (useful for adding normalization layers, like in DroQ or CrossQ)
- Enabled np.ndarray logging for TensorBoardOutputFormat as histogram (see GH#1634) (@iwishwasaneagle)
- Updated env checker to warn users when using multi-dim array to define `MultiDiscrete` spaces
- Added support for Gymnasium v1.0
### Bug Fixes:
- Fixed memory leak when loading learner from storage, `set_parameters()` does not try to load the object data anymore
and only loads the PyTorch parameters (@peteole)
- Cast type in compute gae method to avoid error when using torch compile (@amjames)
- `CallbackList` now sets the `.parent` attribute of child callbacks to its own `.parent`. (will-maclean)
- Fixed error when loading a model that has `net_arch` manually set to `None` (@jak3122)
- Set requirement numpy\<2.0 until PyTorch is compatible ()
- Updated DQN optimizer input to only include q_network parameters, removing the target_q_network ones (@corentinlger)
- Fixed `test_buffers.py::test_device` which was not actually checking the device of tensors (@rhaps0dy)
### [SB3-Contrib]
- Added `CrossQ` algorithm, from "Batch Normalization in Deep Reinforcement Learning" paper (@danielpalen)
- Added `BatchRenorm` PyTorch layer used in `CrossQ` (@danielpalen)
- Updated QR-DQN optimizer input to only include quantile_net parameters (@corentinlger)
- Fixed loading QRDQN changes `target_update_interval` (@jak3122)
### [RL Zoo]
- Updated defaults hyperparameters for TQC/SAC for Swimmer-v4 (decrease gamma for more consistent results)
### [SBX] (SB3 + Jax)
- Added CNN support for DQN
- Bug fix for SAC and related algorithms, optimize log of ent coeff to be consistent with SB3
### Deprecations:
### Others:
- Fixed various typos (@cschindlbeck)
- Remove unnecessary SDE noise resampling in PPO update (@brn-dev)
- Updated PyTorch version on CI to 2.3.1
- Added a warning to recommend using CPU with on policy algorithms (A2C/PPO) and `MlpPolicy`
- Switched to uv to download packages faster on GitHub CI
- Updated dependencies for read the doc
- Removed unnecessary `copy_obs_dict` method for `SubprocVecEnv`, remove the use of ordered dict and rename `flatten_obs` to `stack_obs`
### Documentation:
- Updated PPO doc to recommend using CPU with `MlpPolicy`
- Clarified documentation about planned features and citing software
- Added a note about the fact we are optimizing log of ent coeff for SAC
## Release 2.3.2 (2024-04-27)
### Bug Fixes:
- Reverted `torch.load()` to be called `weights_only=False` as it caused loading issue with old version of PyTorch.
### Documentation:
- Added ER-MRL to the project page (@corentinlger)
- Updated Tensorboard Logging Videos documentation (@NickLucche)
## Release 2.3.1 (2024-04-22)
### Bug Fixes:
- Cast return value of learning rate schedule to float, to avoid issue when loading model because of `weights_only=True` (@markscsmith)
### Documentation:
- Updated SBX documentation (CrossQ and deprecated DroQ)
- Updated RL Tips and Tricks section
## Release 2.3.0 (2024-03-31)
**New defaults hyperparameters for DDPG, TD3 and DQN**
:::{warning}
Because of `weights_only=True`, this release breaks loading of policies when using PyTorch 1.13.
Please upgrade to PyTorch >= 2.0 or upgrade SB3 version (we reverted the change in SB3 2.3.2)
:::
### Breaking Changes:
- The defaults hyperparameters of `TD3` and `DDPG` have been changed to be more consistent with `SAC`
```python
# SB3 < 2.3.0 default hyperparameters
# model = TD3("MlpPolicy", env, train_freq=(1, "episode"), gradient_steps=-1, batch_size=100)
# SB3 >= 2.3.0:
model = TD3("MlpPolicy", env, train_freq=1, gradient_steps=1, batch_size=256)
```
:::{note}
Two inconsistencies remain: the default network architecture for `TD3/DDPG` is `[400, 300]` instead of `[256, 256]` for SAC (for backward compatibility reasons, see [report on the influence of the network size](https://wandb.ai/openrlbenchmark/sbx/reports/SBX-TD3-Influence-of-policy-net--Vmlldzo2NDg1Mzk3)) and the default learning rate is 1e-3 instead of 3e-4 for SAC (for performance reasons, see [W&B report on the influence of the lr](https://wandb.ai/openrlbenchmark/sbx/reports/SBX-TD3-RL-Zoo-v2-3-0a0-vs-SB3-TD3-RL-Zoo-2-2-1---Vmlldzo2MjUyNTQx))
:::
- The default `learning_starts` parameter of `DQN` have been changed to be consistent with the other offpolicy algorithms
```python
# SB3 < 2.3.0 default hyperparameters, 50_000 corresponded to Atari defaults hyperparameters
# model = DQN("MlpPolicy", env, learning_starts=50_000)
# SB3 >= 2.3.0:
model = DQN("MlpPolicy", env, learning_starts=100)
```
- For safety, `torch.load()` is now called with `weights_only=True` when loading torch tensors,
policy `load()` still uses `weights_only=False` as gymnasium imports are required for it to work
- When using `huggingface_sb3`, you will now need to set `TRUST_REMOTE_CODE=True` when downloading models from the hub, as `pickle.load` is not safe.
### New Features:
- Log success rate `rollout/success_rate` when available for on policy algorithms (@corentinlger)
### Bug Fixes:
- Fixed `monitor_wrapper` argument that was not passed to the parent class, and dones argument that wasn't passed to `_update_into_buffer` (@corentinlger)
### [SB3-Contrib]
- Added `rollout_buffer_class` and `rollout_buffer_kwargs` arguments to MaskablePPO
- Fixed `train_freq` type annotation for tqc and qrdqn (@Armandpl)
- Fixed `sb3_contrib/common/maskable/*.py` type annotations
- Fixed `sb3_contrib/ppo_mask/ppo_mask.py` type annotations
- Fixed `sb3_contrib/common/vec_env/async_eval.py` type annotations
- Add some additional notes about `MaskablePPO` (evaluation and multi-process) (@icheered)
### [RL Zoo]
- Updated defaults hyperparameters for TD3/DDPG to be more consistent with SAC
- Upgraded MuJoCo envs hyperparameters to v4 (pre-trained agents need to be updated)
- Added test dependencies to `setup.py` (@power-edge)
- Simplify dependencies of `requirements.txt` (remove duplicates from `setup.py`)
### [SBX] (SB3 + Jax)
- Added support for `MultiDiscrete` and `MultiBinary` action spaces to PPO
- Added support for large values for gradient_steps to SAC, TD3, and TQC
- Fix `train()` signature and update type hints
- Fix replay buffer device at load time
- Added flatten layer
- Added `CrossQ`
### Deprecations:
### Others:
- Updated black from v23 to v24
- Updated ruff to >= v0.3.1
- Updated env checker for (multi)discrete spaces with non-zero start.
### Documentation:
- Added a paragraph on modifying vectorized environment parameters via setters (@fracapuano)
- Updated callback code example
- Updated export to ONNX documentation, it is now much simpler to export SB3 models with newer ONNX Opset!
- Added video link to "Practical Tips for Reliable Reinforcement Learning" video
- Added `render_mode="human"` in the README example (@marekm4)
- Fixed docstring signature for sum_independent_dims (@stagoverflow)
- Updated docstring description for `log_interval` in the base class (@rushitnshah).
## Release 2.2.1 (2023-11-17)
**Support for options at reset, bug fixes and better error messages**
:::{note}
SB3 v2.2.0 was yanked after a breaking change was found in [GH#1751](https://github.com/DLR-RM/stable-baselines3/issues/1751).
Please use SB3 v2.2.1 and not v2.2.0.
:::
### Breaking Changes:
- Switched to `ruff` for sorting imports (isort is no longer needed), black and ruff version now require a minimum version
- Dropped `x is False` in favor of `not x`, which means that callbacks that wrongly returned None (instead of a boolean) will cause the training to stop (@iwishiwasaneagle)
### New Features:
- Improved error message of the `env_checker` for env wrongly detected as GoalEnv (`compute_reward()` is defined)
- Improved error message when mixing Gym API with VecEnv API (see GH#1694)
- Add support for setting `options` at reset with VecEnv via the `set_options()` method. Same as seeds logic, options are reset at the end of an episode (@ReHoss)
- Added `rollout_buffer_class` and `rollout_buffer_kwargs` arguments to on-policy algorithms (A2C and PPO)
### Bug Fixes:
- Prevents using squash_output and not use_sde in ActorCritcPolicy (@PatrickHelm)
- Performs unscaling of actions in collect_rollout in OnPolicyAlgorithm (@PatrickHelm)
- Moves VectorizedActionNoise into `_setup_learn()` in OffPolicyAlgorithm (@PatrickHelm)
- Prevents out of bound error on Windows if no seed is passed (@PatrickHelm)
- Calls `callback.update_locals()` before `callback.on_rollout_end()` in OnPolicyAlgorithm (@PatrickHelm)
- Fixed replay buffer device after loading in OffPolicyAlgorithm (@PatrickHelm)
- Fixed `render_mode` which was not properly loaded when using `VecNormalize.load()`
- Fixed success reward dtype in `SimpleMultiObsEnv` (@NixGD)
- Fixed check_env for Sequence observation space (@corentinlger)
- Prevents instantiating BitFlippingEnv with conflicting observation spaces (@kylesayrs)
- Fixed ResourceWarning when loading and saving models (files were not closed), please note that only path are closed automatically,
the behavior stay the same for tempfiles (they need to be closed manually),
the behavior is now consistent when loading/saving replay buffer
### [SB3-Contrib]
- Added `set_options` for `AsyncEval`
- Added `rollout_buffer_class` and `rollout_buffer_kwargs` arguments to TRPO
### [RL Zoo]
- Removed `gym` dependency, the package is still required for some pretrained agents.
- Added `--eval-env-kwargs` to `train.py` (@Quentin18)
- Added `ppo_lstm` to hyperparams_opt.py (@technocrat13)
- Upgraded to `pybullet_envs_gymnasium>=0.4.0`
- Removed old hacks (for instance limiting offpolicy algorithms to one env at test time)
- Updated docker image, removed support for X server
- Replaced deprecated `optuna.suggest_uniform(...)` by `optuna.suggest_float(..., low=..., high=...)`
### [SBX] (SB3 + Jax)
- Added `DDPG` and `TD3` algorithms
### Deprecations:
### Others:
- Fixed `stable_baselines3/common/callbacks.py` type hints
- Fixed `stable_baselines3/common/utils.py` type hints
- Fixed `stable_baselines3/common/vec_envs/vec_transpose.py` type hints
- Fixed `stable_baselines3/common/vec_env/vec_video_recorder.py` type hints
- Fixed `stable_baselines3/common/save_util.py` type hints
- Updated docker images to Ubuntu Jammy using micromamba 1.5
- Fixed `stable_baselines3/common/buffers.py` type hints
- Fixed `stable_baselines3/her/her_replay_buffer.py` type hints
- Buffers do no call an additional `.copy()` when storing new transitions
- Fixed `ActorCriticPolicy.extract_features()` signature by adding an optional `features_extractor` argument
- Update dependencies (accept newer Shimmy/Sphinx version and remove `sphinx_autodoc_typehints`)
- Fixed `stable_baselines3/common/off_policy_algorithm.py` type hints
- Fixed `stable_baselines3/common/distributions.py` type hints
- Fixed `stable_baselines3/common/vec_env/vec_normalize.py` type hints
- Fixed `stable_baselines3/common/vec_env/__init__.py` type hints
- Switched to PyTorch 2.1.0 in the CI (fixes type annotations)
- Fixed `stable_baselines3/common/policies.py` type hints
- Switched to `mypy` only for checking types
- Added tests to check consistency when saving/loading files
### Documentation:
- Updated RL Tips and Tricks (include recommendation for evaluation, added links to DroQ, ARS and SBX).
- Fixed various typos and grammar mistakes
- Added PokemonRedExperiments to the project page
- Fixed an out-of-date command for installing Atari in examples
## Release 2.1.0 (2023-08-17)
**Float64 actions , Gymnasium 0.29 support and bug fixes**
### Breaking Changes:
- Removed Python 3.7 support
- SB3 now requires PyTorch >= 1.13
### New Features:
- Added Python 3.11 support
- Added Gymnasium 0.29 support (@pseudo-rnd-thoughts)
### [SB3-Contrib]
- Fixed MaskablePPO ignoring `stats_window_size` argument
- Added Python 3.11 support
### [RL Zoo]
- Upgraded to Huggingface-SB3 >= 2.3
- Added Python 3.11 support
### Bug Fixes:
- Relaxed check in logger, that was causing issue on Windows with colorama
- Fixed off-policy algorithms with continuous float64 actions (see #1145) (@tobirohrer)
- Fixed `env_checker.py` warning messages for out of bounds in complex observation spaces (@Gabo-Tor)
### Deprecations:
### Others:
- Updated GitHub issue templates
- Fix typo in gym patch error message (@lukashass)
- Refactor `test_spaces.py` tests
### Documentation:
- Fixed callback example (@BertrandDecoster)
- Fixed policy network example (@kyle-he)
- Added mobile-env as new community project (@stefanbschneider)
- Added \[DeepNetSlice\]() to community projects (@AlexPasqua)
## Release 2.0.0 (2023-06-22)
**Gymnasium support**
:::{warning}
Stable-Baselines3 (SB3) v2.0 will be the last one supporting python 3.7 (end of life in June 2023).
We highly recommended you to upgrade to Python >= 3.8.
:::
### Breaking Changes:
- Switched to Gymnasium as primary backend, Gym 0.21 and 0.26 are still supported via the `shimmy` package (@carlosluis, @arjun-kg, @tlpss)
- The deprecated `online_sampling` argument of `HerReplayBuffer` was removed
- Removed deprecated `stack_observation_space` method of `StackedObservations`
- Renamed environment output observations in `evaluate_policy` to prevent shadowing the input observations during callbacks (@npit)
- Upgraded wrappers and custom environment to Gymnasium
- Refined the `HumanOutputFormat` file check: now it verifies if the object is an instance of `io.TextIOBase` instead of only checking for the presence of a `write` method.
- Because of new Gym API (0.26+), the random seed passed to `vec_env.seed(seed=seed)` will only be effective after then `env.reset()` call.
### New Features:
- Added Gymnasium support (Gym 0.21 and 0.26 are supported via the `shimmy` package)
### [SB3-Contrib]
- Fixed QRDQN update interval for multi envs
### [RL Zoo]
- Gym 0.26+ patches to continue working with pybullet and TimeLimit wrapper
- Renamed `CarRacing-v1` to `CarRacing-v2` in hyperparameters
- Huggingface push to hub now accepts a `--n-timesteps` argument to adjust the length of the video
- Fixed `record_video` steps (before it was stepping in a closed env)
- Dropped Gym 0.21 support
### Bug Fixes:
- Fixed `VecExtractDictObs` does not handle terminal observation (@WeberSamuel)
- Set NumPy version to `>=1.20` due to use of `numpy.typing` (@troiganto)
- Fixed loading DQN changes `target_update_interval` (@tobirohrer)
- Fixed env checker to properly reset the env before calling `step()` when checking
for `Inf` and `NaN` (@lutogniew)
- Fixed HER `truncate_last_trajectory()` (@lbergmann1)
- Fixed HER desired and achieved goal order in reward computation (@JonathanKuelz)
### Deprecations:
### Others:
- Fixed `stable_baselines3/a2c/*.py` type hints
- Fixed `stable_baselines3/ppo/*.py` type hints
- Fixed `stable_baselines3/sac/*.py` type hints
- Fixed `stable_baselines3/td3/*.py` type hints
- Fixed `stable_baselines3/common/base_class.py` type hints
- Fixed `stable_baselines3/common/logger.py` type hints
- Fixed `stable_baselines3/common/envs/*.py` type hints
- Fixed `stable_baselines3/common/vec_env/vec_monitor|vec_extract_dict_obs|util.py` type hints
- Fixed `stable_baselines3/common/vec_env/base_vec_env.py` type hints
- Fixed `stable_baselines3/common/vec_env/vec_frame_stack.py` type hints
- Fixed `stable_baselines3/common/vec_env/dummy_vec_env.py` type hints
- Fixed `stable_baselines3/common/vec_env/subproc_vec_env.py` type hints
- Upgraded docker images to use mamba/micromamba and CUDA 11.7
- Updated env checker to reflect what subset of Gymnasium is supported and improve GoalEnv checks
- Improve type annotation of wrappers
- Tests envs are now checked too
- Added render test for `VecEnv` and `VecEnvWrapper`
- Update issue templates and env info saved with the model
- Changed `seed()` method return type from `List` to `Sequence`
- Updated env checker doc and requirements for tuple spaces/goal envs
### Documentation:
- Added Deep RL Course link to the Deep RL Resources page
- Added documentation about `VecEnv` API vs Gym API
- Upgraded tutorials to Gymnasium API
- Make it more explicit when using `VecEnv` vs Gym env
- Added UAV_Navigation_DRL_AirSim to the project page (@heleidsn)
- Added `EvalCallback` example (@sidney-tio)
- Update custom env documentation
- Added `pink-noise-rl` to projects page
- Fix custom policy example, `ortho_init` was ignored
- Added SBX page
## Release 1.8.0 (2023-04-07)
**Multi-env HerReplayBuffer, Open RL Benchmark, Improved env checker**
:::{warning}
Stable-Baselines3 (SB3) v1.8.0 will be the last one to use Gym as a backend.
Starting with v2.0.0, Gymnasium will be the default backend (though SB3 will have compatibility layers for Gym envs).
You can find a migration guide here: .
If you want to try the SB3 v2.0 alpha version, you can take a look at [PR #1327](https://github.com/DLR-RM/stable-baselines3/pull/1327).
:::
### Breaking Changes:
- Removed shared layers in `mlp_extractor` (@AlexPasqua)
- Refactored `StackedObservations` (it now handles dict obs, `StackedDictObservations` was removed)
- You must now explicitly pass a `features_extractor` parameter when calling `extract_features()`
- Dropped offline sampling for `HerReplayBuffer`
- As `HerReplayBuffer` was refactored to support multiprocessing, previous replay buffer are incompatible with this new version
- `HerReplayBuffer` doesn't require a `max_episode_length` anymore
### New Features:
- Added `repeat_action_probability` argument in `AtariWrapper`.
- Only use `NoopResetEnv` and `MaxAndSkipEnv` when needed in `AtariWrapper`
- Added support for dict/tuple observations spaces for `VecCheckNan`, the check is now active in the `env_checker()` (@DavyMorgan)
- Added multiprocessing support for `HerReplayBuffer`
- `HerReplayBuffer` now supports all datatypes supported by `ReplayBuffer`
- Provide more helpful failure messages when validating the `observation_space` of custom gym environments using `check_env` (@FieteO)
- Added `stats_window_size` argument to control smoothing in rollout logging (@jonasreiher)
### [SB3-Contrib]
- Added warning about potential crashes caused by `check_env` in the `MaskablePPO` docs (@AlexPasqua)
- Fixed `sb3_contrib/qrdqn/*.py` type hints
- Removed shared layers in `mlp_extractor` (@AlexPasqua)
### [RL Zoo]
- [Open RL Benchmark](https://github.com/openrlbenchmark/openrlbenchmark/issues/7)
- Upgraded to new `HerReplayBuffer` implementation that supports multiple envs
- Removed `TimeFeatureWrapper` for Panda and Fetch envs, as the new replay buffer should handle timeout.
- Tuned hyperparameters for RecurrentPPO on Swimmer
- Documentation is now built using Sphinx and hosted on read the doc
- Removed `use_auth_token` for push to hub util
- Reverted from v3 to v2 for HumanoidStandup, Reacher, InvertedPendulum and InvertedDoublePendulum since they were not part of the mujoco refactoring (see )
- Fixed `gym-minigrid` policy (from `MlpPolicy` to `MultiInputPolicy`)
- Replaced deprecated `optuna.suggest_loguniform(...)` by `optuna.suggest_float(..., log=True)`
- Switched to `ruff` and `pyproject.toml`
- Removed `online_sampling` and `max_episode_length` argument when using `HerReplayBuffer`
### Bug Fixes:
- Fixed Atari wrapper that missed the reset condition (@luizapozzobon)
- Added the argument `dtype` (default to `float32`) to the noise for consistency with gym action (@sidney-tio)
- Fixed PPO train/n_updates metric not accounting for early stopping (@adamfrly)
- Fixed loading of normalized image-based environments
- Fixed `DictRolloutBuffer.add` with multidimensional action space (@younik)
### Deprecations:
### Others:
- Fixed `tests/test_tensorboard.py` type hint
- Fixed `tests/test_vec_normalize.py` type hint
- Fixed `stable_baselines3/common/monitor.py` type hint
- Added tests for StackedObservations
- Removed Gitlab CI file
- Moved from `setup.cg` to `pyproject.toml` configuration file
- Switched from `flake8` to `ruff`
- Upgraded AutoROM to latest version
- Fixed `stable_baselines3/dqn/*.py` type hints
- Added `extra_no_roms` option for package installation without Atari Roms
### Documentation:
- Renamed `load_parameters` to `set_parameters` (@DavyMorgan)
- Clarified documentation about subproc multiprocessing for A2C (@Bonifatius94)
- Fixed typo in `A2C` docstring (@AlexPasqua)
- Renamed timesteps to episodes for `log_interval` description (@theSquaredError)
- Removed note about gif creation for Atari games (@harveybellini)
- Added information about default network architecture
- Update information about Gymnasium support
## Release 1.7.0 (2023-01-10)
:::{warning}
Shared layers in MLP policy (`mlp_extractor`) are now deprecated for PPO, A2C and TRPO.
This feature will be removed in SB3 v1.8.0 and the behavior of `net_arch=[64, 64]`
will create **separate** networks with the same architecture, to be consistent with the off-policy algorithms.
:::
:::{note}
A2C and PPO saved with SB3 < 1.7.0 will show a warning about
missing keys in the state dict when loaded with SB3 >= 1.7.0.
To suppress the warning, simply save the model again.
You can find more info in [issue #1233](https://github.com/DLR-RM/stable-baselines3/issues/1233)
:::
### Breaking Changes:
- Removed deprecated `create_eval_env`, `eval_env`, `eval_log_path`, `n_eval_episodes` and `eval_freq` parameters,
please use an `EvalCallback` instead
- Removed deprecated `sde_net_arch` parameter
- Removed `ret` attributes in `VecNormalize`, please use `returns` instead
- `VecNormalize` now updates the observation space when normalizing images
### New Features:
- Introduced mypy type checking
- Added option to have non-shared features extractor between actor and critic in on-policy algorithms (@AlexPasqua)
- Added `with_bias` argument to `create_mlp`
- Added support for multidimensional `spaces.MultiBinary` observations
- Features extractors now properly support unnormalized image-like observations (3D tensor)
when passing `normalize_images=False`
- Added `normalized_image` parameter to `NatureCNN` and `CombinedExtractor`
- Added support for Python 3.10
### [SB3-Contrib]
- Fixed a bug in `RecurrentPPO` where the lstm states where incorrectly reshaped for `n_lstm_layers > 1` (thanks @kolbytn)
- Fixed `RuntimeError: rnn: hx is not contiguous` while predicting terminal values for `RecurrentPPO` when `n_lstm_layers > 1`
### [RL Zoo]
- Added support for python file for configuration
- Added `monitor_kwargs` parameter
### Bug Fixes:
- Fixed `ProgressBarCallback` under-reporting (@dominicgkerr)
- Fixed return type of `evaluate_actions` in `ActorCritcPolicy` to reflect that entropy is an optional tensor (@Rocamonde)
- Fixed type annotation of `policy` in `BaseAlgorithm` and `OffPolicyAlgorithm`
- Allowed model trained with Python 3.7 to be loaded with Python 3.8+ without the `custom_objects` workaround
- Raise an error when the same gym environment instance is passed as separate environments when creating a vectorized environment with more than one environment. (@Rocamonde)
- Fix type annotation of `model` in `evaluate_policy`
- Fixed `Self` return type using `TypeVar`
- Fixed the env checker, the key was not passed when checking images from Dict observation space
- Fixed `normalize_images` which was not passed to parent class in some cases
- Fixed `load_from_vector` that was broken with newer PyTorch version when passing PyTorch tensor
### Deprecations:
- You should now explicitly pass a `features_extractor` parameter when calling `extract_features()`
- Deprecated shared layers in `MlpExtractor` (@AlexPasqua)
### Others:
- Used issue forms instead of issue templates
- Updated the PR template to associate each PR with its peer in RL-Zoo3 and SB3-Contrib
- Fixed flake8 config to be compatible with flake8 6+
- Goal-conditioned environments are now characterized by the availability of the `compute_reward` method, rather than by their inheritance to `gym.GoalEnv`
- Replaced `CartPole-v0` by `CartPole-v1` is tests
- Fixed `tests/test_distributions.py` type hints
- Fixed `stable_baselines3/common/type_aliases.py` type hints
- Fixed `stable_baselines3/common/torch_layers.py` type hints
- Fixed `stable_baselines3/common/env_util.py` type hints
- Fixed `stable_baselines3/common/preprocessing.py` type hints
- Fixed `stable_baselines3/common/atari_wrappers.py` type hints
- Fixed `stable_baselines3/common/vec_env/vec_check_nan.py` type hints
- Exposed modules in `__init__.py` with the `__all__` attribute (@ZikangXiong)
- Upgraded GitHub CI/setup-python to v4 and checkout to v3
- Set tensors construction directly on the device (~8% speed boost on GPU)
- Monkey-patched `np.bool = bool` so gym 0.21 is compatible with NumPy 1.24+
- Standardized the use of `from gym import spaces`
- Modified `get_system_info` to avoid issue linked to copy-pasting on GitHub issue
### Documentation:
- Updated Hugging Face Integration page (@simoninithomas)
- Changed `env` to `vec_env` when environment is vectorized
- Updated custom policy docs to better explain the `mlp_extractor`'s dimensions (@AlexPasqua)
- Updated custom policy documentation (@athatheo)
- Improved tensorboard callback doc
- Clarify doc when using image-like input
- Added RLeXplore to the project page (@yuanmingqi)
## Release 1.6.2 (2022-10-10)
**Progress bar in the learn() method, RL Zoo3 is now a package**
### Breaking Changes:
### New Features:
- Added `progress_bar` argument in the `learn()` method, displayed using TQDM and rich packages
- Added progress bar callback
- The [RL Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) can now be installed as a package (`pip install rl_zoo3`)
### [SB3-Contrib]
### [RL Zoo]
- RL Zoo is now a python package and can be installed using `pip install rl_zoo3`
### Bug Fixes:
- `self.num_timesteps` was initialized properly only after the first call to `on_step()` for callbacks
- Set importlib-metadata version to `~=4.13` to be compatible with `gym=0.21`
### Deprecations:
- Added deprecation warning if parameters `eval_env`, `eval_freq` or `create_eval_env` are used (see #925) (@tobirohrer)
### Others:
- Fixed type hint of the `env_id` parameter in `make_vec_env` and `make_atari_env` (@AlexPasqua)
### Documentation:
- Extended docstring of the `wrapper_class` parameter in `make_vec_env` (@AlexPasqua)
## Release 1.6.1 (2022-09-29)
**Bug fix release**
### Breaking Changes:
- Switched minimum tensorboard version to 2.9.1
### New Features:
- Support logging hyperparameters to tensorboard (@timothe-chaumont)
- Added checkpoints for replay buffer and `VecNormalize` statistics (@anand-bala)
- Added option for `Monitor` to append to existing file instead of overriding (@sidney-tio)
- The env checker now raises an error when using dict observation spaces and observation keys don't match observation space keys
### [SB3-Contrib]
- Fixed the issue of wrongly passing policy arguments when using `CnnLstmPolicy` or `MultiInputLstmPolicy` with `RecurrentPPO` (@mlodel)
### Bug Fixes:
- Fixed issue where `PPO` gives NaN if rollout buffer provides a batch of size 1 (@hughperkins)
- Fixed the issue that `predict` does not always return action as `np.ndarray` (@qgallouedec)
- Fixed division by zero error when computing FPS when a small number of time has elapsed in operating systems with low-precision timers.
- Added multidimensional action space support (@qgallouedec)
- Fixed missing verbose parameter passing in the `EvalCallback` constructor (@burakdmb)
- Fixed the issue that when updating the target network in DQN, SAC, TD3, the `running_mean` and `running_var` properties of batch norm layers are not updated (@honglu2875)
- Fixed incorrect type annotation of the replay_buffer_class argument in `common.OffPolicyAlgorithm` initializer, where an instance instead of a class was required (@Rocamonde)
- Fixed loading saved model with different number of environments
- Removed `forward()` abstract method declaration from `common.policies.BaseModel` (already defined in `torch.nn.Module`) to fix type errors in subclasses (@Rocamonde)
- Fixed the return type of `.load()` and `.learn()` methods in `BaseAlgorithm` so that they now use `TypeVar` (@Rocamonde)
- Fixed an issue where keys with different tags but the same key raised an error in `common.logger.HumanOutputFormat` (@Rocamonde and @AdamGleave)
- Set importlib-metadata version to `~=4.13`
### Deprecations:
### Others:
- Fixed `DictReplayBuffer.next_observations` typing (@qgallouedec)
- Added support for `device="auto"` in buffers and made it default (@qgallouedec)
- Updated `ResultsWriter` (used internally by `Monitor` wrapper) to automatically create missing directories when `filename` is a path (@dominicgkerr)
### Documentation:
- Added an example of callback that logs hyperparameters to tensorboard. (@timothe-chaumont)
- Fixed typo in docstring "nature" -> "Nature" (@Melanol)
- Added info on split tensorboard logs into (@Melanol)
- Fixed typo in ppo doc (@francescoluciano)
- Fixed typo in install doc(@jlp-ue)
- Clarified and standardized verbosity documentation
- Added link to a GitHub issue in the custom policy documentation (@AlexPasqua)
- Update doc on exporting models (fixes and added torch jit)
- Fixed typos (@Akhilez)
- Standardized the use of `"` for string representation in documentation
## Release 1.6.0 (2022-07-11)
**Recurrent PPO (PPO LSTM), better defaults for learning from pixels with SAC/TD3**
### Breaking Changes:
- Changed the way policy "aliases" are handled ("MlpPolicy", "CnnPolicy", ...), removing the former
`register_policy` helper, `policy_base` parameter and using `policy_aliases` static attributes instead (@Gregwar)
- SB3 now requires PyTorch >= 1.11
- Changed the default network architecture when using `CnnPolicy` or `MultiInputPolicy` with SAC or DDPG/TD3,
`share_features_extractor` is now set to False by default and the `net_arch=[256, 256]` (instead of `net_arch=[]` that was before)
### New Features:
### [SB3-Contrib]
- Added Recurrent PPO (PPO LSTM). See
### Bug Fixes:
- Fixed saving and loading large policies greater than 2GB (@jkterry1, @ycheng517)
- Fixed final goal selection strategy that did not sample the final achieved goal (@qgallouedec)
- Fixed a bug with special characters in the tensorboard log name (@quantitative-technologies)
- Fixed a bug in `DummyVecEnv`'s and `SubprocVecEnv`'s seeding function. None value was unchecked (@ScheiklP)
- Fixed a bug where `EvalCallback` would crash when trying to synchronize `VecNormalize` stats when observation normalization was disabled
- Added a check for unbounded actions
- Fixed issues due to newer version of protobuf (tensorboard) and sphinx
- Fix exception causes all over the codebase (@cool-RR)
- Prohibit simultaneous use of optimize_memory_usage and handle_timeout_termination due to a bug (@MWeltevrede)
- Fixed a bug in `kl_divergence` check that would fail when using numpy arrays with MultiCategorical distribution
### Deprecations:
### Others:
- Upgraded to Python 3.7+ syntax using `pyupgrade`
- Removed redundant double-check for nested observations from `BaseAlgorithm._wrap_env` (@TibiGG)
### Documentation:
- Added link to gym doc and gym env checker
- Fix typo in PPO doc (@bcollazo)
- Added link to PPO ICLR blog post
- Added remark about breaking Markov assumption and timeout handling
- Added doc about MLFlow integration via custom logger (@git-thor)
- Updated Huggingface integration doc
- Added copy button for code snippets
- Added doc about EnvPool and Isaac Gym support
## Release 1.5.0 (2022-03-25)
**Bug fixes, early stopping callback**
### Breaking Changes:
- Switched minimum Gym version to 0.21.0
### New Features:
- Added `StopTrainingOnNoModelImprovement` to callback collection (@caburu)
- Makes the length of keys and values in `HumanOutputFormat` configurable,
depending on desired maximum width of output.
- Allow PPO to turn of advantage normalization (see [PR #763](https://github.com/DLR-RM/stable-baselines3/pull/763)) @vwxyzjn
### [SB3-Contrib]
- coming soon: Cross Entropy Method, see
### Bug Fixes:
- Fixed a bug in `VecMonitor`. The monitor did not consider the `info_keywords` during stepping (@ScheiklP)
- Fixed a bug in `HumanOutputFormat`. Distinct keys truncated to the same prefix would overwrite each others value,
resulting in only one being output. This now raises an error (this should only affect a small fraction of use cases
with very long keys.)
- Routing all the `nn.Module` calls through implicit rather than explicit forward as per pytorch guidelines (@manuel-delverme)
- Fixed a bug in `VecNormalize` where error occurs when `norm_obs` is set to False for environment with dictionary observation (@buoyancy99)
- Set default `env` argument to `None` in `HerReplayBuffer.sample` (@qgallouedec)
- Fix `batch_size` typing in `DQN` (@qgallouedec)
- Fixed sample normalization in `DictReplayBuffer` (@qgallouedec)
### Deprecations:
### Others:
- Fixed pytest warnings
- Removed parameter `remove_time_limit_termination` in off policy algorithms since it was dead code (@Gregwar)
### Documentation:
- Added doc on Hugging Face integration (@simoninithomas)
- Added furuta pendulum project to project list (@armandpl)
- Fix indentation 2 spaces to 4 spaces in custom env documentation example (@Gautam-J)
- Update MlpExtractor docstring (@gianlucadecola)
- Added explanation of the logger output
- Update `Directly Accessing The Summary Writer` in tensorboard integration (@xy9485)
## Release 1.4.0 (2022-01-18)
*TRPO, ARS and multi env training for off-policy algorithms*
### Breaking Changes:
- Dropped python 3.6 support (as announced in previous release)
- Renamed `mask` argument of the `predict()` method to `episode_start` (used with RNN policies only)
- local variables `action`, `done` and `reward` were renamed to their plural form for offpolicy algorithms (`actions`, `dones`, `rewards`),
this may affect custom callbacks.
- Removed `episode_reward` field from `RolloutReturn()` type
:::{warning}
An update to the `HER` algorithm is planned to support multi-env training and remove the max episode length constrain.
(see [PR #704](https://github.com/DLR-RM/stable-baselines3/pull/704))
This will be a backward incompatible change (model trained with previous version of `HER` won't work with the new version).
:::
### New Features:
- Added `norm_obs_keys` param for `VecNormalize` wrapper to configure which observation keys to normalize (@kachayev)
- Added experimental support to train off-policy algorithms with multiple envs (note: `HerReplayBuffer` currently not supported)
- Handle timeout termination properly for on-policy algorithms (when using `TimeLimit`)
- Added `skip` option to `VecTransposeImage` to skip transforming the channel order when the heuristic is wrong
- Added `copy()` and `combine()` methods to `RunningMeanStd`
### [SB3-Contrib]
- Added Trust Region Policy Optimization (TRPO) (@cyprienc)
- Added Augmented Random Search (ARS) (@sgillen)
- Coming soon: PPO LSTM, see
### Bug Fixes:
- Fixed a bug where `set_env()` with `VecNormalize` would result in an error with off-policy algorithms (thanks @cleversonahum)
- FPS calculation is now performed based on number of steps performed during last `learn` call, even when `reset_num_timesteps` is set to `False` (@kachayev)
- Fixed evaluation script for recurrent policies (experimental feature in SB3 contrib)
- Fixed a bug where the observation would be incorrectly detected as non-vectorized instead of throwing an error
- The env checker now properly checks and warns about potential issues for continuous action spaces when the boundaries are too small or when the dtype is not float32
- Fixed a bug in `VecFrameStack` with channel first image envs, where the terminal observation would be wrongly created.
### Deprecations:
### Others:
- Added a warning in the env checker when not using `np.float32` for continuous actions
- Improved test coverage and error message when checking shape of observation
- Added `newline="\n"` when opening CSV monitor files so that each line ends with `\r\n` instead of `\r\r\n` on Windows while Linux environments are not affected (@hsuehch)
- Fixed `device` argument inconsistency (@qgallouedec)
### Documentation:
- Add drivergym to projects page (@theDebugger811)
- Add highway-env to projects page (@eleurent)
- Add tactile-gym to projects page (@ac-93)
- Fix indentation in the RL tips page (@cove9988)
- Update GAE computation docstring
- Add documentation on exporting to TFLite/Coral
- Added JMLR paper and updated citation
- Added link to RL Tips and Tricks video
- Updated `BaseAlgorithm.load` docstring (@Demetrio92)
- Added a note on `load` behavior in the examples (@Demetrio92)
- Updated SB3 Contrib doc
- Fixed A2C and migration guide guidance on how to set epsilon with RMSpropTFLike (@thomasgubler)
- Fixed custom policy documentation (@IperGiove)
- Added doc on Weights & Biases integration
## Release 1.3.0 (2021-10-23)
*Bug fixes and improvements for the user*
:::{warning}
This version will be the last one supporting Python 3.6 (end of life in Dec 2021).
We highly recommended you to upgrade to Python >= 3.7.
:::
### Breaking Changes:
- `sde_net_arch` argument in policies is deprecated and will be removed in a future version.
- `_get_latent` (`ActorCriticPolicy`) was removed
- All logging keys now use underscores instead of spaces (@timokau). Concretely this changes:
> - `time/total timesteps` to `time/total_timesteps` for off-policy algorithms (PPO and A2C) and the eval callback (on-policy algorithms already used the underscored version),
> - `rollout/exploration rate` to `rollout/exploration_rate` and
> - `rollout/success rate` to `rollout/success_rate`.
### New Features:
- Added methods `get_distribution` and `predict_values` for `ActorCriticPolicy` for A2C/PPO/TRPO (@cyprienc)
- Added methods `forward_actor` and `forward_critic` for `MlpExtractor`
- Added `sb3.get_system_info()` helper function to gather version information relevant to SB3 (e.g., Python and PyTorch version)
- Saved models now store system information where agent was trained, and load functions have `print_system_info` parameter to help debugging load issues
### Bug Fixes:
- Fixed `dtype` of observations for `SimpleMultiObsEnv`
- Allow `VecNormalize` to wrap discrete-observation environments to normalize reward
when observation normalization is disabled
- Fixed a bug where `DQN` would throw an error when using `Discrete` observation and stochastic actions
- Fixed a bug where sub-classed observation spaces could not be used
- Added `force_reset` argument to `load()` and `set_env()` in order to be able to call `learn(reset_num_timesteps=False)` with a new environment
### Deprecations:
### Others:
- Cap gym max version to 0.19 to avoid issues with atari-py and other breaking changes
- Improved error message when using dict observation with the wrong policy
- Improved error message when using `EvalCallback` with two envs not wrapped the same way.
- Added additional infos about supported python version for PyPi in `setup.py`
### Documentation:
- Add Rocket League Gym to list of supported projects (@AechPro)
- Added gym-electric-motor to project page (@wkirgsn)
- Added policy-distillation-baselines to project page (@CUN-bjy)
- Added ONNX export instructions (@batu)
- Update read the doc env (fixed `docutils` issue)
- Fix PPO environment name (@IljaAvadiev)
- Fix custom env doc and add env registration example
- Update algorithms from SB3 Contrib
- Use underscores for numeric literals in examples to improve clarity
## Release 1.2.0 (2021-09-03)
**Hotfix for VecNormalize, training/eval mode support**
### Breaking Changes:
- SB3 now requires PyTorch >= 1.8.1
- `VecNormalize` `ret` attribute was renamed to `returns`
### New Features:
### Bug Fixes:
- Hotfix for `VecNormalize` where the observation filter was not updated at reset (thanks @vwxyzjn)
- Fixed model predictions when using batch normalization and dropout layers by calling `train()` and `eval()` (@davidblom603)
- Fixed model training for DQN, TD3 and SAC so that their target nets always remain in evaluation mode (@ayeright)
- Passing `gradient_steps=0` to an off-policy algorithm will result in no gradient steps being taken (vs as many gradient steps as steps done in the environment
during the rollout in previous versions)
### Deprecations:
### Others:
- Enabled Python 3.9 in GitHub CI
- Fixed type annotations
- Refactored `predict()` by moving the preprocessing to `obs_to_tensor()` method
### Documentation:
- Updated multiprocessing example
- Added example of `VecEnvWrapper`
- Added a note about logging to tensorboard more often
- Added warning about simplicity of examples and link to RL zoo (@MihaiAnca13)
## Release 1.1.0 (2021-07-01)
**Dict observation support, timeout handling and refactored HER buffer**
### Breaking Changes:
- All customs environments (e.g. the `BitFlippingEnv` or `IdentityEnv`) were moved to `stable_baselines3.common.envs` folder
- Refactored `HER` which is now the `HerReplayBuffer` class that can be passed to any off-policy algorithm
- Handle timeout termination properly for off-policy algorithms (when using `TimeLimit`)
- Renamed `_last_dones` and `dones` to `_last_episode_starts` and `episode_starts` in `RolloutBuffer`.
- Removed `ObsDictWrapper` as `Dict` observation spaces are now supported
```python
her_kwargs = dict(n_sampled_goal=2, goal_selection_strategy="future", online_sampling=True)
# SB3 < 1.1.0
# model = HER("MlpPolicy", env, model_class=SAC, **her_kwargs)
# SB3 >= 1.1.0:
model = SAC("MultiInputPolicy", env, replay_buffer_class=HerReplayBuffer, replay_buffer_kwargs=her_kwargs)
```
- Updated the KL Divergence estimator in the PPO algorithm to be positive definite and have lower variance (@09tangriro)
- Updated the KL Divergence check in the PPO algorithm to be before the gradient update step rather than after end of epoch (@09tangriro)
- Removed parameter `channels_last` from `is_image_space` as it can be inferred.
- The logger object is now an attribute `model.logger` that be set by the user using `model.set_logger()`
- Changed the signature of `logger.configure` and `utils.configure_logger`, they now return a `Logger` object
- Removed `Logger.CURRENT` and `Logger.DEFAULT`
- Moved `warn(), debug(), log(), info(), dump()` methods to the `Logger` class
- `.learn()` now throws an import error when the user tries to log to tensorboard but the package is not installed
### New Features:
- Added support for single-level `Dict` observation space (@JadenTravnik)
- Added `DictRolloutBuffer` `DictReplayBuffer` to support dictionary observations (@JadenTravnik)
- Added `StackedObservations` and `StackedDictObservations` that are used within `VecFrameStack`
- Added simple 4x4 room Dict test environments
- `HerReplayBuffer` now supports `VecNormalize` when `online_sampling=False`
- Added [VecMonitor](https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/vec_env/vec_monitor.py) and
[VecExtractDictObs](https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/vec_env/vec_extract_dict_obs.py) wrappers
to handle gym3-style vectorized environments (@vwxyzjn)
- Ignored the terminal observation if the it is not provided by the environment
such as the gym3-style vectorized environments. (@vwxyzjn)
- Added policy_base as input to the OnPolicyAlgorithm for more flexibility (@09tangriro)
- Added support for image observation when using `HER`
- Added `replay_buffer_class` and `replay_buffer_kwargs` arguments to off-policy algorithms
- Added `kl_divergence` helper for `Distribution` classes (@09tangriro)
- Added support for vector environments with `num_envs > 1` (@benblack769)
- Added `wrapper_kwargs` argument to `make_vec_env` (@amy12xx)
### Bug Fixes:
- Fixed potential issue when calling off-policy algorithms with default arguments multiple times (the size of the replay buffer would be the same)
- Fixed loading of `ent_coef` for `SAC` and `TQC`, it was not optimized anymore (thanks @Atlis)
- Fixed saving of `A2C` and `PPO` policy when using gSDE (thanks @liusida)
- Fixed a bug where no output would be shown even if `verbose>=1` after passing `verbose=0` once
- Fixed observation buffers dtype in DictReplayBuffer (@c-rizz)
- Fixed EvalCallback tensorboard logs being logged with the incorrect timestep. They are now written with the timestep at which they were recorded. (@skandermoalla)
### Deprecations:
### Others:
- Added `flake8-bugbear` to tests dependencies to find likely bugs
- Updated `env_checker` to reflect support of dict observation spaces
- Added Code of Conduct
- Added tests for GAE and lambda return computation
- Updated distribution entropy test (thanks @09tangriro)
- Added sanity check `batch_size > 1` in PPO to avoid NaN in advantage normalization
### Documentation:
- Added gym pybullet drones project (@JacopoPan)
- Added link to SuperSuit in projects (@justinkterry)
- Fixed DQN example (thanks @ltbd78)
- Clarified channel-first/channel-last recommendation
- Update sphinx environment installation instructions (@tom-doerr)
- Clarified pip installation in Zsh (@tom-doerr)
- Clarified return computation for on-policy algorithms (TD(lambda) estimate was used)
- Added example for using `ProcgenEnv`
- Added note about advanced custom policy example for off-policy algorithms
- Fixed DQN unicode checkmarks
- Updated migration guide (@juancroldan)
- Pinned `docutils==0.16` to avoid issue with rtd theme
- Clarified callback `save_freq` definition
- Added doc on how to pass a custom logger
- Remove recurrent policies from `A2C` docs (@bstee615)
## Release 1.0 (2021-03-15)
**First Major Version**
### Breaking Changes:
- Removed `stable_baselines3.common.cmd_util` (already deprecated), please use `env_util` instead
:::{warning}
A refactoring of the `HER` algorithm is planned together with support for dictionary observations
(see [PR #243](https://github.com/DLR-RM/stable-baselines3/pull/243) and [#351](https://github.com/DLR-RM/stable-baselines3/pull/351))
This will be a backward incompatible change (model trained with previous version of `HER` won't work with the new version).
:::
### New Features:
- Added support for `custom_objects` when loading models
### Bug Fixes:
- Fixed a bug with `DQN` predict method when using `deterministic=False` with image space
### Documentation:
- Fixed examples
- Added new project using SB3: rl_reach (@PierreExeter)
- Added note about slow-down when switching to PyTorch
- Add a note on continual learning and resetting environment
### Others:
- Updated RL-Zoo to reflect the fact that is it more than a collection of trained agents
- Added images to illustrate the training loop and custom policies (created with )
- Updated the custom policy section
## Pre-Release 0.11.1 (2021-02-27)
### Bug Fixes:
- Fixed a bug where `train_freq` was not properly converted when loading a saved model
## Pre-Release 0.11.0 (2021-02-27)
### Breaking Changes:
- `evaluate_policy` now returns rewards/episode lengths from a `Monitor` wrapper if one is present,
this allows to return the unnormalized reward in the case of Atari games for instance.
- Renamed `common.vec_env.is_wrapped` to `common.vec_env.is_vecenv_wrapped` to avoid confusion
with the new `is_wrapped()` helper
- Renamed `_get_data()` to `_get_constructor_parameters()` for policies (this affects independent saving/loading of policies)
- Removed `n_episodes_rollout` and merged it with `train_freq`, which now accepts a tuple `(frequency, unit)`:
- `replay_buffer` in `collect_rollout` is no more optional
```python
# SB3 < 0.11.0
# model = SAC("MlpPolicy", env, n_episodes_rollout=1, train_freq=-1)
# SB3 >= 0.11.0:
model = SAC("MlpPolicy", env, train_freq=(1, "episode"))
```
### New Features:
- Add support for `VecFrameStack` to stack on first or last observation dimension, along with
automatic check for image spaces.
- `VecFrameStack` now has a `channels_order` argument to tell if observations should be stacked
on the first or last observation dimension (originally always stacked on last).
- Added `common.env_util.is_wrapped` and `common.env_util.unwrap_wrapper` functions for checking/unwrapping
an environment for specific wrapper.
- Added `env_is_wrapped()` method for `VecEnv` to check if its environments are wrapped
with given Gym wrappers.
- Added `monitor_kwargs` parameter to `make_vec_env` and `make_atari_env`
- Wrap the environments automatically with a `Monitor` wrapper when possible.
- `EvalCallback` now logs the success rate when available (`is_success` must be present in the info dict)
- Added new wrappers to log images and matplotlib figures to tensorboard. (@zampanteymedio)
- Add support for text records to `Logger`. (@lorenz-h)
### Bug Fixes:
- Fixed bug where code added VecTranspose on channel-first image environments (thanks @qxcv)
- Fixed `DQN` predict method when using single `gym.Env` with `deterministic=False`
- Fixed bug that the arguments order of `explained_variance()` in `ppo.py` and `a2c.py` is not correct (@thisray)
- Fixed bug where full `HerReplayBuffer` leads to an index error. (@megan-klaiber)
- Fixed bug where replay buffer could not be saved if it was too big (> 4 Gb) for python\<3.8 (thanks @hn2)
- Added informative `PPO` construction error in edge-case scenario where `n_steps * n_envs = 1` (size of rollout buffer),
which otherwise causes downstream breaking errors in training (@decodyng)
- Fixed discrete observation space support when using multiple envs with A2C/PPO (thanks @ardabbour)
- Fixed a bug for TD3 delayed update (the update was off-by-one and not delayed when `train_freq=1`)
- Fixed numpy warning (replaced `np.bool` with `bool`)
- Fixed a bug where `VecNormalize` was not normalizing the terminal observation
- Fixed a bug where `VecTranspose` was not transposing the terminal observation
- Fixed a bug where the terminal observation stored in the replay buffer was not the right one for off-policy algorithms
- Fixed a bug where `action_noise` was not used when using `HER` (thanks @ShangqunYu)
### Deprecations:
### Others:
- Add more issue templates
- Add signatures to callable type annotations (@ernestum)
- Improve error message in `NatureCNN`
- Added checks for supported action spaces to improve clarity of error messages for the user
- Renamed variables in the `train()` method of `SAC`, `TD3` and `DQN` to match SB3-Contrib.
- Updated docker base image to Ubuntu 18.04
- Set tensorboard min version to 2.2.0 (earlier version are apparently not working with PyTorch)
- Added warning for `PPO` when `n_steps * n_envs` is not a multiple of `batch_size` (last mini-batch truncated) (@decodyng)
- Removed some warnings in the tests
### Documentation:
- Updated algorithm table
- Minor docstring improvements regarding rollout (@stheid)
- Fix migration doc for `A2C` (epsilon parameter)
- Fix `clip_range` docstring
- Fix duplicated parameter in `EvalCallback` docstring (thanks @tfederico)
- Added example of learning rate schedule
- Added SUMO-RL as example project (@LucasAlegre)
- Fix docstring of classes in atari_wrappers.py which were inside the constructor (@LucasAlegre)
- Added SB3-Contrib page
- Fix bug in the example code of DQN (@AptX395)
- Add example on how to access the tensorboard summary writer directly. (@lorenz-h)
- Updated migration guide
- Updated custom policy doc (separate policy architecture recommended)
- Added a note about OpenCV headless version
- Corrected typo on documentation (@mschweizer)
- Provide the environment when loading the model in the examples (@lorepieri8)
## Pre-Release 0.10.0 (2020-10-28)
**HER with online and offline sampling, bug fixes for features extraction**
### Breaking Changes:
- **Warning:** Renamed `common.cmd_util` to `common.env_util` for clarity (affects `make_vec_env` and `make_atari_env` functions)
### New Features:
- Allow custom actor/critic network architectures using `net_arch=dict(qf=[400, 300], pi=[64, 64])` for off-policy algorithms (SAC, TD3, DDPG)
- Added Hindsight Experience Replay `HER`. (@megan-klaiber)
- `VecNormalize` now supports `gym.spaces.Dict` observation spaces
- Support logging videos to Tensorboard (@SwamyDev)
- Added `share_features_extractor` argument to `SAC` and `TD3` policies
### Bug Fixes:
- Fix GAE computation for on-policy algorithms (off-by one for the last value) (thanks @Wovchena)
- Fixed potential issue when loading a different environment
- Fix ignoring the exclude parameter when recording logs using json, csv or log as logging format (@SwamyDev)
- Make `make_vec_env` support the `env_kwargs` argument when using an env ID str (@ManifoldFR)
- Fix model creation initializing CUDA even when `device="cpu"` is provided
- Fix `check_env` not checking if the env has a Dict actionspace before calling `_check_nan` (@wmmc88)
- Update the check for spaces unsupported by Stable Baselines 3 to include checks on the action space (@wmmc88)
- Fixed features extractor bug for target network where the same net was shared instead
of being separate. This bug affects `SAC`, `DDPG` and `TD3` when using `CnnPolicy` (or custom features extractor)
- Fixed a bug when passing an environment when loading a saved model with a `CnnPolicy`, the passed env was not wrapped properly
(the bug was introduced when implementing `HER` so it should not be present in previous versions)
### Deprecations:
### Others:
- Improved typing coverage
- Improved error messages for unsupported spaces
- Added `.vscode` to the gitignore
### Documentation:
- Added first draft of migration guide
- Added intro to [imitation](https://github.com/HumanCompatibleAI/imitation) library (@shwang)
- Enabled doc for `CnnPolicies`
- Added advanced saving and loading example
- Added base doc for exporting models
- Added example for getting and setting model parameters
## Pre-Release 0.9.0 (2020-10-03)
**Bug fixes, get/set parameters and improved docs**
### Breaking Changes:
- Removed `device` keyword argument of policies; use `policy.to(device)` instead. (@qxcv)
- Rename `BaseClass.get_torch_variables` -> `BaseClass._get_torch_save_params` and `BaseClass.excluded_save_params` -> `BaseClass._excluded_save_params`
- Renamed saved items `tensors` to `pytorch_variables` for clarity
- `make_atari_env`, `make_vec_env` and `set_random_seed` must be imported with (and not directly from `stable_baselines3.common`):
```python
from stable_baselines3.common.cmd_util import make_atari_env, make_vec_env
from stable_baselines3.common.utils import set_random_seed
```
### New Features:
- Added `unwrap_vec_wrapper()` to `common.vec_env` to extract `VecEnvWrapper` if needed
- Added `StopTrainingOnMaxEpisodes` to callback collection (@xicocaio)
- Added `device` keyword argument to `BaseAlgorithm.load()` (@liorcohen5)
- Callbacks have access to rollout collection locals as in SB2. (@PartiallyTyped)
- Added `get_parameters` and `set_parameters` for accessing/setting parameters of the agent
- Added actor/critic loss logging for TD3. (@mloo3)
### Bug Fixes:
- Added `unwrap_vec_wrapper()` to `common.vec_env` to extract `VecEnvWrapper` if needed
- Fixed a bug where the environment was reset twice when using `evaluate_policy`
- Fix logging of `clip_fraction` in PPO (@diditforlulz273)
- Fixed a bug where cuda support was wrongly checked when passing the GPU index, e.g., `device="cuda:0"` (@liorcohen5)
- Fixed a bug when the random seed was not properly set on cuda when passing the GPU index
### Deprecations:
### Others:
- Improve typing coverage of the `VecEnv`
- Fix type annotation of `make_vec_env` (@ManifoldFR)
- Removed `AlreadySteppingError` and `NotSteppingError` that were not used
- Fixed typos in SAC and TD3
- Reorganized functions for clarity in `BaseClass` (save/load functions close to each other, private
functions at top)
- Clarified docstrings on what is saved and loaded to/from files
- Simplified `save_to_zip_file` function by removing duplicate code
- Store library version along with the saved models
- DQN loss is now logged
### Documentation:
- Added `StopTrainingOnMaxEpisodes` details and example (@xicocaio)
- Updated custom policy section (added custom features extractor example)
- Re-enable `sphinx_autodoc_typehints`
- Updated doc style for type hints and remove duplicated type hints
## Pre-Release 0.8.0 (2020-08-03)
**DQN, DDPG, bug fixes and performance matching for Atari games**
### Breaking Changes:
- `AtariWrapper` and other Atari wrappers were updated to match SB2 ones
- `save_replay_buffer` now receives as argument the file path instead of the folder path (@tirafesi)
- Refactored `Critic` class for `TD3` and `SAC`, it is now called `ContinuousCritic`
and has an additional parameter `n_critics`
- `SAC` and `TD3` now accept an arbitrary number of critics (e.g. `policy_kwargs=dict(n_critics=3)`)
instead of only 2 previously
### New Features:
- Added `DQN` Algorithm (@Artemis-Skade)
- Buffer dtype is now set according to action and observation spaces for `ReplayBuffer`
- Added warning when allocation of a buffer may exceed the available memory of the system
when `psutil` is available
- Saving models now automatically creates the necessary folders and raises appropriate warnings (@PartiallyTyped)
- Refactored opening paths for saving and loading to use strings, pathlib or io.BufferedIOBase (@PartiallyTyped)
- Added `DDPG` algorithm as a special case of `TD3`.
- Introduced `BaseModel` abstract parent for `BasePolicy`, which critics inherit from.
### Bug Fixes:
- Fixed a bug in the `close()` method of `SubprocVecEnv`, causing wrappers further down in the wrapper stack to not be closed. (@NeoExtended)
- Fix target for updating q values in SAC: the entropy term was not conditioned by terminals states
- Use `cloudpickle.load` instead of `pickle.load` in `CloudpickleWrapper`. (@shwang)
- Fixed a bug with orthogonal initialization when `bias=False` in custom policy (@rk37)
- Fixed approximate entropy calculation in PPO and A2C. (@andyshih12)
- Fixed DQN target network sharing features extractor with the main network.
- Fixed storing correct `dones` in on-policy algorithm rollout collection. (@andyshih12)
- Fixed number of filters in final convolutional layer in NatureCNN to match original implementation.
### Deprecations:
### Others:
- Refactored off-policy algorithm to share the same `.learn()` method
- Split the `collect_rollout()` method for off-policy algorithms
- Added `_on_step()` for off-policy base class
- Optimized replay buffer size by removing the need of `next_observations` numpy array
- Optimized polyak updates (1.5-1.95 speedup) through inplace operations (@PartiallyTyped)
- Switch to `black` codestyle and added `make format`, `make check-codestyle` and `commit-checks`
- Ignored errors from newer pytype version
- Added a check when using `gSDE`
- Removed codacy dependency from Dockerfile
- Added `common.sb2_compat.RMSpropTFLike` optimizer, which corresponds closer to the implementation of RMSprop from Tensorflow.
### Documentation:
- Updated notebook links
- Fixed a typo in the section of Enjoy a Trained Agent, in RL Baselines3 Zoo README. (@blurLake)
- Added Unity reacher to the projects page (@koulakis)
- Added PyBullet colab notebook
- Fixed typo in PPO example code (@joeljosephjin)
- Fixed typo in custom policy doc (@RaphaelWag)
## Pre-Release 0.7.0 (2020-06-10)
**Hotfix for PPO/A2C + gSDE, internal refactoring and bug fixes**
### Breaking Changes:
- `render()` method of `VecEnvs` now only accept one argument: `mode`
- Created new file common/torch_layers.py, similar to SB refactoring
- Contains all PyTorch network layer definitions and features extractors: `MlpExtractor`, `create_mlp`, `NatureCNN`
- Renamed `BaseRLModel` to `BaseAlgorithm` (along with offpolicy and onpolicy variants)
- Moved on-policy and off-policy base algorithms to `common/on_policy_algorithm.py` and `common/off_policy_algorithm.py`, respectively.
- Moved `PPOPolicy` to `ActorCriticPolicy` in common/policies.py
- Moved `PPO` (algorithm class) into `OnPolicyAlgorithm` (`common/on_policy_algorithm.py`), to be shared with A2C
- Moved following functions from `BaseAlgorithm`:
- `_load_from_file` to `load_from_zip_file` (save_util.py)
- `_save_to_file_zip` to `save_to_zip_file` (save_util.py)
- `safe_mean` to `safe_mean` (utils.py)
- `check_env` to `check_for_correct_spaces` (utils.py. Renamed to avoid confusion with environment checker tools)
- Moved static function `_is_vectorized_observation` from common/policies.py to common/utils.py under name `is_vectorized_observation`.
- Removed `{save,load}_running_average` functions of `VecNormalize` in favor of `load/save`.
- Removed `use_gae` parameter from `RolloutBuffer.compute_returns_and_advantage`.
### New Features:
### Bug Fixes:
- Fixed `render()` method for `VecEnvs`
- Fixed `seed()` method for `SubprocVecEnv`
- Fixed loading on GPU for testing when using gSDE and `deterministic=False`
- Fixed `register_policy` to allow re-registering same policy for same sub-class (i.e. assign same value to same key).
- Fixed a bug where the gradient was passed when using `gSDE` with `PPO`/`A2C`, this does not affect `SAC`
### Deprecations:
### Others:
- Re-enable unsafe `fork` start method in the tests (was causing a deadlock with tensorflow)
- Added a test for seeding `SubprocVecEnv` and rendering
- Fixed reference in NatureCNN (pointed to older version with different network architecture)
- Fixed comments saying "CxWxH" instead of "CxHxW" (same style as in torch docs / commonly used)
- Added bit further comments on register/getting policies ("MlpPolicy", "CnnPolicy").
- Renamed `progress` (value from 1 in start of training to 0 in end) to `progress_remaining`.
- Added `policies.py` files for A2C/PPO, which define MlpPolicy/CnnPolicy (renamed ActorCriticPolicies).
- Added some missing tests for `VecNormalize`, `VecCheckNan` and `PPO`.
### Documentation:
- Added a paragraph on "MlpPolicy"/"CnnPolicy" and policy naming scheme under "Developer Guide"
- Fixed second-level listing in changelog
## Pre-Release 0.6.0 (2020-06-01)
**Tensorboard support, refactored logger**
### Breaking Changes:
- Remove State-Dependent Exploration (SDE) support for `TD3`
- Methods were renamed in the logger:
- `logkv` -> `record`, `writekvs` -> `write`, `writeseq` -> `write_sequence`,
- `logkvs` -> `record_dict`, `dumpkvs` -> `dump`,
- `getkvs` -> `get_log_dict`, `logkv_mean` -> `record_mean`,
### New Features:
- Added env checker (Sync with Stable Baselines)
- Added `VecCheckNan` and `VecVideoRecorder` (Sync with Stable Baselines)
- Added determinism tests
- Added `cmd_util` and `atari_wrappers`
- Added support for `MultiDiscrete` and `MultiBinary` observation spaces (@rolandgvc)
- Added `MultiCategorical` and `Bernoulli` distributions for PPO/A2C (@rolandgvc)
- Added support for logging to tensorboard (@rolandgvc)
- Added `VectorizedActionNoise` for continuous vectorized environments (@PartiallyTyped)
- Log evaluation in the `EvalCallback` using the logger
### Bug Fixes:
- Fixed a bug that prevented model trained on cpu to be loaded on gpu
- Fixed version number that had a new line included
- Fixed weird seg fault in docker image due to FakeImageEnv by reducing screen size
- Fixed `sde_sample_freq` that was not taken into account for SAC
- Pass logger module to `BaseCallback` otherwise they cannot write in the one used by the algorithms
### Deprecations:
### Others:
- Renamed to Stable-Baseline3
- Added Dockerfile
- Sync `VecEnvs` with Stable-Baselines
- Update requirement: `gym>=0.17`
- Added `.readthedoc.yml` file
- Added `flake8` and `make lint` command
- Added Github workflow
- Added warning when passing both `train_freq` and `n_episodes_rollout` to Off-Policy Algorithms
### Documentation:
- Added most documentation (adapted from Stable-Baselines)
- Added link to CONTRIBUTING.md in the README (@kinalmehta)
- Added gSDE project and update docstrings accordingly
- Fix `TD3` example code block
## Pre-Release 0.5.0 (2020-05-05)
**CnnPolicy support for image observations, complete saving/loading for policies**
### Breaking Changes:
- Previous loading of policy weights is broken and replace by the new saving/loading for policy
### New Features:
- Added `optimizer_class` and `optimizer_kwargs` to `policy_kwargs` in order to easily
customizer optimizers
- Complete independent save/load for policies
- Add `CnnPolicy` and `VecTransposeImage` to support images as input
### Bug Fixes:
- Fixed `reset_num_timesteps` behavior, so `env.reset()` is not called if `reset_num_timesteps=True`
- Fixed `squashed_output` that was not pass to policy constructor for `SAC` and `TD3` (would result in scaled actions for unscaled action spaces)
### Deprecations:
### Others:
- Cleanup rollout return
- Added `get_device` util to manage PyTorch devices
- Added type hints to logger + use f-strings
### Documentation:
## Pre-Release 0.4.0 (2020-02-14)
**Proper pre-processing, independent save/load for policies**
### Breaking Changes:
- Removed CEMRL
- Model saved with previous versions cannot be loaded (because of the pre-preprocessing)
### New Features:
- Add support for `Discrete` observation spaces
- Add saving/loading for policy weights, so the policy can be used without the model
### Bug Fixes:
- Fix type hint for activation functions
### Deprecations:
### Others:
- Refactor handling of observation and action spaces
- Refactored features extraction to have proper preprocessing
- Refactored action distributions
## Pre-Release 0.3.0 (2020-02-14)
**Bug fixes, sync with Stable-Baselines, code cleanup**
### Breaking Changes:
- Removed default seed
- Bump dependencies (PyTorch and Gym)
- `predict()` now returns a tuple to match Stable-Baselines behavior
### New Features:
- Better logging for `SAC` and `PPO`
### Bug Fixes:
- Synced callbacks with Stable-Baselines
- Fixed colors in `results_plotter`
- Fix entropy computation (now summed over action dim)
### Others:
- SAC with SDE now sample only one matrix
- Added `clip_mean` parameter to SAC policy
- Buffers now return `NamedTuple`
- More typing
- Add test for `expln`
- Renamed `learning_rate` to `lr_schedule`
- Add `version.txt`
- Add more tests for distribution
### Documentation:
- Deactivated `sphinx_autodoc_typehints` extension
## Pre-Release 0.2.0 (2020-02-14)
**Python 3.6+ required, type checking, callbacks, doc build**
### Breaking Changes:
- Python 2 support was dropped, Stable Baselines3 now requires Python 3.6 or above
- Return type of `evaluation.evaluate_policy()` has been changed
- Refactored the replay buffer to avoid transformation between PyTorch and NumPy
- Created `OffPolicyRLModel` base class
- Remove deprecated JSON format for `Monitor`
### New Features:
- Add `seed()` method to `VecEnv` class
- Add support for Callback (cf )
- Add methods for saving and loading replay buffer
- Add `extend()` method to the buffers
- Add `get_vec_normalize_env()` to `BaseRLModel` to retrieve `VecNormalize` wrapper when it exists
- Add `results_plotter` from Stable Baselines
- Improve `predict()` method to handle different type of observations (single, vectorized, ...)
### Bug Fixes:
- Fix loading model on CPU that were trained on GPU
- Fix `reset_num_timesteps` that was not used
- Fix entropy computation for squashed Gaussian (approximate it now)
- Fix seeding when using multiple environments (different seed per env)
### Others:
- Add type check
- Converted all format string to f-strings
- Add test for `OrnsteinUhlenbeckActionNoise`
- Add type aliases in `common.type_aliases`
### Documentation:
- fix documentation build
## Pre-Release 0.1.0 (2020-01-20)
**First Release: base algorithms and state-dependent exploration**
### New Features:
- Initial release of A2C, CEM-RL, PPO, SAC and TD3, working only with `Box` input space
- State-Dependent Exploration (SDE) for A2C, PPO, SAC and TD3
## Maintainers
Stable-Baselines3 is currently maintained by [Antonin Raffin] (aka [@araffin]), [Ashley Hill] (aka @hill-a),
[Maximilian Ernestus] (aka @ernestum), [Adam Gleave] ([@AdamGleave]), [Anssi Kanervisto] (aka [@Miffyli])
and [Quentin Gallouédec] (aka @qgallouedec).
## Contributors:
In random order...
Thanks to the maintainers of V2: @hill-a @ernestum @AdamGleave @Miffyli
And all the contributors:
@taymuur @bjmuld @iambenzo @iandanforth @r7vme @brendenpetersen @huvar @abhiskk @JohannesAck
@EliasHasle @mrakgr @Bleyddyn @antoine-galataud @junhyeokahn @AdamGleave @keshaviyengar @tperol
@XMaster96 @kantneel @Pastafarianist @GerardMaggiolino @PatrickWalter214 @yutingsz @sc420 @Aaahh @billtubbs
@Miffyli @dwiel @miguelrass @qxcv @jaberkow @eavelardev @ruifeng96150 @pedrohbtp @srivatsankrishnan @evilsocket
@MarvineGothic @jdossgollin @stheid @SyllogismRXS @rusu24edward @jbulow @Antymon @seheevic @justinkterry @edbeeching
@flodorner @KuKuXia @NeoExtended @PartiallyTyped @mmcenta @richardwu @kinalmehta @rolandgvc @tkelestemur @mloo3
@tirafesi @blurLake @koulakis @joeljosephjin @shwang @rk37 @andyshih12 @RaphaelWag @xicocaio
@diditforlulz273 @liorcohen5 @ManifoldFR @mloo3 @SwamyDev @wmmc88 @megan-klaiber @thisray
@tfederico @hn2 @LucasAlegre @AptX395 @zampanteymedio @fracapuano @JadenTravnik @decodyng @ardabbour @lorenz-h @mschweizer @lorepieri8 @vwxyzjn
@ShangqunYu @PierreExeter @JacopoPan @ltbd78 @tom-doerr @Atlis @liusida @09tangriro @amy12xx @juancroldan
@benblack769 @bstee615 @c-rizz @skandermoalla @MihaiAnca13 @davidblom603 @ayeright @cyprienc
@wkirgsn @AechPro @CUN-bjy @batu @IljaAvadiev @timokau @kachayev @cleversonahum
@eleurent @ac-93 @cove9988 @theDebugger811 @hsuehch @Demetrio92 @thomasgubler @IperGiove @ScheiklP
@simoninithomas @armandpl @manuel-delverme @Gautam-J @gianlucadecola @buoyancy99 @caburu @xy9485
@Gregwar @ycheng517 @quantitative-technologies @bcollazo @git-thor @TibiGG @cool-RR @MWeltevrede
@carlosluis @arjun-kg @tlpss @JonathanKuelz @Gabo-Tor @iwishiwasaneagle
@Melanol @qgallouedec @francescoluciano @jlp-ue @burakdmb @timothe-chaumont @honglu2875
@anand-bala @hughperkins @sidney-tio @AlexPasqua @dominicgkerr @Akhilez @Rocamonde @tobirohrer @ZikangXiong @ReHoss
@DavyMorgan @luizapozzobon @Bonifatius94 @theSquaredError @harveybellini @DavyMorgan @FieteO @jonasreiher @npit @WeberSamuel @troiganto
@lutogniew @lbergmann1 @lukashass @BertrandDecoster @pseudo-rnd-thoughts @stefanbschneider @kyle-he @PatrickHelm @corentinlger
@marekm4 @stagoverflow @rushitnshah @markscsmith @NickLucche @cschindlbeck @peteole @jak3122 @will-maclean
@brn-dev @jmacglashan @kplers @MarcDcls @chrisgao99 @pstahlhofen @akanto @Trenza1ore @JonathanColetti @unexploredtest @Koustav-github
@m-abr
[@adamgleave]: https://github.com/adamgleave
[@araffin]: https://github.com/araffin
[@miffyli]: https://github.com/Miffyli
[@qgallouedec]: https://github.com/qgallouedec
[adam gleave]: https://gleave.me/
[anssi kanervisto]: https://github.com/Miffyli
[antonin raffin]: https://araffin.github.io/
[ashley hill]: https://github.com/hill-a
[maximilian ernestus]: https://github.com/ernestum
[quentin gallouédec]: https://github.com/qgallouedec
[rl zoo]: https://github.com/DLR-RM/rl-baselines3-zoo
[sb3-contrib]: https://github.com/Stable-Baselines-Team/stable-baselines3-contrib
[sbx]: https://github.com/araffin/sbx
---
### Misc Projects (`docs/misc/projects.md`)
(projects)=
# Projects
This is a list of projects using stable-baselines3.
Please tell us, if you want your project to appear on this page ;)
## DriverGym
An open-source Gym-compatible environment specifically tailored for developing RL algorithms for autonomous driving. DriverGym provides access to more than 1000 hours of expert logged data and also supports reactive and data-driven agent behavior. The performance of an RL policy can be easily validated using an extensive and flexible closed-loop evaluation protocol. We also provide behavior cloning baselines using supervised learning and RL, trained in DriverGym.
Authors: Parth Kothari, Christian Perone, Luca Bergamini, Alexandre Alahi, Peter Ondruska
Github:
Paper:
## RL Reach
A platform for running reproducible reinforcement learning experiments for customizable robotic reaching tasks. This self-contained and straightforward toolbox allows its users to quickly investigate and identify optimal training configurations.
Authors: Pierre Aumjaud, David McAuliffe, Francisco Javier Rodríguez Lera, Philip Cardiff
Github:
Paper:
## Generalized State Dependent Exploration for Deep Reinforcement Learning in Robotics
An exploration method to train RL agent directly on real robots.
It was the starting point of Stable-Baselines3.
Author: Antonin Raffin, Freek Stulp
Github:
Paper:
## Furuta Pendulum Robot
Everything you need to build and train a rotary inverted pendulum, also known as a furuta pendulum! This makes use of gSDE listed above.
The Github repository contains code, CAD files and a bill of materials for you to build the robot. You can watch [a video overview of the project here](https://www.youtube.com/watch?v=Y6FVBbqjR40).
Authors: Armand du Parc Locmaria, Pierre Fabre
Github:
## Reacher
A solution to the second project of the Udacity deep reinforcement learning course.
It is an example of:
- wrapping single and multi-agent Unity environments to make them usable in Stable-Baselines3
- creating experimentation scripts which train and run A2C, PPO, TD3 and SAC models (a better choice for this one is )
- generating several pre-trained models which solve the reacher environment
Author: Marios Koulakis
Github:
## SUMO-RL
A simple interface to instantiate RL environments with SUMO for Traffic Signal Control.
- Supports Multiagent RL
- Compatibility with gym.Env and popular RL libraries such as stable-baselines3 and RLlib
- Easy customization: state and reward definitions are easily modifiable
Author: Lucas Alegre
Github:
## gym-pybullet-drones
PyBullet Gym environments for single and multi-agent reinforcement learning of quadcopter control.
- Physics-based simulation for the development and test of quadcopter control.
- Compatibility with `gym.Env`, RLlib's MultiAgentEnv.
- Learning and testing script templates for stable-baselines3 and RLlib.
Author: Jacopo Panerati
Github:
Paper:
## SuperSuit
SuperSuit contains easy to use wrappers for Gym (and multi-agent PettingZoo) environments to do all forms of common preprocessing (frame stacking, converting graphical observations to greyscale, max-and-skip for Atari, etc.). It also notably includes:
-Wrappers that apply lambda functions to observations, actions, or rewards with a single line of code.
-All wrappers can be used natively on vector environments, wrappers exist to Gym environments to vectorized environments and concatenate multiple vector environments together
-A wrapper is included that allows for using regular single agent RL libraries (e.g. stable baselines) to learn simple multi-agent PettingZoo environments, explained in this tutorial:
Author: Justin Terry
GitHub:
## Rocket League Gym
A fully custom python API and C++ DLL to treat the popular game Rocket League like an OpenAI Gym environment.
- Dramatically increases the rate at which the game runs.
- Supports full configuration of initial states, observations, rewards, and terminal states.
- Supports multiple simultaneous game clients.
- Supports multi-agent training and self-play.
- Provides custom wrappers for easy use with stable-baselines3.
Authors: Lucas Emery, Matthew Allen
GitHub:
## gym-electric-motor
An OpenAI gym environment for the simulation and control of electric drive trains.
Think of Matlab/Simulink for electric motors, inverters, and load profiles, but non-graphical and open-source in Python.
`gym-electric-motor` offers a rich interface for customization, including
\- plug-and-play of different control algorithms ranging from classical controllers (like field-oriented control) up to any RL agent you can find,
\- reward shaping,
\- load profiling,
\- finite-set or continuous-set control,
\- one-phase and three-phase motors such as induction machines and permanent magnet synchronous motors, among others.
SB3 is used as an example in one of many tutorials showcasing the easy usage of `gym-electric-motor`.
Author:
[Paderborn University, LEA department](https://github.com/upb-lea)
GitHub:
SB3 Tutorial:
[Colab Link](https://colab.research.google.com/github/upb-lea/gym-electric-motor/blob/master/examples/reinforcement_learning_controllers/stable_baselines3_dqn_disc_pmsm_example.ipynb)
Paper:
[JOSS](https://joss.theoj.org/papers/10.21105/joss.02498)
,
[TNNLS](https://ieeexplore.ieee.org/document/9241851)
,
[ArXiv](https://arxiv.org/abs/1910.09434)
## policy-distillation-baselines
A PyTorch implementation of Policy Distillation for control, which has well-trained teachers via Stable Baselines3.
- `policy-distillation-baselines` provides some good examples for policy distillation in various environment and using reliable algorithms.
- All well-trained models and algorithms are compatible with Stable Baselines3.
Authors: Junyeob Baek
GitHub:
Demo:
[link](https://github.com/dion-jy/policy-distillation-baselines/issues/3#issuecomment-817730173)
## highway-env
A minimalist environment for decision-making in Autonomous Driving.
Driving policies can be trained in different scenarios, and several notebooks using SB3 are provided as examples.
Author:
[Edouard Leurent](https://edouardleurent.com)
GitHub:
Examples:
[Colab Links](https://github.com/Farama-Foundation/HighwayEnv/tree/master/scripts#using-stable-baselines3)
## tactile-gym
Suite of RL environments focused on using a simulated tactile sensor as the primary source of observations. Sim-to-Real results across 4 out of 5 proposed envs.
Author: Alex Church
GitHub:
Paper:
## RLeXplore
RLeXplore is a set of implementations of intrinsic reward driven-exploration approaches in reinforcement learning using PyTorch, which can be deployed in arbitrary algorithms in a plug-and-play manner. In particular, RLeXplore is designed to be well compatible with Stable-Baselines3, providing more stable exploration benchmarks.
- Support arbitrary RL algorithms;
- Highly modular and high expansibility;
- Keep up with the latest research progress.
Author: Mingqi Yuan
GitHub:
## UAV_Navigation_DRL_AirSim
A platform for training UAV navigation policies in complex unknown environments.
- Based on AirSim and SB3.
- An Open AI Gym env is created including kinematic models for both multirotor and fixed-wing UAVs.
- Some UE4 environments are provided to train and test the navigation policy.
Try to train your own autonomous flight policy and even transfer it to real UAVs! Have fun ^\_^!
Author: Lei He
Github:
## Pink Noise Exploration
A simple library for pink noise exploration with deterministic (DDPG / TD3) and stochastic (SAC) off-policy algorithms. Pink noise has been shown to work better than uncorrelated Gaussian noise (the default choice) and Ornstein-Uhlenbeck noise on a range of continuous control benchmark tasks. This library is designed to work with Stable Baselines3.
Authors: Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, Georg Martius
Github:
Paper:
(Oral at ICLR 2023)
## mobile-env
An open, minimalist Gymnasium environment for autonomous coordination in wireless mobile networks.
It allows simulating various scenarios with moving users in a cellular network with multiple base stations.
- Written in pure Python, easy to modify and extend, and can be installed directly via PyPI.
- Implements the standard Gymnasium interface such that it can be used with all common frameworks for reinforcement learning.
- There are examples for both single-agent and multi-agent RL using either `stable-baselines3` or Ray RLlib.
Authors: Stefan Schneider, Stefan Werner
Github:
Paper:
(2022 IEEE/IFIP Network Operations and Management Symposium (NOMS))
## DeepNetSlice
A Deep Reinforcement Learning Open-Source Toolkit for Network Slice Placement (NSP).
NSP is the problem of deciding which physical servers in a network should host the virtual network functions (VNFs) that make up a network slice, as well as managing the mapping of the virtual links between the VNFs onto the physical infrastructure.
It is a complex optimization problem, as it involves considering the requirements of the network slice and the available resources on the physical network.
The goal is generally to maximize the utilization of the physical resources while ensuring that the network slices meet their performance requirements.
The toolkit includes a customizable simulation environments, as well as some ready-to-use demos for training
intelligent agents to perform network slice placement.
Author: Alex Pasquali
Github:
Paper:
Associated Master's Thesis:
## PokemonRedExperiments
Playing Pokemon Red with Reinforcement Learning.
Author: Peter Whidden
Github:
Video:
## Evolving Reservoirs for Meta Reinforcement Learning
Meta-RL framework to optimize reservoir-like neural structures (special kind of RNNs), and integrate them to RL agents to improve their training.
It enables solving environments involving partial observability or locomotion (e.g MuJoCo), and optimizing reservoirs that can generalize to unseen tasks.
Authors: Corentin Léger, Gautier Hamon, Eleni Nisioti, Xavier Hinaut, Clément Moulin-Frier
Github:
Paper:
## FootstepNet Envs
These environments are dedicated to train efficient agents that can plan and forecast bipedal robot footsteps in order to go to a target location possibly avoiding obstacles. They are designed to be used with Reinforcement Learning (RL) algorithms.
Real world experiments were conducted during RoboCup competitions on the Sigmaban robot, a small-sized humanoid designed by the *Rhoban Team*.
Authors: Clément Gaspard, Grégoire Passault, Mélodie Daniel, Olivier Ly
Github:
Paper:
## FRASA: Fall Recovery And Stand up agent
A Deep Reinforcement Learning agent for a humanoid robot that learns to recover from falls and stand up.
The agent is trained using the MuJoCo physics engine. Real world experiments are conducted on the
Sigmaban humanoid robot, a small-sized humanoid designed by the *Rhoban Team* to compete in the RoboCup Kidsize League.
The results, detailed in the paper and the video, show that the agent is able to recover from
various external disturbances and stand up in a few seconds.
Authors: Marc Duclusaud, Clément Gaspard, Grégoire Passault, Mélodie Daniel, Olivier Ly
Github:
Paper:
Video:
## sb3-extra-buffers: RAM expansions are overrated, just compress your observations!
Reduce the memory consumption of memory buffers in Reinforcement Learning while adding minimal overhead.
Tired of reading a cool RL paper and realizing that the author is storing a **MILLION** observations in their replay buffers? Yeah me too.
This project has implemented several compressed buffer classes that replace Stable Baselines3's standard buffers like ReplayBuffer and
RolloutBuffer. With as simple as 2-5 lines of extra code and **negligible overhead**, memory usage can be reduced by more than **95%**!
Benchmark results and documentations are on Github, feel free to submit feature requests / ask how to use these buffers through issues.
Authors: Hugo Huang
Github:
Relevant project for training RL agents that play Doom with Semantic Segmentation:
## sb3-plus: Multi-Output Policy Support for Stable-Baselines3
An extension to Stable-Baselines3 that implements support for multi-output policies and dictionary action spaces.
This project provides PPO with dict action space support, enabling independent action spaces which is particularly useful
for environments requiring multiple types of actions (e.g., discrete and continuous actions). This addresses the
multi-output policy feature requested in the community and provides a practical solution for complex action scenarios.
Author: Adyson Maia
Github:
---
### Modules A2c (`docs/modules/a2c.md`)
(a2c)=
```{eval-rst}
.. automodule:: stable_baselines3.a2c
```
# A2C
A synchronous, deterministic variant of [Asynchronous Advantage Actor Critic (A3C)](https://arxiv.org/abs/1602.01783).
It uses multiple workers to avoid the use of a replay buffer.
:::{warning}
If you find training unstable or want to match performance of stable-baselines A2C, consider using
`RMSpropTFLike` optimizer from `stable_baselines3.common.sb2_compat.rmsprop_tf_like`.
You can change optimizer with `A2C(policy_kwargs=dict(optimizer_class=RMSpropTFLike, optimizer_kwargs=dict(eps=1e-5)))`.
Read more [here](https://github.com/DLR-RM/stable-baselines3/pull/110#issuecomment-663255241).
:::
## Notes
- Original paper:
- OpenAI blog post:
## Can I use?
- Recurrent policies: ❌
- Multi processing: ✔️
- Gym spaces:
| Space | Action | Observation |
| ------------- | ------ | ----------- |
| Discrete | ✔️ | ✔️ |
| Box | ✔️ | ✔️ |
| MultiDiscrete | ✔️ | ✔️ |
| MultiBinary | ✔️ | ✔️ |
| Dict | ❌ | ✔️ |
## Example
This example is only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in RL Zoo [repository](https://github.com/DLR-RM/rl-baselines3-zoo).
Train a A2C agent on `CartPole-v1` using 4 environments.
```python
from stable_baselines3 import A2C
from stable_baselines3.common.env_util import make_vec_env
# Parallel environments
vec_env = make_vec_env("CartPole-v1", n_envs=4)
model = A2C("MlpPolicy", vec_env, verbose=1)
model.learn(total_timesteps=25000)
model.save("a2c_cartpole")
del model # remove to demonstrate saving and loading
model = A2C.load("a2c_cartpole")
obs = vec_env.reset()
while True:
action, _states = model.predict(obs)
obs, rewards, dones, info = vec_env.step(action)
vec_env.render("human")
```
:::{note}
A2C is meant to be run primarily on the CPU, especially when you are not using a CNN. To improve CPU utilization, try turning off the GPU and using `SubprocVecEnv` instead of the default `DummyVecEnv`:
```python
from stable_baselines3 import A2C
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import SubprocVecEnv
if __name__=="__main__":
env = make_vec_env("CartPole-v1", n_envs=8, vec_env_cls=SubprocVecEnv)
model = A2C("MlpPolicy", env, device="cpu")
model.learn(total_timesteps=25_000)
```
For more information, see [Vectorized Environments](../guide/vec_envs.md), [Issue #1245](https://github.com/DLR-RM/stable-baselines3/issues/1245) or the [Multiprocessing notebook](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb).
:::
:::{note}
Using gSDE (Generalized State-Dependent Exploration) during inference (see [issue #1767](https://github.com/DLR-RM/stable-baselines3/issues/1767)):
When using A2C models trained with `use_sde=True`, the automatic noise resetting that occurs during training (controlled by `sde_sample_freq`) does not happen when using `model.predict()` for inference. This results in deterministic behavior even when `deterministic=False`.
For continuous control tasks, it is recommended to use deterministic behavior during inference (`deterministic=True`). If you need stochastic behavior during inference, you must manually reset the noise by calling `model.policy.reset_noise(env.num_envs)` at appropriate intervals based on your desired `sde_sample_freq`.
:::
## Results
### Atari Games
The complete learning curves are available in the [associated PR #110](https://github.com/DLR-RM/stable-baselines3/pull/110).
### PyBullet Environments
Results on the PyBullet benchmark (2M steps) using 6 seeds.
The complete learning curves are available in the [associated issue #48](https://github.com/DLR-RM/stable-baselines3/issues/48).
:::{note}
Hyperparameters from the [gSDE paper](https://arxiv.org/abs/2005.05719) were used (as they are tuned for PyBullet envs).
:::
*Gaussian* means that the unstructured Gaussian noise is used for exploration,
*gSDE* (generalized State-Dependent Exploration) is used otherwise.
| Environments | A2C | A2C | PPO | PPO |
| ------------ | ------------ | ------------ | ------------ | ----------- |
| | Gaussian | gSDE | Gaussian | gSDE |
| HalfCheetah | 2003 +/- 54 | 2032 +/- 122 | 1976 +/- 479 | 2826 +/- 45 |
| Ant | 2286 +/- 72 | 2443 +/- 89 | 2364 +/- 120 | 2782 +/- 76 |
| Hopper | 1627 +/- 158 | 1561 +/- 220 | 1567 +/- 339 | 2512 +/- 21 |
| Walker2D | 577 +/- 65 | 839 +/- 56 | 1230 +/- 147 | 2019 +/- 64 |
### How to replicate the results?
Clone the [rl-zoo repo](https://github.com/DLR-RM/rl-baselines3-zoo):
```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
```
Run the benchmark (replace `$ENV_ID` by the envs mentioned above):
```bash
python train.py --algo a2c --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```
Plot the results (here for PyBullet envs only):
```bash
python scripts/all_plots.py -a a2c -e HalfCheetah Ant Hopper Walker2D -f logs/ -o logs/a2c_results
python scripts/plot_from_file.py -i logs/a2c_results.pkl -latex -l A2C
```
## Parameters
```{eval-rst}
.. autoclass:: A2C
:members:
:inherited-members:
```
(a2c_policies)=
## A2C Policies
```{eval-rst}
.. autoclass:: MlpPolicy
:members:
:inherited-members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.common.policies.ActorCriticPolicy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: CnnPolicy
:members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.common.policies.ActorCriticCnnPolicy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: MultiInputPolicy
:members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.common.policies.MultiInputActorCriticPolicy
:members:
:noindex:
```
---
### Modules Base (`docs/modules/base.md`)
(base-algo)=
```{eval-rst}
.. automodule:: stable_baselines3.common.base_class
```
# Base RL Class
Common interface for all the RL algorithms
```{eval-rst}
.. autoclass:: BaseAlgorithm
:members:
```
```{eval-rst}
.. automodule:: stable_baselines3.common.off_policy_algorithm
```
## Base Off-Policy Class
The base RL algorithm for Off-Policy algorithm (ex: SAC/TD3)
```{eval-rst}
.. autoclass:: OffPolicyAlgorithm
:members:
```
```{eval-rst}
.. automodule:: stable_baselines3.common.on_policy_algorithm
```
## Base On-Policy Class
The base RL algorithm for On-Policy algorithm (ex: A2C/PPO)
```{eval-rst}
.. autoclass:: OnPolicyAlgorithm
:members:
```
---
### Modules Ddpg (`docs/modules/ddpg.md`)
(ddpg)=
```{eval-rst}
.. automodule:: stable_baselines3.ddpg
```
# DDPG
[Deep Deterministic Policy Gradient (DDPG)](https://spinningup.openai.com/en/latest/algorithms/ddpg.html) combines the
trick for DQN with the deterministic policy gradient, to obtain an algorithm for continuous actions.
:::{note}
As `DDPG` can be seen as a special case of its successor {ref}`TD3 `,
they share the same policies and same implementation.
:::
```{eval-rst}
.. rubric:: Available Policies
```
```{eval-rst}
.. autosummary::
:nosignatures:
MlpPolicy
CnnPolicy
MultiInputPolicy
```
## Notes
- Deterministic Policy Gradient:
- DDPG Paper:
- OpenAI Spinning Guide for DDPG:
## Can I use?
- Recurrent policies: ❌
- Multi processing: ✔️
- Gym spaces:
| Space | Action | Observation |
| ------------- | ------ | ----------- |
| Discrete | ❌ | ✔️ |
| Box | ✔️ | ✔️ |
| MultiDiscrete | ❌ | ✔️ |
| MultiBinary | ❌ | ✔️ |
| Dict | ❌ | ✔️ |
## Example
This example is only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in RL Zoo [repository](https://github.com/DLR-RM/rl-baselines3-zoo).
```python
import gymnasium as gym
import numpy as np
from stable_baselines3 import DDPG
from stable_baselines3.common.noise import NormalActionNoise, OrnsteinUhlenbeckActionNoise
env = gym.make("Pendulum-v1", render_mode="rgb_array")
# The noise objects for DDPG
n_actions = env.action_space.shape[-1]
action_noise = NormalActionNoise(mean=np.zeros(n_actions), sigma=0.1 * np.ones(n_actions))
model = DDPG("MlpPolicy", env, action_noise=action_noise, verbose=1)
model.learn(total_timesteps=10000, log_interval=10)
model.save("ddpg_pendulum")
vec_env = model.get_env()
del model # remove to demonstrate saving and loading
model = DDPG.load("ddpg_pendulum")
obs = vec_env.reset()
while True:
action, _states = model.predict(obs)
obs, rewards, dones, info = vec_env.step(action)
env.render("human")
```
## Results
### PyBullet Environments
Results on the PyBullet benchmark (1M steps) using 6 seeds.
The complete learning curves are available in the [associated issue #48](https://github.com/DLR-RM/stable-baselines3/issues/48).
:::{note}
Hyperparameters of {ref}`TD3 ` from the [gSDE paper](https://arxiv.org/abs/2005.05719) were used for `DDPG`.
:::
*Gaussian* means that the unstructured Gaussian noise is used for exploration,
*gSDE* (generalized State-Dependent Exploration) is used otherwise.
| Environments | DDPG | TD3 | SAC |
| ------------ | ------------ | ------------ | ------------ |
| | Gaussian | Gaussian | gSDE |
| HalfCheetah | 2272 +/- 69 | 2774 +/- 35 | 2984 +/- 202 |
| Ant | 1651 +/- 407 | 3305 +/- 43 | 3102 +/- 37 |
| Hopper | 1201 +/- 211 | 2429 +/- 126 | 2262 +/- 1 |
| Walker2D | 882 +/- 186 | 2063 +/- 185 | 2136 +/- 67 |
### How to replicate the results?
Clone the [rl-zoo repo](https://github.com/DLR-RM/rl-baselines3-zoo):
```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
```
Run the benchmark (replace `$ENV_ID` by the envs mentioned above):
```bash
python train.py --algo ddpg --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```
Plot the results:
```bash
python scripts/all_plots.py -a ddpg -e HalfCheetah Ant Hopper Walker2D -f logs/ -o logs/ddpg_results
python scripts/plot_from_file.py -i logs/ddpg_results.pkl -latex -l DDPG
```
## Parameters
```{eval-rst}
.. autoclass:: DDPG
:members:
:inherited-members:
```
(ddpg_policies)=
## DDPG Policies
```{eval-rst}
.. autoclass:: MlpPolicy
:members:
:inherited-members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.td3.policies.TD3Policy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: CnnPolicy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: MultiInputPolicy
:members:
:noindex:
```
---
### Modules Dqn (`docs/modules/dqn.md`)
(dqn)=
```{eval-rst}
.. automodule:: stable_baselines3.dqn
```
# DQN
[Deep Q Network (DQN)](https://arxiv.org/abs/1312.5602) builds on [Fitted Q-Iteration (FQI)](https://www.jmlr.org/papers/volume6/ernst05a/ernst05a.pdf)
and make use of different tricks to stabilize the learning with neural networks: it uses a replay buffer, a target network and gradient clipping.
```{eval-rst}
.. rubric:: Available Policies
```
```{eval-rst}
.. autosummary::
:nosignatures:
MlpPolicy
CnnPolicy
MultiInputPolicy
```
## Notes
- Original paper:
- Further reference:
- Tutorial "From Tabular Q-Learning to DQN":
:::{note}
This implementation provides only vanilla Deep Q-Learning and has no extensions such as Double-DQN, Dueling-DQN and Prioritized Experience Replay.
:::
## Can I use?
- Recurrent policies: ❌
- Multi processing: ✔️
- Gym spaces:
| Space | Action | Observation |
| ------------- | ------ | ----------- |
| Discrete | ✔️ | ✔️ |
| Box | ❌ | ✔️ |
| MultiDiscrete | ❌ | ✔️ |
| MultiBinary | ❌ | ✔️ |
| Dict | ❌ | ✔️️ |
## Example
This example is only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in RL Zoo [repository](https://github.com/DLR-RM/rl-baselines3-zoo).
```python
import gymnasium as gym
from stable_baselines3 import DQN
env = gym.make("CartPole-v1", render_mode="human")
model = DQN("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10000, log_interval=4)
model.save("dqn_cartpole")
del model # remove to demonstrate saving and loading
model = DQN.load("dqn_cartpole")
obs, info = env.reset()
while True:
action, _states = model.predict(obs, deterministic=True)
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
obs, info = env.reset()
```
## Results
### Atari Games
The complete learning curves are available in the [associated PR #110](https://github.com/DLR-RM/stable-baselines3/pull/110).
### How to replicate the results?
Clone the [rl-zoo repo](https://github.com/DLR-RM/rl-baselines3-zoo):
```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
```
Run the benchmark (replace `$ENV_ID` by the env id, for instance `BreakoutNoFrameskip-v4`):
```bash
python train.py --algo dqn --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```
Plot the results:
```bash
python scripts/all_plots.py -a dqn -e Pong Breakout -f logs/ -o logs/dqn_results
python scripts/plot_from_file.py -i logs/dqn_results.pkl -latex -l DQN
```
## Parameters
```{eval-rst}
.. autoclass:: DQN
:members:
:inherited-members:
```
(dqn_policies)=
## DQN Policies
```{eval-rst}
.. autoclass:: MlpPolicy
:members:
:inherited-members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.dqn.policies.DQNPolicy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: CnnPolicy
:members:
```
```{eval-rst}
.. autoclass:: MultiInputPolicy
:members:
```
---
### Modules Her (`docs/modules/her.md`)
(her)=
```{eval-rst}
.. automodule:: stable_baselines3.her
```
# HER
[Hindsight Experience Replay (HER)](https://arxiv.org/abs/1707.01495)
HER is an algorithm that works with off-policy methods (DQN, SAC, TD3 and DDPG for example).
HER uses the fact that even if a desired goal was not achieved, other goal may have been achieved during a rollout.
It creates "virtual" transitions by relabeling transitions (changing the desired goal) from past episodes.
:::{warning}
Starting from Stable Baselines3 v1.1.0, `HER` is no longer a separate algorithm
but a replay buffer class `HerReplayBuffer` that must be passed to an off-policy algorithm
when using `MultiInputPolicy` (to have Dict observation support).
:::
:::{warning}
HER requires the environment to follow the legacy [gym_robotics.GoalEnv interface](https://github.com/Farama-Foundation/Gymnasium-Robotics/blob/a35b1c1fa669428bf640a2c7101e66eb1627ac3a/gym_robotics/core.py#L8)
In short, the `gym.Env` must have:
\- a vectorized implementation of `compute_reward()`
\- a dictionary observation space with three keys: `observation`, `achieved_goal` and `desired_goal`
:::
:::{warning}
Because it needs access to `env.compute_reward()`
`HER` must be loaded with the env. If you just want to use the trained policy
without instantiating the environment, we recommend saving the policy only.
:::
:::{note}
Compared to other implementations, the `future` goal sampling strategy is inclusive:
the current transition can be used when re-sampling.
:::
## Notes
- Original paper:
- OpenAI paper: [Plappert et al. (2018)]
- OpenAI blog post:
## Can I use?
Please refer to the used model (DQN, QR-DQN, SAC, TQC, TD3, or DDPG) for that section.
## Example
This example is only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in RL Zoo [repository](https://github.com/DLR-RM/rl-baselines3-zoo).
```python
from stable_baselines3 import HerReplayBuffer, DDPG, DQN, SAC, TD3
from stable_baselines3.her.goal_selection_strategy import GoalSelectionStrategy
from stable_baselines3.common.envs import BitFlippingEnv
model_class = DQN # works also with SAC, DDPG and TD3
N_BITS = 15
env = BitFlippingEnv(n_bits=N_BITS, continuous=model_class in [DDPG, SAC, TD3], max_steps=N_BITS)
# Available strategies (cf paper): future, final, episode
goal_selection_strategy = "future" # equivalent to GoalSelectionStrategy.FUTURE
# Initialize the model
model = model_class(
"MultiInputPolicy",
env,
replay_buffer_class=HerReplayBuffer,
# Parameters for HER
replay_buffer_kwargs=dict(
n_sampled_goal=4,
goal_selection_strategy=goal_selection_strategy,
),
verbose=1,
)
# Train the model
model.learn(1000)
model.save("./her_bit_env")
# Because it needs access to `env.compute_reward()`
# HER must be loaded with the env
model = model_class.load("./her_bit_env", env=env)
obs, info = env.reset()
for _ in range(100):
action, _ = model.predict(obs, deterministic=True)
obs, reward, terminated, truncated, _ = env.step(action)
if terminated or truncated:
obs, info = env.reset()
```
## Results
This implementation was tested on the [parking env](https://github.com/eleurent/highway-env)
using 3 seeds.
The complete learning curves are available in the [associated PR #120](https://github.com/DLR-RM/stable-baselines3/pull/120).
### How to replicate the results?
Clone the [rl-zoo repo](https://github.com/DLR-RM/rl-baselines3-zoo):
```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
```
Run the benchmark:
```bash
python train.py --algo tqc --env parking-v0 --eval-episodes 10 --eval-freq 10000
```
Plot the results:
```bash
python scripts/all_plots.py -a tqc -e parking-v0 -f logs/ --no-million
```
## Parameters
## HER Replay Buffer
```{eval-rst}
.. autoclass:: HerReplayBuffer
:members:
:inherited-members:
```
## Goal Selection Strategies
```{eval-rst}
.. autoclass:: GoalSelectionStrategy
:members:
:inherited-members:
:undoc-members:
```
[plappert et al. (2018)]: https://arxiv.org/abs/1802.09464
---
### Modules Ppo (`docs/modules/ppo.md`)
(ppo2)=
```{eval-rst}
.. automodule:: stable_baselines3.ppo
```
# PPO
The [Proximal Policy Optimization](https://arxiv.org/abs/1707.06347) algorithm combines ideas from A2C (having multiple workers)
and TRPO (it uses a trust region to improve the actor).
The main idea is that after an update, the new policy should be not too far from the old policy.
For that, ppo uses clipping to avoid too large update.
:::{note}
PPO contains several modifications from the original algorithm not documented
by OpenAI: advantages are normalized and value function can be also clipped.
:::
## Notes
- Original paper:
- Clear explanation of PPO on Arxiv Insights channel:
- OpenAI blog post:
- Spinning Up guide:
- 37 implementation details blog:
## Can I use?
:::{note}
A recurrent version of PPO is available in our contrib repo:
However we advise users to start with simple frame-stacking as a simpler, faster
and usually competitive alternative, more info in our report:
See also [Procgen paper appendix Fig 11.](https://arxiv.org/abs/1912.01588).
In practice, you can stack multiple observations using `VecFrameStack`.
:::
- Recurrent policies: ❌
- Multi processing: ✔️
- Gym spaces:
| Space | Action | Observation |
| ------------- | ------ | ----------- |
| Discrete | ✔️ | ✔️ |
| Box | ✔️ | ✔️ |
| MultiDiscrete | ✔️ | ✔️ |
| MultiBinary | ✔️ | ✔️ |
| Dict | ❌ | ✔️ |
## Example
This example is only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in RL Zoo [repository](https://github.com/DLR-RM/rl-baselines3-zoo).
Train a PPO agent on `CartPole-v1` using 4 environments.
```python
import gymnasium as gym
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
# Parallel environments
vec_env = make_vec_env("CartPole-v1", n_envs=4)
model = PPO("MlpPolicy", vec_env, verbose=1)
model.learn(total_timesteps=25000)
model.save("ppo_cartpole")
del model # remove to demonstrate saving and loading
model = PPO.load("ppo_cartpole")
obs = vec_env.reset()
while True:
action, _states = model.predict(obs)
obs, rewards, dones, info = vec_env.step(action)
vec_env.render("human")
```
:::{note}
PPO is meant to be run primarily on the CPU, especially when you are not using a CNN. To improve CPU utilization, try turning off the GPU and using `SubprocVecEnv` instead of the default `DummyVecEnv`:
```python
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import SubprocVecEnv
if __name__=="__main__":
env = make_vec_env("CartPole-v1", n_envs=8, vec_env_cls=SubprocVecEnv)
model = PPO("MlpPolicy", env, device="cpu")
model.learn(total_timesteps=25_000)
```
For more information, see [Vectorized Environments](../guide/vec_envs.md), [Issue #1245](https://github.com/DLR-RM/stable-baselines3/issues/1245#issuecomment-1435766949) or the [Multiprocessing notebook](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb).
:::
:::{note}
Using gSDE (Generalized State-Dependent Exploration) during inference (see [issue #1767](https://github.com/DLR-RM/stable-baselines3/issues/1767)):
When using PPO models trained with `use_sde=True`, the automatic noise resetting that occurs during training (controlled by `sde_sample_freq`) does not happen when using `model.predict()` for inference. This results in deterministic behavior even when `deterministic=False`.
For continuous control tasks, it is recommended to use deterministic behavior during inference (`deterministic=True`). If you need stochastic behavior during inference, you must manually reset the noise by calling `model.policy.reset_noise(env.num_envs)` at appropriate intervals based on your desired `sde_sample_freq`.
:::
## Results
### Atari Games
The complete learning curves are available in the [associated PR #110](https://github.com/DLR-RM/stable-baselines3/pull/110).
### PyBullet Environments
Results on the PyBullet benchmark (2M steps) using 6 seeds.
The complete learning curves are available in the [associated issue #48](https://github.com/DLR-RM/stable-baselines3/issues/48).
:::{note}
Hyperparameters from the [gSDE paper](https://arxiv.org/abs/2005.05719) were used (as they are tuned for PyBullet envs).
:::
*Gaussian* means that the unstructured Gaussian noise is used for exploration,
*gSDE* (generalized State-Dependent Exploration) is used otherwise.
| Environments | A2C | A2C | PPO | PPO |
| ------------ | ------------ | ------------ | ------------ | ----------- |
| | Gaussian | gSDE | Gaussian | gSDE |
| HalfCheetah | 2003 +/- 54 | 2032 +/- 122 | 1976 +/- 479 | 2826 +/- 45 |
| Ant | 2286 +/- 72 | 2443 +/- 89 | 2364 +/- 120 | 2782 +/- 76 |
| Hopper | 1627 +/- 158 | 1561 +/- 220 | 1567 +/- 339 | 2512 +/- 21 |
| Walker2D | 577 +/- 65 | 839 +/- 56 | 1230 +/- 147 | 2019 +/- 64 |
### How to replicate the results?
Clone the [rl-zoo repo](https://github.com/DLR-RM/rl-baselines3-zoo):
```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
```
Run the benchmark (replace `$ENV_ID` by the envs mentioned above):
```bash
python train.py --algo ppo --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```
Plot the results (here for PyBullet envs only):
```bash
python scripts/all_plots.py -a ppo -e HalfCheetah Ant Hopper Walker2D -f logs/ -o logs/ppo_results
python scripts/plot_from_file.py -i logs/ppo_results.pkl -latex -l PPO
```
## Parameters
```{eval-rst}
.. autoclass:: PPO
:members:
:inherited-members:
```
(ppo_policies)=
## PPO Policies
```{eval-rst}
.. autoclass:: MlpPolicy
:members:
:inherited-members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.common.policies.ActorCriticPolicy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: CnnPolicy
:members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.common.policies.ActorCriticCnnPolicy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: MultiInputPolicy
:members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.common.policies.MultiInputActorCriticPolicy
:members:
:noindex:
```
---
### Modules Sac (`docs/modules/sac.md`)
(sac)=
```{eval-rst}
.. automodule:: stable_baselines3.sac
```
# SAC
[Soft Actor Critic (SAC)](https://spinningup.openai.com/en/latest/algorithms/sac.html) Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.
SAC is the successor of [Soft Q-Learning SQL](https://arxiv.org/abs/1702.08165) and incorporates the double Q-learning trick from TD3.
A key feature of SAC, and a major difference with common RL algorithms, is that it is trained to maximize a trade-off between expected return and entropy, a measure of randomness in the policy.
```{eval-rst}
.. rubric:: Available Policies
```
```{eval-rst}
.. autosummary::
:nosignatures:
MlpPolicy
CnnPolicy
MultiInputPolicy
```
## Notes
- Original paper:
- OpenAI Spinning Guide for SAC:
- Original Implementation:
- Blog post on using SAC with real robots:
:::{note}
In our implementation, we use an entropy coefficient (as in OpenAI Spinning or Facebook Horizon),
which is the equivalent to the inverse of reward scale in the original SAC paper.
The main reason is that it avoids having too high errors when updating the Q functions.
:::
:::{note}
When automatically adjusting the temperature (alpha/entropy coefficient), we optimize the logarithm of the entropy coefficient instead of the entropy coefficient itself. This is consistent with the original implementation and has proven to be more stable
(see issues [GH#36](https://github.com/DLR-RM/stable-baselines3/issues/36), [#55](https://github.com/araffin/sbx/issues/55) and others).
:::
:::{note}
The default policies for SAC differ a bit from others MlpPolicy: it uses ReLU instead of tanh activation,
to match the original paper
:::
## Can I use?
- Recurrent policies: ❌
- Multi processing: ✔️
- Gym spaces:
| Space | Action | Observation |
| ------------- | ------ | ----------- |
| Discrete | ❌ | ✔️ |
| Box | ✔️ | ✔️ |
| MultiDiscrete | ❌ | ✔️ |
| MultiBinary | ❌ | ✔️ |
| Dict | ❌ | ✔️ |
## Example
This example is only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in RL Zoo [repository](https://github.com/DLR-RM/rl-baselines3-zoo).
```python
import gymnasium as gym
from stable_baselines3 import SAC
env = gym.make("Pendulum-v1", render_mode="human")
model = SAC("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10000, log_interval=4)
model.save("sac_pendulum")
del model # remove to demonstrate saving and loading
model = SAC.load("sac_pendulum")
obs, info = env.reset()
while True:
action, _states = model.predict(obs, deterministic=True)
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
obs, info = env.reset()
```
:::{note}
Using gSDE (Generalized State-Dependent Exploration) during inference (see [PR #1767](https://github.com/DLR-RM/stable-baselines3/issues/1767)):
When using SAC models trained with `use_sde=True`, the automatic noise resetting that occurs during training (controlled by `sde_sample_freq`) does not happen when using `model.predict()` for inference. This results in deterministic behavior even when `deterministic=False`.
For continuous control tasks, it is recommended to use deterministic behavior during inference (`deterministic=True`). If you need stochastic behavior during inference, you must manually reset the noise by calling `model.policy.reset_noise(env.num_envs)` at appropriate intervals based on your desired `sde_sample_freq`.
:::
## Results
### PyBullet Environments
Results on the PyBullet benchmark (1M steps) using 3 seeds.
The complete learning curves are available in the [associated issue #48](https://github.com/DLR-RM/stable-baselines3/issues/48).
:::{note}
Hyperparameters from the [gSDE paper](https://arxiv.org/abs/2005.05719) were used (as they are tuned for PyBullet envs).
:::
*Gaussian* means that the unstructured Gaussian noise is used for exploration,
*gSDE* (generalized State-Dependent Exploration) is used otherwise.
| Environments | SAC | SAC | TD3 |
| ------------ | ------------ | ------------ | ------------ |
| | Gaussian | gSDE | Gaussian |
| HalfCheetah | 2757 +/- 53 | 2984 +/- 202 | 2774 +/- 35 |
| Ant | 3146 +/- 35 | 3102 +/- 37 | 3305 +/- 43 |
| Hopper | 2422 +/- 168 | 2262 +/- 1 | 2429 +/- 126 |
| Walker2D | 2184 +/- 54 | 2136 +/- 67 | 2063 +/- 185 |
### How to replicate the results?
Clone the [rl-zoo repo](https://github.com/DLR-RM/rl-baselines3-zoo):
```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
```
Run the benchmark (replace `$ENV_ID` by the envs mentioned above):
```bash
python train.py --algo sac --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```
Plot the results:
```bash
python scripts/all_plots.py -a sac -e HalfCheetah Ant Hopper Walker2D -f logs/ -o logs/sac_results
python scripts/plot_from_file.py -i logs/sac_results.pkl -latex -l SAC
```
## Parameters
```{eval-rst}
.. autoclass:: SAC
:members:
:inherited-members:
```
(sac_policies)=
## SAC Policies
```{eval-rst}
.. autoclass:: MlpPolicy
:members:
:inherited-members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.sac.policies.SACPolicy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: CnnPolicy
:members:
```
```{eval-rst}
.. autoclass:: MultiInputPolicy
:members:
```
---
### Modules Td3 (`docs/modules/td3.md`)
(td3)=
```{eval-rst}
.. automodule:: stable_baselines3.td3
```
# TD3
[Twin Delayed DDPG (TD3)](https://spinningup.openai.com/en/latest/algorithms/td3.html) Addressing Function Approximation Error in Actor-Critic Methods.
TD3 is a direct successor of {ref}`DDPG ` and improves it using three major tricks: clipped double Q-Learning, delayed policy update and target policy smoothing.
We recommend reading [OpenAI Spinning guide on TD3](https://spinningup.openai.com/en/latest/algorithms/td3.html) to learn more about those.
```{eval-rst}
.. rubric:: Available Policies
```
```{eval-rst}
.. autosummary::
:nosignatures:
MlpPolicy
CnnPolicy
MultiInputPolicy
```
## Notes
- Original paper:
- OpenAI Spinning Guide for TD3:
- Original Implementation:
:::{note}
The default policies for TD3 differ a bit from others MlpPolicy: it uses ReLU instead of tanh activation,
to match the original paper
:::
## Can I use?
- Recurrent policies: ❌
- Multi processing: ✔️
- Gym spaces:
| Space | Action | Observation |
| ------------- | ------ | ----------- |
| Discrete | ❌ | ✔️ |
| Box | ✔️ | ✔️ |
| MultiDiscrete | ❌ | ✔️ |
| MultiBinary | ❌ | ✔️ |
| Dict | ❌ | ✔️ |
## Example
This example is only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in RL Zoo [repository](https://github.com/DLR-RM/rl-baselines3-zoo).
```python
import gymnasium as gym
import numpy as np
from stable_baselines3 import TD3
from stable_baselines3.common.noise import NormalActionNoise, OrnsteinUhlenbeckActionNoise
env = gym.make("Pendulum-v1", render_mode="rgb_array")
# The noise objects for TD3
n_actions = env.action_space.shape[-1]
action_noise = NormalActionNoise(mean=np.zeros(n_actions), sigma=0.1 * np.ones(n_actions))
model = TD3("MlpPolicy", env, action_noise=action_noise, verbose=1)
model.learn(total_timesteps=10000, log_interval=10)
model.save("td3_pendulum")
vec_env = model.get_env()
del model # remove to demonstrate saving and loading
model = TD3.load("td3_pendulum")
obs = vec_env.reset()
while True:
action, _states = model.predict(obs)
obs, rewards, dones, info = vec_env.step(action)
vec_env.render("human")
```
## Results
### PyBullet Environments
Results on the PyBullet benchmark (1M steps) using 3 seeds.
The complete learning curves are available in the [associated issue #48](https://github.com/DLR-RM/stable-baselines3/issues/48).
:::{note}
Hyperparameters from the [gSDE paper](https://arxiv.org/abs/2005.05719) were used (as they are tuned for PyBullet envs).
:::
*Gaussian* means that the unstructured Gaussian noise is used for exploration,
*gSDE* (generalized State-Dependent Exploration) is used otherwise.
| Environments | SAC | SAC | TD3 |
| ------------ | ------------ | ------------ | ------------ |
| | Gaussian | gSDE | Gaussian |
| HalfCheetah | 2757 +/- 53 | 2984 +/- 202 | 2774 +/- 35 |
| Ant | 3146 +/- 35 | 3102 +/- 37 | 3305 +/- 43 |
| Hopper | 2422 +/- 168 | 2262 +/- 1 | 2429 +/- 126 |
| Walker2D | 2184 +/- 54 | 2136 +/- 67 | 2063 +/- 185 |
### How to replicate the results?
Clone the [rl-zoo repo](https://github.com/DLR-RM/rl-baselines3-zoo):
```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
```
Run the benchmark (replace `$ENV_ID` by the envs mentioned above):
```bash
python train.py --algo td3 --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```
Plot the results:
```bash
python scripts/all_plots.py -a td3 -e HalfCheetah Ant Hopper Walker2D -f logs/ -o logs/td3_results
python scripts/plot_from_file.py -i logs/td3_results.pkl -latex -l TD3
```
## Parameters
```{eval-rst}
.. autoclass:: TD3
:members:
:inherited-members:
```
(td3_policies)=
## TD3 Policies
```{eval-rst}
.. autoclass:: MlpPolicy
:members:
:inherited-members:
```
```{eval-rst}
.. autoclass:: stable_baselines3.td3.policies.TD3Policy
:members:
:noindex:
```
```{eval-rst}
.. autoclass:: CnnPolicy
:members:
```
```{eval-rst}
.. autoclass:: MultiInputPolicy
:members:
```
---
--- METRICS ---
- Files Extracted: 47
- Estimated Token Budget: ~80488 tokens
- Canonical Reference: https://codewiki.google/github.com/DLR-RM/stable-baselines3