stable-baselines3 Documentation

Technical Documentation: DLR-RM/stable-baselines3

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Stable Baselines3

Stable Baselines3 (SB3) is a set of reliable implementations of reinforcement learning algorithms in PyTorch. It is the next major version of Stable Baselines.

You can read a detailed presentation of Stable Baselines3 in the v1.0 blog post or our JMLR paper.

These algorithms will make it easier for the research community and industry to replicate, refine, and identify new ideas, and will create good baselines to build projects on top of. We expect these tools will be used as a base around which new ideas can be added, and as a tool for comparing a new approach against existing ones. We also hope that the simplicity of these tools will allow beginners to experiment with a more advanced toolset, without being buried in implementation details.

Note: Despite its simplicity of use, Stable Baselines3 (SB3) assumes you have some knowledge about Reinforcement Learning (RL). You should not utilize this library without some practice. To that extent, we provide good resources in the documentation to get started with RL.

Main Features Copy Chapter

The performance of each algorithm was tested (see Results section in their respective page).
You can take a look at the issues #48 and #49 for more details.

We also provide detailed logs and reports on the OpenRL Benchmark platform.

Planned features Copy Section

Since most of the features from the original roadmap have been implemented, there are no major changes planned for SB3; it is now stable.
If you want to contribute, you can search in the issues for the ones where help is welcomed and the other proposed enhancements.

While SB3 development is now focused on bug fixes and maintenance (doc update, user experience, ...), there is more active development going on in the associated repositories:

• newer algorithms are regularly added to the SB3 Contrib repository
• faster variants are developed in the SBX (SB3 + Jax) repository
• the training framework for SB3, the RL Zoo, has an active roadmap

Migration guide: from Stable-Baselines (SB2) to Stable-Baselines3 (SB3) Copy Chapter

A migration guide from SB2 to SB3 can be found in the documentation.

Documentation Copy Chapter

Documentation is available online: https://stable-baselines3.readthedocs.io/

Integrations Copy Chapter

Stable-Baselines3 has some integration with other libraries/services like Weights & Biases for experiment tracking or Hugging Face for storing/sharing trained models. You can find out more in the dedicated section of the documentation.

RL Baselines3 Zoo: A Training Framework for Stable Baselines3 Reinforcement Learning Agents Copy Chapter

RL Baselines3 Zoo is a training framework for Reinforcement Learning (RL).

It provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.

In addition, it includes a collection of tuned hyperparameters for common environments and RL algorithms, and agents trained with those settings.

Goals of this repository:

1. Provide a simple interface to train and enjoy RL agents
2. Benchmark the different Reinforcement Learning algorithms
3. Provide tuned hyperparameters for each environment and RL algorithm
4. Have fun with the trained agents!

Github repo: https://github.com/DLR-RM/rl-baselines3-zoo

Documentation: https://rl-baselines3-zoo.readthedocs.io/en/master/

SB3-Contrib: Experimental RL Features Copy Chapter

We implement experimental features in a separate contrib repository: SB3-Contrib

This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Recurrent PPO (PPO LSTM), CrossQ, Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO).

Documentation is available online: https://sb3-contrib.readthedocs.io/

Stable-Baselines Jax (SBX) Copy Chapter

Stable Baselines Jax (SBX) is a proof of concept version of Stable-Baselines3 in Jax, with recent algorithms like DroQ or CrossQ.

It provides a minimal number of features compared to SB3 but can be much faster (up to 20x times!): https://twitter.com/araffin2/status/1590714558628253698

Installation Copy Chapter

Note: Stable-Baselines3 supports PyTorch >= 2.8

Prerequisites Copy Section

Stable Baselines3 requires Python 3.10+.

Windows

To install stable-baselines on Windows, please look at the documentation.

Install using pip Copy Section

Install the Stable Baselines3 package:

pip install 'stable-baselines3[extra]'

This includes optional dependencies like Tensorboard, OpenCV, ale-py to train on atari games, as well as pandas and matplotlib for plotting and analyzing results. If you do not need those, you can use:

pip install stable-baselines3

Please read the documentation for more details and alternatives (from source, using docker).

Example Copy Chapter

Most of the code in the library tries to follow a sklearn-like syntax for the Reinforcement Learning algorithms.

Here is a quick example of how to train and run PPO on a cartpole environment:

import gymnasium as gym

from stable_baselines3 import PPO

env = gym.make("CartPole-v1", render_mode="human")

model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10_000)

vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
    actions, _states = model.predict(obs, deterministic=True)
    obs, rewards, dones, info = vec_env.step(actions)
    vec_env.render()
    # VecEnv resets automatically
    # if done:
    #   obs = env.reset()

env.close()

Or just train a model with a one liner if the environment is registered in Gymnasium and if the policy is registered:

from stable_baselines3 import PPO

model = PPO("MlpPolicy", "CartPole-v1").learn(10_000)

Please read the documentation for more examples.

Try it online with Colab Notebooks! Copy Chapter

All the following examples can be executed online using Google Colab notebooks:

• Full Tutorial
• All Notebooks
• Getting Started
• Training, Saving, Loading
• Multiprocessing
• Monitor Training and Plotting
• Atari Games
• RL Baselines Zoo
• PyBullet

Implemented Algorithms Copy Chapter

1: Implemented in SB3 Contrib GitHub repository.

2: Implemented in SBX (SB3 + Jax)

Actions gymnasium.spaces:

• Box: A N-dimensional box that contains every point in the action space.
• Discrete: A list of possible actions, where only one action can be used per timestep.
• MultiDiscrete: A list of possible actions, where each timestep only one action of each discrete set can be used.
• MultiBinary: A list of possible actions, where each timestep any of the actions can be used in any combination.

Testing the installation Copy Chapter

Install dependencies Copy Section

pip install -e '.[docs,tests,extra]'

Run tests Copy Section

All unit tests in stable baselines3 can be run using pytest runner:

make pytest

To run a single test file:

python3 -m pytest -v tests/test_env_checker.py

To run a single test:

python3 -m pytest -v -k 'test_check_env_dict_action'

You can also do a static type check using mypy:

pip install mypy
make type

Codestyle check with ruff:

pip install ruff
make lint

Projects Using Stable-Baselines3 Copy Chapter

We try to maintain a list of projects using stable-baselines3 in the documentation,
please tell us if you want your project to appear on this page ;)

Citing the Project Copy Chapter

To cite this repository in publications:

@article{stable-baselines3,
  author  = {Antonin Raffin and Ashley Hill and Adam Gleave and Anssi Kanervisto and Maximilian Ernestus and Noah Dormann},
  title   = {Stable-Baselines3: Reliable Reinforcement Learning Implementations},
  journal = {Journal of Machine Learning Research},
  year    = {2021},
  volume  = {22},
  number  = {268},
  pages   = {1-8},
  url     = {https://jmlr.org/papers/v22/20-1364.html}
}

Note: If you need to refer to a specific version of SB3, you can also use the Zenodo DOI.

Maintainers Copy Chapter

Stable-Baselines3 is currently maintained by Ashley Hill (aka @hill-a), Antonin Raffin (aka @araffin), Maximilian Ernestus (aka @ernestum), Adam Gleave (@AdamGleave), Anssi Kanervisto (@Miffyli) and Quentin Gallouédec (@qgallouedec).

Important Note: We do not provide technical support, or consulting and do not answer personal questions via email.
Please post your question on the RL Discord, Reddit, or Stack Overflow in that case.

How To Contribute Copy Chapter

For anyone interested in making the baselines better, there is still some documentation that needs to be done.
If you want to contribute, please read CONTRIBUTING.md guide first.

Acknowledgments Copy Chapter

The initial work to develop Stable Baselines3 was partially funded by the project Reduced Complexity Models from the Helmholtz-Gemeinschaft Deutscher Forschungszentren, and by the EU Horizon 2020 Research and Innovation Programme under grant number 951992 (VeriDream).

The original version, Stable Baselines, was created in the robotics lab U2IS (INRIA Flowers team) at ENSTA ParisTech.

Logo credits: L.M. Tenkes

Overview (`docs/README.md`) Copy Section

Stable Baselines3 Documentation

This folder contains documentation for the RL baselines.

Build the Documentation Copy Section

Install Sphinx and Theme

Execute this command in the project root:

pip install -e ".[docs]"

Building the Docs

In the docs/ folder:

make html

for automatic rebuilding each time a file is changed:

sphinx-autobuild . _build/html

Guide Quickstart (`docs/guide/quickstart.md`) Copy Section

(quickstart)=

Getting Started

:::{note}
Stable-Baselines3 (SB3) uses vectorized environments (VecEnv) internally.
Please read the associated section to learn more about its features and differences compared to a single Gym environment.
:::

Most of the library follows a sklearn-like syntax for the reinforcement learning algorithms.

Here is a quick example of how to train and run A2C on a CartPole environment:

import gymnasium as gym

from stable_baselines3 import A2C

env = gym.make("CartPole-v1", render_mode="rgb_array")

model = A2C("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10_000)

vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
    actions, _states = model.predict(obs, deterministic=True)
    obs, rewards, dones, info = vec_env.step(actions)
    vec_env.render("human")
    # VecEnv resets automatically
    # if done:
    #   obs = vec_env.reset()

:::{note}
You can find explanations about the logger output and names in the {ref}Logger <logger> section.
:::

Or just train a model with a one-liner if
the environment is registered in Gymnasium and the
policy is registered:

from stable_baselines3 import A2C

model = A2C("MlpPolicy", "CartPole-v1").learn(10_000)

Conda Env.Yml (`docs/conda_env.yml`) Copy Section

name: root
channels:

• conda-forge
  dependencies:
• pip=26.1.2
• python=3.12
• pytorch-cpu=2.9.1
• pip:
  • gymnasium>=0.29.1,<2.0
  • cloudpickle
  • opencv-python-headless
  • pandas
  • numpy>=1.20,<3.0
  • matplotlib
  • sphinx>=5,<10
  • sphinx_rtd_theme>=3.0
  • sphinx_copybutton
  • myst-parser>=4,<6

Index (`docs/index.rst`) Copy Section

.. Stable Baselines3 documentation master file, created by
sphinx-quickstart on Thu Sep 26 11:06:54 2019.
You can adapt this file completely to your liking, but it should at least
contain the root toctree directive.

Stable-Baselines3 Docs - Reliable Reinforcement Learning Implementations

Stable Baselines3 (SB3) <https://github.com/DLR-RM/stable-baselines3>_ is a set of reliable implementations of reinforcement learning algorithms in PyTorch.
It is the next major version of Stable Baselines <https://github.com/hill-a/stable-baselines>_.

GitHub repository: https://github.com/DLR-RM/stable-baselines3

Paper: https://jmlr.org/papers/volume22/20-1364/20-1364.pdf

RL Baselines3 Zoo (training framework for SB3): https://github.com/DLR-RM/rl-baselines3-zoo

RL Baselines3 Zoo provides a collection of pre-trained agents, scripts for training, evaluating agents, tuning hyperparameters, plotting results, and recording videos.

SB3 Contrib (experimental RL code, latest algorithms): https://github.com/Stable-Baselines-Team/stable-baselines3-contrib

SBX (SB3 + Jax): https://github.com/araffin/sbx

Main Features Copy Chapter

• Unified structure for all algorithms
• PEP 8 compliant (unified code style)
• Documented functions and classes
• Tests, high code coverage, and type hints
• Clean code
• TensorBoard support
• The performance of each algorithm was tested (see Results section in their respective page)

.. toctree::
:maxdepth: 2
:caption: User Guide

guide/install
guide/quickstart
guide/rl_tips
guide/rl
guide/algos
guide/examples
guide/vec_envs
guide/custom_policy
guide/custom_env
guide/callbacks
guide/tensorboard
guide/integrations
guide/rl_zoo
guide/sb3_contrib
guide/sbx
guide/plotting
guide/imitation
guide/migration
guide/checking_nan
guide/developer
guide/save_format
guide/export

.. toctree::
:maxdepth: 1
:caption: RL Algorithms

modules/base
modules/a2c
modules/ddpg
modules/dqn
modules/her
modules/ppo
modules/sac
modules/td3

.. toctree::
:maxdepth: 1
:caption: Common

common/atari_wrappers
common/env_util
common/envs
common/distributions
common/evaluation
common/env_checker
common/monitor
common/logger
common/noise
common/utils

.. toctree::
:maxdepth: 1
:caption: Misc

misc/changelog
misc/projects

Citing Stable Baselines3 Copy Chapter

To cite this project in publications:

.. code-block:: bibtex

@article{stable-baselines3,
author = {Antonin Raffin and Ashley Hill and Adam Gleave and Anssi Kanervisto and Maximilian Ernestus and Noah Dormann},
title = {Stable-Baselines3: Reliable Reinforcement Learning Implementations},
journal = {Journal of Machine Learning Research},
year = {2021},
volume = {22},
number = {268},
pages = {1-8},
url = {https://jmlr.org/papers/v22/20-1364.html}
}

Note: If you need to refer to a specific version of SB3, you can also use the Zenodo DOI <https://doi.org/10.5281/zenodo.8123988>_.

Contributing Copy Chapter

For anyone interested in making the RL baselines better, there are still some improvements
that need to be done.
You can check issues in the repository <https://github.com/DLR-RM/stable-baselines3/labels/help%20wanted>_.

If you want to contribute, please read CONTRIBUTING.md <https://github.com/DLR-RM/stable-baselines3/blob/master/CONTRIBUTING.md>_ first.

Indices and tables Copy Chapter

• :ref:genindex
• :ref:search
• :ref:modindex

Spelling Wordlist (`docs/spelling_wordlist.txt`) Copy Section

py
env
atari
argparse
Argparse
TensorFlow
feedforward
envs
VecEnv
pretrain
petrained
tf
th
nn
np
str
mujoco
cpu
ndarray
ndarrays
timestep
timesteps
stepsize
dataset
adam
fn
normalisation
Kullback
Leibler
boolean
deserialized
pretrained
minibatch
subprocesses
ArgumentParser
Tensorflow
Gaussian
approximator
minibatches
hyperparameters
hyperparameter
vectorized
rl
colab
dataloader
npz
datasets
vf
logits
num
Utils
backpropagate
prepend
NaN
preprocessing
Cloudpickle
async
multiprocess
tensorflow
mlp
cnn
neglogp
tanh
coef
repo
Huber
params
ppo
arxiv
Arxiv
func
DQN
Uhlenbeck
Ornstein
multithread
cancelled
Tensorboard
parallelize
customising
serializable
Multiprocessed
cartpole
toolset
lstm
rescale
ffmpeg
avconv
unnormalized
Github
pre
preprocess
backend
attr
preprocess
Antonin
Raffin
araffin
Homebrew
Numpy
Theano
rollout
kfac
Piecewise
csv
nvidia
visdom
tensorboard
preprocessed
namespace
sklearn
GoalEnv
Torchy
pytorch
dicts
optimizers
Deprecations
forkserver
cuda
Polyak
gSDE
rollouts
Pyro
softmax
stdout
Contrib
Quantile

Common Atari Wrappers (`docs/common/atari_wrappers.md`) Copy Section

(atari-wrapper)=

Atari Wrappers

.. automodule:: stable_baselines3.common.atari_wrappers
  :members:

Common Distributions (`docs/common/distributions.md`) Copy Section

(distributions)=

Probability Distributions

Probability distributions used for the different action spaces:

• CategoricalDistribution -> Discrete
• DiagGaussianDistribution -> Box (continuous actions)
• StateDependentNoiseDistribution -> Box (continuous actions) when use_sde=True

% - MultiCategoricalDistribution -> MultiDiscrete

% - BernoulliDistribution -> MultiBinary

The policy networks output parameters for the distributions (named flat in the methods).
Actions are then sampled from those distributions.

For instance, in the case of discrete actions. The policy network outputs probability
of taking each action. The CategoricalDistribution allows sampling from it,
computes the entropy, the log probability (log_prob) and backpropagate the gradient.

In the case of continuous actions, a Gaussian distribution is used. The policy network outputs
mean and (log) std of the distribution (assumed to be a DiagGaussianDistribution).

.. automodule:: stable_baselines3.common.distributions
  :members:

Common Env Checker (`docs/common/env_checker.md`) Copy Section

(env-checker)=

Gym Environment Checker

.. automodule:: stable_baselines3.common.env_checker
  :members:

Common Env Util (`docs/common/env_util.md`) Copy Section

(env-util)=

Environments Utils

.. automodule:: stable_baselines3.common.env_util
  :members:

Common Envs (`docs/common/envs.md`) Copy Section

(envs)=

.. automodule:: stable_baselines3.common.envs

Custom Environments

Those environments were created for testing purposes.

BitFlippingEnv Copy Chapter

.. autoclass:: BitFlippingEnv
  :members:

SimpleMultiObsEnv Copy Chapter

.. autoclass:: SimpleMultiObsEnv
  :members:

Common Evaluation (`docs/common/evaluation.md`) Copy Section

(eval)=

Evaluation Helper

.. automodule:: stable_baselines3.common.evaluation
  :members:

Common Logger (`docs/common/logger.md`) Copy Section

(logger)=

Logger

To overwrite the default logger, you can pass one to the algorithm.
Available formats are ["stdout", "csv", "log", "tensorboard", "json"].

:::{warning}
When passing a custom logger object,
this will overwrite tensorboard_log and verbose settings
passed to the constructor.
:::

from stable_baselines3 import A2C
from stable_baselines3.common.logger import configure

tmp_path = "/tmp/sb3_log/"
# set up logger
new_logger = configure(tmp_path, ["stdout", "csv", "tensorboard"])

model = A2C("MlpPolicy", "CartPole-v1", verbose=1)
# Set new logger
model.set_logger(new_logger)
model.learn(10000)

Explanation of logger output Copy Chapter

You can find below short explanations of the values logged in Stable-Baselines3 (SB3).
Depending on the algorithm used and of the wrappers/callbacks applied, SB3 only logs a subset of those keys during training.

Below you can find an example of the logger output when training a PPO agent:

-----------------------------------------
| eval/                   |             |
|    mean_ep_length       | 200         |
|    mean_reward          | -157        |
| rollout/                |             |
|    ep_len_mean          | 200         |
|    ep_rew_mean          | -227        |
| time/                   |             |
|    fps                  | 972         |
|    iterations           | 19          |
|    time_elapsed         | 80          |
|    total_timesteps      | 77824       |
| train/                  |             |
|    approx_kl            | 0.037781604 |
|    clip_fraction        | 0.243       |
|    clip_range           | 0.2         |
|    entropy_loss         | -1.06       |
|    explained_variance   | 0.999       |
|    learning_rate        | 0.001       |
|    loss                 | 0.245       |
|    n_updates            | 180         |
|    policy_gradient_loss | -0.00398    |
|    std                  | 0.205       |
|    value_loss           | 0.226       |
-----------------------------------------

eval/ Copy Section

All eval/ values are computed by the EvalCallback.

• mean_ep_length: Mean episode length
• mean_reward: Mean episodic reward (during evaluation)
• success_rate: Mean success rate during evaluation (1.0 means 100% success), the environment info dict must contain an is_success key to compute that value

rollout/ Copy Section

• ep_len_mean: Mean episode length (averaged over stats_window_size episodes, 100 by default)
• ep_rew_mean: Mean episodic training reward (averaged over stats_window_size episodes, 100 by default), a Monitor wrapper is required to compute that value (automatically added by make_vec_env).
• exploration_rate: Current value of the exploration rate when using DQN, it corresponds to the fraction of actions taken randomly (epsilon of the "epsilon-greedy" exploration)
• success_rate: Mean success rate during training (averaged over stats_window_size episodes, 100 by default), you must pass an extra argument to the Monitor wrapper to log that value (info_keywords=("is_success",)) and provide info["is_success"]=True/False on the final step of the episode

time/ Copy Section

• episodes: Total number of episodes
• fps: Number of frames per seconds (includes time taken by gradient update)
• iterations: Number of iterations (data collection + policy update for A2C/PPO)
• time_elapsed: Time in seconds since the beginning of training
• total_timesteps: Total number of timesteps (steps in the environments)

train/ Copy Section

• actor_loss: Current value for the actor loss for off-policy algorithms
• approx_kl: approximate mean KL divergence between old and new policy (for PPO), it is an estimation of how much changes happened in the update
• clip_fraction: mean fraction of surrogate loss that was clipped (above clip_range threshold) for PPO.
• clip_range: Current value of the clipping factor for the surrogate loss of PPO
• critic_loss: Current value for the critic function loss for off-policy algorithms, usually error between value function output and TD(0), temporal difference estimate
• ent_coef: Current value of the entropy coefficient (when using SAC)
• ent_coef_loss: Current value of the entropy coefficient loss (when using SAC)
• entropy_loss: Mean value of the entropy loss (negative of the average policy entropy)
• explained_variance: Fraction of the return variance explained by the value function, see https://scikit-learn.org/stable/modules/model_evaluation.html#explained-variance-score
  (ev=0 => might as well have predicted zero, ev=1 => perfect prediction, ev<0 => worse than just predicting zero)
• learning_rate: Current learning rate value
• loss: Current total loss value
• n_updates: Number of gradient updates applied so far
• policy_gradient_loss: Current value of the policy gradient loss (its value does not have much meaning)
• value_loss: Current value for the value function loss for on-policy algorithms, usually error between value function output and Monte-Carlo estimate (or TD(lambda) estimate)
• std: Current standard deviation of the noise when using generalized State-Dependent Exploration (gSDE)

.. automodule:: stable_baselines3.common.logger
  :members:

Common Monitor (`docs/common/monitor.md`) Copy Section

(monitor)=

Monitor Wrapper

.. automodule:: stable_baselines3.common.monitor
  :members:

Common Noise (`docs/common/noise.md`) Copy Section

(noise)=

Action Noise

.. automodule:: stable_baselines3.common.noise
  :members:

Common Utils (`docs/common/utils.md`) Copy Section

(utils)=

Utils

.. automodule:: stable_baselines3.common.utils
  :members:

Guide Algos (`docs/guide/algos.md`) Copy Section

RL Algorithms

This table displays the RL algorithms that are implemented in the Stable Baselines3 project,
along with some useful characteristics: support for discrete/continuous actions, multiprocessing.

:::{note}
More algorithms (like QR-DQN or TQC) are implemented in our contrib repo
and in our {ref}SBX (SB3 + Jax) repo <sbx> (DroQ, CrossQ, SimBa, ...).
:::

[^f1]: Implemented in SB3 Contrib

[^f2]: Implemented in SBX (SB3 + Jax)

:::{note}
Tuple observation spaces are not supported by any environment,
however, single-level Dict spaces are (cf. {ref}Examples <examples>).
:::

Actions gym.spaces:

• Box: A N-dimensional box that contains every point in the action
  space.
• Discrete: A list of possible actions, where each timestep only
  one of the actions can be used.
• MultiDiscrete: A list of possible actions, where each timestep only one action of each discrete set can be used.
• MultiBinary: A list of possible actions, where each timestep any of the actions can be used in any combination.

:::{note}
Some logging values (like ep_rew_mean, ep_len_mean) are only available when using a Monitor wrapper
See Issue #339 for more info.
:::

:::{note}
When using off-policy algorithms, Time Limits (aka timeouts) are handled
properly (cf. issue #284).
You can revert to SB3 < 2.1.0 behavior by passing handle_timeout_termination=False
via the replay_buffer_kwargs argument.
:::

Reproducibility Copy Chapter

Completely reproducible results are not guaranteed across PyTorch releases or different platforms.
Furthermore, results need not be reproducible between CPU and GPU executions, even when using identical seeds.

In order to make computations deterministic for your specific problem on one specific platform,
you need to pass a seed argument at the creation of a model.
If you pass an environment to the model using set_env(), then you also need to seed the environment first.

Credit: part of the Reproducibility section comes from PyTorch Documentation

Training exceeds `total_timesteps` Copy Chapter

When you train an agent using SB3, you pass a total_timesteps parameter to the learn() method which defines the training budget for the agent (how many interactions with the environment are allowed).
For example:

from stable_baselines3 import PPO

model = PPO("MlpPolicy", "CartPole-v1").learn(total_timesteps=1_000)

Because of the way the algorithms work, total_timesteps is a lower bound (see issue #1150).
In the example above, PPO will effectively collect n_steps * n_envs = 2048 * 1 steps despite total_timesteps=1_000
In more details:

• PPO/A2C and derivates collect n_steps * n_envs of experience
  before performing an update, so if you want to have exactly
  total_timesteps, you will need to adjust those values
• SAC/DQN/TD3 and other off-policy algorithms collect
  train_freq * n_envs steps before doing an update (when train_freq is in steps and not episodes), so if you want to have exactly total_timesteps
  you have to adjust these values (train_freq=4 by default for DQN)
• ARS and other population-based algorithms evaluate the policy for
  n_episodes with n_envs, so unless the number of steps per
  episode is fixed, it is not possible to exactly achieve
  total_timesteps
• when using multiple envs, each call to env.step() corresponds to
  n_envs timesteps, so it is no longer possible to use the
  EvaluationCallback at an exact timestep

Guide Callbacks (`docs/guide/callbacks.md`) Copy Section

(callbacks)=

Callbacks

A callback is a set of functions that will be called at given stages of the training procedure.
You can use callbacks to access internal state of the RL model during training.
It allows one to do monitoring, auto saving, model manipulation, progress bars, ...

Custom Callback Copy Chapter

To build a custom callback, you need to create a class that derives from BaseCallback.
This will give you access to events (_on_training_start, _on_step) and useful variables (like self.model for the RL model).

You can find two examples of custom callbacks in the documentation: one for saving the best model according to the training reward (see {ref}Examples <examples>), and one for logging additional values with Tensorboard (see {ref}Tensorboard section <tensorboard>).

/* Detailed source-code truncated for AI context efficiency. */

:::{note}
self.num_timesteps corresponds to the total number of steps taken in the environment, i.e., it is the number of environments multiplied by the number of time env.step() was called

For the other algorithms, self.num_timesteps is incremented by n_envs (number of environments) after each call to env.step()
:::

:::{note}
For off-policy algorithms like SAC, DDPG, TD3 or DQN, the notion of rollout corresponds to the steps taken in the environment between two updates.
:::

(eventcallback)=

Event Callback Copy Chapter

Compared to Keras, Stable Baselines provides a second type of BaseCallback, named EventCallback that is meant to trigger events. When an event is triggered, then a child callback is called.

As an example, {ref}EvalCallback is an EventCallback that will trigger its child callback when there is a new best model.
A child callback is for instance {ref}StopTrainingOnRewardThreshold <StopTrainingCallback> that stops the training if the mean reward achieved by the RL model is above a threshold.

:::{note}
We recommend taking a look at the source code of {ref}EvalCallback and {ref}StopTrainingOnRewardThreshold <StopTrainingCallback> to have a better overview of what can be achieved with this kind of callbacks.
:::

class EventCallback(BaseCallback):
    """
    Base class for triggering callback on event.

    :param callback: Callback that will be called when an event is triggered.
    :param verbose: Verbosity level: 0 for no output, 1 for info messages, 2 for debug messages
    """
    def __init__(self, callback: BaseCallback, verbose: int = 0):
        super().__init__(verbose=verbose)
        self.callback = callback
        # Give access to the parent
        self.callback.parent = self
    ...

    def _on_event(self) -> bool:
        return self.callback()

Callback Collection Copy Chapter

Stable Baselines provides you with a set of common callbacks for:

• saving the model periodically ({ref}CheckpointCallback)
• evaluating the model periodically and saving the best one ({ref}EvalCallback)
• chaining callbacks ({ref}CallbackList)
• triggering callback on events ({ref}EventCallback, {ref}EveryNTimesteps)
• logging data every N timesteps ({ref}LogEveryNTimesteps)
• stopping the training early based on a reward threshold ({ref}StopTrainingOnRewardThreshold <StopTrainingCallback>)

(checkpointcallback)=

CheckpointCallback Copy Section

Callback for saving a model every save_freq calls to env.step(), you must specify a log folder (save_path)
and optionally a prefix for the checkpoints (rl_model by default).
If you are using this callback to stop and resume training, you may want to optionally save the replay buffer if the
model has one (save_replay_buffer, False by default).
Additionally, if your environment uses a VecNormalize wrapper, you can save the
corresponding statistics using save_vecnormalize (False by default).

:::{warning}
When using multiple environments, each call to env.step() will effectively correspond to n_envs steps.
If you want the save_freq to be similar when using a different number of environments,
you need to account for it using save_freq = max(save_freq // n_envs, 1).
The same goes for the other callbacks.
:::

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import CheckpointCallback

# Save a checkpoint every 1000 steps
checkpoint_callback = CheckpointCallback(
  save_freq=1000,
  save_path="./logs/",
  name_prefix="rl_model",
  save_replay_buffer=True,
  save_vecnormalize=True,
)

model = SAC("MlpPolicy", "Pendulum-v1")
model.learn(2000, callback=checkpoint_callback)

(evalcallback)=

EvalCallback Copy Section

Evaluate periodically the performance of an agent, using a separate test environment.
It will save the best model if best_model_save_path folder is specified and save the evaluations results in a NumPy archive (evaluations.npz) if log_path folder is specified.

:::{note}
You can pass child callbacks via callback_after_eval and callback_on_new_best arguments. callback_after_eval will be triggered after every evaluation, and callback_on_new_best will be triggered each time there is a new best model.
:::

:::{warning}
You need to make sure that eval_env is wrapped the same way as the training environment, for instance using the VecTransposeImage wrapper if you have a channel-last image as input.
The EvalCallback class outputs a warning if it is not the case.
:::

import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback

# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
# Use deterministic actions for evaluation
eval_callback = EvalCallback(eval_env, best_model_save_path="./logs/",
                             log_path="./logs/", eval_freq=500,
                             deterministic=True, render=False)

model = SAC("MlpPolicy", "Pendulum-v1")
model.learn(5000, callback=eval_callback)

(progressbarcallback)=

ProgressBarCallback Copy Section

Display a progress bar with the current progress, elapsed time and estimated remaining time.
This callback is integrated inside SB3 via the progress_bar argument of the learn() method.

:::{note}
ProgressBarCallback callback requires tqdm and rich packages to be installed. This is done automatically when using pip install stable-baselines3[extra]
:::

from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import ProgressBarCallback

model = PPO("MlpPolicy", "Pendulum-v1")
# Display progress bar using the progress bar callback
# this is equivalent to model.learn(100_000, callback=ProgressBarCallback())
model.learn(100_000, progress_bar=True)

(callbacklist)=

CallbackList Copy Section

Class for chaining callbacks, they will be called sequentially.
Alternatively, you can pass directly a list of callbacks to the learn() method, it will be converted automatically to a CallbackList.

import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import CallbackList, CheckpointCallback, EvalCallback

checkpoint_callback = CheckpointCallback(save_freq=1000, save_path="./logs/")
# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
eval_callback = EvalCallback(eval_env, best_model_save_path="./logs/best_model",
                             log_path="./logs/results", eval_freq=500)
# Create the callback list
callback = CallbackList([checkpoint_callback, eval_callback])

model = SAC("MlpPolicy", "Pendulum-v1")
# Equivalent to:
# model.learn(5000, callback=[checkpoint_callback, eval_callback])
model.learn(5000, callback=callback)

(stoptrainingcallback)=

StopTrainingOnRewardThreshold Copy Section

Stop the training once a threshold in episodic reward (mean episode reward over the evaluations) has been reached (i.e., when the model is good enough).
It must be used with the {ref}EvalCallback and use the event triggered by a new best model.

import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback, StopTrainingOnRewardThreshold

# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
# Stop training when the model reaches the reward threshold
callback_on_best = StopTrainingOnRewardThreshold(reward_threshold=-200, verbose=1)
eval_callback = EvalCallback(eval_env, callback_on_new_best=callback_on_best, verbose=1)

model = SAC("MlpPolicy", "Pendulum-v1", verbose=1)
# Almost infinite number of timesteps, but the training will stop
# early as soon as the reward threshold is reached
model.learn(int(1e10), callback=eval_callback)

(everyntimesteps)=

EveryNTimesteps Copy Section

An {ref}EventCallback that will trigger its child callback every n_steps timesteps.

:::{note}
Because of the way VecEnv work, n_steps is a lower bound between two events when using multiple environments.
:::

import gymnasium as gym

from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import CheckpointCallback, EveryNTimesteps

# this is equivalent to defining CheckpointCallback(save_freq=500)
# checkpoint_callback will be triggered every 500 steps
checkpoint_on_event = CheckpointCallback(save_freq=1, save_path="./logs/")
event_callback = EveryNTimesteps(n_steps=500, callback=checkpoint_on_event)

model = PPO("MlpPolicy", "Pendulum-v1", verbose=1)

model.learn(20_000, callback=event_callback)

(logeveryntimesteps)=

LogEveryNTimesteps Copy Section

A callback derived from {ref}EveryNTimesteps that will dump the logged data every n_steps timesteps.

import gymnasium as gym

from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import LogEveryNTimesteps

event_callback = LogEveryNTimesteps(n_steps=1_000)

model = PPO("MlpPolicy", "Pendulum-v1", verbose=1)

# Disable auto-logging by passing `log_interval=None`
model.learn(10_000, callback=event_callback, log_interval=None)

(stoptrainingonmaxepisodes)=

StopTrainingOnMaxEpisodes Copy Section

Stop the training upon reaching the maximum number of episodes, regardless of the model's total_timesteps value.
Also, presumes that, for multiple environments, the desired behavior is that the agent trains on each env for max_episodes
and in total for max_episodes * n_envs episodes.

:::{note}
For multiple environments, the agent will train for a total of max_episodes * n_envs episodes.
However, it can't be guaranteed that this training will occur for an exact number of max_episodes per environment.
Thus, there is an assumption that, on average, each environment ran for max_episodes.
:::

from stable_baselines3 import A2C
from stable_baselines3.common.callbacks import StopTrainingOnMaxEpisodes

# Stops training when the model reaches the maximum number of episodes
callback_max_episodes = StopTrainingOnMaxEpisodes(max_episodes=5, verbose=1)

model = A2C("MlpPolicy", "Pendulum-v1", verbose=1)
# Almost infinite number of timesteps, but the training will stop
# early as soon as the max number of episodes is reached
model.learn(int(1e10), callback=callback_max_episodes)

(stoptrainingonnomodelimprovement)=

StopTrainingOnNoModelImprovement Copy Section

Stop the training if there is no new best model (no new best mean reward) after more than a specific number of consecutive evaluations.
The idea is to save time in experiments when you know that the learning curves are somehow well-behaved and, therefore,
after many evaluations without improvement the learning has probably stabilized.
It must be used with the {ref}EvalCallback and use the event triggered after every evaluation.

import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback, StopTrainingOnNoModelImprovement

# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
# Stop training if there is no improvement after more than 3 evaluations
stop_train_callback = StopTrainingOnNoModelImprovement(max_no_improvement_evals=3, min_evals=5, verbose=1)
eval_callback = EvalCallback(eval_env, eval_freq=1000, callback_after_eval=stop_train_callback, verbose=1)

model = SAC("MlpPolicy", "Pendulum-v1", learning_rate=1e-3, verbose=1)
# Almost infinite number of timesteps, but the training will stop early
# as soon as the number of consecutive evaluations without model
# improvement is greater than 3
model.learn(int(1e10), callback=eval_callback)

.. automodule:: stable_baselines3.common.callbacks
  :members:

Guide Checking Nan (`docs/guide/checking_nan.md`) Copy Section

Dealing with NaNs and infs

During the training of a model on a given environment, it is possible that the RL model becomes completely
corrupted when a NaN or an inf is given or returned from the RL model.

How and why? Copy Chapter

The issue arises when NaNs or infs do not crash, but simply get propagated through the training,
until all the floating point number converge to NaN or inf. This is in line with the
IEEE Standard for Floating-Point Arithmetic (IEEE 754) standard, as it says:

:::{note}
Five possible exceptions can occur:
: - Invalid operation ($\sqrt{-1}$, $\inf \times 1$, $\text{NaN}\ \mathrm{mod}\ 1$, ...) return NaN

• Division by zero:
  : - if the operand is not zero ($1/0$, $-2/0$, ...) returns $\pm\inf$
  • if the operand is zero ($0/0$) returns signaling NaN
• Overflow (exponent too high to represent) returns $\pm\inf$
• Underflow (exponent too low to represent) returns $0$
• Inexact (not representable exactly in base 2, eg: $1/5$) returns the rounded value (ex: {code}assert (1/5) * 3 == 0.6000000000000001)
  :::

And of these, only Division by zero will signal an exception, the rest will propagate invalid values quietly.

In python, dividing by zero will indeed raise the exception: ZeroDivisionError: float division by zero,
but ignores the rest.

The default in numpy, will warn: RuntimeWarning: invalid value encountered
but will not halt the code.

Anomaly detection with PyTorch Copy Chapter

To enable NaN detection in PyTorch you can do

import torch as th
th.autograd.set_detect_anomaly(True)

Numpy parameters Copy Chapter

Numpy has a convenient way of dealing with invalid value: numpy.seterr,
which defines for the python process, how it should handle floating point error.

import numpy as np

np.seterr(all="raise")  # define before your code.

print("numpy test:")

a = np.float64(1.0)
b = np.float64(0.0)
val = a / b  # this will now raise an exception instead of a warning.
print(val)

but this will also avoid overflow issues on floating point numbers:

import numpy as np

np.seterr(all="raise")  # define before your code.

print("numpy overflow test:")

a = np.float64(10)
b = np.float64(1000)
val = a ** b  # this will now raise an exception
print(val)

but will not avoid the propagation issues:

import numpy as np

np.seterr(all="raise")  # define before your code.

print("numpy propagation test:")

a = np.float64("NaN")
b = np.float64(1.0)
val = a + b  # this will neither warn nor raise anything
print(val)

VecCheckNan Wrapper Copy Chapter

In order to find when and from where the invalid value originated from, stable-baselines3 comes with a VecCheckNan wrapper.

It will monitor the actions, observations, and rewards, indicating what action or observation caused it and from what.

import gymnasium as gym
from gymnasium import spaces
import numpy as np

from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv, VecCheckNan

class NanAndInfEnv(gym.Env):
    """Custom Environment that raised NaNs and Infs"""
    metadata = {"render.modes": ["human"]}

    def __init__(self):
        super(NanAndInfEnv, self).__init__()
        self.action_space = spaces.Box(low=-np.inf, high=np.inf, shape=(1,), dtype=np.float64)
        self.observation_space = spaces.Box(low=-np.inf, high=np.inf, shape=(1,), dtype=np.float64)

    def step(self, _action):
        randf = np.random.rand()
        if randf > 0.99:
            obs = float("NaN")
        elif randf > 0.98:
            obs = float("inf")
        else:
            obs = randf
        return [obs], 0.0, False, {}

    def reset(self):
        return [0.0]

    def render(self, close=False):
        pass

# Create environment
env = DummyVecEnv([lambda: NanAndInfEnv()])
env = VecCheckNan(env, raise_exception=True)

# Instantiate the agent
model = PPO("MlpPolicy", env)

# Train the agent
model.learn(total_timesteps=int(2e5))  # this will crash explaining that the invalid value originated from the environment.

RL Model hyperparameters Copy Chapter

Depending on your hyperparameters, NaN can occur much more often.
A great example of this: https://github.com/hill-a/stable-baselines/issues/340

Be aware, the hyperparameters given by default seem to work in most cases,
however your environment might not play nice with them.
If this is the case, try to read up on the effect each hyperparameter has on the model,
so that you can try and tune them to get a stable model. Alternatively, you can try automatic hyperparameter tuning (included in the rl zoo).

Missing values from datasets Copy Chapter

If your environment is generated from an external dataset, do not forget to make sure your dataset does not contain NaNs.
As some datasets will sometimes fill missing values with NaNs as a surrogate value.

Here is some reading material about finding NaNs: https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html

And filling the missing values with something else (imputation): https://towardsdatascience.com/missing-data-in-time-series-machine-learning-techniques-6b2273ff8b45/

Guide Custom Env (`docs/guide/custom_env.md`) Copy Section

(custom-env)=

Using Custom Environments

To use the RL baselines with custom environments, they just need to follow the gymnasium interface.
That is to say, your environment must implement the following methods (and inherits from Gym Class):

:::{note}
If you are using images as input, the observation must be of type np.uint8 and be within a space Box bounded by [0, 255] (Box(low=0, high=255, shape=(<your image shape>))).
By default, the observation is normalized by SB3 pre-processing (dividing by 255 to have values in [0, 1], i.e. Box(low=0, high=1)) when using CNN policies.
Images can be either channel-first or channel-last.

If you want to use CnnPolicy or MultiInputPolicy with image-like observation (3D tensor) that are already normalized, you must pass normalize_images=False
to the policy (using policy_kwargs parameter, policy_kwargs=dict(normalize_images=False))
and make sure your image is in the channel-first format.
:::

:::{note}
Although SB3 supports both channel-last and channel-first images as input, we recommend using the channel-first convention when possible.
Under the hood, when a channel-last image is passed, SB3 uses a VecTransposeImage wrapper to re-order the channels.
:::

:::{note}
SB3 doesn't support Discrete and MultiDiscrete spaces with start!=0. However, you can update your environment or use a wrapper to make your env compatible with SB3:

import gymnasium as gym

class ShiftWrapper(gym.Wrapper):
    """Allow to use Discrete() action spaces with start!=0"""
    def __init__(self, env: gym.Env) -> None:
        super().__init__(env)
        assert isinstance(env.action_space, gym.spaces.Discrete)
        self.action_space = gym.spaces.Discrete(env.action_space.n, start=0)
    
    def step(self, action: int):
        return self.env.step(action + self.env.action_space.start)

:::

:::{note}
SB3 doesn't support MultiDiscrete spaces with multi-dimensional arrays. However, you can update your environment or use a wrapper to make your env compatible with SB3:

import numpy as np
import gymnasium as gym

class ReshapeWrapper(gym.Wrapper):
    """Allow to use MultiDiscrete() action spaces with len(nvec.shape) > 1:"""
    def __init__(self, env: gym.Env) -> None:
        super().__init__(env)
        assert isinstance(env.action_space, gym.spaces.MultiDiscrete)
        self.original_shape = env.action_space.nvec.shape
        self.action_space = gym.spaces.MultiDiscrete(env.action_space.nvec.flatten())
    
    def step(self, action: np.ndarray):
        return self.env.step(action.reshape(self.original_shape))

:::

import gymnasium as gym
import numpy as np
from gymnasium import spaces


class CustomEnv(gym.Env):
    """Custom Environment that follows gym interface."""

    metadata = {"render_modes": ["human"], "render_fps": 30}

    def __init__(self, arg1, arg2, ...):
        super().__init__()
        # Define action and observation space
        # They must be gym.spaces objects
        # Example when using discrete actions:
        self.action_space = spaces.Discrete(N_DISCRETE_ACTIONS)
        # Example for using image as input (channel-first; channel-last also works):
        self.observation_space = spaces.Box(low=0, high=255,
                                            shape=(N_CHANNELS, HEIGHT, WIDTH), dtype=np.uint8)

    def step(self, action):
        ...
        return observation, reward, terminated, truncated, info

    def reset(self, seed=None, options=None):
        ...
        return observation, info

    def render(self):
        ...

    def close(self):
        ...

Then you can define and train a RL agent with:

# Instantiate the env
env = CustomEnv(arg1, ...)
# Define and Train the agent
model = A2C("CnnPolicy", env).learn(total_timesteps=1000)

To check that your environment follows the Gym interface that SB3 supports, please use:

from stable_baselines3.common.env_checker import check_env

env = CustomEnv(arg1, ...)
# It will check your custom environment and output additional warnings if needed
check_env(env)

Gymnasium also has its own env checker but it checks a superset of what SB3 supports (SB3 does not support all Gym features).

We have created a colab notebook for a concrete example of creating a custom environment along with an example of using it with Stable-Baselines3 interface.

Alternatively, you may look at Gymnasium built-in environments.

Optionally, you can also register the environment with gym, that will allow you to create the RL agent in one line (and use gym.make() to instantiate the env):

from gymnasium.envs.registration import register
# Example for the CartPole environment
register(
    # unique identifier for the env `name-version`
    id="CartPole-v1",
    # path to the class for creating the env
    # Note: entry_point also accept a class as input (and not only a string)
    entry_point="gym.envs.classic_control:CartPoleEnv",
    # Max number of steps per episode, using a `TimeLimitWrapper`
    max_episode_steps=500,
)

In the project, for testing purposes, we use a custom environment named IdentityEnv
defined in this file.
An example of how to use it can be found here.

Guide Custom Policy (`docs/guide/custom_policy.md`) Copy Section

(custom-policy)=

Policy Networks

Stable Baselines3 provides policy networks for images (CnnPolicies),
other type of input features (MlpPolicies) and multiple different inputs (MultiInputPolicies).

:::{warning}
For A2C and PPO, continuous actions are clipped during training and testing
(to avoid out of bound error). SAC, DDPG and TD3 squash the action, using a tanh() transformation,
which handles bounds more correctly.
:::

SB3 Policy Copy Chapter

SB3 networks are separated into two main parts (see figure below):

• A features extractor (usually shared between actor and critic when applicable, to save computation)
  whose role is to extract features (i.e. convert to a feature vector) from high-dimensional observations, for instance, a CNN that extracts features from images.
  This is the features_extractor_class parameter. You can change the default parameters of that features extractor
  by passing a features_extractor_kwargs parameter.
• A (fully-connected) network that maps the features to actions/value. Its architecture is controlled by the net_arch parameter.

:::{note}
All observations are first pre-processed (e.g. images are normalized, discrete obs are converted to one-hot vectors, ...) before being fed to the features extractor.
In the case of vector observations, the features extractor is just a Flatten layer.
:::

SB3 policies are usually composed of several networks (actor/critic networks + target networks when applicable) together
with the associated optimizers.

Each of these network have a features extractor followed by a fully-connected network.

:::{note}
When we refer to "policy" in Stable-Baselines3, this is usually an abuse of language compared to RL terminology.
In SB3, "policy" refers to the class that handles all the networks useful for training,
so not only the network used to predict actions (the "learned controller").
:::

Default Network Architecture Copy Chapter

The default network architecture used by SB3 depends on the algorithm and the observation space.
You can visualize the architecture by printing model.policy (see issue #329).

For 1D observation space, a 2 layers fully connected net is used with:

• 64 units (per layer) for PPO/A2C/DQN
• 256 units for SAC
• [400, 300] units for TD3/DDPG (values are taken from the original TD3 paper)

For image observation spaces, the "Nature CNN" (see code for more details) is used for feature extraction, and SAC/TD3 also keeps the same fully connected network after it.
The other algorithms only have a linear layer after the CNN.
The CNN is shared between actor and critic for A2C/PPO (on-policy algorithms) to reduce computation.
Off-policy algorithms (TD3, DDPG, SAC, ...) have separate feature extractors: one for the actor and one for the critic, since the best performance is obtained with this configuration.

For mixed observations (dictionary observations), the two architectures from above are used, i.e., CNN for images and then two layers fully-connected network
(with a smaller output size for the CNN).

Custom Network Architecture Copy Chapter

One way of customising the policy network architecture is to pass arguments when creating the model,
using policy_kwargs parameter:

:::{note}
An extra linear layer will be added on top of the layers specified in net_arch, in order to have the right output dimensions and activation functions (e.g. Softmax for discrete actions).

In the following example, as CartPole's action space has a dimension of 2, the final dimensions of the net_arch's layers will be:

obs
        <4>
   /            \
 <32>          <32>
  |              |
 <32>          <32>
  |              |
 <2>            <1>
action         value

:::

import gymnasium as gym
import torch as th

from stable_baselines3 import PPO

# Custom actor (pi) and value function (vf) networks
# of two layers of size 32 each with Relu activation function
# Note: an extra linear layer will be added on top of the pi and the vf nets, respectively
policy_kwargs = dict(activation_fn=th.nn.ReLU,
                     net_arch=dict(pi=[32, 32], vf=[32, 32]))
# Create the agent
model = PPO("MlpPolicy", "CartPole-v1", policy_kwargs=policy_kwargs, verbose=1)
# Retrieve the environment
env = model.get_env()
# Train the agent
model.learn(total_timesteps=20_000)
# Save the agent
model.save("ppo_cartpole")

del model
# the policy_kwargs are automatically loaded
model = PPO.load("ppo_cartpole", env=env)

Custom Feature Extractor Copy Chapter

If you want to have a custom features extractor (e.g. custom CNN when using images), you can define class
that derives from BaseFeaturesExtractor and then pass it to the model when training.

:::{note}
For on-policy algorithms, the features extractor is shared by default between the actor and the critic to save computation (when applicable).
However, this can be changed setting share_features_extractor=False in the
policy_kwargs (both for on-policy and off-policy algorithms).
:::

/* Detailed source-code truncated for AI context efficiency. */

Multiple Inputs and Dictionary Observations Copy Chapter

Stable Baselines3 supports handling of multiple inputs by using Dict Gym space. This can be done using
MultiInputPolicy, which by default uses the CombinedExtractor features extractor to turn multiple
inputs into a single vector, handled by the net_arch network.

By default, CombinedExtractor processes multiple inputs as follows:

1. If input is an image (automatically detected, see common.preprocessing.is_image_space), process image with Nature Atari CNN network and
   output a latent vector of size 256.
2. If input is not an image, flatten it (no layers).
3. Concatenate all previous vectors into one long vector and pass it to policy.

Much like above, you can define custom features extractors. The following example assumes the environment has two keys in the
observation space dictionary: "image" is a (1,H,W) image (channel first), and "vector" is a (D,) dimensional vector. We process "image" with a simple
downsampling and "vector" with a single linear layer.

/* Detailed source-code truncated for AI context efficiency. */

On-Policy Algorithms Copy Chapter

Custom Networks Copy Section

If you need a network architecture that is different for the actor and the critic when using PPO, A2C or TRPO,
you can pass a dictionary of the following structure: dict(pi=[<actor network architecture>], vf=[<critic network architecture>]).

For example, if you want a different architecture for the actor (aka pi) and the critic (value-function aka vf) networks,
then you can specify net_arch=dict(pi=[32, 32], vf=[64, 64]).

Otherwise, to have actor and critic that share the same network architecture,
you only need to specify net_arch=[128, 128] (here, two hidden layers of 128 units each, this is equivalent to net_arch=dict(pi=[128, 128], vf=[128, 128])).

If shared layers are needed, you need to implement a custom policy network (see advanced example below).

Examples

Same architecture for actor and critic with two layers of size 128: net_arch=[128, 128]

obs
   /            \
 <128>          <128>
  |              |
 <128>          <128>
  |              |
action         value

Different architectures for actor and critic: net_arch=dict(pi=[32, 32], vf=[64, 64])

obs
   /            \
 <32>          <64>
  |              |
 <32>          <64>
  |              |
action         value

Advanced Example

If your task requires even more granular control over the policy/value architecture, you can redefine the policy directly:

/* Detailed source-code truncated for AI context efficiency. */

Off-Policy Algorithms Copy Chapter

If you need a network architecture that is different for the actor and the critic when using SAC, DDPG, TQC or TD3,
you can pass a dictionary of the following structure: dict(pi=[<actor network architecture>], qf=[<critic network architecture>]).

For example, if you want a different architecture for the actor (aka pi) and the critic (Q-function aka qf) networks,
then you can specify net_arch=dict(pi=[64, 64], qf=[400, 300]).

Otherwise, to have actor and critic that share the same network architecture,
you only need to specify net_arch=[256, 256] (here, two hidden layers of 256 units each).

:::{note}
For advanced customization of off-policy algorithms policies, please take a look at the code.
A good understanding of the algorithm used is required, see discussion in issue #425
:::

from stable_baselines3 import SAC

# Custom actor architecture with two layers of 64 units each
# Custom critic architecture with two layers of 400 and 300 units
policy_kwargs = dict(net_arch=dict(pi=[64, 64], qf=[400, 300]))
# Create the agent
model = SAC("MlpPolicy", "Pendulum-v1", policy_kwargs=policy_kwargs, verbose=1)
model.learn(5000)

Guide Developer (`docs/guide/developer.md`) Copy Section

(developer)=

Developer Guide

This guide is meant for those who want to understand the internals and the design choices of Stable-Baselines3.

At first, you should read the two issues where the design choices were discussed:

• https://github.com/hill-a/stable-baselines/issues/576
• https://github.com/hill-a/stable-baselines/issues/733

The library is not meant to be modular, although inheritance is used to reduce code duplication.

Algorithms Structure Copy Chapter

Each algorithm (on-policy and off-policy ones) follows a common structure.
Policy contains code for acting in the environment, and algorithm updates this policy.
There is one folder per algorithm, and in that folder there is the algorithm and the policy definition (policies.py).

Each algorithm has two main methods:

• .collect_rollouts() which defines how new samples are collected, usually inherited from the base class. Those samples are then stored in a RolloutBuffer (discarded after the gradient update) or ReplayBuffer
• .train() which updates the parameters using samples from the buffer

Where to start? Copy Chapter

The first thing you need to read and understand are the base classes in the common/ folder:

• BaseAlgorithm in base_class.py which defines how an RL class should look like.
  It contains also all the "glue code" for saving/loading and the common operations (wrapping environments)
• BasePolicy in policies.py which defines how a policy class should look like.
  It contains also all the magic for the .predict() method, to handle as many spaces/cases as possible
• OffPolicyAlgorithm in off_policy_algorithm.py that contains the implementation of collect_rollouts() for the off-policy algorithms,
  and similarly OnPolicyAlgorithm in on_policy_algorithm.py.

All the environments handled internally are assumed to be VecEnv (gym.Env are automatically wrapped).

Pre-Processing Copy Chapter

To handle different observation spaces, some pre-processing needs to be done (e.g. one-hot encoding for discrete observation).
Most of the code for pre-processing is in common/preprocessing.py and common/policies.py.

For images, environment is automatically wrapped with VecTransposeImage if observations are detected to be images with
channel-last convention to transform it to PyTorch's channel-first convention.

Policy Structure Copy Chapter

When we refer to "policy" in Stable-Baselines3, this is usually an abuse of language compared to RL terminology.
In SB3, "policy" refers to the class that handles all the networks useful for training,
so not only the network used to predict actions (the "learned controller").
For instance, the TD3 policy contains the actor, the critic and the target networks.

To avoid the hassle of importing specific policy classes for specific algorithm (e.g. both A2C and PPO use ActorCriticPolicy),
SB3 uses names like "MlpPolicy" and "CnnPolicy" to refer policies using small feed-forward networks or convolutional networks,
respectively. Importing [algorithm]/policies.py registers an appropriate policy for that algorithm under those names.

Probability distributions Copy Chapter

When needed, the policies handle the different probability distributions.
All distributions are located in common/distributions.py and follow the same interface.
Each distribution corresponds to a type of action space (e.g. Categorical is the one used for discrete actions.
For continuous actions, we can use multiple distributions ("DiagGaussian", "SquashedGaussian" or "StateDependentDistribution")

State-Dependent Exploration Copy Chapter

State-Dependent Exploration (SDE) is a type of exploration that allows to use RL directly on real robots,
that was the starting point for the Stable-Baselines3 library.
I (@araffin) published a paper about a generalized version of SDE (the one implemented in SB3): https://arxiv.org/abs/2005.05719

Misc Copy Chapter

The rest of the common/ is composed of helpers (e.g. evaluation helpers) or basic components (like the callbacks).
The type_aliases.py file contains common type hint aliases like GymStepReturn.

Et voilà?

After reading this guide and the mentioned files, you should be now able to understand the design logic behind the library ;)

Guide Examples (`docs/guide/examples.md`) Copy Section

myst:
substitutions:
colab: |-
{image} ../_static/img/colab.svg

(examples)=

Examples

:::{note}
These examples are only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in the RL Zoo repository.
:::

Try it online with Colab Notebooks! Copy Chapter

All the following examples can be executed online using Google Colab
{{ colab }} notebooks:

• Full Tutorial
• All Notebooks
• Getting Started
• Training, Saving, Loading
• Multiprocessing
• Monitor Training and Plotting
• Atari Games
• RL Baselines zoo
• PyBullet
• Hindsight Experience Replay
• Advanced Saving and Loading

Basic Usage: Training, Saving, Loading Copy Chapter

In the following example, we will train, save and load a DQN model on the Lunar Lander environment.

:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/saving_loading_dqn.ipynb

:::{figure} https://cdn-images-1.medium.com/max/960/1*f4VZPKOI0PYNWiwt0la0Rg.gif
Lunar Lander Environment
:::

:::{note}
LunarLander requires the Python package box2d.
You can install it using apt install swig and then pip install box2d box2d-kengz
:::

:::{warning}
The load method recreates the model from scratch and should be called on the algorithm without instantiating it first,
e.g. model = DQN.load("dqn_lunar", env=env) instead of model = DQN(env=env) followed by model.load("dqn_lunar"). The latter will not work as load is not an in-place operation.
If you want to load parameters without recreating the model, e.g. to evaluate the same model
with multiple different sets of parameters, consider using set_parameters instead.
:::

import gymnasium as gym

from stable_baselines3 import DQN
from stable_baselines3.common.evaluation import evaluate_policy


# Create environment
env = gym.make("LunarLander-v3", render_mode="rgb_array")

# Instantiate the agent
# Note: you can enable multi-step replay buffer by passing `n_steps=3` for instance
model = DQN("MlpPolicy", env, verbose=1)
# Train the agent and display a progress bar
model.learn(total_timesteps=int(2e5), progress_bar=True)
# Save the agent
model.save("dqn_lunar")
del model  # delete trained model to demonstrate loading

# Load the trained agent
# NOTE: if you have loading issue, you can pass `print_system_info=True`
# to compare the system on which the model was trained vs the current one
# model = DQN.load("dqn_lunar", env=env, print_system_info=True)
model = DQN.load("dqn_lunar", env=env)

# Evaluate the agent
# NOTE: If you use wrappers with your environment that modify rewards,
#       this will be reflected here. To evaluate with original rewards,
#       wrap the environment in a "Monitor" wrapper before other wrappers.
mean_reward, std_reward = evaluate_policy(model, model.get_env(), n_eval_episodes=10)

# Enjoy trained agent
vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
    action, _states = model.predict(obs, deterministic=True)
    obs, rewards, dones, info = vec_env.step(action)
    vec_env.render("human")

Multiprocessing: Unleashing the Power of Vectorized Environments Copy Chapter

:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb

:::{figure} https://cdn-images-1.medium.com/max/960/1*h4WTQNVIsvMXJTCpXm_TAw.gif
CartPole Environment
:::

/* Detailed source-code truncated for AI context efficiency. */

Multiprocessing with off-policy algorithms Copy Chapter

:::{warning}
When using multiple environments with off-policy algorithms, you should update the gradient_steps
parameter too. Set it to gradient_steps=-1 to perform as many gradient steps as transitions collected.
There is usually a trade-off between wall-clock time and sample efficiency,
see this example in PR #439
:::

import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.env_util import make_vec_env

vec_env = make_vec_env("Pendulum-v0", n_envs=4, seed=0)

# We collect 4 transitions per call to `env.step()`
# and performs 2 gradient steps per call to `env.step()`
# if gradient_steps=-1, then we would do 4 gradients steps per call to `env.step()`
model = SAC("MlpPolicy", vec_env, train_freq=1, gradient_steps=2, verbose=1)
model.learn(total_timesteps=10_000)

Dict Observations Copy Chapter

You can use environments with dictionary observation spaces. This is useful in the case where one can't directly
concatenate observations such as an image from a camera combined with a vector of servo sensor data (e.g., rotation angles).
Stable Baselines3 provides SimpleMultiObsEnv as an example of this kind of setting.
The environment is a simple grid world, but the observations for each cell come in the form of dictionaries.
These dictionaries are randomly initialized on the creation of the environment and contain a vector observation and an image observation.

from stable_baselines3 import PPO
from stable_baselines3.common.envs import SimpleMultiObsEnv


# Stable Baselines provides SimpleMultiObsEnv as an example environment with Dict observations
env = SimpleMultiObsEnv(random_start=False)

model = PPO("MultiInputPolicy", env, verbose=1)
model.learn(total_timesteps=100_000)

Callbacks: Monitoring Training Copy Chapter

:::{note}
We recommend reading the Callback section
:::

You can define a custom callback function that will be called inside the agent.
This could be useful when you want to monitor training, for instance, display live
learning curves in TensorBoard or save the best agent.
If your callback returns False, training is aborted early.

:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/monitor_training.ipynb

/* Detailed source-code truncated for AI context efficiency. */

Callbacks: Evaluate Agent Performance Copy Chapter

To periodically evaluate an agent's performance on a separate test environment, use EvalCallback.
You can control the evaluation frequency with eval_freq to monitor your agent's progress during training.

import os
import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback
from stable_baselines3.common.env_util import make_vec_env

env_id = "Pendulum-v1"
n_training_envs = 1
n_eval_envs = 5

# Create log dir where evaluation results will be saved
eval_log_dir = "./eval_logs/"
os.makedirs(eval_log_dir, exist_ok=True)

# Initialize a vectorized training environment with default parameters
train_env = make_vec_env(env_id, n_envs=n_training_envs, seed=0)

# Separate evaluation env, with different parameters passed via env_kwargs
# Eval environments can be vectorized to speed up evaluation.
eval_env = make_vec_env(env_id, n_envs=n_eval_envs, seed=0,
                        env_kwargs={'g':0.7})

# Create a callback that evaluates an agent for 5 episodes every 500 training environment steps.
# When using multiple training environments, agent will be evaluated every
# eval_freq calls to train_env.step(), thus it will be evaluated every
# (eval_freq * n_envs) training steps. See EvalCallback doc for more information.
eval_callback = EvalCallback(eval_env, best_model_save_path=eval_log_dir,
                              log_path=eval_log_dir, eval_freq=max(500 // n_training_envs, 1),
                              n_eval_episodes=5, deterministic=True,
                              render=False)

model = SAC("MlpPolicy", train_env)
model.learn(5000, callback=eval_callback)

Atari Games Copy Chapter

:::{figure} ../_static/img/breakout.gif
Trained A2C agent on Breakout
:::

:::{figure} https://cdn-images-1.medium.com/max/960/1*UHYJE7lF8IDZS_U5SsAFUQ.gif
Pong Environment
:::

Training an RL agent on Atari games is straightforward thanks to the make_atari_env helper function.
It will do all the preprocessing
and multiprocessing for you. To install the Atari environments, run the command pip install gymnasium[atari,accept-rom-license] to install the Atari environments and ROMs, or install Stable Baselines3 with pip install stable-baselines3[extra] to install this and other optional dependencies.

:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/atari_games.ipynb

:::{note}
When working with Atari environments, be aware that the default terminal_on_life_loss=True behavior
can cause env.reset() to perform a no-op step instead of truly resetting the environment when
the episode ends due to a life loss (not game over, see issue #666).
To ensure reset() always resets the environment, use:

from stable_baselines3.common.env_util import make_atari_env

import ale_py

env = make_atari_env(
    "BreakoutNoFrameskip-v4",
    n_envs=1,
    wrapper_kwargs=dict(terminal_on_life_loss=False)
)

:::

from stable_baselines3.common.env_util import make_atari_env
from stable_baselines3.common.vec_env import VecFrameStack
from stable_baselines3 import A2C

import ale_py

# There already exists an environment generator
# that will make and wrap atari environments correctly.
# Here we are also multi-worker training (n_envs=4 => 4 environments)
vec_env = make_atari_env("PongNoFrameskip-v4", n_envs=4, seed=0)
# Frame-stacking with 4 frames
vec_env = VecFrameStack(vec_env, n_stack=4)

model = A2C("CnnPolicy", vec_env, verbose=1)
model.learn(total_timesteps=25_000)

obs = vec_env.reset()
while True:
    action, _states = model.predict(obs, deterministic=False)
    obs, rewards, dones, info = vec_env.step(action)
    vec_env.render("human")

PyBullet: Normalizing input features Copy Chapter

Normalizing input features may be essential for successful training of an RL agent
(by default, images are scaled, but other types of input are not),
for instance, when training on PyBullet environments.
For this, there is a wrapper VecNormalize that will compute a running average and standard deviation of the input features (it can do the same for rewards).

:::{note}
You need to install PyBullet environments with pip install pybullet_envs_gymnasium
:::

:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/pybullet.ipynb

from pathlib import Path

import pybullet_envs_gymnasium

from stable_baselines3.common.vec_env import VecNormalize
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3 import PPO

# Alternatively, you can use the MuJoCo equivalent "HalfCheetah-v4"
vec_env = make_vec_env("HalfCheetahBulletEnv-v0", n_envs=1)
# Automatically normalize the input features and reward
vec_env = VecNormalize(vec_env, norm_obs=True, norm_reward=True, clip_obs=10.0)

model = PPO("MlpPolicy", vec_env)
model.learn(total_timesteps=2000)

# Don't forget to save the VecNormalize statistics when saving the agent
log_dir = Path("/tmp/")
model.save(log_dir / "ppo_halfcheetah")
stats_path = log_dir / "vec_normalize.pkl"
vec_env.save(stats_path)

# To demonstrate loading
del model, vec_env

# Load the saved statistics
vec_env = make_vec_env("HalfCheetahBulletEnv-v0", n_envs=1)
vec_env = VecNormalize.load(stats_path, vec_env)
#  do not update them at test time
vec_env.training = False
# reward normalization is not needed at test time
vec_env.norm_reward = False

# Load the agent
model = PPO.load(log_dir / "ppo_halfcheetah", env=vec_env)

Hindsight Experience Replay (HER) Copy Chapter

For this example, we use Highway-Env by @eleurent.

:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/stable_baselines_her.ipynb

:::{figure} https://raw.githubusercontent.com/eleurent/highway-env/gh-media/docs/media/parking-env.gif
The highway-parking-v0 environment.
:::

The parking env is a goal-conditioned continuous control task, in which the vehicle must park in a given space with the appropriate heading.

:::{note}
The hyperparameters in the following example were optimized for that environment.
:::

import gymnasium as gym
import highway_env
import numpy as np

from stable_baselines3 import HerReplayBuffer, SAC, DDPG, TD3
from stable_baselines3.common.noise import NormalActionNoise

env = gym.make("parking-v0")

# Create 4 artificial transitions per real transition
n_sampled_goal = 4

# SAC hyperparams:
model = SAC(
    "MultiInputPolicy",
    env,
    replay_buffer_class=HerReplayBuffer,
    replay_buffer_kwargs=dict(
      n_sampled_goal=n_sampled_goal,
      goal_selection_strategy="future",
    ),
    verbose=1,
    buffer_size=int(1e6),
    learning_rate=1e-3,
    gamma=0.95,
    batch_size=256,
    policy_kwargs=dict(net_arch=[256, 256, 256]),
)

model.learn(int(2e5))
model.save("her_sac_highway")

# Load saved model
# Because it needs access to `env.compute_reward()`
# HER must be loaded with the env
env = gym.make("parking-v0", render_mode="human") # Change the render mode
model = SAC.load("her_sac_highway", env=env)

obs, info = env.reset()

# Evaluate the agent
episode_reward = 0
for _ in range(100):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, info = env.step(action)
    episode_reward += reward
    if terminated or truncated or info.get("is_success", False):
        print("Reward:", episode_reward, "Success?", info.get("is_success", False))
        episode_reward = 0.0
        obs, info = env.reset()

Learning Rate Schedule Copy Chapter

All algorithms allow you to pass a learning rate schedule that takes as input the current progress remaining (from 1 to 0).
PPO's clip_range` parameter also accepts such schedule.

The RL Zoo already includes
linear and constant schedules.

from typing import Callable

from stable_baselines3 import PPO


def linear_schedule(initial_value: float) -> Callable[[float], float]:
    """
    Linear learning rate schedule.

    :param initial_value: Initial learning rate.
    :return: schedule that computes
      current learning rate depending on remaining progress
    """
    def func(progress_remaining: float) -> float:
        """
        Progress will decrease from 1 (beginning) to 0.

        :param progress_remaining:
        :return: current learning rate
        """
        return progress_remaining * initial_value

    return func

# Initial learning rate of 0.001
model = PPO("MlpPolicy", "CartPole-v1", learning_rate=linear_schedule(0.001), verbose=1)
model.learn(total_timesteps=20_000)
# By default, `reset_num_timesteps` is True, in which case the learning rate schedule resets.
# progress_remaining = 1.0 - (num_timesteps / total_timesteps)
model.learn(total_timesteps=10_000, reset_num_timesteps=True)

Advanced Saving and Loading Copy Chapter

In this example, we show how to use a policy independently from a model (and how to save it, load it) and save/load a replay buffer.

By default, the replay buffer is not saved when calling model.save(), in order to save space on the disk (a replay buffer can be up to several GB when using images).
However, SB3 provides a save_replay_buffer() and load_replay_buffer() method to save it separately.

:::{note}
For training model after loading it, we recommend loading the replay buffer to ensure stable learning (for off-policy algorithms).
You also need to pass reset_num_timesteps=True to learn function which initializes the environment
and agent for training if a new environment was created since saving the model.
:::

:target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/advanced_saving_loading.ipynb

/* Detailed source-code truncated for AI context efficiency. */

Accessing and modifying model parameters Copy Chapter

You can access model's parameters via set_parameters and get_parameters functions,
or via model.policy.state_dict() (and load_state_dict()),
which use dictionaries that map variable names to PyTorch tensors.

These functions are useful when you need to e.g. evaluate large set of models with same network structure,
visualize different layers of the network or modify parameters manually.

Policies also offers a simple way to save/load weights as a NumPy vector, using parameters_to_vector()
and load_from_vector() method.

Following example demonstrates reading parameters, modifying some of them and loading them to model
by implementing evolution strategy (es)
for solving the CartPole-v1 environment. The initial guess for parameters is obtained by running
A2C policy gradient updates on the model.

/* Detailed source-code truncated for AI context efficiency. */

SB3 with Isaac Lab, Brax, Procgen, EnvPool Copy Chapter

Some massively parallel simulation environments such as EnvPool, Isaac Lab, Brax or ProcGen already produce a vectorized environment to speed up data collection (see discussion in issue #314).

To use SB3 with these tools, you need to wrap the environment with tool-specific VecEnvWrapper that preprocesses the data for SB3,
you can find links to some of these wrappers in issue #772.

• Isaac Lab wrapper: link
• Brax: link
• EnvPool: link
• Getting SAC to Work on a Massive Parallel Simulator: https://araffin.github.io/post/sac-massive-sim/

SB3 with DeepMind Control (dm_control) Copy Chapter

If you want to use SB3 with dm_control, you need to apply two wrappers (one from shimmy, one pre-built one) to convert it to a Gymnasium compatible environment:

import shimmy
import stable_baselines3 as sb3
from dm_control import suite
from gymnasium.wrappers import FlattenObservation

# Available envs:
# suite._DOMAINS and suite.dog.SUITE

env = suite.load(domain_name="dog", task_name="run")
gym_env = FlattenObservation(shimmy.DmControlCompatibilityV0(env))

model = sb3.PPO("MlpPolicy", gym_env, verbose=1)
model.learn(10_000, progress_bar=True)

Record a Video Copy Chapter

Record an MP4 video (here using a random agent).

:::{note}
It requires ffmpeg or avconv to be installed on the machine.
:::

import gymnasium as gym
from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv

env_id = "CartPole-v1"
video_folder = "logs/videos/"
video_length = 100

vec_env = DummyVecEnv([lambda: gym.make(env_id, render_mode="rgb_array")])

obs = vec_env.reset()

# Record the video starting at the first step
vec_env = VecVideoRecorder(vec_env, video_folder,
                       record_video_trigger=lambda x: x == 0, video_length=video_length,
                       name_prefix=f"random-agent-{env_id}")

vec_env.reset()
for _ in range(video_length + 1):
  action = [vec_env.action_space.sample()]
  obs, _, _, _ = vec_env.step(action)
# Save the video
vec_env.close()

Compiling models Copy Chapter

To make models faster, you can compile them using torch.compile().

import torch as th
from stable_baselines3 import PPO

env_id = "CartPole-v1"
model = PPO("MlpPolicy", env_id, verbose=1)

model.policy = th.compile(model.policy, backend="inductor")
model.learn(total_timesteps=10_000)

:::{note}
To list available backends, you can use torch._dynamo.list_backends().
:::

Bonus: Make a GIF of a Trained Agent Copy Chapter

import imageio
import numpy as np

from stable_baselines3 import A2C

model = A2C("MlpPolicy", "LunarLander-v3").learn(100_000)

images = []
obs = model.env.reset()
img = model.env.render(mode="rgb_array")
for i in range(350):
    images.append(img)
    action, _ = model.predict(obs)
    obs, _, _, _ = model.env.step(action)
    img = model.env.render(mode="rgb_array")

imageio.mimsave("lander_a2c.gif", [np.array(img) for i, img in enumerate(images) if i%2 == 0], fps=29)

Guide Export (`docs/guide/export.md`) Copy Section

(export)=

Exporting models

After training an agent, you may want to deploy/use it in another language
or framework, like tensorflowjs.
Stable Baselines3 does not include tools to export models to other frameworks, but
this document aims to cover parts that are required for exporting along with
more detailed stories from users of Stable Baselines3.

Background Copy Chapter

In Stable Baselines3, the controller is stored inside policies which convert
observations into actions. Each learning algorithm (e.g. DQN, A2C, SAC)
contains a policy object which represents the currently learned behavior,
accessible via model.policy.

Policies hold enough information to do the inference (i.e. predict actions),
so it is enough to export these policies (cf {ref}examples <examples>)
to do inference in another framework.

:::{warning}
When using CNN policies, the observation is normalized during preprocessing.
This pre-processing is done inside the policy (dividing by 255 to have values in [0, 1])
:::

Export to ONNX Copy Chapter

If you are using PyTorch 2.0+ and ONNX Opset 14+, you can easily export SB3 policies using the following code:

:::{warning}
The following returns normalized actions and doesn't include the post-processing step that is done with continuous actions (clip or unscale the action to the correct space).
:::

/* Detailed source-code truncated for AI context efficiency. */

For exporting MultiInputPolicy, please have a look at GH#1873.

For SAC the procedure is similar. The example shown only exports the actor network as the actor is sufficient to roll out the trained policies.

/* Detailed source-code truncated for AI context efficiency. */

For more discussion around the topic, please refer to GH#383 and GH#1349.

Trace/Export to C++ Copy Chapter

You can use PyTorch JIT to trace and save a trained model that can be reused in other applications
(for instance inference code written in C++).

There is a draft PR in the RL Zoo about C++ export: https://github.com/DLR-RM/rl-baselines3-zoo/pull/228

# See "ONNX export" for imports and OnnxablePolicy
jit_path = "sac_traced.pt"

# Trace and optimize the module
traced_module = th.jit.trace(onnxable_model.eval(), dummy_input)
frozen_module = th.jit.freeze(traced_module)
frozen_module = th.jit.optimize_for_inference(frozen_module)
th.jit.save(frozen_module, jit_path)

##### Load and test with torch

import torch as th

dummy_input = th.randn(1, *observation_size)
loaded_module = th.jit.load(jit_path)
action_jit = loaded_module(dummy_input)

Export to ONNX-JS / ONNX Runtime Web Copy Chapter

Official documentation: https://onnxruntime.ai/docs/tutorials/web/build-web-app.html

Full example code: https://github.com/JonathanColetti/CarDodgingGym

Demo: https://jonathancoletti.github.io/CarDodgingGym

The code linked above is a complete example (using car dodging environment) that:

1. Creates/Trains a PPO model
2. Exports the model to ONNX along with normalization stats in JSON
3. Runs in the browser with normalization using onnxruntime-web to achieve similar results

Below is a simple example with converting to ONNX then inferencing without postprocess in ONNX-JS

import torch as th

from stable_baselines3 import SAC


class OnnxablePolicy(th.nn.Module):
    def __init__(self, actor: th.nn.Module):
        super().__init__()
        self.actor = actor

    def forward(self, observation: th.Tensor) -> th.Tensor:
        # NOTE: You may have to postprocess (unnormalize or renormalize)
        return self.actor(observation, deterministic=True)


# Example: model = SAC("MlpPolicy", "Pendulum-v1")
SAC("MlpPolicy", "Pendulum-v1").save("PathToTrainedModel.zip")
model = SAC.load("PathToTrainedModel.zip", device="cpu")
onnxable_model = OnnxablePolicy(model.policy.actor)

observation_size = model.observation_space.shape
dummy_input = th.randn(1, *observation_size)
th.onnx.export(
    onnxable_model,
    dummy_input,
    "my_sac_actor.onnx",
    opset_version=17,
    input_names=["input"],
)

// Install using `npm install onnxruntime-web` (tested with version 1.19) or using cdn
import * as ort from 'onnxruntime-web';

async function runInference() {
  const session = await ort.InferenceSession.create('my_sac_actor.onnx');

  // The observation_size = 3 (for Pendulum-v1)
  const inputData = Float32Array.from([0.1, -0.2, 0.3]);

  const inputTensor = new ort.Tensor('float32', inputData, [1, 3]);

  const results = await session.run({ input: inputTensor });

  const outputName = session.outputNames[0];
  const action = results[outputName].data;

  console.log('Predicted action=', action);
}

runInference();

Export to TensorFlow.js Copy Chapter

:::{warning}
As of November 2025, onnx2tf does not support TensorFlow.js. Therefore, tfjs-converter is used instead. However, tfjs-converter is not currently maintained and requires older opsets and TensorFlow versions.
:::

In order for this to work, you have to do multiple conversions: SB3 => ONNX => TensorFlow => TensorFlow.js.

The opset version needs to be changed for the conversion (opset_version=14 is currently required). Please refer to the code above for more stable usage with a higher opset.

The following is a simple example that showcases the full conversion + inference.

Please refer to the previous sections for the first step (SB3 => ONNX).
The main difference is that you need to specify opset_version=14.

# Tested with python3.10
# Then install these dependencies in a fresh env
"""
pip install --use-deprecated=legacy-resolver tensorflow==2.13.0 keras==2.13.1 onnx==1.16.0 onnx-tf==1.9.0 tensorflow-probability==0.21.0 tensorflowjs==4.15.0 jax==0.4.26 jaxlib==0.4.26
"""
# Then run this codeblock
# If there are no errors (the folder is structure correctly) then
"""
# tensorflowjs_converter --input_format=tf_saved_model --output_format=tfjs_graph_model tf_model tfjs_model
"""

# If you get an error exporting using `tensorflowjs_converter` then upgrade tensorflow
"""
pip install --upgrade tensorflow tensorflow-decision-forests tensorflowjs
"""
# And retry with and it should work (do not rerun this codeblock)
"""
tensorflowjs_converter --input_format=tf_saved_model --output_format=tfjs_graph_model tf_model tfjs_model
"""

import onnx
import onnx_tf.backend
import tensorflow as tf

ONNX_FILE_PATH = "my_sac_actor.onnx"
MODEL_PATH = "tf_model"

onnx_model = onnx.load(ONNX_FILE_PATH)
onnx.checker.check_model(onnx_model)
print(onnx.helper.printable_graph(onnx_model.graph))

print('Converting ONNX to TF...')
tf_rep = onnx_tf.backend.prepare(onnx_model)
tf_rep.export_graph(MODEL_PATH)
# After this do not forget to use `tensorflowjs_converter`

import * as tf from 'https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]/+esm';
// Post processing not included
async function runInference() {
    const MODEL_URL = './tfjs_model/model.json';

    const model = await tf.loadGraphModel(MODEL_URL);

    // Observation_size is 3 for Pendulum-v1
    const inputData = [1.0, 0.0, 0.0];
    const inputTensor = tf.tensor2d([inputData], [1, 3]);

    const resultTensor = model.execute(inputTensor);

    const action = await resultTensor.data();

    console.log('Predicted action=', action);

    inputTensor.dispose();
    resultTensor.dispose();
}

runInference();

Export to TFLite / Coral (Edge TPU) Copy Chapter

Full example code: https://github.com/chunky/sb3_to_coral

Google created a chip called the "Coral" for deploying AI to the
edge. It's available in a variety of form factors, including USB (using
the Coral on a Raspberry Pi, with a SB3-developed model, was the original
motivation for the code example above).

The Coral chip is fast, with very low power consumption, but only has limited
on-device training abilities. More information is on the webpage here:
https://developers.google.com/coral.

To deploy to a Coral, one must work via TFLite, and quantize the
network to reflect the Coral's capabilities. The full chain to go from
SB3 to Coral is: SB3 (Torch) => ONNX => TensorFlow => TFLite => Coral.

The code linked above is a complete, minimal, example that:

1. Creates a model using SB3
2. Follows the path of exports all the way to TFLite and Google Coral
3. Demonstrates the forward pass for most exported variants

There are a number of pitfalls along the way to the complete conversion
that this example covers, including:

• Making the Gym's observation work with ONNX properly
• Quantising the TFLite model appropriately to align with Gym
  while still taking advantage of Coral
• Using OnnxablePolicy as described in the above example

Manual export Copy Chapter

You can also manually export required parameters (weights) and construct the
network in your desired framework.

You can access parameters of the model via agents'
{func}get_parameters <stable_baselines3.common.base_class.BaseAlgorithm.get_parameters> function.
As policies are also PyTorch modules, you can also access model.policy.state_dict() directly.
To find the architecture of the networks for each algorithm, best is to check the policies.py file located
in their respective folders.

:::{note}
In most cases, we recommend using PyTorch methods state_dict() and load_state_dict() from the policy,
unless you need to access the optimizers' state dict too. In that case, you need to call get_parameters().
:::

SBX (SB3 + Jax) Export Copy Chapter

As an example of manual export, {ref}Stable Baselines Jax (SBX) <sbx> policies can be exported to ONNX
by using an intermediate PyTorch representation, as shown in the following example:

/* Detailed source-code truncated for AI context efficiency. */

Guide Imitation (`docs/guide/imitation.md`) Copy Section

(imitation)=

Imitation Learning

The imitation library implements
imitation learning algorithms on top of Stable-Baselines3, including:

• Behavioral Cloning
• DAgger with synthetic examples
• Adversarial Inverse Reinforcement Learning (AIRL)
• Generative Adversarial Imitation Learning (GAIL)
• Deep RL from Human Preferences (DRLHP)

You can install imitation with pip install imitation. The imitation
documentation has more details
on how to use the library, including a quick start guide
for the impatient.

Guide Install (`docs/guide/install.md`) Copy Section

(install)=

Installation

Prerequisites Copy Chapter

Stable-Baselines3 requires python 3.10+ and PyTorch >= 2.8

Windows Copy Section

We recommend using miniforge for Windows users for easier installation of Python packages and required libraries. You need an environment with Python version 3.10 or above.

For a quick start you can move straight to installing Stable-Baselines3 in the next step.

:::{note}
Trying to create Atari environments may result in vague errors related to missing DLL files and modules. This is an
issue with atari-py package. See this discussion for more information.
:::

Stable Release Copy Section

To install Stable Baselines3 with pip, execute:

pip install 'stable-baselines3[extra]'

:::{note}
Some shells such as Zsh require quotation marks around brackets, i.e. pip install 'stable-baselines3[extra]' More information.
:::

This includes optional dependencies like Tensorboard, OpenCV or ale-py to train on Atari games. If you do not need those, you can use:

pip install stable-baselines3

:::{note}
If you need to work with OpenCV on a machine without a X-server (for instance inside a docker image),
you will need to install opencv-python-headless, see issue #298.
:::

Bleeding-edge version Copy Chapter

pip install git+https://github.com/DLR-RM/stable-baselines3

with extras:

pip install "stable_baselines3[extra,tests,docs] @ git+https://github.com/DLR-RM/stable-baselines3"

Development version Copy Chapter

To contribute to Stable-Baselines3, with support for running tests and building the documentation.

git clone https://github.com/DLR-RM/stable-baselines3 && cd stable-baselines3
pip install -e '.[docs,tests,extra]'

Using Docker Images Copy Chapter

If you are looking for docker images with stable-baselines already installed in it,
we recommend using images from RL Baselines3 Zoo.

Otherwise, the following images contained all the dependencies for stable-baselines3 but not the stable-baselines3 package itself.
They are made for development.

Use Built Images Copy Section

GPU image (requires nvidia-docker):

docker pull stablebaselines/stable-baselines3

CPU only:

docker pull stablebaselines/stable-baselines3-cpu

Build the Docker Images Copy Section

Build GPU image (with nvidia-docker):

make docker-gpu

Build CPU image:

make docker-cpu

Note: if you are using a proxy, you need to pass extra params during
build and do some tweaks:

--network=host --build-arg HTTPS_PROXY=https://your.proxy.fr:8080/ --build-arg https_proxy=https://your.proxy.fr:8080/

Run the images (CPU/GPU) Copy Section

Run the nvidia-docker GPU image

docker run -it --runtime=nvidia --rm --network host --ipc=host --name test --mount src="$(pwd)",target=/home/mamba/stable-baselines3,type=bind stablebaselines/stable-baselines3 bash -c 'cd /home/mamba/stable-baselines3/ && pytest tests/'

Or, with the shell file:

./scripts/run_docker_gpu.sh pytest tests/

Run the docker CPU image

docker run -it --rm --network host --ipc=host --name test --mount src="$(pwd)",target=/home/mamba/stable-baselines3,type=bind stablebaselines/stable-baselines3-cpu bash -c 'cd /home/mamba/stable-baselines3/ && pytest tests/'

Or, with the shell file:

./scripts/run_docker_cpu.sh pytest tests/

Explanation of the docker command:

• docker run -it create an instance of an image (=container), and
  run it interactively (so ctrl+c will work)
• --rm option means to remove the container once it exits/stops
  (otherwise, you will have to use docker rm)
• --network host don't use network isolation, this allows to use
  tensorboard/visdom on host machine
• --ipc=host Use the host system’s IPC namespace. IPC (POSIX/SysV IPC) namespace provides
  separation of named shared memory segments, semaphores and message
  queues.
• --name test give explicitly the name test to the container,
  otherwise it will be assigned a random name
• --mount src=... give access of the local directory (pwd
  command) to the container (it will be map to /home/mamba/stable-baselines), so
  all the logs created in the container in this folder will be kept
• bash -c '...' Run command inside the docker image, here run the tests
  (pytest tests/)

Guide Integrations (`docs/guide/integrations.md`) Copy Section

(integrations)=

Integrations

Weights & Biases Copy Chapter

Weights & Biases provides a callback for experiment tracking that allows to visualize and share results.

The full documentation is available here: https://docs.wandb.ai/models/integrations/stable-baselines-3

import gymnasium as gym
import wandb
from wandb.integration.sb3 import WandbCallback

from stable_baselines3 import PPO

config = {
    "policy_type": "MlpPolicy",
    "total_timesteps": 25000,
    "env_id": "CartPole-v1",
}
run = wandb.init(
    project="sb3",
    config=config,
    sync_tensorboard=True,  # auto-upload sb3's tensorboard metrics
    # monitor_gym=True,  # auto-upload the videos of agents playing the game
    # save_code=True,  # optional
)

model = PPO(config["policy_type"], config["env_id"], verbose=1, tensorboard_log=f"runs/{run.id}")
model.learn(
    total_timesteps=config["total_timesteps"],
    callback=WandbCallback(
        model_save_path=f"models/{run.id}",
        verbose=2,
    ),
)
run.finish()

Hugging Face 🤗 Copy Chapter

The Hugging Face Hub 🤗 is a central place where anyone can share and explore models. It allows you to host your saved models 💾.

You can see the list of stable-baselines3 saved models here: https://huggingface.co/models?library=stable-baselines3
Most of them are available via the RL Zoo.

Official pre-trained models are saved in the SB3 organization on the hub: https://huggingface.co/sb3

We wrote a tutorial on how to use 🤗 Hub and Stable-Baselines3
here.

Installation Copy Section

pip install huggingface_sb3

:::{note}
If you use the RL Zoo, pushing/loading models from the hub are already integrated:

# Download model and save it into the logs/ folder
# Only use TRUST_REMOTE_CODE=True with HF models that can be trusted (here the SB3 organization)
TRUST_REMOTE_CODE=True python -m rl_zoo3.load_from_hub --algo a2c --env LunarLander-v3 -orga sb3 -f logs/
# Test the agent
python -m rl_zoo3.enjoy --algo a2c --env LunarLander-v3  -f logs/
# Push model, config and hyperparameters to the hub
python -m rl_zoo3.push_to_hub --algo a2c --env LunarLander-v3 -f logs/ -orga sb3 -m "Initial commit"

:::

Download a model from the Hub Copy Section

You need to copy the repo-id that contains your saved model.
For instance sb3/demo-hf-CartPole-v1:

import os

import gymnasium as gym

from huggingface_sb3 import load_from_hub
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy


# Allow the use of `pickle.load()` when downloading model from the hub
# Please make sure that the organization from which you download can be trusted
os.environ["TRUST_REMOTE_CODE"] = "True"

# Retrieve the model from the hub
## repo_id = id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name})
## filename = name of the model zip file from the repository
checkpoint = load_from_hub(
    repo_id="sb3/demo-hf-CartPole-v1",
    filename="ppo-CartPole-v1.zip",
)
model = PPO.load(checkpoint)

# Evaluate the agent and watch it
eval_env = gym.make("CartPole-v1")
mean_reward, std_reward = evaluate_policy(
    model, eval_env, render=True, n_eval_episodes=5, deterministic=True, warn=False
)
print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")

You need to define two parameters:

• repo-id: the name of the Hugging Face repo you want to download.
• filename: the file you want to download.

Upload a model to the Hub Copy Section

You can easily upload your models using two different functions:

1. package_to_hub(): save the model, evaluate it, generate a model card and record a replay video of your agent before pushing the complete repo to the Hub.
2. push_to_hub(): simply push a file to the Hub.

First, you need to be logged in to Hugging Face to upload a model:

• If you're using Colab/Jupyter Notebooks:

from huggingface_hub import notebook_login
notebook_login()

• Otherwise:

huggingface-cli login

Then, in this example, we train a PPO agent to play CartPole-v1 and push it to a new repo sb3/demo-hf-CartPole-v1

With `package_to_hub()`

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env

from huggingface_sb3 import package_to_hub

# Create the environment
env_id = "CartPole-v1"
env = make_vec_env(env_id, n_envs=1)

# Create the evaluation environment
eval_env = make_vec_env(env_id, n_envs=1)

# Instantiate the agent
model = PPO("MlpPolicy", env, verbose=1)

# Train the agent
model.learn(total_timesteps=int(5000))

# This method saves, evaluates, generates a model card and records a replay video of your agent before pushing the repo to the hub
package_to_hub(model=model,
             model_name="ppo-CartPole-v1",
             model_architecture="PPO",
             env_id=env_id,
             eval_env=eval_env,
             repo_id="sb3/demo-hf-CartPole-v1",
             commit_message="Test commit")

You need to define seven parameters:

• model: your trained model.
• model_architecture: name of the architecture of your model (DQN, PPO, A2C, SAC…).
• env_id: name of the environment.
• eval_env: environment used to evaluate the agent.
• repo-id: the name of the Hugging Face repo you want to create or update. It’s <your huggingface username>/.
• commit-message.
• filename: the file you want to push to the Hub.

With `push_to_hub()`

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env

from huggingface_sb3 import push_to_hub

# Create the environment
env_id = "CartPole-v1"
env = make_vec_env(env_id, n_envs=1)

# Instantiate the agent
model = PPO("MlpPolicy", env, verbose=1)

# Train the agent
model.learn(total_timesteps=int(5000))

# Save the model
model.save("ppo-CartPole-v1")

# Push this saved model .zip file to the hf repo
# If this repo does not exist it will be created
## repo_id = id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name})
## filename: the name of the file == "name" inside model.save("ppo-CartPole-v1")
push_to_hub(
  repo_id="sb3/demo-hf-CartPole-v1",
  filename="ppo-CartPole-v1.zip",
  commit_message="Added CartPole-v1 model trained with PPO",
)

You need to define three parameters:

• repo-id: the name of the Hugging Face repo you want to create or update. It’s <your huggingface username>/.
• filename: the file you want to push to the Hub.
• commit-message.

MLflow Copy Chapter

If you want to use MLflow to track your SB3 experiments,
you can adapt the following code which defines a custom logger output:

import sys
from typing import Any, Dict, Tuple, Union

import mlflow
import numpy as np

from stable_baselines3 import SAC
from stable_baselines3.common.logger import HumanOutputFormat, KVWriter, Logger


class MLflowOutputFormat(KVWriter):
    """
    Dumps key/value pairs into MLflow's numeric format.
    """

    def write(
        self,
        key_values: Dict[str, Any],
        key_excluded: Dict[str, Union[str, Tuple[str, ...]]],
        step: int = 0,
    ) -> None:

        for (key, value), (_, excluded) in zip(
            sorted(key_values.items()), sorted(key_excluded.items())
        ):

            if excluded is not None and "mlflow" in excluded:
                continue

            if isinstance(value, np.ScalarType):
                if not isinstance(value, str):
                    mlflow.log_metric(key, value, step)


loggers = Logger(
    folder=None,
    output_formats=[HumanOutputFormat(sys.stdout), MLflowOutputFormat()],
)

with mlflow.start_run():
    model = SAC("MlpPolicy", "Pendulum-v1", verbose=2)
    # Set custom logger
    model.set_logger(loggers)
    model.learn(total_timesteps=10000, log_interval=1)

Guide Migration (`docs/guide/migration.md`) Copy Section

(migration)=

Migrating from Stable-Baselines

This is a guide to migrate from Stable-Baselines (SB2) to Stable-Baselines3 (SB3).

It also references the main changes.

Overview Copy Chapter

Overall Stable-Baselines3 (SB3) keeps the high-level API of Stable-Baselines (SB2).
Most of the changes are to ensure more consistency and are internal ones.
Because of the backend change, from Tensorflow to PyTorch, the internal code is much more readable and easy to debug
at the cost of some speed (dynamic graph vs static graph., see Issue #90)
However, the algorithms were extensively benchmarked on Atari games and continuous control PyBullet envs
(see Issue #48 and Issue #49)
so you should not expect performance drop when switching from SB2 to SB3.

How to migrate? Copy Chapter

In most cases, replacing from stable_baselines with from stable_baselines3 will be sufficient.
Some files were moved to the common folder (cf below) and could result to import errors.
Some algorithms were removed because of their complexity to improve the maintainability of the project.
We recommend reading this guide carefully to understand all the changes that were made.
You can also take a look at the rl-zoo3 and compare the imports
to the rl-zoo of SB2 to have a concrete example of successful migration.

:::{note}
If you experience massive slow-down switching to PyTorch, you may need to play with the number of threads used,
using torch.set_num_threads(1) or OMP_NUM_THREADS=1, see issue #122
and issue #90.
:::

Breaking Changes Copy Chapter

• SB3 requires python 3.7+ (instead of python 3.5+ for SB2)
• Dropped MPI support
• Dropped layer normalized policies (MlpLnLstmPolicy, CnnLnLstmPolicy)
• LSTM policies (MlpLstmPolicy, CnnLstmPolicy) are not supported for the time being
  (see PR #53 for a recurrent PPO implementation)
• Dropped parameter noise for DDPG and DQN
• PPO is now closer to the original implementation (no clipping of the value function by default), cf PPO section below
• Orthogonal initialization is only used by A2C/PPO
• The features extractor (CNN extractor) is shared between policy and q-networks for DDPG/SAC/TD3 and only the policy loss used to update it (much faster)
• Tensorboard legacy logging was dropped in favor of having one logger for the terminal and Tensorboard (cf {ref}Tensorboard integration <tensorboard>)
• We dropped ACKTR/ACER support because of their complexity compared to simpler alternatives (PPO, SAC, TD3) performing as good.
• We dropped GAIL support as we are focusing on model-free RL only, you can however take a look at the {ref}imitation project which implements
  GAIL and other imitation learning algorithms on top of SB3.
• action_probability is currently not implemented in the base class
• pretrain() method for behavior cloning was removed (see issue #27)

You can take a look at the issue about SB3 implementation design for more details.

Moved Files Copy Section

• bench/monitor.py -> common/monitor.py
• logger.py -> common/logger.py
• results_plotter.py -> common/results_plotter.py
• common/cmd_util.py -> common/env_util.py

Utility functions are no longer exported from common module, you should import them with their absolute path, e.g.:

from stable_baselines3.common.env_util import make_atari_env, make_vec_env
from stable_baselines3.common.utils import set_random_seed

instead of from stable_baselines3.common import make_atari_env

Changes and renaming in parameters Copy Section

Base-class (all algorithms)

• load_parameters -> set_parameters

  • get/set_parameters return a dictionary mapping object names
    to their respective PyTorch tensors and other objects representing
    their parameters, instead of simpler mapping of parameter name to
    a NumPy array. These functions also return PyTorch tensors rather
    than NumPy arrays.

Policies

• cnn_extractor -> features_extractor, as features_extractor is now used with MlpPolicy too

A2C

• epsilon -> rms_prop_eps
• lr_schedule is part of learning_rate (it can be a callable).
• alpha, momentum are modifiable through policy_kwargs key optimizer_kwargs.

:::{warning}
PyTorch implementation of RMSprop differs from Tensorflow's,
which leads to different and potentially more unstable results.
Use stable_baselines3.common.sb2_compat.rmsprop_tf_like.RMSpropTFLike optimizer to match the results
with TensorFlow's implementation. This can be done through policy_kwargs: A2C(policy_kwargs=dict(optimizer_class=RMSpropTFLike, optimizer_kwargs=dict(eps=1e-5)))
:::

PPO

• cliprange -> clip_range
• cliprange_vf -> clip_range_vf
• nminibatches -> batch_size

:::{warning}
nminibatches gave different batch size depending on the number of environments: batch_size = (n_steps * n_envs) // nminibatches
:::

• clip_range_vf behavior for PPO is slightly different: Set it to None (default) to deactivate clipping (in SB2, you had to pass -1, None meant to use clip_range for the clipping)
• lam -> gae_lambda
• noptepochs -> n_epochs

PPO default hyperparameters are the one tuned for continuous control environment.
We recommend taking a look at the RL Zoo for hyperparameters tuned for Atari games.

DQN

Only the vanilla DQN is implemented right now but extensions will follow.
Default hyperparameters are taken from the Nature paper, except for the optimizer and learning rate that were taken from Stable Baselines defaults.

DDPG

DDPG now follows the same interface as SAC/TD3.
For state/reward normalization, you should use VecNormalize as for all other algorithms.

SAC/TD3

SAC/TD3 now accept any number of critics, e.g. policy_kwargs=dict(n_critics=3), instead of only two before.

:::{note}
SAC/TD3 default hyperparameters (including network architecture) now match the ones from the original papers.
DDPG is using TD3 defaults.
:::

SAC

SAC implementation matches the latest version of the original implementation: it uses two Q function networks and two target Q function networks
instead of two Q function networks and one Value function network (SB2 implementation, first version of the original implementation).
Despite this change, no change in performance should be expected.

:::{note}
SAC predict() method has now deterministic=False by default for consistency.
To match SB2 behavior, you need to explicitly pass deterministic=True
:::

HER

The HER implementation now only supports online sampling of the new goals. This is done in a vectorized version.
The goal selection strategy RANDOM is no longer supported.

New logger API Copy Section

• Methods were renamed in the logger:

  • logkv -> record, writekvs -> write, writeseq -> write_sequence,
  • logkvs -> record_dict, dumpkvs -> dump,
  • getkvs -> get_log_dict, logkv_mean -> record_mean,

Internal Changes Copy Section

Please read the {ref}Developer Guide <developer> section.

New Features (SB3 vs SB2) Copy Chapter

• Much cleaner and consistent base code (and no more warnings =D!) and static type checks
• Independent saving/loading/predict for policies
• A2C now supports Generalized Advantage Estimation (GAE) and advantage normalization (both are deactivated by default)
• Generalized State-Dependent Exploration (gSDE) exploration is available for A2C/PPO/SAC. It allows using RL directly on real robots (cf https://arxiv.org/abs/2005.05719)
• Better saving/loading: optimizers are now included in the saved parameters and there are two new methods save_replay_buffer and load_replay_buffer for the replay buffer when using off-policy algorithms (DQN/DDPG/SAC/TD3)
• You can pass optimizer_class and optimizer_kwargs to policy_kwargs in order to easily
  customize optimizers
• Seeding now works properly to have deterministic results
• Replay buffer does not grow, allocate everything at build time (faster)
• We added a memory efficient replay buffer variant (pass optimize_memory_usage=True to the constructor), it reduces drastically the memory used especially when using images
• You can specify an arbitrary number of critics for SAC/TD3 (e.g. policy_kwargs=dict(n_critics=3))

Guide Plotting (`docs/guide/plotting.md`) Copy Section

(plotting)=

Plotting

Stable Baselines3 provides utilities for plotting training results, allowing you to monitor and visualize your agent's learning progress.
The plotting functionality is provided by the results_plotter module, which can load monitor files created during training and generate various plots.

:::{note}
Plotting requires pandas and matplotlib. Install them with:

pip install pandas matplotlib

Or install the extra dependencies:

pip install 'stable-baselines3[extra]'

:::

:::{note}
We recommend using the
RL Baselines3 Zoo plotting scripts
which provide plotting capabilities with confidence intervals, and publication-ready visualizations.
:::

Recommended Approach: RL Baselines3 Zoo Plotting Copy Chapter

The RL Zoo provides scripts that allows to compare results across different environments and have publication-ready plots with confidence intervals.

The three main plotting scripts are:

• plot_train.py: For training plots
• all_plots.py: For evaluation plots, to post-process the result
• plot_from_file.py: For more advanced plotting from post-processed results

These scripts offer features that are not included in the basic SB3 plotting utilities.

Installation Copy Section

First, install RL Baselines3 Zoo:

pip install 'rl_zoo3[plots]'

Basic Training Plot Examples Copy Section

# Train an agent
python -m rl_zoo3.train --algo ppo --env CartPole-v1 -f logs/

# Plot training results for a single algorithm
python -m rl_zoo3.plots.plot_train --algo ppo --env CartPole-v1 --exp-folder logs/

Evaluation and Comparison Plots Copy Section

# Generate evaluation plots and save post-processed results
# in `logs/demo_plots.pkl` in order to use `plot_from_file`
python -m rl_zoo3.plots.all_plots --algo ppo sac -e Pendulum-v1 -f logs/ -o logs/demo_plots

# More advanced plotting from post-processed results (with confidence intervals)
python -m rl_zoo3.plots.plot_from_file -i logs/demo_plots.pkl  --rliable --ci-size 0.95

For more examples, please read the
RL Baselines3 Zoo plotting guide.

Real-Time Monitoring Copy Chapter

For real-time monitoring during training, consider using the plotting functions within callbacks
(see the Callbacks guide) or integrating with tools like Tensorboard or Weights & Biases
(see the Integrations guide).

Monitor File Format Copy Chapter

The Monitor wrapper saves training data in CSV format with the following columns:

• r: Episode return (sum of rewards for one episode)
• l: Episode length (number of steps)
• t: Timestamp (wall-clock time when episode ended)

Additional columns may be present if you log custom metrics in the environment's info dict and pass their names via the info_keywords parameter.

:::{note}
The plotting functions automatically handle multiple monitor files from the same directory.
This occurs when using vectorized environments. Episodes are loaded and sorted by timestamp
to ensure they are in the correct chronological order.
:::

Basic SB3 Plotting (Simple Use Cases) Copy Chapter

Basic Plotting: Single Training Run Copy Section

The simplest way to plot training results is to use the plot_results function after training an agent.
This function reads the monitor files created by the Monitor wrapper and plots the episode rewards over time.

import os
import gymnasium as gym
import matplotlib.pyplot as plt

from stable_baselines3 import PPO
from stable_baselines3.common.monitor import Monitor
from stable_baselines3.common.results_plotter import plot_results
from stable_baselines3.common import results_plotter

# Create log directory
log_dir = "tmp/"
os.makedirs(log_dir, exist_ok=True)

# Create and wrap the environment with Monitor
env = gym.make("CartPole-v1")
env = Monitor(env, log_dir)

# Train the agent
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=20_000)

# Plot the results
plot_results([log_dir], 20_000, results_plotter.X_TIMESTEPS, "PPO CartPole")
plt.show()

Different Plotting Modes Copy Section

The plotting functions support three different x-axis modes:

• X_TIMESTEPS: Plot rewards vs. timesteps (default)
• X_EPISODES: Plot rewards vs. episode number
• X_WALLTIME: Plot rewards vs. wall-clock time in hours

import matplotlib.pyplot as plt
from stable_baselines3.common import results_plotter

# Plot by timesteps (shows sample efficiency)
# plot_results([log_dir], None, results_plotter.X_TIMESTEPS, "Rewards vs Timesteps")
# By Episodes
plot_results([log_dir], None, results_plotter.X_EPISODES, "Rewards vs Episodes")
# plot_results([log_dir], None, results_plotter.X_WALLTIME, "Rewards vs Time")

plt.tight_layout()
plt.show()

Advanced Plotting with Manual Data Processing Copy Section

For more control over the plotting, you can use the underlying functions to process the data manually:

import numpy as np
import matplotlib.pyplot as plt
from stable_baselines3.common.monitor import load_results
from stable_baselines3.common.results_plotter import ts2xy, window_func

# Load the results
df = load_results(log_dir)

# Convert dataframe (x=timesteps, y=episodic return)
x, y = ts2xy(df, "timesteps")

# Plot raw data
plt.figure(figsize=(10, 6))
plt.subplot(2, 1, 1)
plt.scatter(x, y, s=2, alpha=0.6)
plt.xlabel("Timesteps")
plt.ylabel("Episode Reward")
plt.title("Raw Episode Rewards")

# Plot smoothed data with custom window
plt.subplot(2, 1, 2)
if len(x) >= 50:  # Only smooth if we have enough data
    x_smooth, y_smooth = window_func(x, y, 50, np.mean)
    plt.plot(x_smooth, y_smooth, linewidth=2)
    plt.xlabel("Timesteps")
    plt.ylabel("Average Episode Reward (50-episode window)")
    plt.title("Smoothed Episode Rewards")

plt.tight_layout()
plt.show()

Guide Rl (`docs/guide/rl.md`) Copy Section

(rl)=

Reinforcement Learning Resources

Stable-Baselines3 assumes that you already understand the basic concepts of Reinforcement Learning (RL).

However, if you want to learn about RL, there are several good resources to get started:

• OpenAI Spinning Up
• The Deep Reinforcement Learning Course
• David Silver's course
• RL102: From Tabular Q-Learning to Deep Q-Learning (DQN)
• RL103: From Deep Q-Learning (DQN) to Soft Actor-Critic (SAC) and Beyond
• Lilian Weng's blog
• Berkeley's Deep RL Bootcamp
• Berkeley's Deep Reinforcement Learning course
• Decisions & Dragons - FAQ for RL foundations
• More resources

Guide Rl Tips (`docs/guide/rl_tips.md`) Copy Section

(rl-tips)=

Reinforcement Learning Tips and Tricks

The aim of this section is to help you run reinforcement learning experiments.
It covers general advice about RL (where to start, which algorithm to choose, how to evaluate an algorithm, ...),
as well as tips and tricks when using a custom environment or implementing an RL algorithm.

:::{note}
We have a video on YouTube that covers
this section in more detail. You can also find the slides here.
:::

:::{note}
We also have a video on Designing and Running Real-World RL Experiments, slides can be found online, and a video on Recent Advances in RL for Continuous Control (SOTA) with slides (early 2026 update).
:::

General advice when using Reinforcement Learning Copy Chapter

TL;DR Copy Section

1. Read about RL and Stable-Baselines3 (SB3)
2. Do quantitative experiments and hyperparameter tuning if needed
3. Evaluate the performance using a separate test environment (remember to check wrappers!)
4. For better performance, increase the training budget

Like any other subject, if you want to work with RL, you should first read about it (we have a dedicated resource page to get you started) to understand what you are using.
We also recommend that you read the Stable Baselines3 (SB3) documentation and do the tutorial.
It covers basic usage and guides you towards more advanced concepts of the library (e.g. callbacks and wrappers).

Reinforcement Learning differs from other machine learning methods in several ways. The data used to train the agent is collected
through interactions with the environment by the agent itself (as opposed to, for example, supervised learning where you have a fixed dataset).
This dependency can lead to a vicious circle: if the agent collects poor quality data (e.g. trajectories with no rewards), it will not improve and will continue to collect bad trajectories.

This factor, among others, explains that results in RL may vary from one run to another (i.e., when only the seed of the pseudo-random generator changes).
For this reason, you should always do several runs to obtain quantitative results.

Good results in RL generally depend on finding appropriate hyperparameters.
Recent algorithms (PPO, SAC, TD3, DroQ) normally require little hyperparameter tuning, however, don't expect the default ones to work in every environment.

Therefore, we highly recommend that you take a look at the RL zoo (or the original papers) for tuned hyperparameters.
A best practice when you apply RL to a new problem is to do automatic hyperparameter optimization.
Again, this is included in the RL zoo.

When applying RL to a custom problem, you should always normalize the input to the agent (e.g. using VecNormalize for PPO/A2C)
and look at common preprocessing done on other environments (e.g. for Atari, frame-stack, ...).
Please refer to Tips and Tricks when creating a custom environment paragraph below for more advice related to custom environments.

Current Limitations of RL Copy Section

You have to be aware of the current limitations of reinforcement learning.

Model-free RL algorithms (i.e. all the algorithms implemented in SB3) are usually sample inefficient. They require a lot of samples (sometimes millions of interactions) to learn anything useful.
That's why most of the successes in RL were achieved on games or in simulation only.
For instance, in this work by ETH Zurich, the ANYmal robot was trained in simulation only, and then tested in the real world.

As general advice, to obtain better performance, you should increase the budget of the agent (number of training timesteps).

In order to achieve the desired behavior, expert knowledge is often required to design an adequate reward function.
This reward engineering (or RewArt as coined by Freek Stulp), necessitates several iterations. As a good example of reward shaping,
you can take a look at Deep Mimic paper which combines imitation learning and reinforcement learning to do acrobatic moves.

A final limitation of RL is the instability of training. That is, you can observe a huge drop in performance during training.
This behavior is particularly present in DDPG, that's why its extension TD3 tries to tackle that issue.
Other methods, such as TRPO or PPO use a trust region to minimize this problem by avoiding too large updates.

How to evaluate an RL algorithm? Copy Section

:::{note}
Pay attention to environment wrappers when evaluating your agent and comparing results to others' results. Modifications to episode rewards
or lengths may also affect evaluation results which may not be desirable. Check evaluate_policy helper function in {ref}Evaluation Helper <eval> section.
:::

Because most algorithms use exploration noise during training, you need a separate test environment to evaluate the performance of your agent at a given time.
It is recommended to periodically evaluate your agent for n test episodes (n is usually between 5 and 20) and average the reward per episode to have a good estimate.

:::{note}
We provide an EvalCallback for doing such evaluation. You can read more about it in the {ref}Callbacks <callbacks> section.
:::

As some policies are stochastic by default (e.g. A2C or PPO), you should also try to set deterministic=True when calling the .predict() method,
this frequently leads to better performance.
Looking at the training curve (episode reward as a function of timesteps) is a good proxy but underestimates the agent's true performance.

We highly recommend reading Empirical Design in Reinforcement Learning, as it provides valuable insights for best practices when running RL experiments.

We also suggest reading Deep Reinforcement Learning that Matters for a good discussion about RL evaluation,
and Rliable: Better Evaluation for Reinforcement Learning for comparing results.

You can also take a look at this blog post
and this issue by Cédric Colas.

Which algorithm should I use? Copy Chapter

There is no silver bullet in RL, you can choose one or the other depending on your needs and problems.
The first distinction comes from your action space, i.e., do you have discrete (e.g. LEFT, RIGHT, ...)
or continuous actions (ex: go to a certain speed)?

Some algorithms are only tailored for one or the other domain: DQN supports only discrete actions, while SAC is restricted to continuous actions.

The second difference that will help you decide is whether you can parallelize your training or not.
If what matters is the wall clock training time, then you should lean towards A2C and its derivatives (PPO, ...).
Take a look at the Vectorized Environments to learn more about training with multiple workers.

To accelerate training, you can also take a look at SBX, which is SB3 + Jax, it has less features than SB3 but can be up to 20x faster than SB3 PyTorch thanks to JIT compilation of the gradient update.

In sparse reward settings, we recommend using dedicated methods like HER (see below) or population-based algorithms like ARS (available in our contrib repo.

To sum up:

Discrete Actions Copy Section

:::{note}
This covers Discrete, MultiDiscrete, Binary and MultiBinary spaces
:::

Discrete Actions - Single Process

DQN with extensions (double DQN, prioritized replay, ...) are the recommended algorithms.
We notably provide QR-DQN in our contrib repo.
DQN is usually slower to train (regarding wall clock time) but is the most sample efficient (because of its replay buffer).

Discrete Actions - Multiprocessed

You should give a try to PPO or A2C.

Continuous Actions Copy Section

Continuous Actions - Single Process

:::{note}
We have a video on Recent Advances in RL for Continuous Control (SOTA) with slides (early 2026 update).
:::

Current State Of The Art (SOTA) algorithms are SAC, TD3, CrossQ and TQC (available in our contrib repo and SBX (SB3 + Jax) repo).
Please use the hyperparameters in the RL zoo for best results.

If you want an extremely sample-efficient algorithm, we recommend using the DroQ configuration in SBX (it does many gradient steps per step in the environment).

Continuous Actions - Multiprocessed

Take a look at PPO, TRPO (available in our contrib repo) or A2C. Again, don't forget to take the hyperparameters from the RL zoo for continuous actions problems (cf Bullet envs).

:::{note}
Normalization is critical for those algorithms
:::

Goal Environment Copy Section

If your environment follows the GoalEnv interface (cf HER), then you should use
HER + (SAC/TD3/DDPG/DQN/QR-DQN/TQC) depending on the action space.

:::{note}
The batch_size is an important hyperparameter for experiments with HER
:::

Tips and Tricks when creating a custom environment Copy Chapter

If you want to learn about how to create a custom environment, we recommend you read this page.
We also provide a colab notebook for a concrete example of creating a custom gym environment.

Some basic advice:

• always normalize your observation space if you can, i.e. if you know the boundaries
• normalize your action space and make it symmetric if it is continuous (see potential problem below) A good practice is to rescale your actions so that they lie in [-1, 1]. This does not limit you, as you can easily rescale the action within the environment.
• start with a shaped reward (i.e. informative reward) and a simplified version of your problem
• debug with random actions to check if your environment works and follows the gym interface (with check_env, see below)

Two important things to keep in mind when creating a custom environment are avoiding breaking the Markov assumption
and properly handle termination due to a timeout (maximum number of steps in an episode).
For example, if there is a time delay between action and observation (e.g. due to wifi communication), you should provide a history of observations as input.

Termination due to timeout (max number of steps per episode) needs to be handled separately.
You should return truncated = True.
If you are using the gym TimeLimit wrapper, this will be done automatically.
You can read Time Limit in RL, take a look at the Designing and Running Real-World RL Experiments video or RL Tips and Tricks video for more details.

We provide a helper to check that your environment runs without error:

from stable_baselines3.common.env_checker import check_env

env = CustomEnv(arg1, ...)
# It will check your custom environment and output additional warnings if needed
check_env(env)

If you want to quickly try a random agent on your environment, you can also do:

env = YourEnv()
obs, info = env.reset()
n_steps = 10
for _ in range(n_steps):
    # Random action
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()

Why should I normalize the action space?

Most reinforcement learning algorithms rely on a Gaussian distribution (initially centered at 0 with std 1) for continuous actions.
So, if you forget to normalize the action space when using a custom environment, this can harm learning and can be difficult to debug (cf attached image and issue #473).

:::{figure} ../_static/img/mistake.png
:::

Another consequence of using a Gaussian distribution is that the action range is not bounded.
That's why clipping is usually used as a bandage to stay in a valid interval.
A better solution would be to use a squashing function (cf SAC) or a Beta distribution (cf issue #112).

:::{note}
This statement is not true for DDPG or TD3 because they don't rely on any probability distribution.
:::

Tips and Tricks when implementing an RL algorithm Copy Chapter

:::{note}
We have a video on YouTube about reliable RL that covers
this section in more details. You can also find the slides online.
:::

When you try to reproduce a RL paper by implementing the algorithm, the nuts and bolts of RL research
by John Schulman are quite useful (video).

We recommend following those steps to have a working RL algorithm:

1. Read the original paper several times
2. Read existing implementations (if available)
3. Try to have some "signs of life" on toy problems
4. Validate the implementation by making it run on harder and harder envs (you can compare results against the RL zoo).
   You usually need to run hyperparameter optimization for that step.

You need to be particularly careful on the shape of the different objects you are manipulating (a broadcast mistake will fail silently cf. issue #75)
and when to stop the gradient propagation.

Don't forget to handle termination due to timeout separately (see remark in the custom environment section above),
you can also take a look at Issue #284 and Issue #633.

A personal pick (by @araffin) for environments with gradual difficulty in RL with continuous actions:

1. Pendulum (easy to solve)
2. HalfCheetahBullet (medium difficulty with local minima and shaped reward)
3. BipedalWalkerHardcore (if it works on that one, then you can have a cookie)

For RL with discrete actions:

1. CartPole-v1 (easy to be better than random agent, harder to achieve maximal performance)
2. LunarLander
3. Pong (one of the easiest Atari games)
4. other Atari games (e.g. Breakout)

Guide Rl Zoo (`docs/guide/rl_zoo.md`) Copy Section

(rl-zoo)=

RL Baselines3 Zoo

RL Baselines3 Zoo is a training framework for Reinforcement Learning (RL).

It provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.

In addition, it includes a collection of tuned hyperparameters for common environments and RL algorithms, and agents trained with those settings.

Goals of this repository:

1. Provide a simple interface to train and enjoy RL agents
2. Benchmark the different Reinforcement Learning algorithms
3. Provide tuned hyperparameters for each environment and RL algorithm
4. Have fun with the trained agents!

Documentation is available online: https://rl-baselines3-zoo.readthedocs.io/

Installation Copy Chapter

Option 1: install the python package pip install rl_zoo3

or:

1. Clone the repository:

git clone --recursive https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/

:::{note}
You can remove the --recursive option if you don't want to download the trained agents
:::

:::{note}
If you only need the training/plotting scripts and additional callbacks/wrappers from the RL Zoo, you can also install it via pip: pip install rl_zoo3
:::

2. Install dependencies

apt-get install swig cmake ffmpeg
# full dependencies
pip install -r requirements.txt
# minimal dependencies
pip install -e .

Train an Agent Copy Chapter

The hyperparameters for each environment are defined in
hyperparameters/algo_name.yml.

If the environment exists in this file, then you can train an agent
using:

python -m rl_zoo3.train --algo algo_name --env env_id

For example (with evaluation and checkpoints):

python -m rl_zoo3.train --algo ppo --env CartPole-v1 --eval-freq 10000 --save-freq 50000

Continue training (here, load pretrained agent for Breakout and continue
training for 5000 steps):

python -m rl_zoo3.train --algo a2c --env BreakoutNoFrameskip-v4 -i trained_agents/a2c/BreakoutNoFrameskip-v4_1/BreakoutNoFrameskip-v4.zip -n 5000

Enjoy a Trained Agent Copy Chapter

If the trained agent exists, then you can see it in action using:

python -m rl_zoo3.enjoy --algo algo_name --env env_id

For example, enjoy A2C on Breakout during 5000 timesteps:

python -m rl_zoo3.enjoy --algo a2c --env BreakoutNoFrameskip-v4 --folder rl-trained-agents/ -n 5000

Hyperparameter Optimization Copy Chapter

We use Optuna for optimizing the hyperparameters.

Tune the hyperparameters for PPO, using a random sampler and median pruner, 2 parallels jobs,
with a budget of 1000 trials and a maximum of 50000 steps:

python -m rl_zoo3.train --algo ppo --env MountainCar-v0 -n 50000 -optimize --n-trials 1000 --n-jobs 2 \
  --sampler random --pruner median

Colab Notebook: Try it Online! Copy Chapter

You can train agents online using Google colab notebook.

:::{note}
You can find more information about the rl baselines3 zoo in the repo README. For instance, how to record a video of a trained agent.
:::

Guide Save Format (`docs/guide/save_format.md`) Copy Section

(save-format)=

On saving and loading

Stable Baselines3 (SB3) stores both neural network parameters and algorithm-related parameters such as
exploration schedule, number of environments and observation/action space. This allows continual learning and easy
use of trained agents without training, but it is not without its issues. Following describes the format
used to save agents in SB3 along with its pros and shortcomings.

Terminology used in this page:

• parameters refer to neural network parameters (also called "weights"). This is a dictionary
  mapping variable name to a PyTorch tensor.
• data refers to RL algorithm parameters, e.g. learning rate, exploration schedule, action/observation space.
  These depend on the algorithm used. This is a dictionary mapping classes variable names to their values.

Zip-archive Copy Chapter

A zip-archived JSON dump, PyTorch state dictionaries and PyTorch variables. The data dictionary (class parameters)
is stored as a JSON file, model parameters and optimizers are serialized with torch.save() function and these files
are stored under a single .zip archive.

Any objects that are not JSON serializable are serialized with cloudpickle and stored as base64-encoded
string in the JSON file, along with some information that was stored in the serialization. This allows
inspecting stored objects without deserializing the object itself.

This format allows skipping elements in the file, i.e. we can skip deserializing objects that are
broken/non-serializable.
This can be done via custom_objects argument to load functions.

:::{note}
If you encounter loading issue, for instance pickle issues or error after loading
(see #171 or #573),
you can pass print_system_info=True
to compare the system on which the model was trained vs the current one
model = PPO.load("ppo_saved", print_system_info=True)
:::

File structure:

saved_model.zip/
├── data              JSON file of class-parameters (dictionary)
├── *.optimizer.pth   PyTorch optimizers serialized
├── policy.pth        PyTorch state dictionary of the policy saved
├── pytorch_variables.pth Additional PyTorch variables
├── _stable_baselines3_version contains the SB3 version with which the model was saved
├── system_info.txt contains system info (os, python version, ...) on which the model was saved

Pros:

• More robust to unserializable objects (one bad object does not break everything).
• Saved files can be inspected/extracted with zip-archive explorers and by other languages.

Cons:

• More complex implementation.
• Still relies partly on cloudpickle for complex objects (e.g. custom functions)
  which can lead to incompatibilities between Python versions.

Guide Sb3 Contrib (`docs/guide/sb3_contrib.md`) Copy Section

(sb3-contrib)=

SB3 Contrib

We implement experimental features in a separate contrib repository:
SB3-Contrib

This allows Stable-Baselines3 (SB3) to maintain a stable and compact core, while still
providing the latest features, like RecurrentPPO (PPO LSTM), Truncated Quantile Critics (TQC), Augmented Random Search (ARS), Trust Region Policy Optimization (TRPO) or
Quantile Regression DQN (QR-DQN).

Why create this repository? Copy Chapter

Over the span of stable-baselines and stable-baselines3, the community
has been eager to contribute in form of better logging utilities,
environment wrappers, extended support (e.g. different action spaces)
and learning algorithms.

However sometimes these utilities were too niche to be considered for
stable-baselines or proved to be too difficult to integrate well into
the existing code without creating a mess. sb3-contrib aims to fix this by not
requiring the neatest code integration with existing code and not
setting limits on what is too niche: almost everything remotely useful
goes!
We hope this allows us to provide reliable implementations
following stable-baselines usual standards (consistent style, documentation, etc)
beyond the relatively small scope of utilities in the main repository.

Features Copy Section

See documentation for the full list of included features.

RL Algorithms:

• Augmented Random Search (ARS)
• Quantile Regression DQN (QR-DQN)
• PPO with invalid action masking (Maskable PPO)
• PPO with recurrent policy (RecurrentPPO aka PPO LSTM)
• Truncated Quantile Critics (TQC)
• Trust Region Policy Optimization (TRPO)
• Batch Normalization in Deep Reinforcement Learning (CrossQ)

Gym Wrappers:

• Time Feature Wrapper

Documentation Copy Section

Documentation is available online: https://sb3-contrib.readthedocs.io/

Installation Copy Section

To install Stable-Baselines3 contrib with pip, execute:

pip install sb3-contrib

We recommend to use the master version of Stable Baselines3 and SB3-Contrib.

To install Stable Baselines3 master version:

pip install git+https://github.com/DLR-RM/stable-baselines3

To install Stable Baselines3 contrib master version:

pip install git+https://github.com/Stable-Baselines-Team/stable-baselines3-contrib

Example Copy Section

SB3-Contrib follows the SB3 API and folder structure. So, if you are familiar with SB3,
using SB3-Contrib should be easy too.

Here is an example of training a Quantile Regression DQN (QR-DQN) agent on the CartPole environment.

from sb3_contrib import QRDQN

policy_kwargs = dict(n_quantiles=50)
model = QRDQN("MlpPolicy", "CartPole-v1", policy_kwargs=policy_kwargs, verbose=1)
model.learn(total_timesteps=10000, log_interval=4)
model.save("qrdqn_cartpole")

Guide Sbx (`docs/guide/sbx.md`) Copy Section

(sbx)=

Stable Baselines Jax (SBX)

Stable Baselines Jax (SBX) is a proof of concept version of Stable-Baselines3 in Jax.

It provides a minimal number of features compared to SB3 but can be much faster (up to 20x times!): https://twitter.com/araffin2/status/1590714558628253698

Implemented algorithms:

• Soft Actor-Critic (SAC) and SAC-N
• Truncated Quantile Critics (TQC)
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning (DroQ)
• Proximal Policy Optimization (PPO)
• Deep Q Network (DQN)
• Twin Delayed DDPG (TD3)
• Deep Deterministic Policy Gradient (DDPG)
• Batch Normalization in Deep Reinforcement Learning (CrossQ)
• Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning (SimBa)

As SBX follows SB3 API, it is also compatible with the RL Zoo.
For that you will need to create two files:

train_sbx.py:

import rl_zoo3
import rl_zoo3.train
from rl_zoo3.train import train
from sbx import DDPG, DQN, PPO, SAC, TD3, TQC, CrossQ

rl_zoo3.ALGOS["ddpg"] = DDPG
rl_zoo3.ALGOS["dqn"] = DQN
# See SBX readme to use DroQ configuration
# rl_zoo3.ALGOS["droq"] = DroQ
rl_zoo3.ALGOS["sac"] = SAC
rl_zoo3.ALGOS["ppo"] = PPO
rl_zoo3.ALGOS["td3"] = TD3
rl_zoo3.ALGOS["tqc"] = TQC
rl_zoo3.ALGOS["crossq"] = CrossQ
rl_zoo3.train.ALGOS = rl_zoo3.ALGOS
rl_zoo3.exp_manager.ALGOS = rl_zoo3.ALGOS


if __name__ == "__main__":
    train()

Then you can call python train_sbx.py --algo sac --env Pendulum-v1 and use the RL Zoo CLI.

enjoy_sbx.py:

import rl_zoo3
import rl_zoo3.enjoy
from rl_zoo3.enjoy import enjoy
from sbx import DDPG, DQN, PPO, SAC, TD3, TQC, CrossQ

rl_zoo3.ALGOS["ddpg"] = DDPG
rl_zoo3.ALGOS["dqn"] = DQN
# See SBX readme to use DroQ configuration
# rl_zoo3.ALGOS["droq"] = DroQ
rl_zoo3.ALGOS["sac"] = SAC
rl_zoo3.ALGOS["ppo"] = PPO
rl_zoo3.ALGOS["td3"] = TD3
rl_zoo3.ALGOS["tqc"] = TQC
rl_zoo3.ALGOS["crossq"] = CrossQ
rl_zoo3.enjoy.ALGOS = rl_zoo3.ALGOS
rl_zoo3.exp_manager.ALGOS = rl_zoo3.ALGOS


if __name__ == "__main__":
    enjoy()

Guide Tensorboard (`docs/guide/tensorboard.md`) Copy Section

(tensorboard)=

Tensorboard Integration

Basic Usage Copy Chapter

To use Tensorboard with stable baselines3, you simply need to pass the location of the log folder to the RL agent:

from stable_baselines3 import A2C

model = A2C("MlpPolicy", "CartPole-v1", verbose=1, tensorboard_log="./a2c_cartpole_tensorboard/")
model.learn(total_timesteps=10_000)

You can also define custom logging name when training (by default it is the algorithm name)

from stable_baselines3 import A2C

model = A2C("MlpPolicy", "CartPole-v1", verbose=1, tensorboard_log="./a2c_cartpole_tensorboard/")
model.learn(total_timesteps=10_000, tb_log_name="first_run")
# Pass reset_num_timesteps=False to continue the training curve in tensorboard
# By default, it will create a new curve
# Keep tb_log_name constant to have continuous curve (see note below)
model.learn(total_timesteps=10_000, tb_log_name="second_run", reset_num_timesteps=False)
model.learn(total_timesteps=10_000, tb_log_name="third_run", reset_num_timesteps=False)

:::{note}
If you specify different tb_log_name in subsequent runs, you will have split graphs, like in the figure below.
If you want them to be continuous, you must keep the same tb_log_name (see issue #975).
And, if you still managed to get your graphs split by other means, just put tensorboard log files into the same folder.

:alt: split_graph
:width: 330

:::

Once the learn function is called, you can monitor the RL agent during or after the training, with the following bash command:

tensorboard --logdir ./a2c_cartpole_tensorboard/

:::{note}
You can find explanations about the logger output and names in the {ref}Logger <logger> section.
:::

you can also add past logging folders:

tensorboard --logdir ./a2c_cartpole_tensorboard/,./ppo2_cartpole_tensorboard/

It will display information such as the episode reward (when using a Monitor wrapper), the model losses and other parameter unique to some models.

:alt: plotting
:width: 600

Logging More Values Copy Chapter

Using a callback, you can easily log more values with TensorBoard.
Here is a simple example on how to log both additional tensor or arbitrary scalar value:

import numpy as np

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback

model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)


class TensorboardCallback(BaseCallback):
    """
    Custom callback for plotting additional values in tensorboard.
    """

    def __init__(self, verbose=0):
        super().__init__(verbose)

    def _on_step(self) -> bool:
        # Log scalar value (here a random variable)
        value = np.random.random()
        self.logger.record("random_value", value)
        return True


model.learn(50000, callback=TensorboardCallback())

:::{note}
If you want to log values more often than the default to tensorboard, you manually call self.logger.dump(self.num_timesteps) in a callback
(see issue #506).
:::

Logging Images Copy Chapter

TensorBoard supports periodic logging of image data, which helps evaluating agents at various stages during training.

:::{warning}
To support image logging pillow must be installed otherwise, TensorBoard ignores the image and logs a warning.
:::

Here is an example of how to render an image to TensorBoard at regular intervals:

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import Image

model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)


class ImageRecorderCallback(BaseCallback):
    def __init__(self, verbose=0):
        super().__init__(verbose)

    def _on_step(self):
        image = self.training_env.render(mode="rgb_array")
        # "HWC" specify the dataformat of the image, here channel last
        # (H for height, W for width, C for channel)
        # See https://docs.pytorch.org/docs/stable/tensorboard.html
        # for supported formats
        self.logger.record("trajectory/image", Image(image, "HWC"), exclude=("stdout", "log", "json", "csv"))
        return True


model.learn(50000, callback=ImageRecorderCallback())

Logging Figures/Plots Copy Chapter

TensorBoard supports periodic logging of figures/plots created with matplotlib, which helps evaluate agents at various stages during training.

:::{warning}
To support figure logging matplotlib must be installed otherwise, TensorBoard ignores the figure and logs a warning.
:::

Here is an example of how to store a plot in TensorBoard at regular intervals:

import numpy as np
import matplotlib.pyplot as plt

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import Figure

model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)


class FigureRecorderCallback(BaseCallback):
    def __init__(self, verbose=0):
        super().__init__(verbose)

    def _on_step(self):
        # Plot values (here a random variable)
        figure = plt.figure()
        figure.add_subplot().plot(np.random.random(3))
        # Close the figure after logging it
        self.logger.record("trajectory/figure", Figure(figure, close=True), exclude=("stdout", "log", "json", "csv"))
        plt.close()
        return True


model.learn(50000, callback=FigureRecorderCallback())

Logging Videos Copy Chapter

TensorBoard supports periodic logging of video data, which helps evaluate agents at various stages during training.

:::{warning}
To support video logging moviepy must be installed otherwise, TensorBoard ignores the video and logs a warning.
:::

Here is an example of how to render an episode and log the resulting video to TensorBoard at regular intervals:

/* Detailed source-code truncated for AI context efficiency. */

Logging Hyperparameters Copy Chapter

TensorBoard supports logging of hyperparameters in its HPARAMS tab, which helps to compare agents trainings.

:::{warning}
To display hyperparameters in the HPARAMS section, a metric_dict must be given (as well as a hparam_dict).
:::

Here is an example of how to save hyperparameters in TensorBoard:

from stable_baselines3 import A2C
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import HParam


class HParamCallback(BaseCallback):
    """
    Saves the hyperparameters and metrics at the start of the training, and logs them to TensorBoard.
    """

    def _on_training_start(self) -> None:
        hparam_dict = {
            "algorithm": self.model.__class__.__name__,
            "learning rate": self.model.learning_rate,
            "gamma": self.model.gamma,
        }
        # define the metrics that will appear in the `HPARAMS` Tensorboard tab by referencing their tag
        # Tensorbaord will find & display metrics from the `SCALARS` tab
        metric_dict = {
            "rollout/ep_len_mean": 0,
            "train/value_loss": 0.0,
        }
        self.logger.record(
            "hparams",
            HParam(hparam_dict, metric_dict),
            exclude=("stdout", "log", "json", "csv"),
        )

    def _on_step(self) -> bool:
        return True


model = A2C("MlpPolicy", "CartPole-v1", tensorboard_log="runs/", verbose=1)
model.learn(total_timesteps=int(5e4), callback=HParamCallback())

Directly Accessing The Summary Writer Copy Chapter

If you would like to log arbitrary data (in one of the formats supported by PyTorch), you
can get direct access to the underlying SummaryWriter in a callback:

:::{warning}
This method is not recommended and should only be used by advanced users.
:::

:::{note}
If you want a concrete example, you can watch how to log lap time with donkeycar env,
or read the code in the RL Zoo.
You might also want to take a look at issue #1160 and issue #1219.
:::

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import TensorBoardOutputFormat



model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)


class SummaryWriterCallback(BaseCallback):

    def _on_training_start(self):
        self._log_freq = 1000  # log every 1000 calls

        output_formats = self.logger.output_formats
        # Save reference to tensorboard formatter object
        # note: the failure case (not formatter found) is not handled here, should be done with try/except.
        self.tb_formatter = next(formatter for formatter in output_formats if isinstance(formatter, TensorBoardOutputFormat))

    def _on_step(self) -> bool:
        if self.n_calls % self._log_freq == 0:
            # You can have access to info from the env using self.locals.
            # for instance, when using one env (index 0 of locals["infos"]):
            # lap_count = self.locals["infos"][0]["lap_count"]
            # self.tb_formatter.writer.add_scalar("train/lap_count", lap_count, self.num_timesteps)

            self.tb_formatter.writer.add_text("direct_access", "this is a value", self.num_timesteps)
            self.tb_formatter.writer.flush()


model.learn(50000, callback=SummaryWriterCallback())

Guide Vec Envs (`docs/guide/vec_envs.md`) Copy Section

(vec-env)=

.. automodule:: stable_baselines3.common.vec_env

Vectorized Environments

Vectorized Environments are a method for stacking multiple independent environments into a single environment.
Instead of training an RL agent on 1 environment per step, it allows us to train it on n environments per step.
Because of this, actions passed to the environment are now a vector (of dimension n).
It is the same for observations, rewards and end of episode signals (dones).
In the case of non-array observation spaces such as Dict or Tuple, where different sub-spaces
may have different shapes, the sub-observations are vectors (of dimension n).

:::{note}
Vectorized environments are required when using wrappers for frame-stacking or normalization.
:::

:::{note}
When using vectorized environments, the environments are automatically reset at the end of each episode.
Thus, the observation returned for the i-th environment when done[i] is true will in fact be the first observation of the next episode, not the last observation of the episode that has just terminated.
You can access the "real" final observation of the terminated episode—that is, the one that accompanied the done event provided by the underlying environment—using the terminal_observation keys in the info dicts returned by the VecEnv.
:::

:::{warning}
When defining a custom VecEnv (for instance, using gym3 ProcgenEnv), you should provide terminal_observation keys in the info dicts returned by the VecEnv
(cf. note above).
:::

:::{warning}
When using SubprocVecEnv, users must wrap the code in an if __name__ == "__main__": if using the forkserver or spawn start method (default on Windows).
On Linux, the default start method is fork which is not thread safe and can create deadlocks.

For more information, see Python's multiprocessing guidelines.
:::

VecEnv API vs Gym API Copy Chapter

For consistency across Stable-Baselines3 (SB3) versions and because of its special requirements and features,
SB3 VecEnv API is not the same as Gym API.
SB3 VecEnv API is actually close to Gym 0.21 API but differs to Gym 0.26+ API:

• the reset() method only returns the observation (obs = vec_env.reset()) and not a tuple, the info at reset are stored in vec_env.reset_infos.

• only the initial call to vec_env.reset() is required, environments are reset automatically afterward (and reset_infos is updated automatically).

• the vec_env.step(actions) method expects an array as input
  (with a batch size corresponding to the number of environments) and returns a 4-tuple (and not a 5-tuple): obs, rewards, dones, infos instead of obs, reward, terminated, truncated, info
  where dones = terminated or truncated (for each env).
  obs, rewards, dones are NumPy arrays with shape (n_envs, shape_for_single_env) (so with a batch dimension).
  Additional information is passed via the infos value which is a list of dictionaries.

• at the end of an episode, infos[env_idx]["TimeLimit.truncated"] = truncated and not terminated
  tells the user if an episode was truncated or not:
  you should bootstrap if infos[env_idx]["TimeLimit.truncated"] is True (episode over due to a timeout/truncation)
  or dones[env_idx] is False (episode not finished).
  Note: compared to Gym 0.26+ infos[env_idx]["TimeLimit.truncated"] and terminated are mutually exclusive.
  The conversion from SB3 to Gym API is

  python Copy

  # done is True at the end of an episode
  # dones[env_idx] = terminated[env_idx] or truncated[env_idx]
  # In SB3, truncated and terminated are mutually exclusive
  # infos[env_idx]["TimeLimit.truncated"] = truncated and not terminated
  # terminated[env_idx] tells you whether you should bootstrap or not:
  # when the episode has not ended or when the termination was a timeout/truncation
  terminated[env_idx] = dones[env_idx] and not infos[env_idx]["TimeLimit.truncated"]
  should_bootstrap[env_idx] = not terminated[env_idx]

• at the end of an episode, because the environment resets automatically,
  we provide infos[env_idx]["terminal_observation"] which contains the last observation
  of an episode (and can be used when bootstrapping, see note in the previous section)

• to overcome the current Gymnasium limitation (only one render mode allowed per env instance, see issue #100),
  we recommend using render_mode="rgb_array" since we can both have the image as a numpy array and display it with OpenCV.
  if no mode is passed or mode="rgb_array" is passed when calling vec_env.render then we use the default mode, otherwise, we use the OpenCV display.
  Note that if render_mode != "rgb_array", you can only call vec_env.render() (without argument or with mode=env.render_mode).

• the reset() method doesn't take any parameter. If you want to seed the pseudo-random generator or pass options,
  you should call vec_env.seed(seed=seed)/vec_env.set_options(options) and obs = vec_env.reset() afterward (seed and options are discarded after each call to reset()).

• methods and attributes of the underlying Gym envs can be accessed, called and set using vec_env.get_attr("attribute_name"),
  vec_env.env_method("method_name", args1, args2, kwargs1=kwargs1) and vec_env.set_attr("attribute_name", new_value).

Modifying Vectorized Environments Attributes Copy Chapter

If you plan to modify the attributes of an environment while it is used (e.g., modifying an attribute specifying the task carried out for a portion of training when doing multi-task learning, or
a parameter of the environment dynamics), you must expose a setter method.
In fact, directly accessing the environment attribute in the callback can lead to unexpected behavior because environments can be wrapped (using gym or VecEnv wrappers, the Monitor wrapper being one example).

Consider the following example for a custom env:

/* Detailed source-code truncated for AI context efficiency. */

In this example env.mu cannot be accessed/changed directly because it is wrapped in a VecEnv and because it could be wrapped with other wrappers (see GH#1573 for a longer explanation).
Instead, the callback should use the set_mu method via the env_method method for Vectorized Environments.

from itertools import cycle

class ChangeMuCallback(BaseCallback):
  """
  This callback changes the value of mu during training looping
  through a list of values until training is aborted.
  The environment is implemented so that the impact of changing
  the value of mu mid-episode is visible only after the episode is over
  and the reset method has been called.
  """
  def __init__(self):
    super().__init__()
    # An iterator that contains the different of the friction coefficient
    self.mus = cycle([0.1, 0.2, 0.5, 0.13, 0.9])

  def _on_step(self):
    # Note: in practice, you should not change this value at every step
    # but rather depending on some events/metrics like agent performance/episode termination
    # both accessible via the `self.logger` or `self.locals` variables
    self.training_env.env_method("set_mu", next(self.mus))

This callback can then be used to safely modify environment attributes during training since
it calls the environment setter method.

Vectorized Environments Wrappers Copy Chapter

If you want to alter or augment a VecEnv without redefining it completely (e.g. stack multiple frames, monitor the VecEnv, normalize the observation, ...), you can use VecEnvWrapper for that.
They are the vectorized equivalents (i.e., they act on multiple environments at the same time) of gym.Wrapper.

You can find below an example for extracting one key from the observation:

import numpy as np

from stable_baselines3.common.vec_env.base_vec_env import VecEnv, VecEnvStepReturn, VecEnvWrapper


class VecExtractDictObs(VecEnvWrapper):
    """
    A vectorized wrapper for filtering a specific key from dictionary observations.
    Similar to Gym's FilterObservation wrapper:
        https://github.com/openai/gym/blob/master/gym/wrappers/filter_observation.py

    :param venv: The vectorized environment
    :param key: The key of the dictionary observation
    """

    def __init__(self, venv: VecEnv, key: str):
        self.key = key
        super().__init__(venv=venv, observation_space=venv.observation_space.spaces[self.key])

    def reset(self) -> np.ndarray:
        obs = self.venv.reset()
        return obs[self.key]

    def step_async(self, actions: np.ndarray) -> None:
        self.venv.step_async(actions)

    def step_wait(self) -> VecEnvStepReturn:
        obs, rewards, dones, info = self.venv.step_wait()
        return obs[self.key], rewards, dones, info

env = DummyVecEnv([lambda: gym.make("FetchReach-v1")])
# Wrap the VecEnv
env = VecExtractDictObs(env, key="observation")

:::{note}
When creating a vectorized environment, you can also specify ordinary gymnasium
wrappers to wrap each of the sub-environments. See the
{func}make_vec_env <stable_baselines3.common.env_util.make_vec_env>
documentation for details.
Example:

from gymnasium.wrappers import RescaleAction
from stable_baselines3.common.env_util import make_vec_env

# Use gym wrapper for each sub-env of the VecEnv
wrapper_kwargs = dict(min_action=-1.0, max_action=1.0)
vec_env = make_vec_env(
    "Pendulum-v1", n_envs=2, wrapper_class=RescaleAction, wrapper_kwargs=wrapper_kwargs
)

:::

VecEnv Copy Chapter

.. autoclass:: VecEnv
  :members:

DummyVecEnv Copy Chapter

.. autoclass:: DummyVecEnv
  :members:

SubprocVecEnv Copy Chapter

.. autoclass:: SubprocVecEnv
  :members:

Wrappers Copy Chapter

VecFrameStack Copy Section

.. autoclass:: VecFrameStack
  :members:

StackedObservations Copy Section

.. autoclass:: stable_baselines3.common.vec_env.stacked_observations.StackedObservations
  :members:

VecNormalize Copy Section

.. autoclass:: VecNormalize
  :members:

VecVideoRecorder Copy Section

.. autoclass:: VecVideoRecorder
  :members:

VecCheckNan Copy Section

.. autoclass:: VecCheckNan
  :members:

VecTransposeImage Copy Section

.. autoclass:: VecTransposeImage
  :members:

VecMonitor Copy Section

.. autoclass:: VecMonitor
  :members:

VecExtractDictObs Copy Section

.. autoclass:: VecExtractDictObs
  :members:

Misc Changelog (`docs/misc/changelog.md`) Copy Section

(changelog)=

Changelog

Release 2.9.2 (2026-10-26) Copy Chapter

:::{warning}
Stable-Baselines3 (SB3) v2.9.2 will be the last one supporting Python 3.10 (end of life in October 2026).
We highly recommended you to upgrade to Python >= 3.11.
:::

Breaking Changes: Copy Section

New Features: Copy Section

Bug Fixes: Copy Section

• Fixed file descriptor leak in SubprocVecEnv.close() by closing parent-side pipes to prevent "too many open files" errors in long-running loops
• Fixed Pandas deprecation warnings on inplace operations

[SB3-Contrib] Copy Section

• Upgraded to Stable-Baselines3 >= 2.9.2

[RL Zoo] Copy Section

• Set learning_starts to 500 for parking-v0 in TQC hyperparameters to ensure the first episode is finished
• Fixed "too many open files" issue when doing hyperparameter optimization: the trial environments are now closed in a finally block and gc_after_trial=True is passed to Optuna to free them between trials (also needs latest SB3 master version to fully work)
• Set use_expln=True for pybullet envs when using gSDE to avoid NaNs
• Add a note regarding pybullet source installation error on macOS due to legacy code
• Migrate packaging setup from setup.py to pyproject.toml and integrate requirements.txt as a new extra

[SBX] (SB3 + Jax) Copy Section

• Updated Jax dependency version range

Deprecations: Copy Section

Others: Copy Section

• Fixed moviepy test compatibility with moviepy >= 2.0 (PyTorch doesn't support it yet)
• Added filter for numpy newshape deprecation warning from torch tensorboard
• Fixed mypy error in test_vec_envs.py by wrapping itertools.product with list()
• Improved tests coverage for RMSpropTFLike optimizer (invalid params, centered/momentum/weight_decay branches, pickle roundtrip)
• Improved test coverage for save_util.py: added tests for BadZipFile error handling in load_from_zip_file and IsADirectoryError/FileNotFoundError handling in open_path
• Fixed Docker build by adding the missing --system flag to uv pip uninstall opencv-python (required by recent uv)
• Added a warning when n_steps > 1 is passed together with a custom replay_buffer_class, as n_steps is only applied when the replay buffer class is selected automatically (@Koustav-github)
• Updated tests dependencies

Documentation: Copy Section

• Clarified in the n_steps docstring that it only applies when replay_buffer_class is None, and that a custom replay buffer must be configured with replay_buffer_kwargs={"n_steps": ..., "gamma": ...} (@Koustav-github)
• Updated algorithm table (added the ones from SBX and sorted the table)
• Fixed typos, grammar, and Gymnasium API inconsistencies across documentation

Release 2.9.0 (2026-06-15) Copy Chapter

Updated dependencies (pandas is now optional, gymnasium 1.3.0 support, torch>=2.8)

Breaking Changes: Copy Section

• Relaxed Gymnasium version range (from "gymnasium>=0.29.1,<1.3.0" to "gymnasium>=0.29.1,<2.0")
• pandas and matplotlib are no longer core dependencies; they are now optional and only required for loading results and plotting (moved to stable-baselines3[extra]).
• Moved read_json and read_csv helper functions to test files
• Raised torch minimum version from 2.3 to 2.8 to mitigate https://github.com/advisories/GHSA-887c-mr87-cxwp

New Features: Copy Section

Bug Fixes: Copy Section

• Fixed deprecated error Taxi-v3 from gymnasium v1.3.0 in tests

[SB3-Contrib] Copy Section

• Optimized tests (faster to run)
• Fixed dead link for RecurrentPPO.

[RL Zoo] Copy Section

[SBX] (SB3 + Jax) Copy Section

• Added support for rollout_buffer_class and rollout_buffer_kwargs arguments in PPO and OnPolicyAlgorithmJax constructors, as in Stable Baselines3. (@Trenza1ore)
• Updated Jax dependency

Deprecations: Copy Section

Others: Copy Section

• Optimized tests (faster to run)
• Add workflow to automatically publish to PyPi when a new tag is created

Documentation: Copy Section

• Added example for using torch.compile
• Fixed many broken links and updated links to https whenever possible

Release 2.8.0 (2026-04-01) Copy Chapter

Dropped Python 3.9, added Python 3.13 support, MaskablePPO bug fix, default hyperparams for unlisted env in the RL Zoo, Markdown doc

Breaking Changes: Copy Section

• Removed support for Python 3.9, please upgrade to Python >= 3.10
• Set strict=True for every call to zip(...)
• Switched to pygame-ce when installing extras

New Features: Copy Section

• Added official support for Python 3.13

Bug Fixes: Copy Section

• Fixed saving and loading of Torch compiled models (using th.compile()) by updating get_parameters()
• Added a warning to env-checker if a multidiscrete space has multi-dimensional array (@unexploredtest)
• Fixed pandas.concat futurewarnings occuring when dataframes are empty by removing empty frames from the list before concatenating

[SB3-Contrib] Copy Section

• Set strict=True for every call to zip(...)
• Fixed MaskablePPO and RecurrentPPO inaccurate n_updates counting when target_kl early exits the training loop
• Fixed RecurrentPPO and MaskablePPO forward and predict not reshaping the action before clipping it (@immortal-boy)
• Do not call forward() method directly in RecurrentPPO (@immortal-boy)
• Fixed MaskableCategorical.apply_masking() crashing with ValueError: Simplex when cached probs deviate from sum=1 in float32 with large action spaces (torch 2.9+) (@kirann-05)
• Switched to Markdown documentation (using MyST parser)

[RL Zoo] Copy Section

• Set strict=True for every call to zip(...)
• Allow to specify env_kwargs in the hyperparam config
• Switched to Markdown documentation (using MyST parser)
• Allow to use default hyperparameters for any environment
• Save training command in Weights & Biases (Wandb)
• Save training command and default hyperparameters as study attributes

[SBX] (SB3 + Jax) Copy Section

• Increased Jax version range and use tf-nightly

Deprecations: Copy Section

• zip_strict() is not needed anymore since Python 3.10, please use zip(..., strict=True) instead

Others: Copy Section

• Updated to Python 3.10+ annotations
• Removed some unused variables (@unexploredtest)
• Improved type hints for distributions
• Simplified zip file loading by removing Python 3.6 workaround and enabling weights_only=True (PyTorch 2.x)
• Sped up saving/loading tests
• Updated black from v25 to v26
• Updated monitor test to check handling of empty monitor files

Documentation: Copy Section

• Added a note on MultiDiscrete spaces with multi-dimensional arrays and a wrapper to fix the issue (@unexploredtest)
• Added an example of manual export of SBX (SB3 + Jax) model to ONNX (@m-abr)
• Switched to Markdown documentation (using MyST parser)

Release 2.7.1 (2025-12-05) Copy Chapter

:::{warning}
Stable-Baselines3 (SB3) v2.7.1 will be the last one supporting Python 3.9 (end of life in October 2025).
We highly recommended you to upgrade to Python >= 3.10.
:::

Breaking Changes: Copy Section

New Features: Copy Section

• RolloutBuffer and DictRolloutBuffer now uses the actual observation / action space dtype (instead of float32), this should save memory (@Trenza1ore)

Bug Fixes: Copy Section

• Fixed env checker to properly handle Sequence observation spaces when nested inside composite spaces (Dict, Tuple, OneOf) (@copilot)
• Update env checker to warn users when using Graph space (@dhruvmalik007).
• Fixed memory leak in VecVideoRecorder where recorded_frames stayed in memory due to reference in the moviepy clip (@copilot)
• Remove double space in StopTrainingOnRewardThreshold callback message (@sea-bass)

[SB3-Contrib] Copy Section

• Fixed tensorboard log name for MaskablePPO

[SBX] (SB3 + Jax) Copy Section

• Added CnnPolicy to PPO

Documentation: Copy Section

• Added plotting documentation and examples
• Added documentation clarifying gSDE (Generalized State-Dependent Exploration) inference behavior for PPO, SAC, and A2C algorithms
• Documented Atari wrapper reset behavior where env.reset() may perform a no-op step instead of truly resetting when terminal_on_life_loss=True (default), and how to avoid this behavior by setting terminal_on_life_loss=False
• Clarified comment in _sample_action() method to better explain action scaling behavior for off-policy algorithms (@copilot)
• Added sb3-plus to projects page
• Added example usage of ONNX JS
• Updated link to paper of community project DeepNetSlice (@AlexPasqua)
• Added example usage of Tensorflow JS
• Included exact versions in ONNX JS and example project
• Made step 2 (pip install) of CONTRIBUTING.md more robust

Release 2.7.0 (2025-07-25) Copy Chapter

n-step returns for all off-policy algorithms

Breaking Changes: Copy Section

New Features: Copy Section

• Added support for n-step returns for off-policy algorithms via the n_steps parameter
• Added NStepReplayBuffer that allows to compute n-step returns without additional memory requirement (and without for loops)
• Added Gymnasium v1.2 support

Bug Fixes: Copy Section

• Fixed docker GPU image (PyTorch GPU was not installed)
• Fixed segmentation faults caused by non-portable schedules during model loading (@akanto)

[SB3-Contrib] Copy Section

• Added support for n-step returns for off-policy algorithms via the n_steps parameter
• Use the FloatSchedule and LinearSchedule classes instead of lambdas in the ARS, PPO, and QRDQN implementations to improve model portability across different operating systems

[RL Zoo] Copy Section

• linear_schedule now returns a SimpleLinearSchedule object for better portability
• Renamed LunarLander-v2 to LunarLander-v3 in hyperparameters
• Renamed CarRacing-v2 to CarRacing-v3 in hyperparameters
• Docker GPU images are now working again
• Use ConstantSchedule, and SimpleLinearSchedule instead of constant_fn and linear_schedule
• Fixed CarRacing-v3 hyperparameters for newer Gymnasium version

[SBX] (SB3 + Jax) Copy Section

• Added support for n-step returns for off-policy algorithms via the n_steps parameter
• Added KL Adaptive LR for PPO and LR schedule for SAC/TQC

Deprecations: Copy Section

• get_schedule_fn(), get_linear_fn(), constant_fn() are deprecated, please use FloatSchedule(), LinearSchedule(), ConstantSchedule() instead

Others: Copy Section

Documentation: Copy Section

• Clarify evaluate_policy documentation
• Added doc about training exceeding the total_timesteps parameter
• Updated LunarLander and LunarLanderContinuous environment versions to v3 (@j0m0k0)
• Added sb3-extra-buffers to the project page (@Trenza1ore)

Release 2.6.0 (2025-03-24) Copy Chapter

New ``LogEveryNTimesteps`` callback and ``has_attr`` method, refactored hyperparameter optimization

Breaking Changes: Copy Section

New Features: Copy Section

• Added has_attr method for VecEnv to check if an attribute exists
• Added LogEveryNTimesteps callback to dump logs every N timesteps (note: you need to pass log_interval=None to avoid any interference)
• Added Gymnasium v1.1 support

Bug Fixes: Copy Section

• SubProcVecEnv will now exit gracefully (without big traceback) when using KeyboardInterrupt

[SB3-Contrib] Copy Section

• Renamed _dump_logs() to dump_logs()
• Fixed issues with SubprocVecEnv and MaskablePPO by using vec_env.has_attr() (pickling issues, mask function not present)

[RL Zoo] Copy Section

• Refactored hyperparameter optimization. The Optuna Journal storage backend is now supported (recommended default) and you can easily load tuned hyperparameter via the new --trial-id argument of train.py.
• Save the exact command line used to launch a training
• Added support for special vectorized env (e.g. Brax, IsaacSim) by allowing to override the VecEnv class use to instantiate the env in the ExperimentManager
• Allow to disable auto-logging by passing --log-interval -2 (useful when logging things manually)
• Added Gymnasium v1.1 support
• Fixed use of old HF api in get_hf_trained_models()

[SBX] (SB3 + Jax) Copy Section

• Updated PPO to support net_arch, and additional fixes
• Fixed entropy coeff wrongly logged for SAC and derivatives.
• Fixed PPO predict() for env that were not normalized (action spaces with limits != [-1, 1])
• PPO now logs the standard deviation

Deprecations: Copy Section

• algo._dump_logs() is deprecated in favor of algo.dump_logs() and will be removed in SB3 v2.7.0

Others: Copy Section

• Updated black from v24 to v25
• Improved error messages when checking Box space equality (loading VecNormalize)
• Updated test to reflect how set_wrapper_attr should be used now

Documentation: Copy Section

• Clarify the use of Gym wrappers with make_vec_env in the section on Vectorized Environments (@pstahlhofen)
• Updated callback doc for EveryNTimesteps
• Added doc on how to set env attributes via VecEnv calls
• Added ONNX export example for MultiInputPolicy (@darkopetrovic)

Release 2.5.0 (2025-01-27) Copy Chapter

New algorithm: SimBa in SBX, NumPy 2.0 support

Breaking Changes: Copy Section

• Increased minimum required version of PyTorch to 2.3.0
• Removed support for Python 3.8

New Features: Copy Section

• Added support for NumPy v2.0: VecNormalize now cast normalized rewards to float32, updated bit flipping env to avoid overflow issues too
• Added official support for Python 3.12

[SBX] (SB3 + Jax) Copy Section

• Added SimBa Policy: Simplicity Bias for Scaling Up Parameters in DRL
• Added support for parameter resets

Others: Copy Section

• Updated Dockerfile

Documentation: Copy Section

• Added Decisions and Dragons to resources. (@jmacglashan)
• Updated PyBullet example, now compatible with Gymnasium
• Added link to policies for policy_kwargs parameter (@kplers)
• Add FootstepNet Envs to the project page (@cgaspard3333)
• Added FRASA to the project page (@MarcDcls)
• Fixed atari example (@chrisgao99)
• Add a note about Discrete action spaces with start!=0
• Update doc for massively parallel simulators (Isaac Lab, Brax, ...)
• Add dm_control example

Release 2.4.1 (2024-12-20) Copy Chapter

Bug Fixes: Copy Section

• Fixed a bug introduced in v2.4.0 where the VecVideoRecorder would override videos

Release 2.4.0 (2024-11-18) Copy Chapter

New algorithm: CrossQ in SB3 Contrib, Gymnasium v1.0 support

:::{note}
DQN (and QR-DQN) models saved with SB3 < 2.4.0 will show a warning about
truncation of optimizer state when loaded with SB3 >= 2.4.0.
To suppress the warning, simply save the model again.
You can find more info in PR #1963
:::

:::{warning}
Stable-Baselines3 (SB3) v2.4.0 will be the last one supporting Python 3.8 (end of life in October 2024)
and PyTorch < 2.3.
We highly recommended you to upgrade to Python >= 3.9 and PyTorch >= 2.3 (compatible with NumPy v2).
:::

Breaking Changes: Copy Section

• Increased minimum required version of Gymnasium to 0.29.1

New Features: Copy Section

• Added support for pre_linear_modules and post_linear_modules in create_mlp (useful for adding normalization layers, like in DroQ or CrossQ)
• Enabled np.ndarray logging for TensorBoardOutputFormat as histogram (see GH#1634) (@iwishwasaneagle)
• Updated env checker to warn users when using multi-dim array to define MultiDiscrete spaces
• Added support for Gymnasium v1.0

Bug Fixes: Copy Section

• Fixed memory leak when loading learner from storage, set_parameters() does not try to load the object data anymore
  and only loads the PyTorch parameters (@peteole)
• Cast type in compute gae method to avoid error when using torch compile (@amjames)
• CallbackList now sets the .parent attribute of child callbacks to its own .parent. (will-maclean)
• Fixed error when loading a model that has net_arch manually set to None (@jak3122)
• Set requirement numpy<2.0 until PyTorch is compatible (https://github.com/pytorch/pytorch/issues/107302)
• Updated DQN optimizer input to only include q_network parameters, removing the target_q_network ones (@corentinlger)
• Fixed test_buffers.py::test_device which was not actually checking the device of tensors (@rhaps0dy)

[SB3-Contrib] Copy Section

• Added CrossQ algorithm, from "Batch Normalization in Deep Reinforcement Learning" paper (@danielpalen)
• Added BatchRenorm PyTorch layer used in CrossQ (@danielpalen)
• Updated QR-DQN optimizer input to only include quantile_net parameters (@corentinlger)
• Fixed loading QRDQN changes target_update_interval (@jak3122)

[RL Zoo] Copy Section

• Updated defaults hyperparameters for TQC/SAC for Swimmer-v4 (decrease gamma for more consistent results)

[SBX] (SB3 + Jax) Copy Section

• Added CNN support for DQN
• Bug fix for SAC and related algorithms, optimize log of ent coeff to be consistent with SB3

Deprecations: Copy Section

Others: Copy Section

• Fixed various typos (@cschindlbeck)
• Remove unnecessary SDE noise resampling in PPO update (@brn-dev)
• Updated PyTorch version on CI to 2.3.1
• Added a warning to recommend using CPU with on policy algorithms (A2C/PPO) and MlpPolicy
• Switched to uv to download packages faster on GitHub CI
• Updated dependencies for read the doc
• Removed unnecessary copy_obs_dict method for SubprocVecEnv, remove the use of ordered dict and rename flatten_obs to stack_obs

Documentation: Copy Section

• Updated PPO doc to recommend using CPU with MlpPolicy
• Clarified documentation about planned features and citing software
• Added a note about the fact we are optimizing log of ent coeff for SAC

Release 2.3.2 (2024-04-27) Copy Chapter

Bug Fixes: Copy Section

• Reverted torch.load() to be called weights_only=False as it caused loading issue with old version of PyTorch.

Documentation: Copy Section

• Added ER-MRL to the project page (@corentinlger)
• Updated Tensorboard Logging Videos documentation (@NickLucche)

Release 2.3.1 (2024-04-22) Copy Chapter

Bug Fixes: Copy Section

• Cast return value of learning rate schedule to float, to avoid issue when loading model because of weights_only=True (@markscsmith)

Documentation: Copy Section

• Updated SBX documentation (CrossQ and deprecated DroQ)
• Updated RL Tips and Tricks section

Release 2.3.0 (2024-03-31) Copy Chapter

New defaults hyperparameters for DDPG, TD3 and DQN

:::{warning}
Because of weights_only=True, this release breaks loading of policies when using PyTorch 1.13.
Please upgrade to PyTorch >= 2.0 or upgrade SB3 version (we reverted the change in SB3 2.3.2)
:::

Breaking Changes: Copy Section

• The defaults hyperparameters of TD3 and DDPG have been changed to be more consistent with SAC

# SB3 < 2.3.0 default hyperparameters
# model = TD3("MlpPolicy", env, train_freq=(1, "episode"), gradient_steps=-1, batch_size=100)
# SB3 >= 2.3.0:
model = TD3("MlpPolicy", env, train_freq=1, gradient_steps=1, batch_size=256)

:::{note}
Two inconsistencies remain: the default network architecture for TD3/DDPG is [400, 300] instead of [256, 256] for SAC (for backward compatibility reasons, see report on the influence of the network size) and the default learning rate is 1e-3 instead of 3e-4 for SAC (for performance reasons, see W&B report on the influence of the lr)
:::

• The default learning_starts parameter of DQN have been changed to be consistent with the other offpolicy algorithms

# SB3 < 2.3.0 default hyperparameters, 50_000 corresponded to Atari defaults hyperparameters
# model = DQN("MlpPolicy", env, learning_starts=50_000)
# SB3 >= 2.3.0:
model = DQN("MlpPolicy", env, learning_starts=100)

• For safety, torch.load() is now called with weights_only=True when loading torch tensors,
  policy load() still uses weights_only=False as gymnasium imports are required for it to work
• When using huggingface_sb3, you will now need to set TRUST_REMOTE_CODE=True when downloading models from the hub, as pickle.load is not safe.

New Features: Copy Section

• Log success rate rollout/success_rate when available for on policy algorithms (@corentinlger)

Bug Fixes: Copy Section

• Fixed monitor_wrapper argument that was not passed to the parent class, and dones argument that wasn't passed to _update_into_buffer (@corentinlger)

[SB3-Contrib] Copy Section

• Added rollout_buffer_class and rollout_buffer_kwargs arguments to MaskablePPO
• Fixed train_freq type annotation for tqc and qrdqn (@Armandpl)
• Fixed sb3_contrib/common/maskable/*.py type annotations
• Fixed sb3_contrib/ppo_mask/ppo_mask.py type annotations
• Fixed sb3_contrib/common/vec_env/async_eval.py type annotations
• Add some additional notes about MaskablePPO (evaluation and multi-process) (@icheered)

[RL Zoo] Copy Section

• Updated defaults hyperparameters for TD3/DDPG to be more consistent with SAC
• Upgraded MuJoCo envs hyperparameters to v4 (pre-trained agents need to be updated)
• Added test dependencies to setup.py (@power-edge)
• Simplify dependencies of requirements.txt (remove duplicates from setup.py)

[SBX] (SB3 + Jax) Copy Section

• Added support for MultiDiscrete and MultiBinary action spaces to PPO
• Added support for large values for gradient_steps to SAC, TD3, and TQC
• Fix train() signature and update type hints
• Fix replay buffer device at load time
• Added flatten layer
• Added CrossQ

Deprecations: Copy Section

Others: Copy Section

• Updated black from v23 to v24
• Updated ruff to >= v0.3.1
• Updated env checker for (multi)discrete spaces with non-zero start.

Documentation: Copy Section

• Added a paragraph on modifying vectorized environment parameters via setters (@fracapuano)
• Updated callback code example
• Updated export to ONNX documentation, it is now much simpler to export SB3 models with newer ONNX Opset!
• Added video link to "Practical Tips for Reliable Reinforcement Learning" video
• Added render_mode="human" in the README example (@marekm4)
• Fixed docstring signature for sum_independent_dims (@stagoverflow)
• Updated docstring description for log_interval in the base class (@rushitnshah).

Release 2.2.1 (2023-11-17) Copy Chapter

Support for options at reset, bug fixes and better error messages

:::{note}
SB3 v2.2.0 was yanked after a breaking change was found in GH#1751.
Please use SB3 v2.2.1 and not v2.2.0.
:::

Breaking Changes: Copy Section

• Switched to ruff for sorting imports (isort is no longer needed), black and ruff version now require a minimum version
• Dropped x is False in favor of not x, which means that callbacks that wrongly returned None (instead of a boolean) will cause the training to stop (@iwishiwasaneagle)

New Features: Copy Section

• Improved error message of the env_checker for env wrongly detected as GoalEnv (compute_reward() is defined)
• Improved error message when mixing Gym API with VecEnv API (see GH#1694)
• Add support for setting options at reset with VecEnv via the set_options() method. Same as seeds logic, options are reset at the end of an episode (@ReHoss)
• Added rollout_buffer_class and rollout_buffer_kwargs arguments to on-policy algorithms (A2C and PPO)

Bug Fixes: Copy Section

• Prevents using squash_output and not use_sde in ActorCritcPolicy (@PatrickHelm)
• Performs unscaling of actions in collect_rollout in OnPolicyAlgorithm (@PatrickHelm)
• Moves VectorizedActionNoise into _setup_learn() in OffPolicyAlgorithm (@PatrickHelm)
• Prevents out of bound error on Windows if no seed is passed (@PatrickHelm)
• Calls callback.update_locals() before callback.on_rollout_end() in OnPolicyAlgorithm (@PatrickHelm)
• Fixed replay buffer device after loading in OffPolicyAlgorithm (@PatrickHelm)
• Fixed render_mode which was not properly loaded when using VecNormalize.load()
• Fixed success reward dtype in SimpleMultiObsEnv (@NixGD)
• Fixed check_env for Sequence observation space (@corentinlger)
• Prevents instantiating BitFlippingEnv with conflicting observation spaces (@kylesayrs)
• Fixed ResourceWarning when loading and saving models (files were not closed), please note that only path are closed automatically,
  the behavior stay the same for tempfiles (they need to be closed manually),
  the behavior is now consistent when loading/saving replay buffer

[SB3-Contrib] Copy Section

• Added set_options for AsyncEval
• Added rollout_buffer_class and rollout_buffer_kwargs arguments to TRPO

[RL Zoo] Copy Section

• Removed gym dependency, the package is still required for some pretrained agents.
• Added --eval-env-kwargs to train.py (@Quentin18)
• Added ppo_lstm to hyperparams_opt.py (@technocrat13)
• Upgraded to pybullet_envs_gymnasium>=0.4.0
• Removed old hacks (for instance limiting offpolicy algorithms to one env at test time)
• Updated docker image, removed support for X server
• Replaced deprecated optuna.suggest_uniform(...) by optuna.suggest_float(..., low=..., high=...)

[SBX] (SB3 + Jax) Copy Section

• Added DDPG and TD3 algorithms

Deprecations: Copy Section

Others: Copy Section

• Fixed stable_baselines3/common/callbacks.py type hints
• Fixed stable_baselines3/common/utils.py type hints
• Fixed stable_baselines3/common/vec_envs/vec_transpose.py type hints
• Fixed stable_baselines3/common/vec_env/vec_video_recorder.py type hints
• Fixed stable_baselines3/common/save_util.py type hints
• Updated docker images to Ubuntu Jammy using micromamba 1.5
• Fixed stable_baselines3/common/buffers.py type hints
• Fixed stable_baselines3/her/her_replay_buffer.py type hints
• Buffers do no call an additional .copy() when storing new transitions
• Fixed ActorCriticPolicy.extract_features() signature by adding an optional features_extractor argument
• Update dependencies (accept newer Shimmy/Sphinx version and remove sphinx_autodoc_typehints)
• Fixed stable_baselines3/common/off_policy_algorithm.py type hints
• Fixed stable_baselines3/common/distributions.py type hints
• Fixed stable_baselines3/common/vec_env/vec_normalize.py type hints
• Fixed stable_baselines3/common/vec_env/__init__.py type hints
• Switched to PyTorch 2.1.0 in the CI (fixes type annotations)
• Fixed stable_baselines3/common/policies.py type hints
• Switched to mypy only for checking types
• Added tests to check consistency when saving/loading files

Documentation: Copy Section

• Updated RL Tips and Tricks (include recommendation for evaluation, added links to DroQ, ARS and SBX).
• Fixed various typos and grammar mistakes
• Added PokemonRedExperiments to the project page
• Fixed an out-of-date command for installing Atari in examples

Release 2.1.0 (2023-08-17) Copy Chapter

Float64 actions , Gymnasium 0.29 support and bug fixes

Breaking Changes: Copy Section

• Removed Python 3.7 support
• SB3 now requires PyTorch >= 1.13

New Features: Copy Section

• Added Python 3.11 support
• Added Gymnasium 0.29 support (@pseudo-rnd-thoughts)

[SB3-Contrib] Copy Section

• Fixed MaskablePPO ignoring stats_window_size argument
• Added Python 3.11 support

[RL Zoo] Copy Section

• Upgraded to Huggingface-SB3 >= 2.3
• Added Python 3.11 support

Bug Fixes: Copy Section

• Relaxed check in logger, that was causing issue on Windows with colorama
• Fixed off-policy algorithms with continuous float64 actions (see #1145) (@tobirohrer)
• Fixed env_checker.py warning messages for out of bounds in complex observation spaces (@Gabo-Tor)

Deprecations: Copy Section

Others: Copy Section

• Updated GitHub issue templates
• Fix typo in gym patch error message (@lukashass)
• Refactor test_spaces.py tests

Documentation: Copy Section

• Fixed callback example (@BertrandDecoster)
• Fixed policy network example (@kyle-he)
• Added mobile-env as new community project (@stefanbschneider)
• Added [DeepNetSlice](https://github.com/AlexPasqua/DeepNetSlice) to community projects (@AlexPasqua)

Release 2.0.0 (2023-06-22) Copy Chapter

Gymnasium support

:::{warning}
Stable-Baselines3 (SB3) v2.0 will be the last one supporting python 3.7 (end of life in June 2023).
We highly recommended you to upgrade to Python >= 3.8.
:::

Breaking Changes: Copy Section

• Switched to Gymnasium as primary backend, Gym 0.21 and 0.26 are still supported via the shimmy package (@carlosluis, @arjun-kg, @tlpss)
• The deprecated online_sampling argument of HerReplayBuffer was removed
• Removed deprecated stack_observation_space method of StackedObservations
• Renamed environment output observations in evaluate_policy to prevent shadowing the input observations during callbacks (@npit)
• Upgraded wrappers and custom environment to Gymnasium
• Refined the HumanOutputFormat file check: now it verifies if the object is an instance of io.TextIOBase instead of only checking for the presence of a write method.
• Because of new Gym API (0.26+), the random seed passed to vec_env.seed(seed=seed) will only be effective after then env.reset() call.

New Features: Copy Section

• Added Gymnasium support (Gym 0.21 and 0.26 are supported via the shimmy package)

[SB3-Contrib] Copy Section

• Fixed QRDQN update interval for multi envs

[RL Zoo] Copy Section

• Gym 0.26+ patches to continue working with pybullet and TimeLimit wrapper
• Renamed CarRacing-v1 to CarRacing-v2 in hyperparameters
• Huggingface push to hub now accepts a --n-timesteps argument to adjust the length of the video
• Fixed record_video steps (before it was stepping in a closed env)
• Dropped Gym 0.21 support

Bug Fixes: Copy Section

• Fixed VecExtractDictObs does not handle terminal observation (@WeberSamuel)
• Set NumPy version to >=1.20 due to use of numpy.typing (@troiganto)
• Fixed loading DQN changes target_update_interval (@tobirohrer)
• Fixed env checker to properly reset the env before calling step() when checking
  for Inf and NaN (@lutogniew)
• Fixed HER truncate_last_trajectory() (@lbergmann1)
• Fixed HER desired and achieved goal order in reward computation (@JonathanKuelz)

Deprecations: Copy Section

Others: Copy Section

• Fixed stable_baselines3/a2c/*.py type hints
• Fixed stable_baselines3/ppo/*.py type hints
• Fixed stable_baselines3/sac/*.py type hints
• Fixed stable_baselines3/td3/*.py type hints
• Fixed stable_baselines3/common/base_class.py type hints
• Fixed stable_baselines3/common/logger.py type hints
• Fixed stable_baselines3/common/envs/*.py type hints
• Fixed stable_baselines3/common/vec_env/vec_monitor|vec_extract_dict_obs|util.py type hints
• Fixed stable_baselines3/common/vec_env/base_vec_env.py type hints
• Fixed stable_baselines3/common/vec_env/vec_frame_stack.py type hints
• Fixed stable_baselines3/common/vec_env/dummy_vec_env.py type hints
• Fixed stable_baselines3/common/vec_env/subproc_vec_env.py type hints
• Upgraded docker images to use mamba/micromamba and CUDA 11.7
• Updated env checker to reflect what subset of Gymnasium is supported and improve GoalEnv checks
• Improve type annotation of wrappers
• Tests envs are now checked too
• Added render test for VecEnv and VecEnvWrapper
• Update issue templates and env info saved with the model
• Changed seed() method return type from List to Sequence
• Updated env checker doc and requirements for tuple spaces/goal envs

Documentation: Copy Section

• Added Deep RL Course link to the Deep RL Resources page
• Added documentation about VecEnv API vs Gym API
• Upgraded tutorials to Gymnasium API
• Make it more explicit when using VecEnv vs Gym env
• Added UAV_Navigation_DRL_AirSim to the project page (@heleidsn)
• Added EvalCallback example (@sidney-tio)
• Update custom env documentation
• Added pink-noise-rl to projects page
• Fix custom policy example, ortho_init was ignored
• Added SBX page

Release 1.8.0 (2023-04-07) Copy Chapter

Multi-env HerReplayBuffer, Open RL Benchmark, Improved env checker

:::{warning}
Stable-Baselines3 (SB3) v1.8.0 will be the last one to use Gym as a backend.
Starting with v2.0.0, Gymnasium will be the default backend (though SB3 will have compatibility layers for Gym envs).
You can find a migration guide here: https://gymnasium.farama.org/introduction/migration_guide/.
If you want to try the SB3 v2.0 alpha version, you can take a look at PR #1327.
:::

Breaking Changes: Copy Section

• Removed shared layers in mlp_extractor (@AlexPasqua)
• Refactored StackedObservations (it now handles dict obs, StackedDictObservations was removed)
• You must now explicitly pass a features_extractor parameter when calling extract_features()
• Dropped offline sampling for HerReplayBuffer
• As HerReplayBuffer was refactored to support multiprocessing, previous replay buffer are incompatible with this new version
• HerReplayBuffer doesn't require a max_episode_length anymore

New Features: Copy Section

• Added repeat_action_probability argument in AtariWrapper.
• Only use NoopResetEnv and MaxAndSkipEnv when needed in AtariWrapper
• Added support for dict/tuple observations spaces for VecCheckNan, the check is now active in the env_checker() (@DavyMorgan)
• Added multiprocessing support for HerReplayBuffer
• HerReplayBuffer now supports all datatypes supported by ReplayBuffer
• Provide more helpful failure messages when validating the observation_space of custom gym environments using check_env (@FieteO)
• Added stats_window_size argument to control smoothing in rollout logging (@jonasreiher)

[SB3-Contrib] Copy Section

• Added warning about potential crashes caused by check_env in the MaskablePPO docs (@AlexPasqua)
• Fixed sb3_contrib/qrdqn/*.py type hints
• Removed shared layers in mlp_extractor (@AlexPasqua)

[RL Zoo] Copy Section

• Open RL Benchmark
• Upgraded to new HerReplayBuffer implementation that supports multiple envs
• Removed TimeFeatureWrapper for Panda and Fetch envs, as the new replay buffer should handle timeout.
• Tuned hyperparameters for RecurrentPPO on Swimmer
• Documentation is now built using Sphinx and hosted on read the doc
• Removed use_auth_token for push to hub util
• Reverted from v3 to v2 for HumanoidStandup, Reacher, InvertedPendulum and InvertedDoublePendulum since they were not part of the mujoco refactoring (see https://github.com/openai/gym/pull/1304)
• Fixed gym-minigrid policy (from MlpPolicy to MultiInputPolicy)
• Replaced deprecated optuna.suggest_loguniform(...) by optuna.suggest_float(..., log=True)
• Switched to ruff and pyproject.toml
• Removed online_sampling and max_episode_length argument when using HerReplayBuffer

Bug Fixes: Copy Section

• Fixed Atari wrapper that missed the reset condition (@luizapozzobon)
• Added the argument dtype (default to float32) to the noise for consistency with gym action (@sidney-tio)
• Fixed PPO train/n_updates metric not accounting for early stopping (@adamfrly)
• Fixed loading of normalized image-based environments
• Fixed DictRolloutBuffer.add with multidimensional action space (@younik)

Deprecations: Copy Section

Others: Copy Section

• Fixed tests/test_tensorboard.py type hint
• Fixed tests/test_vec_normalize.py type hint
• Fixed stable_baselines3/common/monitor.py type hint
• Added tests for StackedObservations
• Removed Gitlab CI file
• Moved from setup.cg to pyproject.toml configuration file
• Switched from flake8 to ruff
• Upgraded AutoROM to latest version
• Fixed stable_baselines3/dqn/*.py type hints
• Added extra_no_roms option for package installation without Atari Roms

Documentation: Copy Section

• Renamed load_parameters to set_parameters (@DavyMorgan)
• Clarified documentation about subproc multiprocessing for A2C (@Bonifatius94)
• Fixed typo in A2C docstring (@AlexPasqua)
• Renamed timesteps to episodes for log_interval description (@theSquaredError)
• Removed note about gif creation for Atari games (@harveybellini)
• Added information about default network architecture
• Update information about Gymnasium support

Release 1.7.0 (2023-01-10) Copy Chapter

:::{warning}
Shared layers in MLP policy (mlp_extractor) are now deprecated for PPO, A2C and TRPO.
This feature will be removed in SB3 v1.8.0 and the behavior of net_arch=[64, 64]
will create separate networks with the same architecture, to be consistent with the off-policy algorithms.
:::

:::{note}
A2C and PPO saved with SB3 < 1.7.0 will show a warning about
missing keys in the state dict when loaded with SB3 >= 1.7.0.
To suppress the warning, simply save the model again.
You can find more info in issue #1233
:::

Breaking Changes: Copy Section

• Removed deprecated create_eval_env, eval_env, eval_log_path, n_eval_episodes and eval_freq parameters,
  please use an EvalCallback instead
• Removed deprecated sde_net_arch parameter
• Removed ret attributes in VecNormalize, please use returns instead
• VecNormalize now updates the observation space when normalizing images

New Features: Copy Section

• Introduced mypy type checking
• Added option to have non-shared features extractor between actor and critic in on-policy algorithms (@AlexPasqua)
• Added with_bias argument to create_mlp
• Added support for multidimensional spaces.MultiBinary observations
• Features extractors now properly support unnormalized image-like observations (3D tensor)
  when passing normalize_images=False
• Added normalized_image parameter to NatureCNN and CombinedExtractor
• Added support for Python 3.10

[SB3-Contrib] Copy Section

• Fixed a bug in RecurrentPPO where the lstm states where incorrectly reshaped for n_lstm_layers > 1 (thanks @kolbytn)
• Fixed RuntimeError: rnn: hx is not contiguous while predicting terminal values for RecurrentPPO when n_lstm_layers > 1

[RL Zoo] Copy Section

• Added support for python file for configuration
• Added monitor_kwargs parameter

Bug Fixes: Copy Section

• Fixed ProgressBarCallback under-reporting (@dominicgkerr)
• Fixed return type of evaluate_actions in ActorCritcPolicy to reflect that entropy is an optional tensor (@Rocamonde)
• Fixed type annotation of policy in BaseAlgorithm and OffPolicyAlgorithm
• Allowed model trained with Python 3.7 to be loaded with Python 3.8+ without the custom_objects workaround
• Raise an error when the same gym environment instance is passed as separate environments when creating a vectorized environment with more than one environment. (@Rocamonde)
• Fix type annotation of model in evaluate_policy
• Fixed Self return type using TypeVar
• Fixed the env checker, the key was not passed when checking images from Dict observation space
• Fixed normalize_images which was not passed to parent class in some cases
• Fixed load_from_vector that was broken with newer PyTorch version when passing PyTorch tensor

Deprecations: Copy Section

• You should now explicitly pass a features_extractor parameter when calling extract_features()
• Deprecated shared layers in MlpExtractor (@AlexPasqua)

Others: Copy Section

• Used issue forms instead of issue templates
• Updated the PR template to associate each PR with its peer in RL-Zoo3 and SB3-Contrib
• Fixed flake8 config to be compatible with flake8 6+
• Goal-conditioned environments are now characterized by the availability of the compute_reward method, rather than by their inheritance to gym.GoalEnv
• Replaced CartPole-v0 by CartPole-v1 is tests
• Fixed tests/test_distributions.py type hints
• Fixed stable_baselines3/common/type_aliases.py type hints
• Fixed stable_baselines3/common/torch_layers.py type hints
• Fixed stable_baselines3/common/env_util.py type hints
• Fixed stable_baselines3/common/preprocessing.py type hints
• Fixed stable_baselines3/common/atari_wrappers.py type hints
• Fixed stable_baselines3/common/vec_env/vec_check_nan.py type hints
• Exposed modules in __init__.py with the __all__ attribute (@ZikangXiong)
• Upgraded GitHub CI/setup-python to v4 and checkout to v3
• Set tensors construction directly on the device (~8% speed boost on GPU)
• Monkey-patched np.bool = bool so gym 0.21 is compatible with NumPy 1.24+
• Standardized the use of from gym import spaces
• Modified get_system_info to avoid issue linked to copy-pasting on GitHub issue

Documentation: Copy Section

• Updated Hugging Face Integration page (@simoninithomas)
• Changed env to vec_env when environment is vectorized
• Updated custom policy docs to better explain the mlp_extractor's dimensions (@AlexPasqua)
• Updated custom policy documentation (@athatheo)
• Improved tensorboard callback doc
• Clarify doc when using image-like input
• Added RLeXplore to the project page (@yuanmingqi)

Release 1.6.2 (2022-10-10) Copy Chapter

Progress bar in the learn() method, RL Zoo3 is now a package

Breaking Changes: Copy Section

New Features: Copy Section

• Added progress_bar argument in the learn() method, displayed using TQDM and rich packages
• Added progress bar callback
• The RL Zoo can now be installed as a package (pip install rl_zoo3)

[SB3-Contrib] Copy Section

[RL Zoo] Copy Section

• RL Zoo is now a python package and can be installed using pip install rl_zoo3

Bug Fixes: Copy Section

• self.num_timesteps was initialized properly only after the first call to on_step() for callbacks
• Set importlib-metadata version to ~=4.13 to be compatible with gym=0.21

Deprecations: Copy Section

• Added deprecation warning if parameters eval_env, eval_freq or create_eval_env are used (see #925) (@tobirohrer)

Others: Copy Section

• Fixed type hint of the env_id parameter in make_vec_env and make_atari_env (@AlexPasqua)

Documentation: Copy Section

• Extended docstring of the wrapper_class parameter in make_vec_env (@AlexPasqua)

Release 1.6.1 (2022-09-29) Copy Chapter

Bug fix release

Breaking Changes: Copy Section

• Switched minimum tensorboard version to 2.9.1

New Features: Copy Section

• Support logging hyperparameters to tensorboard (@timothe-chaumont)
• Added checkpoints for replay buffer and VecNormalize statistics (@anand-bala)
• Added option for Monitor to append to existing file instead of overriding (@sidney-tio)
• The env checker now raises an error when using dict observation spaces and observation keys don't match observation space keys

[SB3-Contrib] Copy Section

• Fixed the issue of wrongly passing policy arguments when using CnnLstmPolicy or MultiInputLstmPolicy with RecurrentPPO (@mlodel)

Bug Fixes: Copy Section

• Fixed issue where PPO gives NaN if rollout buffer provides a batch of size 1 (@hughperkins)
• Fixed the issue that predict does not always return action as np.ndarray (@qgallouedec)
• Fixed division by zero error when computing FPS when a small number of time has elapsed in operating systems with low-precision timers.
• Added multidimensional action space support (@qgallouedec)
• Fixed missing verbose parameter passing in the EvalCallback constructor (@burakdmb)
• Fixed the issue that when updating the target network in DQN, SAC, TD3, the running_mean and running_var properties of batch norm layers are not updated (@honglu2875)
• Fixed incorrect type annotation of the replay_buffer_class argument in common.OffPolicyAlgorithm initializer, where an instance instead of a class was required (@Rocamonde)
• Fixed loading saved model with different number of environments
• Removed forward() abstract method declaration from common.policies.BaseModel (already defined in torch.nn.Module) to fix type errors in subclasses (@Rocamonde)
• Fixed the return type of .load() and .learn() methods in BaseAlgorithm so that they now use TypeVar (@Rocamonde)
• Fixed an issue where keys with different tags but the same key raised an error in common.logger.HumanOutputFormat (@Rocamonde and @AdamGleave)
• Set importlib-metadata version to ~=4.13

Deprecations: Copy Section

Others: Copy Section

• Fixed DictReplayBuffer.next_observations typing (@qgallouedec)
• Added support for device="auto" in buffers and made it default (@qgallouedec)
• Updated ResultsWriter (used internally by Monitor wrapper) to automatically create missing directories when filename is a path (@dominicgkerr)

Documentation: Copy Section

• Added an example of callback that logs hyperparameters to tensorboard. (@timothe-chaumont)
• Fixed typo in docstring "nature" -> "Nature" (@Melanol)
• Added info on split tensorboard logs into (@Melanol)
• Fixed typo in ppo doc (@francescoluciano)
• Fixed typo in install doc(@jlp-ue)
• Clarified and standardized verbosity documentation
• Added link to a GitHub issue in the custom policy documentation (@AlexPasqua)
• Update doc on exporting models (fixes and added torch jit)
• Fixed typos (@Akhilez)
• Standardized the use of " for string representation in documentation

Release 1.6.0 (2022-07-11) Copy Chapter

Recurrent PPO (PPO LSTM), better defaults for learning from pixels with SAC/TD3

Breaking Changes: Copy Section

• Changed the way policy "aliases" are handled ("MlpPolicy", "CnnPolicy", ...), removing the former
  register_policy helper, policy_base parameter and using policy_aliases static attributes instead (@Gregwar)
• SB3 now requires PyTorch >= 1.11
• Changed the default network architecture when using CnnPolicy or MultiInputPolicy with SAC or DDPG/TD3,
  share_features_extractor is now set to False by default and the net_arch=[256, 256] (instead of net_arch=[] that was before)

New Features: Copy Section

[SB3-Contrib] Copy Section

• Added Recurrent PPO (PPO LSTM). See https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/53

Bug Fixes: Copy Section

• Fixed saving and loading large policies greater than 2GB (@jkterry1, @ycheng517)
• Fixed final goal selection strategy that did not sample the final achieved goal (@qgallouedec)
• Fixed a bug with special characters in the tensorboard log name (@quantitative-technologies)
• Fixed a bug in DummyVecEnv's and SubprocVecEnv's seeding function. None value was unchecked (@ScheiklP)
• Fixed a bug where EvalCallback would crash when trying to synchronize VecNormalize stats when observation normalization was disabled
• Added a check for unbounded actions
• Fixed issues due to newer version of protobuf (tensorboard) and sphinx
• Fix exception causes all over the codebase (@cool-RR)
• Prohibit simultaneous use of optimize_memory_usage and handle_timeout_termination due to a bug (@MWeltevrede)
• Fixed a bug in kl_divergence check that would fail when using numpy arrays with MultiCategorical distribution

Deprecations: Copy Section

Others: Copy Section

• Upgraded to Python 3.7+ syntax using pyupgrade
• Removed redundant double-check for nested observations from BaseAlgorithm._wrap_env (@TibiGG)

Documentation: Copy Section

• Added link to gym doc and gym env checker
• Fix typo in PPO doc (@bcollazo)
• Added link to PPO ICLR blog post
• Added remark about breaking Markov assumption and timeout handling
• Added doc about MLFlow integration via custom logger (@git-thor)
• Updated Huggingface integration doc
• Added copy button for code snippets
• Added doc about EnvPool and Isaac Gym support

Release 1.5.0 (2022-03-25) Copy Chapter

Bug fixes, early stopping callback

Breaking Changes: Copy Section

• Switched minimum Gym version to 0.21.0

New Features: Copy Section

• Added StopTrainingOnNoModelImprovement to callback collection (@caburu)
• Makes the length of keys and values in HumanOutputFormat configurable,
  depending on desired maximum width of output.
• Allow PPO to turn of advantage normalization (see PR #763) @vwxyzjn

[SB3-Contrib] Copy Section

• coming soon: Cross Entropy Method, see https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/62

Bug Fixes: Copy Section

• Fixed a bug in VecMonitor. The monitor did not consider the info_keywords during stepping (@ScheiklP)
• Fixed a bug in HumanOutputFormat. Distinct keys truncated to the same prefix would overwrite each others value,
  resulting in only one being output. This now raises an error (this should only affect a small fraction of use cases
  with very long keys.)
• Routing all the nn.Module calls through implicit rather than explicit forward as per pytorch guidelines (@manuel-delverme)
• Fixed a bug in VecNormalize where error occurs when norm_obs is set to False for environment with dictionary observation (@buoyancy99)
• Set default env argument to None in HerReplayBuffer.sample (@qgallouedec)
• Fix batch_size typing in DQN (@qgallouedec)
• Fixed sample normalization in DictReplayBuffer (@qgallouedec)

Deprecations: Copy Section

Others: Copy Section

• Fixed pytest warnings
• Removed parameter remove_time_limit_termination in off policy algorithms since it was dead code (@Gregwar)

Documentation: Copy Section

• Added doc on Hugging Face integration (@simoninithomas)
• Added furuta pendulum project to project list (@armandpl)
• Fix indentation 2 spaces to 4 spaces in custom env documentation example (@Gautam-J)
• Update MlpExtractor docstring (@gianlucadecola)
• Added explanation of the logger output
• Update Directly Accessing The Summary Writer in tensorboard integration (@xy9485)

Release 1.4.0 (2022-01-18) Copy Chapter

TRPO, ARS and multi env training for off-policy algorithms

Breaking Changes: Copy Section

• Dropped python 3.6 support (as announced in previous release)
• Renamed mask argument of the predict() method to episode_start (used with RNN policies only)
• local variables action, done and reward were renamed to their plural form for offpolicy algorithms (actions, dones, rewards),
  this may affect custom callbacks.
• Removed episode_reward field from RolloutReturn() type

:::{warning}
An update to the HER algorithm is planned to support multi-env training and remove the max episode length constrain.
(see PR #704)
This will be a backward incompatible change (model trained with previous version of HER won't work with the new version).
:::

New Features: Copy Section

• Added norm_obs_keys param for VecNormalize wrapper to configure which observation keys to normalize (@kachayev)
• Added experimental support to train off-policy algorithms with multiple envs (note: HerReplayBuffer currently not supported)
• Handle timeout termination properly for on-policy algorithms (when using TimeLimit)
• Added skip option to VecTransposeImage to skip transforming the channel order when the heuristic is wrong
• Added copy() and combine() methods to RunningMeanStd

[SB3-Contrib] Copy Section

• Added Trust Region Policy Optimization (TRPO) (@cyprienc)
• Added Augmented Random Search (ARS) (@sgillen)
• Coming soon: PPO LSTM, see https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/53

Bug Fixes: Copy Section

• Fixed a bug where set_env() with VecNormalize would result in an error with off-policy algorithms (thanks @cleversonahum)
• FPS calculation is now performed based on number of steps performed during last learn call, even when reset_num_timesteps is set to False (@kachayev)
• Fixed evaluation script for recurrent policies (experimental feature in SB3 contrib)
• Fixed a bug where the observation would be incorrectly detected as non-vectorized instead of throwing an error
• The env checker now properly checks and warns about potential issues for continuous action spaces when the boundaries are too small or when the dtype is not float32
• Fixed a bug in VecFrameStack with channel first image envs, where the terminal observation would be wrongly created.

Deprecations: Copy Section

Others: Copy Section

• Added a warning in the env checker when not using np.float32 for continuous actions
• Improved test coverage and error message when checking shape of observation
• Added newline="\n" when opening CSV monitor files so that each line ends with \r\n instead of \r\r\n on Windows while Linux environments are not affected (@hsuehch)
• Fixed device argument inconsistency (@qgallouedec)

Documentation: Copy Section

• Add drivergym to projects page (@theDebugger811)
• Add highway-env to projects page (@eleurent)
• Add tactile-gym to projects page (@ac-93)
• Fix indentation in the RL tips page (@cove9988)
• Update GAE computation docstring
• Add documentation on exporting to TFLite/Coral
• Added JMLR paper and updated citation
• Added link to RL Tips and Tricks video
• Updated BaseAlgorithm.load docstring (@Demetrio92)
• Added a note on load behavior in the examples (@Demetrio92)
• Updated SB3 Contrib doc
• Fixed A2C and migration guide guidance on how to set epsilon with RMSpropTFLike (@thomasgubler)
• Fixed custom policy documentation (@IperGiove)
• Added doc on Weights & Biases integration

Release 1.3.0 (2021-10-23) Copy Chapter

Bug fixes and improvements for the user

:::{warning}
This version will be the last one supporting Python 3.6 (end of life in Dec 2021).
We highly recommended you to upgrade to Python >= 3.7.
:::

Breaking Changes: Copy Section

• sde_net_arch argument in policies is deprecated and will be removed in a future version.

• _get_latent (ActorCriticPolicy) was removed

• All logging keys now use underscores instead of spaces (@timokau). Concretely this changes:

  • time/total timesteps to time/total_timesteps for off-policy algorithms (PPO and A2C) and the eval callback (on-policy algorithms already used the underscored version),
  • rollout/exploration rate to rollout/exploration_rate and
  • rollout/success rate to rollout/success_rate.

New Features: Copy Section

• Added methods get_distribution and predict_values for ActorCriticPolicy for A2C/PPO/TRPO (@cyprienc)
• Added methods forward_actor and forward_critic for MlpExtractor
• Added sb3.get_system_info() helper function to gather version information relevant to SB3 (e.g., Python and PyTorch version)
• Saved models now store system information where agent was trained, and load functions have print_system_info parameter to help debugging load issues

Bug Fixes: Copy Section

• Fixed dtype of observations for SimpleMultiObsEnv
• Allow VecNormalize to wrap discrete-observation environments to normalize reward
  when observation normalization is disabled
• Fixed a bug where DQN would throw an error when using Discrete observation and stochastic actions
• Fixed a bug where sub-classed observation spaces could not be used
• Added force_reset argument to load() and set_env() in order to be able to call learn(reset_num_timesteps=False) with a new environment

Deprecations: Copy Section

Others: Copy Section

• Cap gym max version to 0.19 to avoid issues with atari-py and other breaking changes
• Improved error message when using dict observation with the wrong policy
• Improved error message when using EvalCallback with two envs not wrapped the same way.
• Added additional infos about supported python version for PyPi in setup.py

Documentation: Copy Section

• Add Rocket League Gym to list of supported projects (@AechPro)
• Added gym-electric-motor to project page (@wkirgsn)
• Added policy-distillation-baselines to project page (@CUN-bjy)
• Added ONNX export instructions (@batu)
• Update read the doc env (fixed docutils issue)
• Fix PPO environment name (@IljaAvadiev)
• Fix custom env doc and add env registration example
• Update algorithms from SB3 Contrib
• Use underscores for numeric literals in examples to improve clarity

Release 1.2.0 (2021-09-03) Copy Chapter

Hotfix for VecNormalize, training/eval mode support

Breaking Changes: Copy Section

• SB3 now requires PyTorch >= 1.8.1
• VecNormalize ret attribute was renamed to returns

New Features: Copy Section

Bug Fixes: Copy Section

• Hotfix for VecNormalize where the observation filter was not updated at reset (thanks @vwxyzjn)
• Fixed model predictions when using batch normalization and dropout layers by calling train() and eval() (@davidblom603)
• Fixed model training for DQN, TD3 and SAC so that their target nets always remain in evaluation mode (@ayeright)
• Passing gradient_steps=0 to an off-policy algorithm will result in no gradient steps being taken (vs as many gradient steps as steps done in the environment
  during the rollout in previous versions)

Deprecations: Copy Section

Others: Copy Section

• Enabled Python 3.9 in GitHub CI
• Fixed type annotations
• Refactored predict() by moving the preprocessing to obs_to_tensor() method

Documentation: Copy Section

• Updated multiprocessing example
• Added example of VecEnvWrapper
• Added a note about logging to tensorboard more often
• Added warning about simplicity of examples and link to RL zoo (@MihaiAnca13)

Release 1.1.0 (2021-07-01) Copy Chapter

Dict observation support, timeout handling and refactored HER buffer

Breaking Changes: Copy Section

• All customs environments (e.g. the BitFlippingEnv or IdentityEnv) were moved to stable_baselines3.common.envs folder
• Refactored HER which is now the HerReplayBuffer class that can be passed to any off-policy algorithm
• Handle timeout termination properly for off-policy algorithms (when using TimeLimit)
• Renamed _last_dones and dones to _last_episode_starts and episode_starts in RolloutBuffer.
• Removed ObsDictWrapper as Dict observation spaces are now supported

her_kwargs = dict(n_sampled_goal=2, goal_selection_strategy="future", online_sampling=True)
# SB3 < 1.1.0
# model = HER("MlpPolicy", env, model_class=SAC, **her_kwargs)
# SB3 >= 1.1.0:
model = SAC("MultiInputPolicy", env, replay_buffer_class=HerReplayBuffer, replay_buffer_kwargs=her_kwargs)

• Updated the KL Divergence estimator in the PPO algorithm to be positive definite and have lower variance (@09tangriro)
• Updated the KL Divergence check in the PPO algorithm to be before the gradient update step rather than after end of epoch (@09tangriro)
• Removed parameter channels_last from is_image_space as it can be inferred.
• The logger object is now an attribute model.logger that be set by the user using model.set_logger()
• Changed the signature of logger.configure and utils.configure_logger, they now return a Logger object
• Removed Logger.CURRENT and Logger.DEFAULT
• Moved warn(), debug(), log(), info(), dump() methods to the Logger class
• .learn() now throws an import error when the user tries to log to tensorboard but the package is not installed

New Features: Copy Section

• Added support for single-level Dict observation space (@JadenTravnik)
• Added DictRolloutBuffer DictReplayBuffer to support dictionary observations (@JadenTravnik)
• Added StackedObservations and StackedDictObservations that are used within VecFrameStack
• Added simple 4x4 room Dict test environments
• HerReplayBuffer now supports VecNormalize when online_sampling=False
• Added VecMonitor and
  VecExtractDictObs wrappers
  to handle gym3-style vectorized environments (@vwxyzjn)
• Ignored the terminal observation if the it is not provided by the environment
  such as the gym3-style vectorized environments. (@vwxyzjn)
• Added policy_base as input to the OnPolicyAlgorithm for more flexibility (@09tangriro)
• Added support for image observation when using HER
• Added replay_buffer_class and replay_buffer_kwargs arguments to off-policy algorithms
• Added kl_divergence helper for Distribution classes (@09tangriro)
• Added support for vector environments with num_envs > 1 (@benblack769)
• Added wrapper_kwargs argument to make_vec_env (@amy12xx)

Bug Fixes: Copy Section

• Fixed potential issue when calling off-policy algorithms with default arguments multiple times (the size of the replay buffer would be the same)
• Fixed loading of ent_coef for SAC and TQC, it was not optimized anymore (thanks @Atlis)
• Fixed saving of A2C and PPO policy when using gSDE (thanks @liusida)
• Fixed a bug where no output would be shown even if verbose>=1 after passing verbose=0 once
• Fixed observation buffers dtype in DictReplayBuffer (@c-rizz)
• Fixed EvalCallback tensorboard logs being logged with the incorrect timestep. They are now written with the timestep at which they were recorded. (@skandermoalla)

Deprecations: Copy Section

Others: Copy Section

• Added flake8-bugbear to tests dependencies to find likely bugs
• Updated env_checker to reflect support of dict observation spaces
• Added Code of Conduct
• Added tests for GAE and lambda return computation
• Updated distribution entropy test (thanks @09tangriro)
• Added sanity check batch_size > 1 in PPO to avoid NaN in advantage normalization

Documentation: Copy Section

• Added gym pybullet drones project (@JacopoPan)
• Added link to SuperSuit in projects (@justinkterry)
• Fixed DQN example (thanks @ltbd78)
• Clarified channel-first/channel-last recommendation
• Update sphinx environment installation instructions (@tom-doerr)
• Clarified pip installation in Zsh (@tom-doerr)
• Clarified return computation for on-policy algorithms (TD(lambda) estimate was used)
• Added example for using ProcgenEnv
• Added note about advanced custom policy example for off-policy algorithms
• Fixed DQN unicode checkmarks
• Updated migration guide (@juancroldan)
• Pinned docutils==0.16 to avoid issue with rtd theme
• Clarified callback save_freq definition
• Added doc on how to pass a custom logger
• Remove recurrent policies from A2C docs (@bstee615)

Release 1.0 (2021-03-15) Copy Chapter

First Major Version

Breaking Changes: Copy Section

• Removed stable_baselines3.common.cmd_util (already deprecated), please use env_util instead

:::{warning}
A refactoring of the HER algorithm is planned together with support for dictionary observations
(see PR #243 and #351)
This will be a backward incompatible change (model trained with previous version of HER won't work with the new version).
:::

New Features: Copy Section

• Added support for custom_objects when loading models

Bug Fixes: Copy Section

• Fixed a bug with DQN predict method when using deterministic=False with image space

Documentation: Copy Section

• Fixed examples
• Added new project using SB3: rl_reach (@PierreExeter)
• Added note about slow-down when switching to PyTorch
• Add a note on continual learning and resetting environment

Others: Copy Section

• Updated RL-Zoo to reflect the fact that is it more than a collection of trained agents
• Added images to illustrate the training loop and custom policies (created with https://excalidraw.com/)
• Updated the custom policy section

Pre-Release 0.11.1 (2021-02-27) Copy Chapter

Bug Fixes: Copy Section

• Fixed a bug where train_freq was not properly converted when loading a saved model

Pre-Release 0.11.0 (2021-02-27) Copy Chapter

Breaking Changes: Copy Section

• evaluate_policy now returns rewards/episode lengths from a Monitor wrapper if one is present,
  this allows to return the unnormalized reward in the case of Atari games for instance.
• Renamed common.vec_env.is_wrapped to common.vec_env.is_vecenv_wrapped to avoid confusion
  with the new is_wrapped() helper
• Renamed _get_data() to _get_constructor_parameters() for policies (this affects independent saving/loading of policies)
• Removed n_episodes_rollout and merged it with train_freq, which now accepts a tuple (frequency, unit):
• replay_buffer in collect_rollout is no more optional

# SB3 < 0.11.0
# model = SAC("MlpPolicy", env, n_episodes_rollout=1, train_freq=-1)
# SB3 >= 0.11.0:
model = SAC("MlpPolicy", env, train_freq=(1, "episode"))

New Features: Copy Section

• Add support for VecFrameStack to stack on first or last observation dimension, along with
  automatic check for image spaces.
• VecFrameStack now has a channels_order argument to tell if observations should be stacked
  on the first or last observation dimension (originally always stacked on last).
• Added common.env_util.is_wrapped and common.env_util.unwrap_wrapper functions for checking/unwrapping
  an environment for specific wrapper.
• Added env_is_wrapped() method for VecEnv to check if its environments are wrapped
  with given Gym wrappers.
• Added monitor_kwargs parameter to make_vec_env and make_atari_env
• Wrap the environments automatically with a Monitor wrapper when possible.
• EvalCallback now logs the success rate when available (is_success must be present in the info dict)
• Added new wrappers to log images and matplotlib figures to tensorboard. (@zampanteymedio)
• Add support for text records to Logger. (@lorenz-h)

Bug Fixes: Copy Section

• Fixed bug where code added VecTranspose on channel-first image environments (thanks @qxcv)
• Fixed DQN predict method when using single gym.Env with deterministic=False
• Fixed bug that the arguments order of explained_variance() in ppo.py and a2c.py is not correct (@thisray)
• Fixed bug where full HerReplayBuffer leads to an index error. (@megan-klaiber)
• Fixed bug where replay buffer could not be saved if it was too big (> 4 Gb) for python<3.8 (thanks @hn2)
• Added informative PPO construction error in edge-case scenario where n_steps * n_envs = 1 (size of rollout buffer),
  which otherwise causes downstream breaking errors in training (@decodyng)
• Fixed discrete observation space support when using multiple envs with A2C/PPO (thanks @ardabbour)
• Fixed a bug for TD3 delayed update (the update was off-by-one and not delayed when train_freq=1)
• Fixed numpy warning (replaced np.bool with bool)
• Fixed a bug where VecNormalize was not normalizing the terminal observation
• Fixed a bug where VecTranspose was not transposing the terminal observation
• Fixed a bug where the terminal observation stored in the replay buffer was not the right one for off-policy algorithms
• Fixed a bug where action_noise was not used when using HER (thanks @ShangqunYu)

Deprecations: Copy Section

Others: Copy Section

• Add more issue templates
• Add signatures to callable type annotations (@ernestum)
• Improve error message in NatureCNN
• Added checks for supported action spaces to improve clarity of error messages for the user
• Renamed variables in the train() method of SAC, TD3 and DQN to match SB3-Contrib.
• Updated docker base image to Ubuntu 18.04
• Set tensorboard min version to 2.2.0 (earlier version are apparently not working with PyTorch)
• Added warning for PPO when n_steps * n_envs is not a multiple of batch_size (last mini-batch truncated) (@decodyng)
• Removed some warnings in the tests

Documentation: Copy Section

• Updated algorithm table
• Minor docstring improvements regarding rollout (@stheid)
• Fix migration doc for A2C (epsilon parameter)
• Fix clip_range docstring
• Fix duplicated parameter in EvalCallback docstring (thanks @tfederico)
• Added example of learning rate schedule
• Added SUMO-RL as example project (@LucasAlegre)
• Fix docstring of classes in atari_wrappers.py which were inside the constructor (@LucasAlegre)
• Added SB3-Contrib page
• Fix bug in the example code of DQN (@AptX395)
• Add example on how to access the tensorboard summary writer directly. (@lorenz-h)
• Updated migration guide
• Updated custom policy doc (separate policy architecture recommended)
• Added a note about OpenCV headless version
• Corrected typo on documentation (@mschweizer)
• Provide the environment when loading the model in the examples (@lorepieri8)

Pre-Release 0.10.0 (2020-10-28) Copy Chapter

HER with online and offline sampling, bug fixes for features extraction

Breaking Changes: Copy Section

• Warning: Renamed common.cmd_util to common.env_util for clarity (affects make_vec_env and make_atari_env functions)

New Features: Copy Section

• Allow custom actor/critic network architectures using net_arch=dict(qf=[400, 300], pi=[64, 64]) for off-policy algorithms (SAC, TD3, DDPG)
• Added Hindsight Experience Replay HER. (@megan-klaiber)
• VecNormalize now supports gym.spaces.Dict observation spaces
• Support logging videos to Tensorboard (@SwamyDev)
• Added share_features_extractor argument to SAC and TD3 policies

Bug Fixes: Copy Section

• Fix GAE computation for on-policy algorithms (off-by one for the last value) (thanks @Wovchena)
• Fixed potential issue when loading a different environment
• Fix ignoring the exclude parameter when recording logs using json, csv or log as logging format (@SwamyDev)
• Make make_vec_env support the env_kwargs argument when using an env ID str (@ManifoldFR)
• Fix model creation initializing CUDA even when device="cpu" is provided
• Fix check_env not checking if the env has a Dict actionspace before calling _check_nan (@wmmc88)
• Update the check for spaces unsupported by Stable Baselines 3 to include checks on the action space (@wmmc88)
• Fixed features extractor bug for target network where the same net was shared instead
  of being separate. This bug affects SAC, DDPG and TD3 when using CnnPolicy (or custom features extractor)
• Fixed a bug when passing an environment when loading a saved model with a CnnPolicy, the passed env was not wrapped properly
  (the bug was introduced when implementing HER so it should not be present in previous versions)

Deprecations: Copy Section

Others: Copy Section

• Improved typing coverage
• Improved error messages for unsupported spaces
• Added .vscode to the gitignore

Documentation: Copy Section

• Added first draft of migration guide
• Added intro to imitation library (@shwang)
• Enabled doc for CnnPolicies
• Added advanced saving and loading example
• Added base doc for exporting models
• Added example for getting and setting model parameters

Pre-Release 0.9.0 (2020-10-03) Copy Chapter

Bug fixes, get/set parameters and improved docs

Breaking Changes: Copy Section

• Removed device keyword argument of policies; use policy.to(device) instead. (@qxcv)
• Rename BaseClass.get_torch_variables -> BaseClass._get_torch_save_params and BaseClass.excluded_save_params -> BaseClass._excluded_save_params
• Renamed saved items tensors to pytorch_variables for clarity
• make_atari_env, make_vec_env and set_random_seed must be imported with (and not directly from stable_baselines3.common):

from stable_baselines3.common.cmd_util import make_atari_env, make_vec_env
from stable_baselines3.common.utils import set_random_seed

New Features: Copy Section

• Added unwrap_vec_wrapper() to common.vec_env to extract VecEnvWrapper if needed
• Added StopTrainingOnMaxEpisodes to callback collection (@xicocaio)
• Added device keyword argument to BaseAlgorithm.load() (@liorcohen5)
• Callbacks have access to rollout collection locals as in SB2. (@PartiallyTyped)
• Added get_parameters and set_parameters for accessing/setting parameters of the agent
• Added actor/critic loss logging for TD3. (@mloo3)

Bug Fixes: Copy Section

• Added unwrap_vec_wrapper() to common.vec_env to extract VecEnvWrapper if needed
• Fixed a bug where the environment was reset twice when using evaluate_policy
• Fix logging of clip_fraction in PPO (@diditforlulz273)
• Fixed a bug where cuda support was wrongly checked when passing the GPU index, e.g., device="cuda:0" (@liorcohen5)
• Fixed a bug when the random seed was not properly set on cuda when passing the GPU index

Deprecations: Copy Section

Others: Copy Section

• Improve typing coverage of the VecEnv
• Fix type annotation of make_vec_env (@ManifoldFR)
• Removed AlreadySteppingError and NotSteppingError that were not used
• Fixed typos in SAC and TD3
• Reorganized functions for clarity in BaseClass (save/load functions close to each other, private
  functions at top)
• Clarified docstrings on what is saved and loaded to/from files
• Simplified save_to_zip_file function by removing duplicate code
• Store library version along with the saved models
• DQN loss is now logged

Documentation: Copy Section

• Added StopTrainingOnMaxEpisodes details and example (@xicocaio)
• Updated custom policy section (added custom features extractor example)
• Re-enable sphinx_autodoc_typehints
• Updated doc style for type hints and remove duplicated type hints

Pre-Release 0.8.0 (2020-08-03) Copy Chapter

DQN, DDPG, bug fixes and performance matching for Atari games

Breaking Changes: Copy Section

• AtariWrapper and other Atari wrappers were updated to match SB2 ones
• save_replay_buffer now receives as argument the file path instead of the folder path (@tirafesi)
• Refactored Critic class for TD3 and SAC, it is now called ContinuousCritic
  and has an additional parameter n_critics
• SAC and TD3 now accept an arbitrary number of critics (e.g. policy_kwargs=dict(n_critics=3))
  instead of only 2 previously

New Features: Copy Section

• Added DQN Algorithm (@Artemis-Skade)
• Buffer dtype is now set according to action and observation spaces for ReplayBuffer
• Added warning when allocation of a buffer may exceed the available memory of the system
  when psutil is available
• Saving models now automatically creates the necessary folders and raises appropriate warnings (@PartiallyTyped)
• Refactored opening paths for saving and loading to use strings, pathlib or io.BufferedIOBase (@PartiallyTyped)
• Added DDPG algorithm as a special case of TD3.
• Introduced BaseModel abstract parent for BasePolicy, which critics inherit from.

Bug Fixes: Copy Section

• Fixed a bug in the close() method of SubprocVecEnv, causing wrappers further down in the wrapper stack to not be closed. (@NeoExtended)
• Fix target for updating q values in SAC: the entropy term was not conditioned by terminals states
• Use cloudpickle.load instead of pickle.load in CloudpickleWrapper. (@shwang)
• Fixed a bug with orthogonal initialization when bias=False in custom policy (@rk37)
• Fixed approximate entropy calculation in PPO and A2C. (@andyshih12)
• Fixed DQN target network sharing features extractor with the main network.
• Fixed storing correct dones in on-policy algorithm rollout collection. (@andyshih12)
• Fixed number of filters in final convolutional layer in NatureCNN to match original implementation.

Deprecations: Copy Section

Others: Copy Section