tianshou Documentation

File: README.md Copy Chapter

[!NOTE]
Tianshou version 2 is here!

We have released the new major version of Tianshou on PyPI.
Version 2 is a complete overhaul of the software design of the procedural API, in which

• we establish a clear separation between learning algorithms and policies (via the separate abstractions Algorithm and Policy).
• we provide more well-defined, more usable interfaces with extensive documentation of all algorithm and trainer parameters,
  renaming some parameters to make their names more consistent and intuitive.
• the class hierarchy is fully revised, establishing a clear separation between on-policy, off-policy and offline algorithms
  at the type level and ensuring that all inheritance relationships are meaningful.

Because of the extent of the changes, this version is not backwards compatible with previous versions of Tianshou.
For migration information, please see the change log.

Tianshou (天授) is a reinforcement learning (RL) library based on pure PyTorch and Gymnasium. Tianshou's main features at a glance are:

1. Modular low-level interfaces for algorithm developers (RL researchers) that are both flexible, hackable and type-safe.
2. Convenient high-level interfaces for applications of RL (training an implemented algorithm on a custom environment).
3. Large scope: online (on- and off-policy) and offline RL, experimental support for multi-agent RL (MARL), experimental support for model-based RL, and more

Unlike other reinforcement learning libraries, which may have complex codebases,
unfriendly high-level APIs, or are not optimized for speed, Tianshou provides a high-performance, modularized framework
and user-friendly interfaces for building deep reinforcement learning agents. One more aspect that sets Tianshou apart is its
generality: it supports online and offline RL, multi-agent RL, and model-based algorithms.

Tianshou aims at enabling concise implementations, both for researchers and practitioners, without sacrificing flexibility.

Supported algorithms include:

• Deep Q-Network (DQN)
• Double DQN
• Dueling DQN
• Branching DQN
• Categorical DQN (C51)
• Rainbow DQN (Rainbow)
• Quantile Regression DQN (QRDQN)
• Implicit Quantile Network (IQN)
• Fully-parameterized Quantile Function (FQF)
• Policy Gradient (PG)
• Natural Policy Gradient (NPG)
• Advantage Actor-Critic (A2C)
• Trust Region Policy Optimization (TRPO)
• Proximal Policy Optimization (PPO)
• Deep Deterministic Policy Gradient (DDPG)
• Twin Delayed DDPG (TD3)
• Soft Actor-Critic (SAC)
• Randomized Ensembled Double Q-Learning (REDQ)
• Discrete Soft Actor-Critic (SAC-Discrete)
• Vanilla Imitation Learning
• Batch-Constrained deep Q-Learning (BCQ)
• Conservative Q-Learning (CQL)
• Twin Delayed DDPG with Behavior Cloning (TD3+BC)
• Discrete Batch-Constrained deep Q-Learning (BCQ-Discrete)
• Discrete Conservative Q-Learning (CQL-Discrete)
• Discrete Critic Regularized Regression (CRR-Discrete)
• Generative Adversarial Imitation Learning (GAIL)
• Prioritized Experience Replay (PER)
• Generalized Advantage Estimator (GAE)
• Posterior Sampling Reinforcement Learning (PSRL)
• Intrinsic Curiosity Module (ICM)
• Hindsight Experience Replay (HER)

Other noteworthy features:

• Elegant framework with dual APIs:
  • Tianshou's high-level API maximizes ease of use for application development while still retaining a high degree
    of flexibility.
  • The fundamental procedural API provides a maximum of flexibility for algorithm development without being
    overly verbose.
• State-of-the-art results in MuJoCo benchmarks for REINFORCE/A2C/TRPO/PPO/DDPG/TD3/SAC algorithms
• Support for vectorized environments (synchronous or asynchronous) for all algorithms (see usage)
• Support for super-fast vectorized environments based on EnvPool for all algorithms (see usage)
• Support for recurrent state representations in actor networks and critic networks (RNN-style training for POMDPs) (see usage)
• Support any type of environment state/action (e.g. a dict, a self-defined class, ...) Usage
• Support for customized training processes (see usage)
• Support n-step returns estimation and prioritized experience replay for all Q-learning based algorithms; GAE, nstep and PER are highly optimized thanks to numba's just-in-time compilation and vectorized numpy operations
• Support for multi-agent RL (see usage)
• Support for logging based on both TensorBoard and W&B
• Support for multi-GPU training (see usage)
• Comprehensive documentation, PEP8 code-style checking, type checking and thorough tests

In Chinese, Tianshou means divinely ordained, being derived to the gift of being born.
Tianshou is a reinforcement learning platform, and the nature of RL is not learn from humans.
So taking "Tianshou" means that there is no teacher to learn from, but rather to learn by oneself through constant interaction with the environment.

“天授”意指上天所授，引申为与生具有的天赋。天授是强化学习平台，而强化学习算法并不是向人类学习的，所以取“天授”意思是没有老师来教，而是自己通过跟环境不断交互来进行学习。

Installation Copy Chapter

Tianshou is currently hosted on PyPI and conda-forge. It requires Python >= 3.11.

For installing the most recent version of Tianshou, the best way is clone the repository and install it with poetry
(which you need to install on your system first)

git clone [email protected]:thu-ml/tianshou.git
cd tianshou
poetry install

You can also install the dev requirements by adding --with dev or the extras
for say mujoco and acceleration by envpool
by adding --extras "mujoco envpool"

If you wish to install multiple extras, ensure that you include them in a single command. Sequential calls to poetry install --extras xxx will overwrite prior installations, leaving only the last specified extras installed.
Or you may install all the following extras by adding --all-extras.

Available extras are:

• atari (for Atari environments)
• box2d (for Box2D environments)
• classic_control (for classic control (discrete) environments)
• mujoco (for MuJoCo environments)
• mujoco-py (for legacy mujoco-py environments[^1])
• pybullet (for pybullet environments)
• robotics (for gymnasium-robotics environments)
• vizdoom (for ViZDoom environments)
• envpool (for envpool integration)
• argparse (in order to be able to run the high level API examples)

[^1]:
mujoco-py is a legacy package and is not recommended for new projects.
It is only included for compatibility with older projects.
Also note that there may be compatibility issues with macOS newer than
Monterey.

Otherwise, you can install the latest release from PyPI (currently
far behind the master) with the following command:

$ pip install tianshou

If you are using Anaconda or Miniconda, you can install Tianshou from conda-forge:

$ conda install tianshou -c conda-forge

Alternatively to the poetry install, you can also install the latest source version through GitHub:

$ pip install git+https://github.com/thu-ml/tianshou.git@master --upgrade

Finally, you may check the installation via your Python console as follows:

import tianshou
print(tianshou.__version__)

If no errors are reported, you have successfully installed Tianshou.

Documentation Copy Chapter

Find example scripts in the test/ and examples/ folders.

Tutorials and API documentation are hosted on tianshou.readthedocs.io.

Why Tianshou? Copy Chapter

Comprehensive Functionality Copy Section

High Software Engineering Standards Copy Section

^{(1): it has continuous integration but the coverage rate is not available}

Reproducible, High-Quality Results Copy Section

Tianshou is rigorously tested. In contrast to other RL platforms, our tests include the full agent training procedure for all of the implemented algorithms. Our tests would fail once if any of the agents failed to achieve a consistent level of performance on limited epochs.
Our tests thus ensure reproducibility.
Check out the GitHub Actions page for more detail.

Atari and MuJoCo benchmark results can be found in the examples/atari/ and examples/mujoco/ folders respectively. Our MuJoCo results reach or exceed the level of performance of most existing benchmarks.

Algorithm Abstraction Copy Section

Reinforcement learning algorithms are build on abstractions for

• on-policy algorithms (OnPolicyAlgorithm),
• off-policy algorithms (OffPolicyAlgorithm), and
• offline algorithms (OfflineAlgorithm),

all of which clearly separate the core algorithm from the training process and the respective environment interactions.

In each case, the implementation of an algorithm necessarily involves only the implementation of methods for

• pre-processing a batch of data, augmenting it with necessary information/sufficient statistics for learning (_preprocess_batch),
• updating model parameters based on an augmented batch of data (_update_with_batch).

The implementation of these methods suffices for a new algorithm to be applicable within Tianshou,
making experimentation with new approaches particularly straightforward.

Quick Start Copy Chapter

Tianshou provides two API levels:

• the high-level interface, which provides ease of use for end users seeking to run deep reinforcement learning applications
• the procedural interface, which provides a maximum of control, especially for very advanced users and developers of reinforcement learning algorithms.

In the following, let us consider an example application using the CartPole gymnasium environment.
We shall apply the deep Q-network (DQN) learning algorithm using both APIs.

High-Level API Copy Section

In the high-level API, the basis for an RL experiment is an ExperimentBuilder
with which we can build the experiment we then seek to run.
Since we want to use DQN, we use the specialization DQNExperimentBuilder.

The high-level API provides largely declarative semantics, i.e. the code is
almost exclusively concerned with configuration that controls what to do
(rather than how to do it).

from tianshou.highlevel.config import OffPolicyTrainingConfig
from tianshou.highlevel.env import (
    EnvFactoryRegistered,
    VectorEnvType,
)
from tianshou.highlevel.experiment import DQNExperimentBuilder, ExperimentConfig
from tianshou.highlevel.params.algorithm_params import DQNParams
from tianshou.highlevel.trainer import (
    EpochStopCallbackRewardThreshold,
)

experiment = (
    DQNExperimentBuilder(
        EnvFactoryRegistered(
            task="CartPole-v1",
            venv_type=VectorEnvType.DUMMY,
            training_seed=0,
            test_seed=10,
        ),
        ExperimentConfig(
            persistence_enabled=False,
            watch=True,
            watch_render=1 / 35,
            watch_num_episodes=100,
        ),
        OffPolicyTrainingConfig(
            max_epochs=10,
            epoch_num_steps=10000,
            batch_size=64,
            num_training_envs=10,
            num_test_envs=100,
            buffer_size=20000,
            collection_step_num_env_steps=10,
            update_step_num_gradient_steps_per_sample=1 / 10,
        ),
    )
    .with_dqn_params(
        DQNParams(
            lr=1e-3,
            gamma=0.9,
            n_step_return_horizon=3,
            target_update_freq=320,
            eps_training=0.3,
            eps_inference=0.0,
        ),
    )
    .with_model_factory_default(hidden_sizes=(64, 64))
    .with_epoch_stop_callback(EpochStopCallbackRewardThreshold(195))
    .build()
)
experiment.run()

The experiment builder takes three arguments:

• the environment factory for the creation of environments. In this case,
  we use an existing factory implementation for gymnasium environments.
• the experiment configuration, which controls persistence and the overall
  experiment flow. In this case, we have configured that we want to observe
  the agent's behavior after it is trained (watch=True) for a number of
  episodes (watch_num_episodes=100). We have disabled persistence, because
  we do not want to save training logs, the agent or its configuration for
  future use.
• the training configuration, which controls fundamental training parameters,
  such as the total number of epochs we run the experiment for (num_epochs=10)
  and the number of environment steps each epoch shall consist of
  (epoch_num_steps=10000).
  Every epoch consists of a series of data collection (rollout) steps and
  training steps.
  The parameter collection_step_num_env_steps controls the amount of data that is
  collected in each collection step and after each collection step, we
  perform a training step, applying a gradient-based update based on a sample
  of data (batch_size=64) taken from the buffer of data that has been
  collected. For further details, see the documentation of configuration class.

We then proceed to configure some of the parameters of the DQN algorithm itself:
For instance, we control the epsilon parameter for exploration.
We want to use random exploration during rollouts for training (eps_training),
but we don't when evaluating the agent's performance in the test environments
(eps_inference).
Furthermore, we configure model parameters of the network for the Q function,
parametrising the number of hidden layers of the default MLP factory.

Find the script in examples/discrete/discrete_dqn_hl.py.
Here's a run (with the training time cut short):

Find many further applications of the high-level API in the examples/ folder;
look for scripts ending with _hl.py.
Note that most of these examples require the extra argparse
(install it by adding --extras argparse when invoking poetry).

Procedural API Copy Section

Let us now consider an analogous example in the procedural API.
Find the full script in examples/discrete/discrete_dqn.py.

First, import the relevant packages:

import gymnasium as gym
import tianshou as ts
from tianshou.algorithm.modelfree.dqn import DiscreteQLearningPolicy
from tianshou.algorithm.optim import AdamOptimizerFactory
from tianshou.data import CollectStats
from tianshou.trainer import OffPolicyTrainerParams
from tianshou.utils.net.common import Net
from tianshou.utils.space_info import SpaceInfo
from torch.utils.tensorboard import SummaryWriter

Define hyper-parameters:

task = 'CartPole-v1'
lr, epoch, batch_size = 1e-3, 10, 64
num_training_envs, num_test_envs = 10, 100
gamma, n_step, target_freq = 0.9, 3, 320
buffer_size = 20000
eps_train, eps_test = 0.1, 0.05
epoch_num_steps, collection_step_num_env_steps = 10000, 10

Initialize the logger:

logger = ts.utils.TensorboardLogger(SummaryWriter('log/dqn'))

Create the environments:

# You can also try SubprocVectorEnv, which will use parallelization
training_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_training_envs)])
test_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_test_envs)])

Create the network, policy, and algorithm:

# Create the network
# Note: You can easily define other networks.
# See https://tianshou.readthedocs.io/en/master/01_tutorials/00_dqn.html#build-the-network
env = gym.make(task, render_mode="human")
assert isinstance(env.action_space, gym.spaces.Discrete)
space_info = SpaceInfo.from_env(env)
state_shape = space_info.observation_info.obs_shape
action_shape = space_info.action_info.action_shape
net = Net(state_shape=state_shape, action_shape=action_shape, hidden_sizes=[128, 128, 128])
optim = AdamOptimizerFactory(lr=lr)

# Create the policy
policy = DiscreteQLearningPolicy(
    model=net,
    action_space=env.action_space,
    eps_training=eps_train,
    eps_inference=eps_test
)

# Create the algorithm with the policy and optimizer factory
algorithm = DQN(
    policy=policy,
    optim=AdamOptimizerFactory(lr=lr),
    gamma=gamma,
    n_step_return_horizon=n_step,
    target_update_freq=target_freq
)

Set up the collectors:

training_collector = ts.data.Collector[CollectStats](
  algorithm,
  training_envs,
  ts.data.VectorReplayBuffer(buffer_size, num_training_envs),
  exploration_noise=True,
)
test_collector = ts.data.Collector[CollectStats](
  algorithm,
  test_envs,
  exploration_noise=True,
)

Let's train the model using the algorithm:

result = algorithm.run_training(
  OffPolicyTrainerParams(
    training_collector=training_collector,
    test_collector=test_collector,
    max_epochs=epoch,
    epoch_num_steps=epoch_num_steps,
    collection_step_num_env_steps=collection_step_num_env_steps,
    test_step_num_episodes=num_test_envs,
    batch_size=batch_size,
    update_step_num_gradient_steps_per_sample=1 / collection_step_num_env_steps,
    stop_fn=lambda mean_rewards: mean_rewards >= env.spec.reward_threshold,
    logger=logger,
    test_in_training=True,
  )
)
print(f"Finished training in {result.timing.total_time} seconds")

This is how you could manually save/load the trained policy (it's exactly the same as loading a torch.nn.module):

torch.save(policy.state_dict(), 'dqn.pth')
policy.load_state_dict(torch.load('dqn.pth'))

Now let's watch the agent with 35 FPS:

collector = ts.data.Collector(policy, env, exploration_noise=True)
collector.collect(n_episode=1, render=1 / 35)

Inspect the data saved in TensorBoard:

$ tensorboard --logdir log/dqn

Please read the documentation for advanced usage.

Contributing Copy Chapter

Tianshou is still under development.
Further algorithms and features are continuously being added, and we always welcome contributions to help make Tianshou better.
If you would like to contribute, please check out this link.

Citing Tianshou Copy Chapter

If you find Tianshou useful, please cite it in your publications.

@article{tianshou,
  author  = {Jiayi Weng and Huayu Chen and Dong Yan and Kaichao You and Alexis Duburcq and Minghao Zhang and Yi Su and Hang Su and Jun Zhu},
  title   = {Tianshou: A Highly Modularized Deep Reinforcement Learning Library},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {267},
  pages   = {1--6},
  url     = {http://jmlr.org/papers/v23/21-1127.html}
}

Acknowledgments Copy Chapter

Tianshou is supported by appliedAI Institute for Europe,
a non-profit organization committed to providing long-term support and development.

Tianshou was previously a reinforcement learning platform based on TensorFlow. You can check out the branch priv for more detail. Many thanks to Haosheng Zou's pioneering work for Tianshou before version 0.1.1.

We would like to thank TSAIL and Institute for Artificial Intelligence, Tsinghua University for providing such an excellent AI research platform.

File: CHANGELOG.md Copy Chapter

Release 2.0.1 (2026-04-02)

This is a maintenance release.

Dependencies:

• pandas compatibility was adjusted to require version < 3 (incompatibility detected) #1289

Fixes:

• algorithm:
  • TRPO: Fix original policy parameters not being restored upon line search failure #1287
• highlevel.experiment:
  • Fix Experiment.run returning ExperimentResult with trainer_result unset (None) #1288

Release 2.0.0 (2025-12-01)

This major release of Tianshou is a big step towards cleaner design and improved usability.

Given the large extent of the changes, it was not possible to maintain compatibility with the previous version.

• Persisted agents that were created with earlier versions cannot be loaded in v2.
• Source code from v1 can, however, be migrated to v2 with minimal effort.
  See migration information below. For concrete examples, you may use git to diff individual
  example scripts with the corresponding ones in v1.2.0.

This release is brought to you by Applied AI Institute gGmbH.

Developers:

• Dr. Dominik Jain (@opcode81)
• Michael Panchenko (@MischaPanch)

Runtime Environment Compatibility Copy Chapter

Tianshou v2 is now compatible with

• Python 3.12 and Python 3.13 #1274
• newer versions of gymnasium (v1+) and numpy (v2+)

Our main test environment remains Python 3.11-based for the time being (see poetry.lock).

Trainer Abstraction Copy Chapter

• The trainer logic and configuration is now properly separated between the three cases of on-policy, off-policy
  and offline learning: The base class is no longer a "God" class (formerly BaseTrainer) which does it all; logic and functionality has moved
  to the respective subclasses (OnPolicyTrainer, OffPolicyTrainer and OfflineTrainer, with OnlineTrainer
  being introduced as a base class for the two former specialisations).

• The trainers now use configuration objects with central documentation (which has been greatly improved to enhance
  clarity and usability in general); every type of trainer now has a dedicated configuration class which provides
  precisely the options that are applicable.

• The interface has been streamlined with improved naming of functions/parameters and limiting the public interface to purely
  the methods and attributes a user should reasonably access.

• Further changes potentially affecting usage:

  • We dropped the iterator semantics: Method __next__ has been replaced by execute_epoch. #913
  • We no longer report outdated statistics (e.g. on rewards/returns when a training step does not collect any full
    episodes)
  • See also "Issues resolved" below (as issue resolution can result in usage changes)
  • The default value for test_in_train was changed from True to False (updating all usage sites to explicitly
    set the parameter), because False is the more natural default, which does not make assumptions about
    returns/score values computed for the data from a collection step being at all meaningful for early stopping
  • The management of epsilon-greedy exploration for discrete Q-learning algorithms has been simplified:
    • All respective Policy implementations (e.g. DQNPolicy, C51Policy, etc.) now accept two parameters
      eps_training and eps_inference, which allows the training and test collection cases to be sufficiently
      differentiated and makes the use of callback functions (train_fn, test_fn) unnecessary if only
      constants are to be set.
    • The setter method set_eps has been replaced with set_eps_training and set_eps_inference accordingly.

• Further internal changes unlikely to affect usage:

  • Module trainer.utils was removed and the functions therein where moved to class Trainer
  • The two places that collected and evaluated test episodes (_test_in_train and _reset) in addition to
    _test_step were unified to use _test_step (with some minor parametrisation) and now log the results
    of the test step accordingly.

• Issues resolved:

  • Methods run and reset: Parameter reset_prior_to_run of run was never respected if it was set to False,
    because the implementation of __iter__ (now removed) would call reset regardless - and calling reset
    is indeed necessary, because it initializes the training. The parameter was removed and replaced by
    reset_collectors (such that run now replicates the parameters of reset).
  • Inconsistent configuration options now raise exceptions rather than silently ignoring the issue in the
    hope that default behaviour will achieve what the user intended.
    One condition where test_in_train was silently set to False was removed and replaced by a warning.
  • The stop criterion stop_fn did not consider scores as computed by compute_score_fn but instead always used
    mean returns (i.e. it was assumed that the default implementation of compute_score_fn applies).
    This is an inconsistency which has been resolved.
  • The gradient_step counter was flawed (as it made assumptions about the underlying algorithms, which were
    not valid). It has been replaced with an update step counter.
    Members of InfoStats and parameters of Logger (and subclasses) were changed accordingly.

• Migration information at a glance:

  • Training parameters are now passed via instances of configuration objects instead of directly as keyword arguments:
    OnPolicyTrainerParams, OffPolicyTrainerParams, OfflineTrainerParams.
    • Changed parameter default: Default for test_in_train was changed from True to False.
    • Changed parameter names to improve clarity:
      • max_epoch (num_epochs in high-level API) -> max_epochs
      • step_per_epoch -> epoch_num_steps
      • episode_per_test (num_test_episodes in high-level API) -> test_step_num_episodes
      • step_per_collect -> collection_step_num_env_steps
      • episode_per_collect -> collection_step_num_episodes`
      • update_per_step -> update_step_num_gradient_steps_per_sample
      • repeat_per_collect -> update_step_num_repetitions
  • Trainer classes have been renamed:
    • OnpolicyTrainer -> OnPolicyTrainer
    • OffpolicyTrainer -> OffPolicyTrainer
  • Method run: The parameter reset_prior_to_run was removed and replaced by reset_collectors (see above).
  • Methods run and reset: The parameter reset_buffer was renamed to reset_collector_buffers for clarity
  • Trainers are no longer iterators; manual usage (not using run) should simply call reset followed by
    calls of execute_epoch.

Algorithms and Policies Copy Chapter

• We now conceptually differentiate between the learning algorithm and the policy being optimised:

  • The abstraction BasePolicy is thus replaced by Algorithm and Policy, and the package was renamed
    from tianshou.policy to tianshou.algorithm.

  • Migration information: The instantiation of a policy is replaced by the instantiation of an Algorithm,
    which is passed a Policy. In most cases, the former policy class name <Name>Policy is replaced by algorithm
    class <Name>; exceptions are noted below.

    • ImitationPolicy -> OffPolicyImitationLearning, OfflineImitationLearning
    • PGPolicy -> Reinforce
    • MultiAgentPolicyManager -> MultiAgentOnPolicyAlgorithm, MultiAgentOffPolicyAlgorithm
    • MARLRandomPolicy -> MARLRandomDiscreteMaskedOffPolicyAlgorithm

    For the respective subtype of Policy to use, see the respective algorithm class' constructor.

• Interface changes/improvements:

  • Core methods have been renamed (and removed from the public interface; #898):
    • process_fn -> _preprocess_batch
    • post_process_fn -> _postprocess_batch
    • learn -> _update_with_batch
  • The updating interface has been cleaned up (#949):
    • Functions update and _update_with_batch (formerly learn) no longer have *args and **kwargs.
    • Instead, the interfaces for the offline, off-policy and on-policy cases are properly differentiated.
  • New method run_training: The Algorithm abstraction can now directly initiate the learning process via this method.
  • Algorithms no longer require torch.optim.Optimizer instances and instead require OptimizerFactory
    instances, which create the actual optimizers internally. #959
    The new OptimizerFactory abstraction simultaneously handles the creation of learning rate schedulers
    for the optimizers created (via method with_lr_scheduler_factory and accompanying factory abstraction
    LRSchedulerFactory).
    The parameter lr_scheduler has thus been removed from all algorithm constructors.
  • The flag updating has been removed (no internal usage, general usefulness questionable).
  • Removed max_action_num, instead read it off from action_space
  • Parameter changes:
    • actor_step_size -> trust_region_size in NP
    • discount_factor -> gamma (was already used internally almost everywhere)
    • reward_normalization -> return_standardization or return_scaling (more precise naming) or removed (was actually unsupported by Q-learning algorithms)
      • return_standardization in Reinforce and DiscreteCRR (as it applies standardization of returns)
      • return_scaling in actor-critic on-policy algorithms (A2C, PPO, GAIL, NPG, TRPO)
      • removed from Q-learning algorithms, where it was actually unsupported (DQN, C561, etc.)
    • clip_grad -> max_grad_norm (for consistency)
    • clip_loss_grad -> huber_loss_delta (allowing to control not only the use of the Huber loss but also its essential parameter)
    • estimation_step -> n_step_return_horizon (more precise naming)

• Internal design improvements:

  • Introduced an abstraction for the alpha parameter (coefficient of the entropy term)
    in SAC, DiscreteSAC and other algorithms.

    • Class hierarchy:
      • Abstract base class Alpha base class with value property and update method
      • FixedAlpha for constant entropy coefficients
      • AutoAlpha for automatic entropy tuning (replaces the old tuple-based representation)
    • The (auto-)updating logic is now completely encapsulated, reducing the complexity of the algorithms.
    • Implementations for continuous and discrete cases now share the same abstraction,
      making the codebase more consistent while preserving the original functionality.

  • Introduced a policy base class ContinuousPolicyWithExplorationNoise which encapsulates noise generation
    for continuous action spaces (e.g. relevant to DDPG, SAC and REDQ).

  • Multi-agent RL methods are now differentiated by the type of the sub-algorithms being employed
    (MultiAgentOnPolicyAlgorithm, MultiAgentOffPolicyAlgorithm), which renders all interfaces clean.
    Helper class MARLDispatcher has been factored out to manage the dispatching of data to the respective agents.

  • Algorithms now internally use a wrapper (Algorithm.Optimizer) around the optimizers; creation is handled
    by method _create_optimizer.

    • This facilitates backpropagation steps with gradient clipping.
    • The optimizers of an Algorithm instance are now centrally tracked, such that we can ensure that the
      optimizers' states are handled alongside the model parameters when calling state_dict or load_state_dict
      on the Algorithm instance.
      Special handling of the restoration of optimizers' state dicts was thus removed from examples and tests.

  • Lagged networks (target networks) are now conveniently handled via the new algorithm mixins
    LaggedNetworkPolyakUpdateAlgorithmMixin and LaggedNetworkFullUpdateAlgorithmMixin.
    Using these mixins,

    • a lagged network can simply be added by calling _add_lagged_network
    • the torch method train must no longer be overridden to ensure that the target networks
      are never set to train mode/remain in eval mode (which was prone to errors),
    • a method which updates all target networks with their source networks is automatically
      provided and does not need to be implemented specifically for every algorithm
      (_update_lagged_network_weights).

    All classes which make use of lagged networks were updated to use these mixins, simplifying
    the implementations and reducing the potential for implementation errors.
    (In the BCQ implementation, the VAE network was not correctly handled, but due to the way
    in which examples were structured, it did not result in an error.)

• Fixed issues in the class hierarchy (particularly critical violations of the Liskov substitution principle):

  • Introduced base classes (to retain factorization without abusive inheritance):
    • ActorCriticOnPolicyAlgorithm
    • ActorCriticOffPolicyAlgorithm
    • ActorDualCriticsOffPolicyAlgorithm (extends ActorCriticOffPolicyAlgorithm)
    • QLearningOffPolicyAlgorithm
  • A2C: Inherit from ActorCriticOnPolicyAlgorithm instead of Reinforce
  • BDQN:
    • Inherit from QLearningOffPolicyAlgorithm instead of DQN
    • Remove parameter clip_loss_grad (unused; only passed on to former base class)
    • Remove parameter estimation_step, for which only one option was valid
  • C51:
    • Inherit from QLearningOffPolicyAlgorithm instead of DQN
    • Remove parameters clip_loss_grad and is_double (unused; only passed on to former base class)
  • CQL:
    • Inherit directly from OfflineAlgorithm instead of SAC (off-policy).
    • Remove parameter estimation_step (now n_step_return_horizon), which was not actually used (it was only passed it on to its
      superclass).
  • DiscreteBCQ:
    • Inherit directly from OfflineAlgorithm instead of DQN
    • Remove unused parameters clip_loss_grad and is_double, which were only passed on to
      former the base class but actually unused.
  • DiscreteCQL: Remove unused parameters clip_loss_grad and is_double, which were only passed on to
    base class QRDQN (and unused by it).
  • DiscreteCRR: Inherit directly from OfflineAlgorithm instead of Reinforce (on-policy)
  • FQF: Remove unused parameters clip_loss_grad and is_double, which were only passed on to
    base class QRDQN (and unused by it).
  • IQN: Remove unused parameters clip_loss_grad and is_double, which were only passed on to
    base class QRDQN (and unused by it).
  • NPG: Inherit from ActorCriticOnPolicyAlgorithm instead of A2C
  • QRDQN:
    • Inherit from QLearningOffPolicyAlgorithm instead of DQN
    • Remove parameters clip_loss_grad and is_double (unused; only passed on to former base class)
  • REDQ: Inherit from ActorCriticOffPolicyAlgorithm instead of DDPG
  • SAC: Inherit from ActorDualCriticsOffPolicyAlgorithm instead of DDPG
  • TD3: Inherit from ActorDualCriticsOffPolicyAlgorithm instead of DDPG

High-Level API Copy Chapter

• Detailed optimizer configuration (analogous to the procedural API) is now possible:

  • All optimizers can be configured in the respective algorithm-specific Params object by using
    OptimizerFactoryFactory instances as parameter values (e.g. optim, actor_optim, critic_optim, etc.).
  • Learning rate schedulers remain separate parameters and now use LRSchedulerFactoryFactory
    instances. The respective parameter names now use the suffix lr_scheduler instead of lr_scheduler_factory
    (as the precise nature need not be reflected in the name; brevity is preferable).

• SamplingConfig is replaced by TrainingConfig and subclasses differentiating off-policy and on-policy cases
  appropriately (OnPolicyTrainingConfig, OffPolicyTrainingConfig).

  • The test_in_train parameter is now exposed (default False).
  • Inapplicable arguments can no longer be set in the respective subclass (e.g. OffPolicyTrainingConfig does not
    contain parameter repeat_per_collect).
  • All parameter names have been aligned with the new names used by TrainerParams (see above).

• Add option to customize the factory for the collector (ExperimentBuilder.with_collector_factory),
  adding the abstraction CollectorFactory. #1256

Peripheral Changes Copy Chapter

• The Actor classes have been renamed for clarity (#1091):
  • BaseActor -> Actor
  • continuous.ActorProb -> ContinuousActorProbabilistic
  • coninuous.Actor -> ContinuousActorDeterministic
  • discrete.Actor -> DiscreteActor
• The Critic classes have been renamed for clarity (#1091):
  • continuous.Critic -> ContinuousCritic
  • discrete.Critic -> DiscreteCritic
• Moved Atari helper modules atari_network and atari_wrapper to the library under tianshou.env.atari.
• Fix issues pertaining to the torch device assignment of network components (#810):
  • Remove 'device' member (and the corresponding constructor argument) from the following classes:
    BranchingNet, C51Net, ContinuousActorDeterministic, ContinuousActorProbabilistic, ContinuousCritic,
    DiscreteActor, DiscreteCritic, DQNet, FullQuantileFunction, ImplicitQuantileNetwork,
    IntrinsicCuriosityModule, MLPActor, MLP, Perturbation, QRDQNet, Rainbow, Recurrent,
    RecurrentActorProb, RecurrentCritic, VAE
  • (Peripheral change:) Require the use of keyword arguments for the constructors of all of these classes
• Clean up handling of modules that define attribute output_dim, introducing the explicit base class
  ModuleWithVectorOutput
  • Interfaces where one could specify either a module with output_dim or additionally provide the output
    dimension as an argument were changed to use ModuleWithVectorOutput.
  • The high-level API class IntermediateModule can now provide a ModuleWithVectorOutput instance
    (via adaptation if necessary).
• The class hierarchy of supporting nn.Module implementations was cleaned up (#1091):
  • With the fundamental base classes ActionReprNet and ActionReprNetWithVectorOutput, we etablished a
    well-defined interface for the most commonly used forward interface in Tianshou's algorithms & policies. #948
  • Some network classes were renamed:
    • ScaledObsInputModule -> ScaledObsInputActionReprNet
    • Rainbow -> RainbowNet
• All modules containing base classes were renamed from base to a more descriptive name, rendering
  file names unique.

Release 1.2.0 (2025-06-23)

This is the final release in the 1.x series before Tianshou v2.0.0.
It resolves performance regressions introduced in v1.1.0 and resolves several issues,
partly by backporting improvements from the upcoming v2.0.0 release.

This release is brought to you by Applied AI Institute gGmbH.

Core developers:

• Dr. Dominik Jain (@opcode81)
• Michael Panchenko (@MischaPanch)

Changes/Improvements Copy Chapter

• trainer:
  • Custom scoring now supported for selecting the best model. #1202
• highlevel:
  • DiscreteSACExperimentBuilder: Expose method with_actor_factory_default #1248 #1250
  • ActorFactoryDefault: Fix parameters for hidden sizes and activation not being
    passed on in the discrete case (affects with_actor_factory_default method of experiment builders)
  • ExperimentConfig: Do not inherit from other classes, as this breaks automatic handling by
    jsonargparse when the class is used to define interfaces (as in high-level API examples)
  • AutoAlphaFactoryDefault: Differentiate discrete and continuous action spaces
    and allow coefficient to be modified, adding an informative docstring
    (previous implementation was reasonable only for continuous action spaces)
    • Adjust usage in atari_sac_hl example accordingly.
  • NPGAgentFactory, TRPOAgentFactory: Fix optimizer instantiation including the actor parameters
    (which was misleadingly suggested in the docstring in the respective policy classes; docstrings were fixed),
    as the actor parameters are intended to be handled via natural gradients internally
• data:
  • ReplayBuffer: Fix collection of empty episodes being disallowed
  • Collection was slow due to isinstance checks on Protocols and due to Buffer integrity validation. This was solved
    by no longer performing isinstance on Protocols and by making the integrity validation disabled by default.
• Tests:
  • We have introduced extensive determinism tests which allow to validate whether
    training processes deterministically compute the same results across different development branches.
    This is an important step towards ensuring reproducibility and consistency, which will be
    instrumental in supporting Tianshou developers in their work, especially in the context of
    algorithm development and evaluation.

Breaking Changes Copy Chapter

• trainer:
  • BaseTrainer.run and __iter__: Resetting was never optional prior to running the trainer,
    yet the recently introduced parameter reset_prior_to_run of run suggested that it was optional.
    Yet the parameter was ultimately not respected, because __iter__ would always call reset(reset_collectors=True, reset_buffer=False)
    regardless. The parameter was removed; instead, the parameters of run now mirror the parameters of reset,
    and the implicit reset call in __iter__ was removed.
    This aligns with upcoming changes in Tianshou v2.0.0.
    • NOTE: If you have been using a trainer without calling run but by directly iterating over it, you
      will need to call reset on the trainer explicitly before iterating over the trainer.
    • Using a trainer as an iterator is considered deprecated and support for this will be removed in Tianshou v2.0.0.
• data:
  • InfoStats has a new non-optional field best_score which is used
    for selecting the best model. #1202
• highlevel:
  • Change the way in which seeding is handled: The mechanism introduced in v1.1.0
    was completely revised:
    • The training_seed and test_seed attributes were removed from SamplingConfig.
      Instead, the seeds are derived from the seed defined in ExperimentConfig.
    • Seed attributes of EnvFactory classes were removed.
      Instead, seeds are passed to methods of EnvFactory.

Release 1.1.0 (2024-08-10)

NOTE: This release introduced (potentially severe) performance regressions in data collection, please switch to a newer release for better performance.

Highlights Copy Chapter

Evaluation Package Copy Section

This release introduces a new package evaluation that integrates best
practices for running experiments (seeding test and train environmets) and for
evaluating them using the rliable
library. This should be especially useful for algorithm developers for comparing
performances and creating meaningful visualizations. This functionality is
currently in alpha state and will be further improved in the next releases.
You will need to install tianshou with the extra eval to use it.

The creation of multiple experiments with varying random seeds has been greatly
facilitated. Moreover, the ExpLauncher interface has been introduced and
implemented with several backends to support the execution of multiple
experiments in parallel.

An example for this using the high-level interfaces can be found
here, examples that use low-level
interfaces will follow soon.

Improvements in Batch Copy Section

Apart from that, several important
extensions have been added to internal data structures, most notably to Batch.
Batches now implement __eq__ and can be meaningfully compared. Applying
operations in a nested fashion has been significantly simplified, and checking
for NaNs and dropping them is now possible.

One more notable change is that torch Distribution objects are now sliced when
slicing a batch. Previously, when a Batch with say 10 actions and a dist
corresponding to them was sliced to [:3], the dist in the result would still
correspond to all 10 actions. Now, the dist is also "sliced" to be the
distribution of the first 3 actions.

A detailed list of changes can be found below.

Changes/Improvements Copy Chapter

• evaluation: New package for repeating the same experiment with multiple
  seeds and aggregating the results. #1074 #1141 #1183
• data:
  • Batch:
    • Add methods to_dict and to_list_of_dicts. #1063 #1098
    • Add methods to_numpy_ and to_torch_. #1098, #1117
    • Add __eq__ (semantic equality check). #1098
    • keys() deprecated in favor of get_keys() (needed to make iteration
      consistent with naming) #1105.
    • Major: new methods for applying functions to values, to check for NaNs
      and drop them, and to set values. #1181
    • Slicing a batch with a torch distribution now also slices the
      distribution. #1181
  • data.collector:
    • Collector:
      • Introduced BaseCollector as a base class for all collectors.
        #1123
      • Add method close #1063
      • Method reset is now more granular (new flags controlling
        behavior). #1063
    • CollectStats: Add convenience
      constructor with_autogenerated_stats. #1063
• trainer:
  • Trainers can now control whether collectors should be reset prior to
    training. #1063
• policy:
  • introduced attribute in_training_step that is controlled by the trainer.
    #1123
  • policy automatically set to eval mode when collecting and to train
    mode when updating. #1123
  • Extended interface of compute_action to also support array-like inputs
    #1169
• highlevel:
  • SamplingConfig:
    • Add support for batch_size=None. #1077
    • Add training_seed for explicit seeding of training and test
      environments, the test_seed is inferred from training_seed. #1074
  • experiment:
    • Experiment now has a name attribute, which can be set
      using ExperimentBuilder.with_name and
      which determines the default run name and therefore the persistence
      subdirectory.
      It can still be overridden in Experiment.run(), the new parameter
      name being run_name rather than
      experiment_name (although the latter will still be interpreted
      correctly). #1074 #1131
    • Add class ExperimentCollection for the convenient execution of
      multiple experiment runs #1131
    • The World object, containing all low-level objects needed for
      experimentation,
      can now be extracted from an Experiment instance. This enables
      customizing
      the experiment prior to its execution, bridging the low and high-level
      interfaces. #1187
    • ExperimentBuilder:
      • Add method build_seeded_collection for the sound creation of
        multiple
        experiments with varying random seeds #1131
      • Add method copy to facilitate the creation of multiple
        experiments from a single builder #1131
  • env:
    • Added new VectorEnvType called SUBPROC_SHARED_MEM_AUTO and used in
      for Atari and Mujoco venv creation. #1141
• utils:
  • logger:
    • Loggers can now restore the logged data into python by using the
      new restore_logged_data method. #1074
    • Wandb logger extended #1183
  • net.continuous.Critic:
    • Add flag apply_preprocess_net_to_obs_only to allow the
      preprocessing network to be applied to the observations only (without
      the actions concatenated), which is essential for the case where we
      want
      to reuse the actor's preprocessing network #1128
  • torch_utils (new module)
    • Added context managers torch_train_mode
      and policy_within_training_step #1123
  • print
    • DataclassPPrintMixin now supports outputting a string, not just
      printing the pretty repr. #1141

Fixes Copy Chapter

• highlevel:
  • CriticFactoryReuseActor: Enable the Critic
    flag apply_preprocess_net_to_obs_only for continuous critics,
    fixing the case where we want to reuse an actor's preprocessing network
    for the critic (affects usages
    of the experiment builder method with_critic_factory_use_actor with
    continuous environments) #1128
  • Policy parameter action_scaling value "default" was not correctly
    transformed to a Boolean value for
    algorithms SAC, DDPG, TD3 and REDQ. The value "default" being truthy
    caused action scaling to be enabled
    even for discrete action spaces. #1191
• atari_network.DQN:
  • Fix constructor input validation #1128
  • Fix output_dim not being set if features_only=True
    and output_dim_added_layer is not None #1128
• PPOPolicy:
  • Fix max_batchsize not being used in logp_old computation
    inside process_fn #1168
• Fix Batch.__eq__ to allow comparing Batches with scalar array values #1185

Internal Improvements Copy Chapter

• Collectors rely less on state, the few stateful things are stored explicitly
  instead of through a .data attribute. #1063
• Introduced a first iteration of a naming convention for vars in Collectors.
  #1063
• Generally improved readability of Collector code and associated tests (still
  quite some way to go). #1063
• Improved typing for exploration_noise and within Collector. #1063
• Better variable names related to model outputs (logits, dist input etc.).
  #1032
• Improved typing for actors and critics, using Tianshou classes
  like Actor, ActorProb, etc.,
  instead of just nn.Module. #1032
• Added interfaces for most Actor and Critic classes to enforce the presence
  of forward methods. #1032
• Simplified PGPolicy forward by unifying the dist_fn interface (see
  associated breaking change). #1032
• Use .mode of distribution instead of relying on knowledge of the
  distribution type. #1032
• Exception no longer raised on len of empty Batch. #1084
• tests and examples are covered by mypy. #1077
• Actor is more used, stricter typing by making it generic. #1077
• Use explicit multiprocessing context for creating Pipe in subproc.py.
  #1102

Breaking Changes Copy Chapter

• data:
  • Collector:
    • Removed .data attribute. #1063
    • Collectors no longer reset the environment on initialization.
      Instead, the user might have to call reset expicitly or
      pass reset_before_collect=True . #1063
    • Removed no_grad argument from collect method (was unused in
      tianshou). #1123
  • Batch:
    • Fixed iter(Batch(...) which now behaves the same way
      as Batch(...).__iter__().
      Can be considered a bugfix. #1063
    • The methods to_numpy and to_torch in are not in-place anymore
      (use to_numpy_ or to_torch_ instead). #1098, #1117
    • The method Batch.is_empty has been removed. Instead, the user can
      simply check for emptiness of Batch by using len on dicts. #1144
    • Stricter cat_, only concatenation of batches with the same structure
      is allowed. #1181
    • to_torch and to_numpy are no longer static methods.
      So Batch.to_numpy(batch) should be replaced by batch.to_numpy().
      #1200
• utils:
  • logger:
    • BaseLogger.prepare_dict_for_logging is now abstract. #1074
    • Removed deprecated and unused BasicLogger (only affects users who
      subclassed it). #1074
  • utils.net:
    • Recurrent now receives and returns
      a RecurrentStateBatch instead of a dict. #1077
  • Modules with code that was copied from sensAI have been replaced by
    imports from new dependency sensAI-utils:
    • tianshou.utils.logging is replaced with sensai.util.logging
    • tianshou.utils.string is replaced with sensai.util.string
    • tianshou.utils.pickle is replaced with sensai.util.pickle
• env:
  • All VectorEnvs now return a numpy array of info-dicts on reset instead of
    a list. #1063
• policy:
  • Changed interface of dist_fn in PGPolicy and all subclasses to take a
    single argument in both
    continuous and discrete cases. #1032
• AtariEnvFactory constructor (in examples, so not really breaking) now
  requires explicit train and test seeds. #1074
• EnvFactoryRegistered now requires an explicit test_seed in the
  constructor. #1074
• highlevel:
  • params: The parameter dist_fn has been removed from the parameter
    objects (PGParams, A2CParams, PPOParams, NPGParams, TRPOParams).
    The correct distribution is now determined automatically based on the
    actor factory being used, avoiding the possibility of
    misspecification. Persisted configurations/policies continue to work as
    expected, but code must not specify the dist_fn parameter.
    #1194 #1195
  • env:
    • EnvFactoryRegistered: parameter seed has been replaced by the pair
      of parameters training_seed and test_seed
      Persisted instances will continue to work correctly.
      Subclasses such as AtariEnvFactory are also affected requires
      explicit train and test seeds. #1074
    • VectorEnvType: SUBPROC_SHARED_MEM has been replaced
      by SUBPROC_SHARED_MEM_DEFAULT. It is recommended to
      use SUBPROC_SHARED_MEM_AUTO instead. However, persisted configs will
      continue working. #1141

Tests Copy Chapter

• Fixed env seeding it test_sac_with_il.py so that the test doesn't fail
  randomly. #1081
• Improved CI triggers and added telemetry (if requested by user) #1177
• Improved environment used in tests.
• Improved tests bach equality to check with scalar values #1185

Dependencies Copy Chapter

• DeepDiff added to help with diffs of
  batches in tests. #1098
• Bumped black, idna, pillow
• New extra "eval"
• Bumped numba to >=60.0.0, permitting installation on python 3.12 # 1177
• New dependency sensai-utils

Release 1.0.0 (2024-03-20)

This release focuses on updating and improving Tianshou internals (in particular, code quality) while creating relatively few breaking changes (apart from things like the python and dependencies' versions).

We view it as a significant step for transforming Tianshou into the go-to place both for RL researchers, as well as for RL practitioners working on industry projects.
 
This is the first release after the appliedAI Institute (the TransferLab division) has decided to further develop Tianshou and provide long-term support. 

Breaking Changes Copy Chapter

• dropped support of python<3.11
• dropped support of gym, from now on only Gymnasium envs are supported
• removed functions like offpolicy_trainer in favor of OffpolicyTrainer(...).run() (this affects all example scripts)
• several breaking changes related to removing **kwargs from signatures, renamings of internal attributes (like critic1 -> critic)
• Outputs of training methods are now dataclasses instead of dicts

Functionality Extensions Copy Chapter

Major Copy Section

• High level interfaces for experiments, demonstrated by the new example scripts with names ending in _hl.py

Minor Copy Section

• Method to compute action directly from a policy's observation, can be used for unrolling
• Support for custom keys in ReplayBuffer
• Support for CalQL as part of CQL
• Support for explicit setting of multiprocessing context for SubprocEnvWorker
• critic2 no longer has to be explicitly constructed and passed if it is supposed to be the same network as critic (formerly critic1)

Internal Improvements Copy Chapter

Build and Docs Copy Section

• Completely changed the build pipeline. Tianshou now uses poetry, black, ruff, poethepoet, nbqa and other niceties.
• Notebook tutorials are now part of the repository (previously they were in a drive). They were fixed and are executed during the build as integration tests, in addition to serving as documentation. Parts of the content have been improved.
• Documentation is now built with jupyter book. JavaScript code has been slightly improved, JS dependencies are included as part of the repository.
• Many improvements in docstrings

Typing Copy Section

• Adding BatchPrototypes to cover the fields needed and returned by methods relying on batches in a backwards compatible way
• Removing **kwargs from policies' constructors
• Overall, much stricter and more correct typing. Removing kwargs and replacing dicts by dataclasses in several places.
• Making use of Generic to express different kinds of stats that can be returned by learn and update
• Improved typing in tests and examples, close to passing mypy

General Copy Section

• Reduced duplication, improved readability and simplified code in several places
• Use dist.mode instead of inferring loc or argmax from the dist_fn input

Contributions Copy Chapter

The OG creators Copy Section

• @Trinkle23897 participated in almost all aspects of the coordination and reviewed most of the merged PRs
• @nuance1979 participated in several discussions

From appliedAI Copy Section

The team working on this release of Tianshou consisted of @opcode81 @MischaPanch @maxhuettenrauch @carlocagnetta @bordeauxred

External contributions Copy Section

• @BFAnas participated in several discussions and contributed the CalQL implementation, extending the pre-processing logic.
• @dantp-ai fixed many mypy issues and improved the tests
• @arnaujc91 improved the logic of computing deterministic actions
• Many other contributors, among them many new ones participated in this release. The Tianshou team is very grateful for your contributions!

Older Releases

See releases on GitHub

File: CONTRIBUTING.md Copy Chapter

Contributing to Tianshou

Please refer to the 'Developer Guide' on tianshou.org.

File: .pre-commit-config.yaml Copy Chapter

default_install_hook_types: [commit-msg, pre-commit]
default_stages: [commit, manual]
fail_fast: false
repos:

• repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v4.4.0
  hooks:
  • id: check-added-large-files
• repo: local
  hooks:
  • id: ruff
    name: ruff
    entry: poetry run ruff
    require_serial: true
    language: system
    types: [python]
  • id: ruff-nb
    name: ruff-nb
    entry: poetry run nbqa ruff .
    require_serial: true
    language: system
    pass_filenames: false
    types: [python]
  • id: black
    name: black
    entry: poetry run black
    require_serial: true
    language: system
    types: [python]
  • id: poetry-check
    name: poetry check
    entry: poetry check
    language: system
    files: pyproject.toml
    pass_filenames: false
  • id: poetry-lock-check
    name: poetry lock check
    entry: poetry check
    args: [--lock]
    language: system
    pass_filenames: false
  • id: mypy
    name: mypy
    entry: poetry run mypy tianshou examples test

    filenames should not be passed as they would collide with the config in pyproject.toml

    pass_filenames: false
    files: '^tianshou(/[^/])/[^/]*.py$'
    language: system
  • id: mypy-nb
    name: mypy-nb
    entry: poetry run nbqa mypy
    language: system

File: .readthedocs.yaml Copy Chapter

.readthedocs.yaml

Read the Docs configuration file

See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

Required

version: 2

Set the version of Python and other tools you might need

build:
os: ubuntu-22.04
tools:
python: "3.11"
commands:
- mkdir -p $READTHEDOCS_OUTPUT/html
- curl -sSL https://install.python-poetry.org | python -

- ~/.local/bin/poetry config virtualenvs.create false

- ~/.local/bin/poetry install --with dev -E eval

Same as poe tasks, but unfortunately poe doesn't work with poetry not creating virtualenvs Copy Chapter

- ~/.local/bin/poetry run python docs/autogen_rst.py
- ~/.local/bin/poetry run which jupyter-book
- ~/.local/bin/poetry run python docs/create_toc.py
- ~/.local/bin/poetry run jupyter-book config sphinx docs/
- ~/.local/bin/poetry run sphinx-build -W -b html docs $READTHEDOCS_OUTPUT/html

File: docs/_config.yml Copy Chapter

Book settings

Learn more at https://jupyterbook.org/customize/config.html

#######################################################################################

A default configuration that will be loaded for all jupyter books

Users are expected to override these values in their own `_config.yml` file.

This is also the "master list" of all allowed keys and values.

#######################################################################################

Book settings

title : Tianshou Documentation # The title of the book. Will be placed in the left navbar.
author : Tianshou contributors # The author of the book
copyright : "2020, Tianshou contributors." # Copyright year to be placed in the footer
logo : _static/images/tianshou-logo.png # A path to the book logo

Patterns to skip when building the book. Can be glob-style (e.g. "*skip.ipynb")

exclude_patterns : ['**.ipynb_checkpoints', '.DS_Store', 'Thumbs.db', '_build', 'jupyter_execute', '.jupyter_cache', '.pytest_cache', 'docs/autogen_rst.py', 'docs/create_toc.py']

Auto-exclude files not in the toc

only_build_toc_files : false

#######################################################################################

Execution settings

execute:
execute_notebooks : cache # Whether to execute notebooks at build time. Must be one of ("auto", "force", "cache", "off")
cache : "" # A path to the jupyter cache that will be used to store execution artifacts. Defaults to _build/.jupyter_cache/
exclude_patterns : [] # A list of patterns to skip in execution (e.g. a notebook that takes a really long time)
timeout : -1 # The maximum time (in seconds) each notebook cell is allowed to run.
run_in_temp : false # If True, then a temporary directory will be created and used as the command working directory (cwd),
# otherwise the notebook's parent directory will be the cwd.
allow_errors : false # If False, when a code cell raises an error the execution is stopped, otherwise all cells are always run.
stderr_output : show # One of 'show', 'remove', 'remove-warn', 'warn', 'error', 'severe'

#######################################################################################

Parse and render settings

parse:
myst_enable_extensions: # default extensions to enable in the myst parser. See https://myst-parser.readthedocs.io/en/latest/using/syntax-optional.html
- amsmath
- colon_fence
# - deflist
- dollarmath
- html_admonition
# - html_image
- linkify
# - replacements
# - smartquotes
- substitution
- tasklist
myst_url_schemes: [ mailto, http, https ] # URI schemes that will be recognised as external URLs in Markdown links
myst_dmath_double_inline: true # Allow display math ($$) within an inline context

#######################################################################################

HTML-specific settings

html:
favicon : "_static/images/tianshou-favicon.png" # A path to a favicon image
use_edit_page_button : false # Whether to add an "edit this page" button to pages. If true, repository information in repository: must be filled in
use_repository_button : false # Whether to add a link to your repository button
use_issues_button : false # Whether to add an "open an issue" button
use_multitoc_numbering : true # Continuous numbering across parts/chapters
extra_footer : ""
google_analytics_id : "" # A GA id that can be used to track book views.
home_page_in_navbar : true # Whether to include your home page in the left Navigation Bar
baseurl : "https://tianshou.readthedocs.io/en/master/"
analytics:

comments:
hypothesis : false
utterances : false
announcement : "" # A banner announcement at the top of the site.

#######################################################################################

LaTeX-specific settings

latex:
latex_engine : pdflatex # one of 'pdflatex', 'xelatex' (recommended for unicode), 'luatex', 'platex', 'uplatex'
use_jupyterbook_latex : true # use sphinx-jupyterbook-latex for pdf builds as default
targetname : book.tex

Add a bibtex file so that we can create citations

bibtex_bibfiles:

• refs.bib

#######################################################################################

Launch button settings

launch_buttons:
notebook_interface : classic # The interface interactive links will activate ["classic", "jupyterlab"]
binderhub_url : "" # The URL of the BinderHub (e.g., https://mybinder.org)
jupyterhub_url : "" # The URL of the JupyterHub (e.g., https://datahub.berkeley.edu)
thebe : false # Add a thebe button to pages (requires the repository to run on Binder)
colab_url : "https://colab.research.google.com"
url : https://github.com/thu-ml/tianshou # The URL to your book's repository
path_to_book : docs # A path to your book's folder, relative to the repository root.
branch : master # Which branch of the repository should be used when creating links

#######################################################################################

Advanced and power-user settings

sphinx:
extra_extensions :
- sphinx.ext.autodoc
- sphinx.ext.viewcode
- sphinx_toolbox.more_autodoc.sourcelink
- sphinxcontrib.spelling
- sphinxcontrib.mermaid
local_extensions : # A list of local extensions to load by sphinx specified by "name: path" items
recursive_update : false # A boolean indicating whether to overwrite the Sphinx config (true) or recursively update (false)
config : # key-value pairs to directly over-ride the Sphinx configuration
autodoc_typehints_format: "short"
autodoc_member_order: "bysource"
autodoc_mock_imports:
# mock imports for optional dependencies (e.g. dependencies of atari/atari_wrapper)
- cv2
autoclass_content: "both"
autodoc_default_options:
show-inheritance: True
html_js_files:

We have to list them explicitly because they need to be loaded in a specific order

- js/[email protected]
  - js/[email protected]
  - js/[email protected]
autodoc_show_sourcelink: True
add_module_names: False
github_username: thu-ml
github_repository: tianshou
python_use_unqualified_type_names: True
nb_mime_priority_overrides: [
  [ 'html', 'application/vnd.jupyter.widget-view+json', 10 ],
  [ 'html', 'application/javascript', 20 ],
  [ 'html', 'text/html', 30 ],
  [ 'html', 'text/latex', 40 ],
  [ 'html', 'image/svg+xml', 50 ],
  [ 'html', 'image/png', 60 ],
  [ 'html', 'image/jpeg', 70 ],
  [ 'html', 'text/markdown', 80 ],
  [ 'html', 'text/plain', 90 ],
  [ 'spelling', 'application/vnd.jupyter.widget-view+json', 10 ],
  [ 'spelling', 'application/javascript', 20 ],
  [ 'spelling', 'text/html', 30 ],
  [ 'spelling', 'text/latex', 40 ],
  [ 'spelling', 'image/svg+xml', 50 ],
  [ 'spelling', 'image/png', 60 ],
  [ 'spelling', 'image/jpeg', 70 ],
  [ 'spelling', 'text/markdown', 80 ],
  [ 'spelling', 'text/plain', 90 ],
]
mathjax_path: https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js
myst_fence_as_directive: ["mermaid"]
mathjax3_config:
  loader: { load: [ '[tex]/configmacros' ] }
  tex:
    packages: { '[+]': [ 'configmacros' ] }
    macros:
      vect: ["{\\mathbf{\\boldsymbol{#1}} }", 1]
      E: "{\\mathbb{E}}"
      P: "{\\mathbb{P}}"
      R: "{\\mathbb{R}}"
      abs: ["{\\left| #1 \\right|}", 1]
      simpl: ["{\\Delta^{#1} }", 1]
      amax: "{\\text{argmax}}"

File: docs/bibtex.json Copy Chapter

{
"cited": {
"tutorials/dqn": [
"DQN",
"DDPG",
"PPO"
]
}
}

File: docs/index.rst Copy Chapter

Welcome to Tianshou!

Tianshou (天授 <https://baike.baidu.com/item/%E5%A4%A9%E6%8E%88>_) is a reinforcement learning platform based on pure PyTorch. Unlike existing reinforcement learning libraries, which are mainly based on TensorFlow, have many nested classes, unfriendly API, or slow-speed, Tianshou provides a fast-speed framework and pythonic API for building the deep reinforcement learning agent. The supported interface algorithms include:

• :class:~tianshou.algorithm.modelfree.dqn.DQN Deep Q-Network <https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf>_
• :class:~tianshou.algorithm.modelfree.dqn.DQN Double DQN <https://arxiv.org/pdf/1509.06461.pdf>_
• :class:~tianshou.algorithm.modelfree.dqn.DQN Dueling DQN <https://arxiv.org/pdf/1511.06581.pdf>_
• :class:~tianshou.algorithm.modelfree.bdqn.BDQN Branching DQN <https://arxiv.org/pdf/1711.08946.pdf>_
• :class:~tianshou.algorithm.modelfree.c51.C51 Categorical DQN <https://arxiv.org/pdf/1707.06887.pdf>_
• :class:~tianshou.algorithm.modelfree.rainbow.RainbowDQN Rainbow DQN <https://arxiv.org/pdf/1710.02298.pdf>_
• :class:~tianshou.algorithm.modelfree.qrdqn.QRDQN Quantile Regression DQN <https://arxiv.org/pdf/1710.10044.pdf>_
• :class:~tianshou.algorithm.modelfree.iqn.IQN Implicit Quantile Network <https://arxiv.org/pdf/1806.06923.pdf>_
• :class:~tianshou.algorithm.modelfree.fqf.FQF Fully-parameterized Quantile Function <https://arxiv.org/pdf/1911.02140.pdf>_
• :class:~tianshou.algorithm.modelfree.reinforce.Reinforce Reinforce/Vanilla Policy Gradients <https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf>_
• :class:~tianshou.algorithm.modelfree.npg.NPG Natural Policy Gradient <https://proceedings.neurips.cc/paper/2001/file/4b86abe48d358ecf194c56c69108433e-Paper.pdf>_
• :class:~tianshou.algorithm.modelfree.a2c.A2C Advantage Actor-Critic <https://openai.com/blog/baselines-acktr-a2c/>_
• :class:~tianshou.algorithm.modelfree.trpo.TRPO Trust Region Policy Optimization <https://arxiv.org/pdf/1502.05477.pdf>_
• :class:~tianshou.algorithm.modelfree.ppo.PPO Proximal Policy Optimization <https://arxiv.org/pdf/1707.06347.pdf>_
• :class:~tianshou.algorithm.modelfree.ddpg.DDPG Deep Deterministic Policy Gradient <https://arxiv.org/pdf/1509.02971.pdf>_
• :class:~tianshou.algorithm.modelfree.td3.TD3 Twin Delayed DDPG <https://arxiv.org/pdf/1802.09477.pdf>_
• :class:~tianshou.algorithm.modelfree.sac.SAC Soft Actor-Critic <https://arxiv.org/pdf/1812.05905.pdf>_
• :class:~tianshou.algorithm.modelfree.redq.REDQ Randomized Ensembled Double Q-Learning <https://arxiv.org/pdf/2101.05982.pdf>_
• :class:~tianshou.algorithm.modelfree.discrete_sac.DiscreteSAC Discrete Soft Actor-Critic <https://arxiv.org/pdf/1910.07207.pdf>_
• :class:~tianshou.algorithm.imitation.imitation_base.ImitationPolicy Imitation Learning
• :class:~tianshou.algorithm.imitation.bcq.BCQ Batch-Constrained deep Q-Learning <https://arxiv.org/pdf/1812.02900.pdf>_
• :class:~tianshou.algorithm.imitation.cql.CQL Conservative Q-Learning <https://arxiv.org/pdf/2006.04779.pdf>_
• :class:~tianshou.algorithm.imitation.td3_bc.TD3BC Twin Delayed DDPG with Behavior Cloning <https://arxiv.org/pdf/2106.06860.pdf>_
• :class:~tianshou.algorithm.imitation.discrete_cql.DiscreteCQL Discrete Conservative Q-Learning <https://arxiv.org/pdf/2006.04779.pdf>_
• :class:~tianshou.algorithm.imitation.discrete_bcq.DiscreteBCQ Discrete Batch-Constrained deep Q-Learning <https://arxiv.org/pdf/1910.01708.pdf>_
• :class:~tianshou.algorithm.imitation.discrete_crr.DiscreteCRR Critic Regularized Regression <https://arxiv.org/pdf/2006.15134.pdf>_
• :class:~tianshou.algorithm.imitation.gail.GAIL Generative Adversarial Imitation Learning <https://arxiv.org/pdf/1606.03476.pdf>_
• :class:~tianshou.algorithm.modelbased.psrl.PSRLPolicy Posterior Sampling Reinforcement Learning <https://www.ece.uvic.ca/~bctill/papers/learning/Strens_2000.pdf>_
• :class:~tianshou.algorithm.modelbased.icm.ICMOffPolicyWrapper, :class:~tianshou.algorithm.modelbased.icm.ICMOnPolicyWrapper Intrinsic Curiosity Module <https://arxiv.org/pdf/1705.05363.pdf>_
• :class:~tianshou.data.buffer.prio.PrioritizedReplayBuffer Prioritized Experience Replay <https://arxiv.org/pdf/1511.05952.pdf>_
• :meth:~tianshou.algorithm.algorithm_base.Algorithm.compute_episodic_return Generalized Advantage Estimator <https://arxiv.org/pdf/1506.02438.pdf>_
• :class:~tianshou.data.buffer.her.HERReplayBuffer Hindsight Experience Replay <https://arxiv.org/pdf/1707.01495.pdf>_

Installation Copy Chapter

Tianshou is available through PyPI <https://pypi.org/project/tianshou/>_.
New releases require Python >= 3.11.

Install Tianshou with the following command:

.. code-block:: bash

$ pip install tianshou

Alternatively, install the current version on GitHub:

.. code-block:: bash

$ pip install git+https://github.com/thu-ml/tianshou.git@master --upgrade

After installation, open your python console and type
::

import tianshou
print(tianshou.__version__)

If no error occurs, you have successfully installed Tianshou.

Indices and tables Copy Chapter

• :ref:genindex
• :ref:modindex
• :ref:search

File: docs/01_user_guide/00_training_process.md Copy Chapter

The Reinforcement Learning Process

The following diagram illustrates the key mechanisms underlying the learning process in model-free reinforcement learning algorithms.
It shows how the agent interacts with the environment, collects experiences, and periodically updates its policy based on those experiences.

⇦ Prev
Next ⇨

Accordingly, the key entities involved in the learning process are:

• The environment: This is the system the agent interacts with.
  It provides the agent with observable states and rewards based on the actions taken by the agent.
• The agent's policy: This is the strategy used by the agent to decide which action to take in a given state.
  The policy can be deterministic or stochastic and is typically represented by a neural network in deep reinforcement learning.
• The replay buffer: This is a data structure used to store the agent's experiences, which consist of state transitions,
  actions taken, and rewards received.
  The agent learns from past experience by sampling mini-batches from the buffer during the policy update phase.
• The learning algorithm: This defines how the agent updates its policy based on the experiences stored in the replay buffer.
  Different algorithms have different update mechanisms, which can significantly affect the learning performance.
  In some cases, the algorithm may also involve additional components (specifically neural networks), such as target networks or value
  functions.

These entities have direct correspondences in Tianshou's codebase:

• The environment is represented by an instance of a class that inherits from gymnasium.Env, which is a standard interface for
  reinforcement learning environments.
  In practice, environments are typically vectorized to enable parallel interactions, increasing efficiency.
• The policy is encapsulated in the {class}~tianshou.algorithm.algorithm_base.Policy class, which provides methods for action selection.
• The replay buffer is implemented in the {class}~tianshou.data.buffer.buffer_base.ReplayBuffer class.
  A {class}~tianshou.data.collector.Collector instance is used to manage the addition of new experiences to the replay buffer as the agent interacts with the
  environment.
  During the learning phase, the replay buffer may be sampled, providing an instance of {class}~tianshou.data.batch.Batch for the policy update.
• The abstraction for learning algorithms is given by the {class}~tianshou.algorithm.algorithm_base.Algorithm class, which defines how to update the policy using data from the
  replay buffer.

(structuring-the-process)=

Structuring the Process Copy Chapter

The learning process itself is reified in Tianshou's {class}~tianshou.trainer.trainer.Trainer class, which orchestrates the interaction between the agent and the
environment, manages the replay buffer, and coordinates the policy updates according to the specified learning algorithm.

In general, the process can be described as executing a number of epochs as follows:

• epoch:
  • repeat until a sufficient number of steps is reached (for online learning, typically environment step count)
    • training step:
      • for online learning algorithms …
        • collection step: collect state transitions in the environment by running the agent
        • (optionally) conduct a test step if collected data indicates promising behaviour
      • update step: apply gradient updates using the algorithm’s update logic.
        The update is based on …
        • data from the preceding collection step only (on-policy learning)
        • data from the collection step and previous data (off-policy learning)
        • data from a user-provided replay buffer (offline learning)
  • test step
    • collect test episodes from dedicated test environments and evaluate agent performance
    • (optionally) stop training early if performance is sufficiently high

:class: note
The above introduces some of the key terms used throughout Tianshou.

Note that the above description encompasses several modes of model-free reinforcement learning, including:

• online learning (where the agent continuously interacts with the environment in order to collect new experiences)
  • on-policy learning (where the policy is updated based on data collected using the current policy only)
  • off-policy learning (where the policy is updated based on data collected using the current and previous policies)
• offline learning (where the replay buffer is pre-filled and not updated during training)

In Tianshou, the {class}~tianshou.trainer.trainer.Trainer and {class}~tianshou.algorithm.algorithm_base.Algorithm classes are specialised to handle these different modes accordingly.

File: docs/01_user_guide/01_apis.md Copy Chapter

Dual APIs

Tianshou provides two distinct APIs to serve different use cases and user preferences:

1. high-level API: a declarative, configuration-based interface designed for ease of use
2. procedural API: a flexible, imperative interface providing maximum control

Both APIs access the same underlying algorithm implementations, allowing you to choose the level
of abstraction that best fits your needs without sacrificing functionality.

Overview Copy Chapter

High-Level API Copy Section

The high-level API is built around the builder pattern and declarative semantics.
Instead of writing procedural code that sequentially constructs and connects components,
you declare what you want through configuration objects and let Tianshou handle how to
build and execute the experiment.

Key characteristics:

• centered around {class}~tianshou.highlevel.experiment.ExperimentBuilder classes (e.g., {class}~tianshou.highlevel.experiment.DQNExperimentBuilder, {class}~tianshou.highlevel.experiment.PPOExperimentBuilder, etc.)
• uses configuration dataclasses and factories for all relevant parameters
• automatically handles component creation and "wiring"
• provides sensible defaults that adapt to the nature of your environment
• includes built-in persistence, logging, and experiment management
• full type hints (but object structure is not flat; a proper IDE is required for seamless user experience)

Procedural API Copy Section

The procedural API provides explicit control over every component in the RL pipeline.
You manually create environments, networks, policies, algorithms, collectors, and
trainers, then wire them together.

Key characteristics:

• direct instantiation of all components
• explicit control over the training loop
• lower-level access to internal mechanisms
• minimal abstraction (closer to the implementation)
• ideal for algorithm development and research

When to Use Which API Copy Chapter

Use the high-level API when ...

• you're applying existing algorithms to new problems
• you want to get started quickly with minimal boilerplate
• you need experiment management with persistence, logging, and reproducibility
• you prefer declarative code that focuses on configuration
• you're building applications rather than developing new algorithms

Use the procedural API when:

• you're developing new algorithms or modifying existing ones
• you need fine-grained control over the training process
• you want to understand the internal workings of Tianshou
• you're implementing custom components not supported by the high-level API
• you prefer imperative programming where each step is explicit
• you need maximum flexibility for experimental research

Comparison by Example Copy Chapter

Let's compare both APIs by implementing the same DQN learning task on the CartPole environment.

High-Level API Example Copy Section

from tianshou.highlevel.config import OffPolicyTrainingConfig
from tianshou.highlevel.env import EnvFactoryRegistered, VectorEnvType
from tianshou.highlevel.experiment import DQNExperimentBuilder, ExperimentConfig
from tianshou.highlevel.params.algorithm_params import DQNParams
from tianshou.highlevel.trainer import EpochStopCallbackRewardThreshold

# Build the experiment through configuration
experiment = (
    DQNExperimentBuilder(
        # Environment configuration
        EnvFactoryRegistered(
            task="CartPole-v1",
            venv_type=VectorEnvType.DUMMY,
            training_seed=0,
            test_seed=10,
        ),
        # Experiment settings
        ExperimentConfig(
            persistence_enabled=False,
            watch=True,
            watch_render=1 / 35,
            watch_num_episodes=100,
        ),
        # Training configuration
        OffPolicyTrainingConfig(
            max_epochs=10,
            epoch_num_steps=10000,
            batch_size=64,
            num_training_envs=10,
            num_test_envs=100,
            buffer_size=20000,
            collection_step_num_env_steps=10,
            update_step_num_gradient_steps_per_sample=1 / 10,
        ),
    )
    # Algorithm-specific parameters
    .with_dqn_params(
        DQNParams(
            lr=1e-3,
            gamma=0.9,
            n_step_return_horizon=3,
            target_update_freq=320,
            eps_training=0.3,
            eps_inference=0.0,
        ),
    )
    # Network architecture
    .with_model_factory_default(hidden_sizes=(64, 64))
    # Stop condition
    .with_epoch_stop_callback(EpochStopCallbackRewardThreshold(195))
    .build()
)

# Run the experiment
experiment.run()

What's happening here:

1. We create an {class}~tianshou.highlevel.experiment.ExperimentBuilder with three main configuration objects
2. We chain builder methods to specify algorithm parameters, model architecture, and callbacks
3. We call .build() to construct the experiment
4. We call .run() to execute the entire training pipeline

The high-level API handles ...

• creating and configuring environments
• building the neural network
• instantiating the policy and algorithm
• setting up collectors and replay buffer
• managing the training loop
• watching the trained agent

Procedural API Example Copy Section

import gymnasium as gym
import tianshou as ts
from tianshou.algorithm.modelfree.dqn import DiscreteQLearningPolicy
from tianshou.algorithm.optim import AdamOptimizerFactory
from tianshou.data import CollectStats
from tianshou.trainer import OffPolicyTrainerParams
from tianshou.utils.net.common import Net
from tianshou.utils.space_info import SpaceInfo
from torch.utils.tensorboard import SummaryWriter

# Define hyperparameters
task = "CartPole-v1"
lr, epoch, batch_size = 1e-3, 10, 64
num_training_envs, num_test_envs = 10, 100
gamma, n_step, target_freq = 0.9, 3, 320
buffer_size = 20000
eps_train, eps_test = 0.1, 0.05
epoch_num_steps, collection_step_num_env_steps = 10000, 10

# Set up logging
logger = ts.utils.TensorboardLogger(SummaryWriter("log/dqn"))

# Create environments
training_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_training_envs)])
test_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_test_envs)])

# Build the network
env = gym.make(task, render_mode="human")
space_info = SpaceInfo.from_env(env)
state_shape = space_info.observation_info.obs_shape
action_shape = space_info.action_info.action_shape
net = Net(state_shape=state_shape, action_shape=action_shape, hidden_sizes=[128, 128, 128])

# Create policy and algorithm
policy = DiscreteQLearningPolicy(
    model=net,
    action_space=env.action_space,
    eps_training=eps_train,
    eps_inference=eps_test,
)
algorithm = ts.algorithm.DQN(
    policy=policy,
    optim=AdamOptimizerFactory(lr=lr),
    gamma=gamma,
    n_step_return_horizon=n_step,
    target_update_freq=target_freq,
)

# Set up collectors
training_collector = ts.data.Collector[CollectStats](
    algorithm,
    training_envs,
    ts.data.VectorReplayBuffer(buffer_size, num_training_envs),
    exploration_noise=True,
)
test_collector = ts.data.Collector[CollectStats](
    algorithm,
    test_envs,
    exploration_noise=True,
)


# Define stop condition
def stop_fn(mean_rewards: float) -> bool:
    if env.spec and env.spec.reward_threshold:
        return mean_rewards >= env.spec.reward_threshold
    return False


# Train the algorithm
result = algorithm.run_training(
    OffPolicyTrainerParams(
        training_collector=training_collector,
        test_collector=test_collector,
        max_epochs=epoch,
        epoch_num_steps=epoch_num_steps,
        collection_step_num_env_steps=collection_step_num_env_steps,
        test_step_num_episodes=num_test_envs,
        batch_size=batch_size,
        update_step_num_gradient_steps_per_sample=1 / collection_step_num_env_steps,
        stop_fn=stop_fn,
        logger=logger,
        test_in_training=True,
    )
)
print(f"Finished training in {result.timing.total_time} seconds")

# Watch the trained agent
collector = ts.data.Collector[CollectStats](algorithm, env, exploration_noise=True)
collector.collect(n_episode=100, render=1 / 35)

What's happening here:

1. We explicitly define all hyperparameters as variables
2. We manually create the logger
3. We construct training and test environments
4. We build the neural network by extracting space information from the environment
5. We create the policy and algorithm objects
6. We set up collectors with a replay buffer
7. We define callback functions
8. We call algorithm.run_training() with explicit parameters
9. We manually set up and run the evaluation collector

The procedural API requires ...

• explicit creation of every component
• manual extraction of environment properties
• direct specification of all connections

Key Concepts in the High-Level API Copy Chapter

ExperimentBuilder Copy Section

The {class}~tianshou.highlevel.experiment.ExperimentBuilder is the core abstraction.
Each algorithm has its own builder (e.g., {class}~tianshou.highlevel.experiment.DQNExperimentBuilder, {class}~tianshou.highlevel.experiment.PPOExperimentBuilder, {class}~tianshou.highlevel.experiment.SACExperimentBuilder).

Some methods you will find in experiment builders:

• .with_<algorithm>_params() - Set algorithm-specific parameters
• .with_model_factory(), .with_model_factory_default() - Configure network architecture
• .with_critic_factory() - Configure critic network (for actor-critic methods)
• .with_epoch_train_callback() - Add function to be called at the beginning of the training step in each epoch
• .with_epoch_test_callback() - Add function to be called at the beginning of the test step in each epoch
• .with_epoch_stop_callback() - Define stopping conditions
• .with_algorithm_wrapper_factory() - Add algorithm wrappers (e.g., ICM)

Configuration Objects Copy Section

Three main configuration objects are required when constructing an experiment builder:

1. Environment Configuration ({class}~tianshou.highlevel.env.EnvFactory subclasses)

   • Defines how to create and configure environments
   • Existing factories:
     • {class}~tianshou.highlevel.env.EnvFactoryRegistered - For the creation of environments registered in Gymnasium
     • {class}~tianshou.highlevel.env.atari.atari_wrapper.AtariEnvFactory - For Atari environments with preprocessing
   • Custom factories for your own environments can be created by subclassing {class}~tianshou.highlevel.env.EnvFactory

2. Experiment Configuration ({class}~tianshou.highlevel.experiment.ExperimentConfig):
   General settings for the experiment, particularly related to

   • logging
   • randomization
   • persistence
   • watching the trained agent's performance after training

3. Training Configuration ({class}~tianshou.highlevel.config.OffPolicyTrainingConfig, {class}~tianshou.highlevel.config.OnPolicyTrainingConfig):
   Defines all parameters related to the training process

Parameter Classes Copy Section

Algorithm parameters are defined in dataclasses specific to each algorithm (e.g., {class}~tianshou.highlevel.params.algorithm_params.DQNParams, {class}~tianshou.highlevel.params.algorithm_params.PPOParams).
The parameters are extensively documented.

Make sure to use a modern IDE to take advantage of auto-completion and inline documentation!

Factories Copy Section

The high-level API uses factories extensively:

• Model Factories: Create neural networks (e.g., {class}~tianshou.highlevel.module.intermediate.IntermediateModuleFactoryAtariDQN)
• Environment Factories: Create and configure environments
• Optimizer Factories: Create optimizers with specific configurations

Extensibility Copy Section

The high-level API is designed to be extensible.
You can create custom factories (e.g. for your own models or your own environments) by subclassing the appropriate base classes
and then use them in the experiment builder.

If we have created a torch module in CustomNetwork, which we want to use within our policy,
we simply need to define a factory for it in order to apply it in the high-level API:

from tianshou.highlevel.env import Environments
from tianshou.highlevel.module.core import TDevice
from tianshou.highlevel.module.intermediate import IntermediateModuleFactory, IntermediateModule

class CustomNetFactory(IntermediateModuleFactory):
    def __init__(self, hidden_sizes: tuple[int, ...] = (128, 128)):
        self.hidden_sizes = hidden_sizes
    
    def create_intermediate_module(self, envs: Environments, device: TDevice) -> IntermediateModule:
        obs_shape = envs.get_observation_shape()
        action_shape = envs.get_action_shape()
        
        # Your custom network creation logic
        net = CustomNetwork(
            obs_shape=obs_shape,
            action_shape=action_shape,
            hidden_sizes=self.hidden_sizes,
        ).to(device)
        
        return IntermediateModule(net, net.output_dim)

experiment = (
    DQNExperimentBuilder(...)
    .with_model_factory(CustomNetFactory(hidden_sizes=(256, 256)))
    .build()
)

Key Concepts in the Procedural API Copy Chapter

Core Components Copy Section

You manually create and connect ...

1. environments: e.g. using gym.make() and vectorization ({class}~tianshou.env.DummyVectorEnv, {class}~tianshou.env.SubprocVectorEnv)
2. networks: using {class}~tianshou.utils.net.common.Net or other PyTorch modules
3. policies: using algorithm-specific policy classes (e.g., {class}~tianshou.algorithm.modelfree.dqn.DiscreteQLearningPolicy)
4. algorithms: using algorithm classes (e.g., {class}~tianshou.algorithm.modelfree.dqn.DQN, {class}~tianshou.algorithm.modelfree.ppo.PPO, {class}~tianshou.algorithm.modelfree.sac.SAC)
5. collectors: using {class}~tianshou.data.Collector to gather experience
6. buffers: using {class}~tianshou.data.buffer.VectorReplayBuffer or {class}~tianshou.data.buffer.ReplayBuffer
7. trainers: using the respective trainer class and corresponding parameter class (e.g., {class}~tianshou.trainer.OffPolicyTrainer and {class}~tianshou.trainer.OffPolicyTrainerParams)

Training Loop Copy Section

The training is executed via algorithm.run_training(), which takes a trainer parameter object.
You can alternatively implement custom training loops (or even your own trainer class) for maximum flexibility.

Additional Resources Copy Chapter

• high-Level API examples: See examples/ directory (scripts ending in _hl.py)
• procedural API examples: See examples/ directory (scripts without suffix)

File: docs/01_user_guide/02_core_abstractions.md Copy Chapter

Core Abstractions

Tianshou's architecture is built around a number of key abstractions that work together to provide a modular and flexible reinforcement learning framework.
This document describes the conceptual foundation and functionality of each abstraction, helping you understand how they interact to enable RL agent training.

Knowing these abstractions is primarily relevant when using the procedural API – and particularly when implementing one's own learning algoriithms.

Algorithm Copy Chapter

The {class}~tianshou.algorithm.algorithm_base.Algorithm is the central abstraction representing the core of a reinforcement learning method (such as DQN, PPO, or SAC).
It implements the key steps within the {ref}learning process <structuring-the-process>, containing a {ref}policy and defining how to update it from experience data.

Since an Algorithm contains neural networks and manages their training, the class inherits from torch.nn.Module.

Core Responsibilities Copy Section

An Algorithm implements the details of an {ref}update step <structuring-the-process>:

1. preprocessing: Before the actual update begins, the algorithm prepares the training data.
   This includes computing derived quantities that depend on temporal sequences, such as n-step returns, GAE advantages, or terminal state handling.
   The {meth}~tianshou.algorithm.algorithm_base.Algorithm._preprocess_batch method handles this phase, often leveraging static methods like
   {meth}~tianshou.algorithm.algorithm_base.Algorithm.compute_nstep_return and
   {meth}~tianshou.algorithm.algorithm_base.Algorithm.compute_episodic_return to
   efficiently compute returns using the buffer's temporal structure.

2. network update: The algorithm performs the actual neural network updates based on its specific learning method.
   Each algorithm implements its own {meth}~tianshou.algorithm.algorithm_base.Algorithm._update_with_batch logic that defines how to update
   the policy networks using the preprocessed batch data.

3. postprocessing: After the update, the algorithm may perform cleanup operations, such as updating prioritized replay buffer weights or other
   algorithm-specific bookkeeping.

Learning Orchestration Copy Section

The Algorithm orchestrates the {ref}update step <structuring-the-process> through its
{meth}~tianshou.algorithm.algorithm_base.Algorithm.update method, which ensures these three phases execute in proper sequence.
It also manages optimizer state and learning rate schedulers, making them available for state persistence through
{meth}~tianshou.algorithm.algorithm_base.Algorithm.state_dict and
{meth}~tianshou.algorithm.algorithm_base.Algorithm.load_state_dict methods.

Each algorithm type (on-policy, off-policy, offline) creates its appropriate trainer through the
{meth}~tianshou.algorithm.algorithm_base.Algorithm.create_trainer method,
establishing the connection between the learning logic and the training loop.

(policy)=

Policy Copy Chapter

The {class}~tianshou.algorithm.algorithm_base.Policy represents the agent's decision-making component, i.e. the mapping from observations to actions.
While the Algorithm defines how to learn, the Policy defines what is learned and how to act.

Like Algorithm, the class inherits from torch.nn.Module.

States of Operation Copy Section

A Policy operates in two main modes:

• training mode: During training, the policy may employ exploration strategies, sample from action distributions, or add noise to encourage discovery.
  Training mode is further divided into:

  • collecting state: When gathering experience from environment interaction
  • updating state: When performing network updates during learning

• testing/inference mode: During evaluation, the policy typically acts deterministically or uses the mode of predicted distributions to showcase
  learned behavior without exploration.

The flag is_within_training_step controls the collection strategy, distinguishing between training and inference behavior.

Key Methods Copy Section

The Policy provides several essential methods:

• {meth}~tianshou.algorithm.algorithm_base.Policy.forward:
  The core computation method that processes batched observations to produce action distributions or Q-values.
  It takes a batch of environment data and optional hidden state (for recurrent policies),
  returning a batch containing at minimum the "act" key,
  and potentially "state" (hidden state) and "policy" (intermediate results to be stored in the buffer).

• {meth}~tianshou.algorithm.algorithm_base.Policy.compute_action:
  A convenient method for inference that takes a single observation and returns a concrete action suitable for the environment.
  This method internally calls forward with proper batching and unbatching.

• {meth}~tianshou.algorithm.algorithm_base.Policy.map_action: Transforms the raw neural network output to the environment's action space format, handling any necessary scaling or discretization.

The separation between forward (which works with batches) and
{meth}~tianshou.algorithm.algorithm_base.Policy.compute_action (which works with single observations) provides efficiency
during training and convenience during inference.

Collector Copy Chapter

The class {class}~tianshou.data.Collector bridges the gap between the policy and the environment(s),
managing the process of gathering experience data.
It enables efficient interaction with both single environments and vectorized environments (multiple parallel environments).

Data Collection Copy Section

The Collector's primary method, {meth}~tianshou.data.Collector.collect, orchestrates the environment interaction loop. It can collect either:

• a specified number of steps (n_step): useful for maintaining consistent training batch sizes
• a specified number of episodes (n_episode): useful for evaluation or when episode-level statistics are important

During collection, the Collector ...

1. obtains observations from the environment(s),
2. calls the policy to compute actions,
3. steps the environment(s) with these actions,
4. stores the resulting transitions (observation, action, reward, next observation, termination flags, and info) in the replay buffer,
5. manages episode boundaries and reset logic,
6. collects statistics such as episode returns, lengths, and collection speed.

Hooks and Extensibility Copy Section

The Collector supports customization through hooks that can be triggered at different points in the collection process:

• step hooks: called after each environment step
• episode done hooks: called when episodes complete

These hooks enable custom logging, curriculum learning, or other dynamic behaviors during data collection.

Vectorized Environments Copy Section

The Collector seamlessly handles vectorized environments, where multiple environment instances run in parallel.
This significantly speeds up data collection while maintaining correct episode boundaries and statistics for each environment instance.

Trainer Copy Chapter

The {class}~tianshou.trainer.Trainer orchestrates the complete training loop, coordinating data collection, policy updates, and evaluation.
It provides the high-level control flow that brings all components together.

Trainer Types Copy Section

Tianshou provides three main trainer types, each suited to different algorithm families:

• {class}~tianshou.trainer.OnPolicyTrainer: for algorithms that must learn from freshly collected data (e.g., PPO, A2C).
  After each collection phase, the buffer is used for updates and thereafter is cleared.

• {class}~tianshou.trainer.OffPolicyTrainer: for algorithms that can learn from any past experience (e.g., DQN, SAC, DDPG).
  Data accumulates in the replay buffer over time, and updates sample from this growing pool of experience.

• {class}~tianshou.trainer.OfflineTrainer: for algorithms that learn exclusively from a fixed dataset without any environment interaction (e.g., BCQ, CQL).

Training Loop Structure Copy Section

The training process is organized into epochs, where each epoch consists of:

1. data collection: The trainer uses the train collector to gather experience according to its algorithm type's needs
2. policy update: The algorithm performs one or more update steps using the collected data
3. evaluation: Periodically, the trainer uses the test collector to evaluate the current policy's performance
4. logging: Statistics from collection, updates, and evaluation are logged
5. checkpointing: The best policy (according to a scoring function) is saved

The trainer handles the detailed choreography of these steps, including determining when to collect more data,
how many update steps to perform, when to evaluate, and when to stop training (based on maximum epochs, timesteps, or early stopping criteria).

Configuration Copy Section

Trainers are configured through parameter dataclasses
({class}~tianshou.trainer.OnPolicyTrainerParams, {class}~tianshou.trainer.OffPolicyTrainerParams, {class}~tianshou.trainer.OfflineTrainerParams)
that specify in particular:

• training duration (number of epochs, steps per epoch)
• collectors for training and testing
• update frequency and batch size
• evaluation frequency
• logging and checkpointing settings
• early stopping criteria

Batch Copy Chapter

The class {class}~tianshou.data.Batch is Tianshou's flexible data structure for passing information between components.
It serves as the lingua franca of the framework, carrying everything from raw environment observations to computed returns and policy outputs.

Design Philosophy Copy Section

Batch is designed to be ...

• flexible: can contain any key-value pairs, with nested structures supported,
• numpy/torch-compatible: automatically converts lists to arrays and seamlessly works with both NumPy arrays and PyTorch tensors,
• sliceable: supports indexing and slicing operations that work across all contained data,
• composable: can be concatenated, stacked, and split to support batching operations.

Type Safety with BatchProtocol Copy Section

While Batch provides a flexible, dictionary-like structure for holding arbitrary data, this flexibility can make it challenging to statically type-check which attributes are present in a batch at any given point in the code. To address this, Tianshou uses {class}~tianshou.data.batch.BatchProtocol and derived protocols to specify the expected attributes while keeping the actual runtime type as Batch.

BatchProtocol is a Python Protocol (from typing.Protocol) that defines the interface of a Batch object, specifying which operations and attributes should be available. More importantly, Tianshou provides a rich set of derived protocols in {mod}tianshou.data.types that describe batches with specific sets of attributes commonly used throughout the framework:

• {class}~tianshou.data.types.ObsBatchProtocol: Contains obs and info - the minimal batch for policy forward passes
• {class}~tianshou.data.types.RolloutBatchProtocol: Adds obs_next, act, rew, terminated, and truncated - typical data from replay buffer sampling
• {class}~tianshou.data.types.BatchWithReturnsProtocol: Extends RolloutBatchProtocol with returns computed from rewards
• {class}~tianshou.data.types.BatchWithAdvantagesProtocol: Includes adv (advantages) and v_s (value estimates) for policy gradient methods
• {class}~tianshou.data.types.ActStateBatchProtocol: Contains act and state for policy outputs, especially with RNN support
• {class}~tianshou.data.types.ModelOutputBatchProtocol: Adds logits to action and state information
• {class}~tianshou.data.types.DistBatchProtocol: Contains action distributions (dist) for stochastic policies
• {class}~tianshou.data.types.PrioBatchProtocol: Includes weight for prioritized experience replay

These protocols serve as type hints in function signatures throughout Tianshou, making it explicit what attributes are expected and available. For example, a policy's forward method might accept an ObsBatchProtocol and return an ActStateBatchProtocol, clearly documenting the data contract. Despite these type annotations, the actual objects remain flexible Batch instances at runtime, preserving Tianshou's dynamic nature while improving code clarity and IDE support.

Common Use Cases Copy Section

Batches flow through the system carrying different types of information:

1. environment data: observations, rewards, done flags, and info from environment steps
2. policy outputs: actions, hidden states, and intermediate computations
3. training data: returns, advantages, and other computed quantities needed for learning
4. sampling results: batches sampled from the replay buffer for training

Operations Copy Section

Key operations on batches include:

• attribute access: dot notation (batch.obs) or dictionary-style access (batch['obs'])
• slicing: extract subsets with standard indexing (batch[0:10], batch[[1,3,5]])
• stacking: combine multiple batches along a new dimension
• type conversion: convert between NumPy and PyTorch with to_numpy() and to_torch()
• null handling: detect and remove null values with hasnull(), isnull(), and dropnull()

The first dimension of all data in a Batch represents the batch size, enabling vectorized operations.

Buffer Copy Chapter

A buffer (i.e. class {class}~tianshou.data.buffer.ReplayBuffer and its variants) manages the storage and retrieval of experience data.
It acts as the memory of the learning system, preserving the temporal structure of episodes while providing efficient access patterns.

Storage Structure Copy Section

Buffers store data in a circular queue fashion with a fixed maximum size. When the buffer fills, new data overwrites the oldest stored experiences.
All data is stored within a single underlying Batch object, with the buffer managing:

• pointer tracking: current insertion position
• episode boundaries: which transitions belong to which episodes
• temporal relationships: the sequential order of transitions

Reserved Keys Copy Section

Buffers use a standard set of keys for storing transitions:

• obs: Observation at time t
• act: Action taken at time t
• rew: Reward received at time t
• terminated: True if the episode ended naturally at time t
• truncated: True if the episode was cut off at time t (e.g., time limit)
• done: Automatically inferred as terminated or truncated
• obs_next: Observation at time t+1
• info: Additional information from the environment
• policy: Intermediate policy computations to be stored

Core Operations Copy Section

adding data: The {meth}~tianshou.data.buffer.buffer_base.ReplayBuffer.add method stores new transitions,
automatically handling episode boundaries and computing episode statistics (return, length)
when episodes complete.

sampling: The {meth}~tianshou.data.buffer.buffer_base.ReplayBuffer.sample method retrieves batches of experiences for training,
returning both the sampled batch and the corresponding indices.
The sample size can be specified, or set to 0 to retrieve all available data.

temporal navigation: The {meth}~tianshou.data.buffer.buffer_base.ReplayBuffer.prev and {meth}~tianshou.data.buffer.ReplayBuffer.next
methods enable traversal along the temporal sequence, respecting episode boundaries.
This is essential for computing n-step returns and other time-dependent quantities.

persistence: Buffers support saving and loading via pickle or HDF5 format, enabling dataset collection and offline learning.

Buffer Variants Copy Section

Tianshou provides specialized buffer types:

• {class}~tianshou.data.buffer.buffer_base.ReplayBuffer: the standard buffer for single environments
• {class}~tianshou.data.buffer.vecbuf.VectorReplayBuffer: manages separate sub-buffers for multiple parallel environments while maintaining chronological order
• {class}~tianshou.data.buffer.prio.PrioritizedReplayBuffer: samples transitions based on their TD-error or other priority metrics, using an efficient segment tree implementation

Advanced Features Copy Section

Buffers support sophisticated use cases:

• frame stacking: automatically stacks consecutive observations (useful for RNN inputs or Atari)
• memory optimization: option to skip storing next observations (useful for Atari where they can be inferred)
• multi-modal observations: handle observations with multiple components (e.g., image + vector)

Logger Copy Chapter

The {class}~tianshou.utils.logger.logger_base.BaseLogger abstraction provides a unified interface for recording and tracking training progress, metrics, and statistics.
It decouples the training loop from the specifics of where and how data is logged.

Purpose Copy Section

Loggers serve several essential functions:

• progress tracking: record timesteps, episodes, and epochs as training progresses
• metric collection: store performance indicators like rewards, losses, and success rates
• experiment organization: manage different data scopes (training, testing, updating)
• reproducibility: save training curves and hyperparameters for later analysis

Logging Scopes Copy Section

The framework organizes logged data into distinct scopes:

• train data: metrics from the training collector (episode returns, steps, collection speed)
• test data: evaluation metrics from the test collector
• update data: learning statistics from the algorithm (losses, gradients, learning rates)
• info data: additional custom metrics or metadata

Each scope has a corresponding log method (log_train_data, log_test_data, log_update_data, log_info_data) that the trainer calls at appropriate times.

Implementations Copy Section

Tianshou provides several logger implementations:

• {class}~tianshou.utils.logger.tensorboard.TensorboardLogger: writes to TensorBoard format for visualization with TensorBoard
• {class}~tianshou.utils.logger.wandb.WandbLogger: integrates with Weights & Biases for cloud-based experiment tracking

All implementations inherit from {class}~tianshou.utils.logger.logger_base.BaseLogger and share a common interface,
making it easy to switch between logging backends or use multiple loggers simultaneously.

Data Preparation Copy Section

Before writing, loggers prepare data through the prepare_dict_for_logging method, which can filter, transform, or aggregate metrics.
The write method then persists the prepared data to the logging backend with an associated step count.

How They Work Together Copy Chapter

These seven abstractions collaborate to enable reinforcement learning:

1. The Trainer initializes and orchestrates the training process.
2. The Collector uses the Policy to gather experience from environments.
3. Collected transitions are stored in the Buffer extracted as Batches.
4. The Algorithm samples from the Buffer, preprocesses the data, and updates the Policy.
5. The Logger records metrics throughout the process.
6. The cycle repeats until training completes.

This modular design allows each component to focus on its specific responsibility while maintaining clean interfaces.
You can customize individual components (e.g., implementing a new Algorithm or Buffer) without affecting the others,
making Tianshou both powerful and flexible.

File: docs/01_user_guide/index.rst Copy Chapter

User Guide

The user guide provides an introduction to core concepts, establishes the glossary of terms,
introduces Tianshou's dual API architecture and provides an overview of important abstractions.

File: docs/02_deep_dives/0_intro.md Copy Chapter

Deep Dives

Our deep dives are a collection of executable tutorials on some of the internal representations used by Tianshou.
Provided as notebooks, you can run them directly in Colab or download them to run them locally.

File: docs/04_benchmarks/benchmarks.rst Copy Chapter

Benchmarks

Tianshou's algorithm implementations lead to state-of-the-art results on standard benchmarks.

An efficient parallel implementation for
evaluating algorithms on mujoco or atari is in benchmark/run_benchmark.py. It can easily be adapted for custom
benchmarks as well. The reported results are thus completely reproducible.

The evaluation code uses Tianshou's integration with the rliable <https://github.com/google-research/rliable>_ framework,
which supports best practices for trustworthy RL evaluation.
Each experiment is conducted under 5 random seeds, we report the interquartile mean (IQM) and 95% confidence intervals over these seeds.

.. raw:: html

<select id="env-mujoco" onchange="showMujocoResults(this)"></select>

File: docs/05_developer_guide/developer_guide.md Copy Chapter

Developer Guide

The section addresses developers of Tianshou, providing information for
both casual contributors and maintainers alike.

Python Virtual Environment Copy Chapter

Tianshou is built and managed by poetry.

The development environment uses Python 3.11.

To install all relevant requirements (as well as Tianshou itself in editable mode)
you can simply call

poetry install --with dev

Depending on your setup, you may need to create and activate an empty virtual environment
using the right Python version beforehand. For instance, to do this with conda, use:

    conda create -n tianshou python=3.11
    conda activate tianshou

Code Style and Auto-Formatting Copy Chapter

When editing code in Tianshou, strive for local consistency, i.e.
adhere to the style already present in the codebase.

Tianshou uses an auto-formatting for consistency.
To apply it, call

poe format

To check whether your formatting is compliant without applying the
auto-formatter, call

poe lint

Type Checking Copy Chapter

We use mypy to perform static type analysis.
To check typing, run

poe type-check

Tests Copy Chapter

Running the Test Suite Locally Copy Section

Tianshou uses pytest. Tests are located in ./test.

To run the full set of tests locally, run

poe test

Determinism Tests Copy Section

We implemented determinism tests for Tianshou's algorithms, which allow us to determine
whether algorithms still compute exactly the same results even after large refactorings.
These tests are applied by

1. creating a behavior snapshot in the old code branch before the changes and then
2. running the respective determinism test in the new branch to ensure that the behavior is the same.

Unfortunately, full determinism is difficult to achieve across different platforms and even different
machines using the same platform an Python environment.
Therefore, these tests are not carried out in the CI pipeline.
Instead, it is up to the developer to run them locally and check the results whenever a change
is made to the codebase that could affect algorithm behavior.

Technically, the two steps are handled by setting static flags in class AlgorithmDeterminismTest and then
running either the full test suite or a specific determinism test (test_*_determinism, e.g. test_ddpg_determinism)
in the two branches to be compared.

1. On the old branch: (Temporarily) set ENABLED=True and FORCE_SNAPSHOT_UPDATE=True and run the test(s).
2. On the new branch: (Temporarily) set ENABLED=True and FORCE_SNAPSHOT_UPDATE=False and run the test(s).
3. Inspect the test results; find a summary in determinism_tests.log

Tests in CI (GitHub Actions) Copy Section

CI tests will extensively test Tianshou's functionality in multiple environments.

In particular, we test

• on Ubuntu (full functionality tested)
  • py_pinned: using the pinned development environment (Python 3.11, known versions of all dependencies)
  • py_latest: using a more recent Python version with the newest set of compatible dependencies (automatically resolved)
• on Windows and macOS (core functionality tested)

Principle of Maximum Compatibility

The idea behind testing with dynamically resolved dependencies is that we want to maximize the applicability
of Tianshou: For important dependencies that could conflict with environments used by our users, we do not restrict the version of a dependency unless there is a known incompatibility.

If incompatibilities should arise (e.g. by the "py_latest" test failing), we either

• resolve them by making the code compatible with both old and new versions OR
• add an upper bound to our dependency declarations (excluding the incompatible versions) and release a new
  version of Tianshou to make these exclusions explicit.

High-Level API Copy Chapter

The high-level API provides a declarative, user-friendly interface for setting up reinforcement learning experiments. From a library developer's perspective, it is important that this API be clearly structured and maintainable. This section explains the architectural principles and how to extend the API to support new algorithms.

Core Abstractions Copy Section

The high-level API is built around a clear separation of concerns:

Parameter Classes are dataclasses (inheriting from Params) that represent algorithm-specific configuration.
They capture hyperparameters in a high-level, user-friendly form.
Because the high-level interface must abstract away from low-level details, parameters may need transformation before being passed to policy classes.
This is handled via ParamTransformer instances, which successively transform the parameter dictionary representation.
To maintain clarity and reduce coupling, parameter transformers are co-located with the parameters they affect.
The system uses inheritance and mixins extensively to reduce duplication while maintaining flexibility.

Factories embody the principle of declarative configuration.
Because object creation may depend on other objects that don't yet exist at configuration time (e.g., neural networks depend on environment properties),
the API transitions from objects to factories.
Key factory types include:

• EnvFactory for creating training, test, and watch environments
• AgentFactory as the central factory that creates policies, trainers, and collectors
• Various specialized factories for optimizers, actors, critics, noise, distributions, learning rate schedulers, and policy wrappers

Algorithm Factories (subclasses of AlgorithmFactory) are the core components responsible for orchestrating the creation of all algorithm-specific objects.
They handle the creation of neural network architectures, apply parameter transformations, instantiate policies, and create trainers with appropriate collectors.
To support a new algorithm, this is the primary extension point.

Experiment Builders (subclasses of ExperimentBuilder) provide the user-facing interface following the builder pattern.
They contain sensible defaults while allowing customization through fluent with_* methods.
Builder mixins provide composable functionality for common patterns (e.g., actor/critic configuration), avoiding code duplication across algorithm-specific implementations.

Supporting a New Algorithm Copy Section

Extending the high-level API to support a new algorithm involves creating three main components:

Parameter Class: Define a dataclass in tianshou/highlevel/params/algorithm_params.py that inherits from appropriate base classes and mixins.
The choice of base class depends on the algorithm's architecture (actor-critic, single network, etc.) and learning paradigm (on-policy, off-policy).
Override _get_param_transformers() to specify how high-level parameters should be transformed for the low-level policy API.
Common transformers handle optimizer creation, noise instantiation, and environment-dependent parameter resolution.

Algorithm Factory: Implement a subclass of AlgorithmFactory in tianshou/highlevel/algorithm.py.
In most cases, inherit from existing base factories like ActorCriticOnPolicyAlgorithmFactory, ActorCriticOffPolicyAlgorithmFactory,
or DiscreteCriticOnlyOffPolicyAlgorithmFactory, which handle common creation patterns.
The primary requirement is implementing _get_algorithm_class() to return the appropriate algorithm class.
For algorithms with non-standard requirements, override _create_algorithm(), _create_kwargs(), etc. to customize the instantiation logic.

Experiment Builder: Add a builder class in tianshou/highlevel/experiment.py that inherits from OnPolicyExperimentBuilder or OffPolicyExperimentBuilder
along with appropriate mixins. The mixins provide standard functionality for configuring actors and critics
(single critic, dual critics, critic ensembles, parameter sharing patterns, etc.).
The main responsibility is implementing _create_algorithm_factory() to instantiate the algorithm factory with appropriate parameters and network factories.
Optionally provide with_* methods for algorithm-specific configuration.

Export the new classes in tianshou/highlevel/__init__.py to make them available to users.

Design Principles Copy Section

The architecture follows several key principles:

Separation of Concerns: Configuration is cleanly separated from implementation.
The transformation system bridges these layers while maintaining independence.

Declarative Configuration: Factories enable a declarative style where experiments are defined by what should be created rather than imperative steps.
This makes experiments easily serializable and reproducible.

Composition and Inheritance: Mixins and inheritance reduce code duplication.
Common functionality is factored into reusable components while maintaining flexibility for algorithm-specific requirements.

Progressive Disclosure: The API provides sensible defaults for simple use cases while allowing deep customization when needed.
Users can progress from simple configurations to advanced setups without fighting the abstractions.

Co-location: Related code is kept together. Parameter transformers are defined near the parameters they transform,
maintaining clarity about dependencies and making the codebase easier to navigate.

Type Safety: Extensive use of generics and type hints ensures that type checkers can catch configuration errors at development time rather than runtime.

Documentation Copy Chapter

Documentation is in the docs/ directory, using Markdown (.md), ReStructuredText (.rst) and notebook files.
index.rst is the main page.

API References are automatically generated by Sphinx according to the outlines under docs/api/ and should be modified when any code changes.

To compile documentation into webpage, run

poe doc-build

The generated webpages can subsequently be found in docs/_build and can be viewed with any browser.

Verifications Copy Section

We have several automated verification methods for documentation:

1. pydocstyle (as part of ruff): tests all docstring under tianshou/;

2. doc8 (as part of ruff): tests ReStructuredText format;

3. sphinx spelling and test: test if there is any error/warning when generating front-end html documentation.

Creating a Release Copy Chapter

To release a new version on PyPI,

• set the version to be released in tianshou/__init__.py and in pyproject.toml, creating a commit
• tag the commit with the version (using the format v1.2.3)
• push the commit (git push) and the tag (git push --tags)
• create a new release on GitHub based on the tag; this will trigger the release job for PyPI.

In the past, we provided releases to conda-forge as well, but this is currently not maintained.

File: docs/06_contributors/contributors.rst Copy Chapter

Contributors

We always welcome contributions to help make Tianshou better!
Tianshou was originally created by the THU-ML Group <https://ml.cs.tsinghua.edu.cn>_ at Tsinghua University.

Today, it is backed by the appliedAI Institute for Europe <https://www.appliedai-institute.de/en/>_,
a non-profit organization committed to making Tianshou the go-to resource for reinforcement learning research and development,
guaranteeing its long-term maintenance and support.

The original creator Jiayi Weng (Trinkle23897 <https://github.com/Trinkle23897>_) continues
to be involved in Tianshou development.

The current core developers, who are behind the v1.0 and v2.0 releases of Tianshou, are:

• Dominik Jain (opcode81 <https://github.com/opcode81>_)
• Michael Panchenko (MischaPanch <https://github.com/MischaPanch>_)

An incomplete list of early contributors is:

• Alexis Duburcq (duburcqa <https://github.com/duburcqa>_)
• Kaichao You (youkaichao <https://github.com/youkaichao>_)
• Huayu Chen (ChenDRAG <https://github.com/ChenDRAG>_)
• Yi Su (nuance1979 <https://github.com/nuance1979>_)

You can find more information about contributors here <https://github.com/thu-ml/tianshou/graphs/contributors>_.

File: examples/atari/README.md Copy Chapter

Atari Environment

EnvPool Copy Chapter

We highly recommend using envpool to run the following experiments. To install, in a linux machine, type:

pip install envpool

After that, atari_wrapper will automatically switch to envpool's Atari env. EnvPool's implementation is much faster (
about 2~3x faster for pure execution speed, 1.5x for overall RL training pipeline) than python vectorized env
implementation, and it's behavior is consistent to that approach (OpenAI wrapper), which will describe below.

For more information, please refer to
EnvPool's GitHub, Docs,
and 3rd-party report.

ALE-py Copy Chapter

The sample speed is ~3000 env step per second (~12000 Atari frame per second in fact since we use frame_stack=4) under
the normal mode (use a CNN policy and a collector, also storing data into the buffer).

The env wrapper is a crucial thing. Without wrappers, the agent cannot perform well enough on Atari games. Many existing
RL codebases use OpenAI wrapper,
but it is not the original DeepMind version (related issue). Dopamine
has a different wrapper but
unfortunately it cannot work very well in our codebase.

DQN (single run)

One epoch here is equal to 100,000 env step, 100 epochs stand for 10M.

Note: The eps_train_final and eps_test in the original DQN paper is 0.1 and 0.01,
but some works found that smaller eps helps improve the
performance. Also, a large batchsize (say 64 instead of 32) will help faster convergence but will slow down the training
speed.

We haven't tuned this result to the best, so have fun with playing these hyperparameters!

C51 (single run)

One epoch here is equal to 100,000 env step, 100 epochs stand for 10M.

Note: The selection of n_step is based on Figure 6 in the Rainbow paper.

QRDQN (single run)

One epoch here is equal to 100,000 env step, 100 epochs stand for 10M.

IQN (single run)

One epoch here is equal to 100,000 env step, 100 epochs stand for 10M.

FQF (single run)

One epoch here is equal to 100,000 env step, 100 epochs stand for 10M.

Rainbow (single run)

One epoch here is equal to 100,000 env step, 100 epochs stand for 10M.

PPO (single run)

One epoch here is equal to 100,000 env step, 100 epochs stand for 10M.

SAC (single run)

One epoch here is equal to 100,000 env step, 100 epochs stand for 10M.

File: examples/box2d/README.md Copy Chapter

Bipedal-Hardcore-SAC

• Our default choice: remove the done flag penalty, will soon converge to ~280 reward within 100 epochs (10M env steps,
  3~4 hours, see the image below)
• If the done penalty is not removed, it converges much slower than before, about 200 epochs (20M env steps) to reach
  the same performance (~200 reward)

BipedalWalker-BDQ

• To demonstrate the cpabilities of the BDQ to scale up to big discrete action spaces, we run it on a discretized
  version of the BipedalWalker-v3 environment, where the number of possible actions in each dimension is 25, for a total
  of 25^4 = 390 625 possible actions. A usaual DQN architecture would use 25^4 output neurons for the Q-network, thus
  scaling exponentially with the number of action space dimensions, while the Branching architecture scales linearly and
  uses only 25*4 output neurons.

File: examples/inverse/README.md Copy Chapter

Inverse Reinforcement Learning

In inverse reinforcement learning setting, the agent learns a policy from interaction with an environment without reward
and a fixed dataset which is collected with an expert policy.

Continuous control Copy Chapter

Once the dataset is collected, it will not be changed during training. We
use d4rl datasets to train agent for continuous control. You can refer
to d4rl to see how to use d4rl datasets.

We provide implementation of GAIL algorithm for continuous control.

Train Copy Section

You can parse d4rl datasets into a ReplayBuffer , and set it as the parameter expert_buffer of GAILPolicy.
irl_gail.py is an example of inverse RL using the d4rl dataset.

To train an agent with BCQ algorithm:

python irl_gail.py --task HalfCheetah-v2 --expert-data-task halfcheetah-expert-v2

GAIL (single run) Copy Chapter

File: examples/modelbased/README.md Copy Chapter

PSRL

NChain-v0: python3 psrl.py --task NChain-v0 --epoch_num_steps 10 --rew-mean-prior 0 --rew-std-prior 1

FrozenLake-v0:
python3 psrl.py --task FrozenLake-v0 --epoch_num_steps 1000 --rew-mean-prior 0 --rew-std-prior 1 --add-done-loop --epoch 20

Taxi-v3: python3 psrl.py --task Taxi-v3 --epoch_num_steps 1000 --rew-mean-prior 0 --rew-std-prior 2 --epoch 20

File: examples/mujoco/README.md Copy Chapter

Tianshou's Mujoco Benchmark

We benchmarked Tianshou algorithm implementations in 9 out of 13 environments from the MuJoCo Gym task
suite^[1].

For each supported algorithm and supported mujoco environments, we provide:

• Default hyperparameters used for benchmark and scripts to reproduce the benchmark;
• A comparison of performance (or code level details) with other open source implementations or classic papers;
• Graphs and raw data that can be used for research purposes^[2];
• Log details obtained during training^[2];
• Pretrained agents^[2];
• Some hints on how to tune the algorithm.

Supported algorithms are listed below:

• Deep Deterministic Policy Gradient (DDPG), commit id
• Twin Delayed DDPG (TD3), commit id
• Soft Actor-Critic (SAC), commit id
• REINFORCE algorithm, commit id
• Natural Policy Gradient, commit id
• Advantage Actor-Critic (A2C), commit id
• Proximal Policy Optimization (PPO), commit id
• Trust Region Policy Optimization (TRPO), commit id
• Hindsight Experience Replay (HER)

EnvPool Copy Chapter

We highly recommend using envpool to run the following experiments. To install, in a linux machine, type:

pip install envpool

After that, make_mujoco_env will automatically switch to envpool's Mujoco env. EnvPool's implementation is much
faster (about 2~3x faster for pure execution speed, 1.5x for overall RL training pipeline in average) than python
vectorized env implementation, and it's behavior is consistent to gym's Mujoco env.

For more information, please refer to EnvPool's GitHub
and Docs.

Usage Copy Chapter

Run

$ python mujoco_sac.py --task Ant-v3

Logs is saved in ./log/ and can be monitored with tensorboard.

$ tensorboard --logdir log

You can also reproduce the benchmark (e.g. SAC in Ant-v3) with the example script we provide under examples/mujoco/:

$ ./run_experiments.sh Ant-v3 sac

This will start 10 experiments with different seeds.

Now that all the experiments are finished, we can convert all tfevent files into csv files and then try plotting the
results.

# generate csv
$ ./tools.py --root-dir ./results/Ant-v3/sac
# generate figures
$ ./plotter.py --root-dir ./results/Ant-v3 --shaded-std --legend-pattern "\\w+"
# generate numerical result (support multiple groups: `--root-dir ./` instead of single dir)
$ ./analysis.py --root-dir ./results --norm

Example benchmark Copy Chapter

Other graphs can be found under examples/mujuco/benchmark/

For pretrained agents, detailed graphs (single agent, single game) and log details, please refer
to https://cloud.tsinghua.edu.cn/d/f45fcfc5016043bc8fbc/.

Offpolicy algorithms Copy Chapter

Notes Copy Section

1. In offpolicy algorithms (DDPG, TD3, SAC), the shared hyperparameters are almost the same, and unless otherwise
   stated, hyperparameters are consistent with those used for benchmark in SpinningUp's implementations (e.g. we use
   batchsize 256 in DDPG/TD3/SAC while SpinningUp use 100. Minor difference also lies with start-timesteps, data loop
   method collection_step_num_env_steps, method to deal with/bootstrap truncated steps because of timelimit and
   unfinished/collecting episodes (contribute to performance improvement), etc.).
2. By comparison to both classic literature and open source implementations (e.g.,
   SpinningUp)^[1][2], Tianshou's implementations of DDPG, TD3, and SAC
   are roughly at-parity with or better than the best reported results for these algorithms, so you can definitely use
   Tianshou's benchmark for research purposes.
3. We didn't compare offpolicy algorithms to OpenAI
   baselines benchmark, because for now it
   seems that they haven't provided benchmark for offpolicy algorithms, but
   in SpinningUp docs they stated that "SpinningUp
   implementations of DDPG, TD3, and SAC are roughly at-parity with the best-reported results for these algorithms", so
   we think lack of comparisons with OpenAI baselines is okay.

DDPG Copy Section

* details^[4][5][6]

TD3 Copy Section

* details^[4][5][6]

Hints for TD3

1. TD3's learning rate is set to 3e-4 while it is 1e-3 for DDPG/SAC. However, there is NO enough evidence to support our
   choice of such hyperparameters (we simply choose them because SpinningUp do so) and you can try playing with those
   hyperparameters to see if you can improve performance. Do tell us if you can!

SAC Copy Section

* details^[4][5]

Hints for SAC

1. SAC's start-timesteps is set to 10000 by default while it is 25000 is DDPG/TD3. However, there is NO enough evidence
   to support our choice of such hyperparameters (we simply choose them because SpinningUp do so) and you can try
   playing with those hyperparameters to see if you can improve performance. Do tell us if you can!
2. DO NOT share the same network with two critic networks.
3. The sigma (of the Gaussian policy) should be conditioned on input.
4. The deterministic evaluation helps a lot :)

Onpolicy Algorithms Copy Chapter

Notes Copy Section

1. In A2C and PPO, unless otherwise stated, most hyperparameters are consistent with those used for benchmark
   in ikostrikov/pytorch-a2c-ppo-acktr-gail.
2. Gernally speaking, by comparison to both classic literature and open source implementations (e.g., OPENAI
   Baselines)^[1][2], Tianshou's implementations of REINFORCE, A2C, PPO
   are better than the best reported results for these algorithms, so you can definitely use Tianshou's benchmark for
   research purposes.

REINFORCE Copy Section

* details^[4][5]

Hints for REINFORCE Copy Section

1. Following Andrychowicz, Marcin, et al, we downscale last layer of policy network
   by a factor of 0.01 after orthogonal initialization.
2. We choose "tanh" function to squash sampled action from range (-inf, inf) to (-1, 1) rather than usually used
   clipping method (As in StableBaselines3). We did full scale ablation studies and results show that tanh squashing
   performs a tiny little bit better than clipping overall, and is much better than no action bounding. However, "clip"
   method is still a very good method, considering its simplicity.
3. We use global observation normalization and global rew-to-go (value) normalization by default. Both are crucial to
   good performance of REINFORCE algorithm. Since we minus mean when doing rew-to-go normalization, you can treat global
   mean of rew-to-go as a naive version of "baseline".
4. Since we do not have a value estimator, we use global rew-to-go mean to bootstrap truncated steps because of
   timelimit and unfinished collecting, while most other implementations use 0. We feel this would help because mean is
   more likely a better estimate than 0 (no ablation study has been done).
5. We have done full scale ablation study on learning rate and lr decay strategy. We experiment with lr of 3e-4, 5e-4,
   1e-3, each have 2 options: no lr decay or linear decay to 0. Experiments show that 3e-4 learning rate will cause
   slowly learning and make agent step in local optima easily for certain environments like InvertedDoublePendulum, Ant,
   HalfCheetah, and 1e-3 lr helps a lot. However, after training agents with lr 1e-3 for 5M steps or so, agents in
   certain environments like InvertedPendulum will become unstable. Conclusion is that we should start with a large
   learning rate and linearly decay it, but for a small initial learning rate or if you only train agents for limited
   timesteps, DO NOT decay it.
6. We didn't tune step-per-collect option and training-num option. Default values are finetuned with PPO algorithm
   so we assume they are also good for REINFORCE. You can play with them if you want, but remember that buffer-size
   should always be larger than step-per-collect, and if step-per-collect is too small and training-num too large,
   episodes will be truncated and bootstrapped very often, which will harm performance. If training-num is too small (
   e.g., less than 8), speed will go down.
7. Sigma of action is not fixed (normally seen in other implementation) or conditioned on observation, but is an
   independent parameter which can be updated by gradient descent. We choose this setting because it works well in PPO,
   and is recommended by Andrychowicz, Marcin, et al. See Fig. 23.

A2C Copy Section

* details^[4][5]

Hints for A2C

1. We choose clip action method in A2C instead of tanh option as used in REINFORCE simply to be consistent with
   original implementation. tanh may be better or equally well but we didn't have a try.
2. (Initial) learning rate, lr_decay, step-per-collect and training-num affect the performance of A2C to a great
   extend. These 4 hyperparameters also affect each other and should be tuned together. We have done full scale ablation
   studies on these 4 hyperparameters (more than 800 agents have been trained). Below are our findings.
3. step-per-collect / training-num are equal to bootstrap-lenghth, which is the max length of an "episode" used in
   GAE estimator and 80/16=5 in default settings. When bootstrap-lenghth is small, (maybe) because GAE can look
   forward at most 5 steps and use bootstrap strategy very often, the critic is less well-trained leading the actor to a
   not very high score. However, if we increase step-per-collect to increase bootstrap-lenghth (e.g. 256/16=16),
   actor/critic will be updated less often, resulting in low sample efficiency and slow training process. To conclude,
   If you don't restrict env timesteps, you can try using larger bootstrap-lenghth and train with more steps to get a
   better converged score. Train slower, achieve higher.
4. The learning rate 7e-4 with decay strategy is appropriate for step-per-collect=80 and training-num=16. But if you
   use a larger step-per-collect(e.g. 256 - 2048), 7e-4 is a little bit small for lr because each update will have
   more data, less noise and thus smaller deviation in this case. So it is more appropriate to use a higher learning
   rate (e.g. 1e-3) to boost performance in this setting. If plotting results arise fast in early stages and become
   unstable later, consider lr decay first before decreasing lr.
5. max-grad-norm didn't really help in our experiments. We simply keep it for consistency with other open-source
   implementations (e.g. SB3).
6. Although original paper of A3C uses RMSprop optimizer, we found that Adam with the same learning rate worked equally
   well. We use RMSprop anyway. Again, for consistency.
7. We noticed that the implementation of A2C in SB3 sets gae-lambda to 1 by default for no reason, and our experiments
   showed better results overall when gae-lambda was set to 0.95.
8. We found out that step-per-collect=256 and training-num=8 are also good settings. You can have a try.

PPO Copy Section

* details^[4][5]

Hints for PPO

1. Following Andrychowicz, Marcin, et al Sec 3.5, we use "recompute advantage"
   strategy, which contributes a lot to our SOTA benchmark. However, I personally don't quite agree with their
   explanation about why "recompute advantage" helps. They stated that it's because old strategy "makes it impossible to
   compute advantages as the temporal structure is broken", but PPO's update equation is designed to learn from
   slightly-outdated advantages. I think the only reason "recompute advantage" works is that it update the critic
   several times rather than just one time per update, which leads to a better value function estimation.
2. We have done full scale ablation studies of PPO algorithm's hyperparameters. Here are our findings: In Mujoco
   settings, value-clip and norm-adv may help a litte bit in some games (e.g. norm-adv helps stabilize training in
   InvertedPendulum-v2), but they make no difference to overall performance. So in our benchmark we do not use such
   tricks. We validate that setting ent-coef to 0.0 rather than 0.01 will increase overall performance in mujoco
   environments. max-grad-norm still offers no help for PPO algorithm, but we still keep it for consistency.
3. Andrychowicz, Marcin, et al's work indicates that using gae-lambda 0.9 and
   changing policy network's width based on which game you play (e.g. use [16, 16] hidden-sizes for actor network in
   HalfCheetah and [256, 256] for Ant) may help boost performance. Our ablation studies say otherwise: both options may
   lead to equal or lower performance overall in our experiments. We are not confident about this claim because we
   didn't change learning rate and other maybe-correlated factors in our experiments. So if you want, you can still have
   a try.
4. batch-size 128 and 64 (default) work equally well. Changing training-num alone slightly (maybe in range [8, 128])
   won't affect performance. For bound action method, both clip and tanh work quite well.
5. In OPENAI implementations of PPO, they multiply value loss with a factor of 0.5 for no good reason (see
   this issue). We do not do so and therefore
   make our vf-coef 0.25 (half of standard 0.5). However, since value loss is only used to optimize critic network,
   setting different vf-coef should in theory make no difference if using Adam optimizer.

TRPO Copy Section

* details^[4][5]

Hints for TRPO

1. We have tried step-per-collect in (80, 1024, 2048, 4096), and training-num in (4, 16, 32, 64), and found out 1024
   for step-per-collect (same as OpenAI Baselines) and smaller training-num (below 16) are good choices. Set
   training-num to 4 is actually better but we still use 16 considering the boost of training speed.
2. Advantage normalization is a standard trick in TRPO, but we found it of minor help, just like in PPO.
3. Larger optim-critic-iters (than 5, as used in OpenAI Baselines) helps in most environments. Smaller lr and lr_decay
   strategy also help a tiny little bit for performance.
4. gae-lambda 0.98 and 0.95 work equally well.
5. We use GAE returns (GAE advantage + value) as the target of critic network when updating, while people usually tend
   to use reward to go (lambda = 0.) as target. We found that they work equally well although using GAE returns is a
   little bit inaccurate (biased) by math.
6. Empirically, Swimmer-v3 usually requires larger bootstrap lengths and learning rate. Humanoid-v3 and
   InvertedPendulum-v2, however, are on the opposite.
7. In contrast, with the statement made in TRPO paper, we found that backtracking in line search is rarely used at least
   in Mujoco settings, which is actually unimportant. This makes TRPO algorithm actually the same as TNPG algorithm (
   described in this paper). This also explains why TNPG and TRPO's
   plotting results look so similar in that paper.
8. "recompute advantage" is helpful in PPO but doesn't help in TRPO.

NPG Copy Section

* details^[4][5]

Hints for NPG

1. All shared hyperparameters are exactly the same as TRPO, regarding how similar these two algorithms are.
2. We found different games in Mujoco may require quite different actor-step-size: Reacher/Swimmer are insensitive to
   step-size in range (0.11.0), while InvertedDoublePendulum / InvertedPendulum / Humanoid are quite sensitive to step
   size, and even 0.1 is too large. Other games may require actor-step-size in range (0.10.4), but aren't that
   sensitive in general.

Others Copy Chapter

HER Copy Section

Hints for HER

1. The HER technique is proposed for solving task-based environments, so it cannot be compared with non-task-based
   mujoco benchmarks. The environment used in this evaluation is FetchReach-v3 which requires an
   extra installation.
2. Simple hyperparameters optimizations are done for both settings, DDPG with and without HER. However, since DDPG
   without HER failed in every experiment, the best hyperparameters for DDPG with HER are used in the evaluation of
   both settings.
3. The scores are the mean reward ± 1 standard deviation of 16 seeds. The minimum reward for FetchReach-v3 is -50
   which we can imply that DDPG without HER performs as good as a random policy. DDPG with HER although has a better
   mean reward, the standard deviation is quite high. This is because in this setting, the agent will either fail
   completely (-50 reward) or successfully learn the task (close to 0 reward). This means that the agent successfully
   learned in about 70% of the 16 seeds.

Note Copy Chapter

[1] Supported environments include HalfCheetah-v3, Hopper-v3, Swimmer-v3, Walker2d-v3, Ant-v3,
Humanoid-v3, Reacher-v2, InvertedPendulum-v2 and InvertedDoublePendulum-v2. Pusher, Thrower, Striker and HumanoidStandup
are not supported because they are not commonly seen in literatures.

[2] Pretrained agents, detailed graphs (single agent, single game) and log details can all be
found at Google Drive.

[3] We used the latest version of all mujoco environments in gym (0.17.3 with mujoco==2.0.2.13),
but it's not often the case with other benchmarks. Please check for details yourself in the original paper. (Different
version's outcomes are usually similar, though)

[4] ~ means the number is approximated from the graph because accurate numbers is not provided
in the paper. N means graphs not provided.

[5] Reward metric: The meaning of the table value is the max average return over 10 trails (
different seeds) ± a single standard deviation over trails. Each trial is averaged on another 10 test seeds. Only the
first 1M steps data will be considered, if not otherwise stated. The shaded region on the graph also represents a single
standard deviation. It is the same as TD3 evaluation method.

[6] In TD3 paper, shaded region represents only half of standard deviation.

[7] Comparing Tianshou's REINFORCE algorithm with SpinningUp's VPG is quite unfair because
SpinningUp's VPG uses a generative advantage estimator (GAE) which requires a dnn value predictor (critic network),
which makes so called "VPG" more like A2C (advantage actor critic) algorithm. Even so, you can see that we are roughly
at-parity with each other even if tianshou's REINFORCE do not use a critic or GAE.

File: examples/offline/README.md Copy Chapter

Offline

In offline reinforcement learning setting, the agent learns a policy from a fixed dataset which is collected once with
any policy. And the agent does not interact with environment anymore.

Continuous control Copy Chapter

Once the dataset is collected, it will not be changed during training. We
use d4rl datasets to train offline agent for continuous control. You can refer
to d4rl to see how to use d4rl datasets.

We provide implementation of BCQ and CQL algorithm for continuous control.

Train Copy Section

Tianshou provides an offline_trainer for offline reinforcement learning. You can parse d4rl datasets into a
ReplayBuffer , and set it as the parameter buffer of offline_trainer. d4rl_bcq.py is an example of offline RL
using the d4rl dataset.

Results Copy Chapter

IL (Imitation Learning, aka, Behavior Cloning) Copy Section

BCQ Copy Section

CQL Copy Section

TD3+BC Copy Section

Observation normalization

Following the original paper, we use observation normalization by default. You can turn it off by setting
--norm-obs 0. The difference are small but consistent.

Discrete control Copy Chapter

For discrete control, we currently use ad hoc Atari data generated from a trained QRDQN agent.

Gather Data Copy Section

To running CQL algorithm on Atari, you need to do the following things:

• Train an expert, by using the command listed in the QRDQN section of Atari examples:
  python3 atari_qrdqn.py --task {your_task}
• Generate buffer with noise:
  python3 atari_qrdqn.py --task {your_task} --watch --resume-path log/{your_task}/qrdqn/policy.pth --eps-test 0.2 --buffer-size 1000000 --save-buffer-name expert.hdf5 (
  note that 1M Atari buffer cannot be saved as .pkl format because it is too large and will cause error);
• Train offline model: python3 atari_{bcq,cql,crr}.py --task {your_task} --load-buffer-name expert.hdf5.

IL Copy Section

We test our IL implementation on two example tasks (different from author's version, we use v4 instead of v0; one epoch
means 10k gradient step):

BCQ Copy Section

We test our BCQ implementation on two example tasks (different from author's version, we use v4 instead of v0; one epoch
means 10k gradient step):

CQL Copy Section

We test our CQL implementation on two example tasks (different from author's version, we use v4 instead of v0; one epoch
means 10k gradient step):

We reduce the size of the offline data to 10% and 1% of the above and get:

Buffer size 100000:

Buffer size 10000:

CRR Copy Section

We test our CRR implementation on two example tasks (different from author's version, we use v4 instead of v0; one epoch
means 10k gradient step):

Note that CRR itself does not work well in Atari tasks but adding CQL loss/regularizer helps.

RL Unplugged Data Copy Section

We provide a script to convert the Atari datasets
of RL Unplugged to Tianshou ReplayBuffer.

For example, the following command will download the first shard of the first run of Breakout game to
~/.rl_unplugged/datasets/Breakout/run_1-00001-of-00100 then convert it to a tianshou.data.ReplayBuffer and save it
to ~/.rl_unplugged/buffers/Breakout/run_1-00001-of-00100.hdf5 (use --dataset-dir and --buffer-dir to change the
default directories):

python3 convert_rl_unplugged_atari.py --task Breakout --run-id 1 --shard-id 1

Then you can use it to train an agent by:

python3 atari_bcq.py --task BreakoutNoFrameskip-v4 --load-buffer-name ~/.rl_unplugged/datasets/Breakout/run_1-00001-of-00100.hdf5 --buffer-from-rl-unplugged --epoch 12

Note:

• Each shard contains about 500k transitions.
• This conversion script depends on Tensorflow.
• It takes about 1 hour to process one shard on my machine. YMMV.

File: examples/vizdoom/README.md Copy Chapter

ViZDoom

ViZDoom is a popular RL env for a famous first-person shooting game Doom. Here we
provide some results and intuitions for this scenario.

EnvPool Copy Chapter

We highly recommend using envpool to run the following experiments. To install, in a linux machine, type:

pip install envpool

After that, make_vizdoom_env will automatically switch to envpool's ViZDoom env. EnvPool's implementation is much
faster (about 2~3x faster for pure execution speed, 1.5x for overall RL training pipeline) than python vectorized env
implementation.

For more information, please refer to EnvPool's GitHub
and Docs.

Train Copy Chapter

To train an agent:

python3 vizdoom_c51.py --task {D1_basic|D2_navigation|D3_battle|D4_battle2}

D1 (health gathering) should finish training (no death) in less than 500k env step (5 epochs);

D3 can reach 1600+ reward (75+ killcount in 5 minutes);

D4 can reach 700+ reward. Here is the result:

(episode length, the maximum length is 2625 because we use frameskip=4, that is 10500/4=2625)

(episode reward)

To evaluate an agent's performance:

python3 vizdoom_c51.py --num_test_envs 100 --resume-path policy.pth --watch --task {D1_basic|D3_battle|D4_battle2}

To save .lmp files for recording:

python3 vizdoom_c51.py --save-lmp --num_test_envs 100 --resume-path policy.pth --watch --task {D1_basic|D3_battle|D4_battle2}

it will store lmp file in lmps/ directory. To watch these lmp files (for example, d3 lmp):

python3 replay.py maps/D3_battle.cfg episode_8_25.lmp

We provide two lmp files (d3 best and d4 best) under results/c51, you can use the following command to enjoy:

python3 replay.py maps/D3_battle.cfg results/c51/d3.lmp
python3 replay.py maps/D4_battle2.cfg results/c51/d4.lmp

Maps Copy Chapter

See maps/README.md

Reward Copy Chapter

1. living reward is bad
2. combo-action is really important
3. negative reward for health and ammo2 is really helpful for d3/d4
4. only with positive reward for health is really helpful for d1
5. remove MOVE_BACKWARD may converge faster but the final performance may be lower

Algorithms Copy Chapter

The setting is exactly the same as Atari. You can definitely try more algorithms listed in Atari example.

C51 (single run) Copy Section

PPO (single run) Copy Section

PPO with ICM (single run) Copy Section

File: examples/vizdoom/maps/README.md Copy Chapter

D1-D4 maps are from https://github.com/intel-isl/DirectFuturePrediction/

More maps and cfgs: https://github.com/mwydmuch/ViZDoom/tree/master/scenarios

File: .devcontainer/devcontainer.json Copy Chapter

{
"name": "Tianshou",
"dockerFile": "../Dockerfile",
"workspaceFolder": "/workspaces/tianshou",
"runArgs": ["--shm-size=1g"],
"customizations": {
"vscode": {
"settings": {
"terminal.integrated.shell.linux": "/bin/bash",
"python.pythonPath": "/usr/local/bin/python"
},
"extensions": [
"ms-python.python",
"ms-toolsai.jupyter",
"ms-python.vscode-pylance"
]
}
},
"forwardPorts": [],
"postCreateCommand": "poetry install --with dev",
"remoteUser": "root"
}

File: .github/ISSUE_TEMPLATE.md Copy Chapter

• I have marked all applicable categories:
  • exception-raising bug
  • RL algorithm bug
  • documentation request (i.e. "X is missing from the documentation.")
  • new feature request
  • design request (i.e. "X should be changed to Y.")
• I have visited the source website
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and environment, where applicable:
  python Copy

  import tianshou, gymnasium as gym, torch, numpy, sys
  print(tianshou.__version__, gym.__version__, torch.__version__, numpy.__version__, sys.version, sys.platform)

File: .github/PULL_REQUEST_TEMPLATE.md Copy Chapter

• I have added the correct label(s) to this Pull Request or linked the relevant issue(s)
• I have provided a description of the changes in this Pull Request
• I have added documentation for my changes and have listed relevant changes in CHANGELOG.md
• If applicable, I have added tests to cover my changes.
• If applicable, I have made sure that the determinism tests run through, meaning that my changes haven't influenced any aspect of training. See info in the contributing documentation.
• I have reformatted the code using poe format
• I have checked style and types with poe lint and poe type-check
• (Optional) I ran tests locally with poe test
  (or a subset of them with poe test-reduced) ,and they pass
• (Optional) I have tested that documentation builds correctly with poe doc-build

File: .github/workflows/extra_sys.yml Copy Chapter

name: Windows/MacOS

on:
pull_request:
branches:
- master
push:
branches:
- master
workflow_dispatch:
inputs:
debug_enabled:
type: boolean
required: false
default: false
jobs:
cpu-extra:
runs-on: ${{ matrix.os }}
if: "!contains(github.event.head_commit.message, 'ci skip')"
strategy:
matrix:
os: [macos-latest, windows-latest]
python-version: [3.11]
steps:
- name: Setup tmate session
uses: mxschmitt/action-tmate@v3
if: ${{ github.event_name == 'workflow_dispatch' && inputs.debug_enabled }}
- name: Cancel previous run
uses: styfle/[email protected]
with:
access_token: ${{ github.token }}
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
# use poetry and cache installed packages, see https://github.com/marketplace/actions/python-poetry-action
- name: Install poetry
uses: abatilo/actions-poetry@v2
- name: Setup a local virtual environment (if no poetry.toml file)
run: |
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
- uses: actions/cache@v3
name: Define a cache for the virtual environment based on the dependencies lock file
with:
path: ./.venv
key: venv-${{ hashFiles('poetry.lock') }}
- name: Install the project dependencies
# ugly as hell, but well...
# see https://github.com/python-poetry/poetry/issues/7611
run: poetry install --with dev || poetry install --with dev || poetry install --with dev
- name: wandb login
run: poetry run wandb login e2366d661b89f2bee877c40bee15502d67b7abef
- name: Test with pytest
run: poetry run poe test-reduced

File: .github/workflows/gputest.yml Copy Chapter

name: Ubuntu GPU

on:
pull_request:
branches:
- master
push:
branches:
- master
workflow_dispatch:
inputs:
debug_enabled:
type: boolean
required: false
default: false

jobs:
gpu:
runs-on: [self-hosted, Linux, X64]
if: "!contains(github.event.head_commit.message, 'ci skip')"
steps:
- name: Setup tmate session
uses: mxschmitt/action-tmate@v3
if: ${{ github.event_name == 'workflow_dispatch' && inputs.debug_enabled }}
- name: Cancel previous run
uses: styfle/[email protected]
with:
access_token: ${{ github.token }}
- uses: actions/checkout@v3
- name: Set up Python 3.11
uses: actions/setup-python@v4
with:
python-version: "3.11"
# use poetry and cache installed packages, see https://github.com/marketplace/actions/python-poetry-action
- name: Install poetry
uses: abatilo/actions-poetry@v2
- name: Setup a local virtual environment (if no poetry.toml file)
run: |
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
- uses: actions/cache@v3
name: Define a cache for the virtual environment based on the dependencies lock file
with:
path: ./.venv
key: venv-${{ hashFiles('poetry.lock') }}
- name: Install the project dependencies
run: |
poetry install --with dev --extras "envpool"
- name: wandb login
run: |
poetry run wandb login e2366d661b89f2bee877c40bee15502d67b7abef
- name: Test with pytest
run: |
poetry run poe test

File: .github/workflows/lint_and_docs.yml Copy Chapter

name: Check Formatting/Typing and Build Docs

on:
pull_request:
branches:
- master
push:
branches:
- master
workflow_dispatch:
inputs:
debug_enabled:
type: boolean
required: false
default: false

jobs:
check:
runs-on: ubuntu-latest
steps:
- name: Setup tmate session
uses: mxschmitt/action-tmate@v3
if: ${{ github.event_name == 'workflow_dispatch' && inputs.debug_enabled }}
- name: Cancel previous run
uses: styfle/[email protected]
with:
access_token: ${{ github.token }}
- uses: actions/checkout@v3
- name: Set up Python 3.11
uses: actions/setup-python@v4
with:
python-version: 3.11
# use poetry and cache installed packages, see https://github.com/marketplace/actions/python-poetry-action
- name: Install poetry
uses: abatilo/actions-poetry@v2
- name: Setup a local virtual environment (if no poetry.toml file)
run: |
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
- uses: actions/cache@v3
name: Define a cache for the virtual environment based on the dependencies lock file
with:
path: ./.venv
key: venv-${{ hashFiles('poetry.lock') }}
- name: Install the project dependencies
run: |
poetry install --with dev --extras "eval"
- name: Check formatting
run: poetry run poe lint
- name: Check typing
run: poetry run poe type-check
- name: Build docs
run: MYSTNB_DEBUG=1 poetry run poe doc-build
- name: Show errors (if any)
if: failure()
run: find docs/_build/reports -name "*.err.log" -exec echo "--- {} ---" ; -exec cat {} ;

File: .github/workflows/publish.yaml Copy Chapter

name: Upload Python Package

on:
release:
types: [created]

jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: 3.11
# use poetry and cache installed packages, see https://github.com/marketplace/actions/python-poetry-action
- name: Install poetry
uses: abatilo/actions-poetry@v2
- name: Setup a local virtual environment (if no poetry.toml file)
run: |
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
- name: Build and publish
env:
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_TOKEN }}
run: |
if [ -z "${POETRY_PYPI_TOKEN_PYPI}" ]; then echo "Set the PYPI_TOKEN variable in your repository secrets"; exit 1; fi
poetry publish --build

File: .github/workflows/pytest.yml Copy Chapter

name: Ubuntu

on:
pull_request:
branches:
- master
push:
branches:
- master
workflow_dispatch:
inputs:
debug_enabled:
type: boolean
required: false
default: false

This job runs the test suite in two environments:

- py_pinned: uses Python 3.11 with the existing poetry.lock file (our stable, pinned dev environment)

- py_latest: latest Python version we want to support, without the lock file to furthermore install the newest dependency versions

This ensures compatibility with both our controlled dev setup and the latest upstream packages,

helping catch issues introduced by dependency updates or newer Python versions.

jobs:
cpu:
runs-on: ubuntu-latest
if: "!contains(github.event.head_commit.message, 'ci skip')"
strategy:
fail-fast: false
matrix:
include:
- env_name: py_pinned
python-version: "3.11"
use_lock: true
- env_name: py_latest
python-version: "3.13"
use_lock: false

steps:
  - name: Setup tmate session
    uses: mxschmitt/action-tmate@v3
    if: ${{ github.event_name == 'workflow_dispatch' && inputs.debug_enabled }}

  - name: Cancel previous run
    uses: styfle/[email protected]
    with:
      access_token: ${{ github.token }}

  - uses: actions/checkout@v3

  - name: Set up Python ${{ matrix.python-version }}
    uses: actions/setup-python@v4
    with:
      python-version: ${{ matrix.python-version }}

  - name: Install poetry
    uses: abatilo/actions-poetry@v2

  - name: Setup a local virtual environment (if no poetry.toml file)
    run: |
      poetry config virtualenvs.create true --local
      poetry config virtualenvs.in-project true --local

  - name: Remove poetry.lock for latest dependency test
    if: ${{ !matrix.use_lock }}
    run: rm -f poetry.lock

  - name: Define a cache for the virtual environment based on the dependencies lock file
    if: matrix.use_lock
    uses: actions/cache@v3
    with:
      path: ./.venv
      key: venv-${{ matrix.env_name }}-${{ hashFiles('poetry.lock') }}
      restore-keys: |
        venv-${{ matrix.env_name }}-

  - name: Install the project dependencies
    run: |
      if [ "${{ matrix.env_name }}" = "py_latest" ]; then
        poetry install --with dev --extras "eval"
      else
        poetry install --with dev --extras "envpool eval"
      fi

  - name: List installed packages
    run: |
      poetry run pip list

  - name: wandb login
    run: |
      poetry run wandb login e2366d661b89f2bee877c40bee15502d67b7abef

  - name: Test with pytest
    run: |
      if [ "${{ matrix.env_name }}" = "py_pinned" ]; then
        poetry run poe test
      else
        poetry run poe test-nocov
      fi
  - name: Upload coverage to Codecov
    if: matrix.env_name == 'py_pinned'
    uses: codecov/codecov-action@v1
    with:
      token: ${{ secrets.CODECOV }}
      file: ./coverage.xml
      flags: ${{ matrix.env_name }}
      name: codecov-${{ matrix.env_name }}
      fail_ci_if_error: false