## File: README.md

## ElegantRL “小雅”: Massively Parallel Deep Reinforcement Learning

[](https://pepy.tech/project/elegantrl)
[](https://pepy.tech/project/elegantrl)
[](https://www.python.org/downloads/release/python-360/)
[](https://pypi.org/project/elegantrl/)







<a href="https://github.com/AI4Finance-Foundation/ElegantRL" target="\_blank">
	

</a>


ElegantRL is a lightweight and structurally clean reinforcement learning framework designed to express core RL algorithms with minimal complexity and maximal clarity.

The name “Elegant” reflects its philosophy:
small in dependency footprint, yet elegant in code structure.
The framework avoids unnecessary third-party libraries while maintaining modular design, mathematical transparency, and engineering readability.

ElegantRL focuses on implementing reinforcement learning algorithms in their pure form — clear, extensible, and efficient — without sacrificing performance or simplicity.



[](https://discord.gg/trsr8SXpW5)



ElegantRL ([website](https://elegantrl.readthedocs.io/en/latest/index.html)) is developed for users/developers with the following advantages:

- **Cloud-native**: follows a cloud-native paradigm through micro-service architecture and containerization, and supports [ElegantRL-Podracer](https://elegantrl.readthedocs.io/en/latest/tutorial/elegantrl-podracer.html) and [FinRL-Podracer](https://elegantrl.readthedocs.io/en/latest/tutorial/finrl-podracer.html).

- **Scalable**: fully exploits the parallelism of DRL algorithms, making it easily scale out to hundreds or thousands of computing nodes on a cloud platform, say, a [DGX SuperPOD platform](https://www.nvidia.com/en-us/data-center/dgx-superpod/) with thousands of GPUs.

- **Elastic**: allows to elastically and automatically allocate computing resources on the cloud.

- **Lightweight**: the core codes have <1,000 lines (check [Elegantrl_Helloworld](https://github.com/AI4Finance-Foundation/ElegantRL/tree/master/helloworld)).

- **Efficient**: in many testing cases (e.g., single-GPU/multi-GPU/GPU-cloud), we find it more efficient than [Ray RLlib](https://github.com/ray-project/ray). 

- **Stable**: much much much more stable than [Stable Baselines 3](https://github.com/DLR-RM/stable-baselines3) by utilizing various methods such as the Hamiltonian term.
  
- **Practical**: used in multipe projects ([FinRL](https://github.com/AI4Finance-Foundation/FinRL), [FinRL-Meta](https://github.com/AI4Finance-Foundation/FinRL-Meta), etc.)

- **Massively parallel simulations** are used in multipe projects ([FinRL](https://github.com/AI4Finance-Foundation/FinRL), etc.); therefore, the sampling speed is high since we can build many many GPU-based environments. 
  
ElegantRL implements the following model-free deep reinforcement learning (DRL) algorithms:

- **DDPG, TD3, SAC, PPO, REDQ** for continuous actions in single-agent environment,
- **DQN, Double DQN, D3QN** for discrete actions in single-agent environment,
- **QMIX, VDN, MADDPG, MAPPO, MATD3** in multi-agent environment.

For more details of DRL algorithms, please refer to the educational webpage [OpenAI Spinning Up](https://spinningup.openai.com/en/latest/).

ElegantRL supports the following simulators:

- **Isaac Gym** for massively parallel simulations,
- **OpenAI Gym, MuJoCo, PyBullet, FinRL** for benchmarking.


## Contents

- [News](#News)
- [ElegantRL-Helloworld](#ElegantRL-Helloworld)
- [File Structure](#File-Structure)
- [Experimental Demos](#Experimental-Demos)
- [Requirements](#Requirements)
- [Citation](#Citation)


## Tutorials

- [Towardsdatascience] [A New Era of Massively Parallel Simulation: A Practical Tutorial Using ElegantRL](https://medium.com/towards-data-science/a-new-era-of-massively-parallel-simulation-a-practical-tutorial-using-elegantrl-5ebc483c3385), Nov. 2, 2022.
- [MLearning.ai] [ElegantRL: Much More Stable Deep Reinforcement Learning Algorithms than Stable-Baseline3](https://medium.com/mlearning-ai/elegantrl-much-much-more-stable-than-stable-baseline3-f096533c26db), Mar. 3, 2022.
- [Towardsdatascience] [ElegantRL-Podracer: A Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning](https://elegantrl.medium.com/elegantrl-podracer-scalable-and-elastic-library-for-cloud-native-deep-reinforcement-learning-bafda6f7fbe0), Dec. 11, 2021.
- [Towardsdatascience] [ElegantRL: Mastering PPO Algorithms](https://medium.com/@elegantrl/elegantrl-mastering-the-ppo-algorithm-part-i-9f36bc47b791), May. 3, 2021.
- [MLearning.ai] [ElegantRL Demo: Stock Trading Using DDPG (Part II)](https://medium.com/mlearning-ai/elegantrl-demo-stock-trading-using-ddpg-part-ii-d3d97e01999f), Apr. 19, 2021.
- [MLearning.ai] [ElegantRL Demo: Stock Trading Using DDPG (Part I)](https://elegantrl.medium.com/elegantrl-demo-stock-trading-using-ddpg-part-i-e77d7dc9d208), Mar. 28, 2021.
- [Towardsdatascience] [ElegantRL-Helloworld: A Lightweight and Stable Deep Reinforcement Learning Library](https://towardsdatascience.com/elegantrl-a-lightweight-and-stable-deep-reinforcement-learning-library-95cef5f3460b), Mar. 4, 2021.

## ElegantRL-Helloworld



For beginners, we maintain [ElegantRL-Helloworld](https://github.com/AI4Finance-Foundation/ElegantRL/tree/master/helloworld) as a tutorial. Its goal is to get hands-on experience with ELegantRL.

- Run the [tutorial code and learn about RL algorithms in this order: DQN -> DDPG -> PPO](https://github.com/AI4Finance-Foundation/ElegantRL/tree/master/elegantrl_helloworld)
- Write the [suggestion for Eleagant_HelloWorld in github issue](https://github.com/AI4Finance-Foundation/ElegantRL/issues/135).

One sentence summary: an agent (_agent.py_) with Actor-Critic networks (_net.py_) is trained (_run.py_) by interacting with an environment (_env.py_).

## File Structure

- **elegantrl** # main folder
  - **agents**  # a collection of DRL algorithms
  	- AgentXXX.py  # a collection of one kind of DRL algorithms
  	- **net.py**  # a collection of network architectures
  - **envs** # a collection of environments
  	- XxxEnv.py  # a training environment for RL
  - train # a collection of training programs
      	- demo.py  # a collection of demos
  	- config.py  # configurations (hyper-parameter)
  	- **run.py**  # training loop
  	- worker.py  # the worker class (explores the env, saving the data to replay buffer)
  	- learner.py  # the learner class (update the networks, using the data in replay buffer)
  	- evaluator.py  # the evaluator class (evaluate the cumulative rewards of policy network)
  	- replay_buffer.py # the buffer class (save sequences of transitions for training)
- **elegantrl_helloworld** # tutorial version
  - config.py  # configurations (hyper-parameter)
  - **agent.py**  # DRL algorithms
  - **net.py**  # network architectures 
  - **run.py**  # training loop
  - **env.py**  # environments for RL training

- **examples** # a collection of example codes
- **ready-to-run Google-Colab notebooks**
  - quickstart_Pendulum_v1.ipynb
  - tutorial_BipedalWalker_v3.ipynb
  - tutorial_Creating_ChasingVecEnv.ipynb
  - tutorial_LunarLanderContinuous_v2.ipynb
- **unit_tests** # a collection of tests

## Experimental Demos

### More efficient than Ray RLlib

Experiments on Ant (MuJoCo), Humainoid (MuJoCo), Ant (Isaac Gym), Humanoid (Isaac Gym) # from left to right



ElegantRL fully supports Isaac Gym that runs massively parallel simulation (e.g., 4096 sub-envs) on one GPU.

### More stable than Stable-baseline 3

Experiment on Hopper-v2 # ElegantRL achieves much smaller variance (average over 8 runs).

Also, PPO+H in ElegantRL completed the training process of 5M samples about 6x faster than Stable-Baseline3.



## Testing and Contributing

Our tests are written with the built-in `unittest` Python module for easy access. In order to run a specific test file (for example, `test_training_agents.py`), use the following command from the root directory:

    python -m unittest unit_tests/test_training_agents.py

In order to run all the tests sequentially, you can use the following command:

    python -m unittest discover

Please note that some of the tests require [Isaac Gym](https://developer.nvidia.com/isaac-gym) to be installed on your system. If it is not, any tests related to Isaac Gym will fail.

We welcome any contributions to the codebase, but we ask that you please **do not** submit/push code that breaks the tests. Also, please shy away from modifying the tests just to get your proposed changes to pass them. As it stands, the tests on their own are quite minimal (instantiating environments, training agents for one step, etc.), so if they're breaking, it's almost certainly a problem with your code and not with the tests.

We're actively working on refactoring and trying to make the codebase cleaner and more performant as a whole. If you'd like to help us clean up some code, we'd strongly encourage you to also watch [Uncle Bob's clean coding lessons](https://www.youtube.com/playlist?list=PLmmYSbUCWJ4x1GO839azG_BBw8rkh-zOj) if you haven't already.

## Requirements

    Necessary:
    | Python 3.6+     |
    | PyTorch 1.6+    |

    Not necessary:
    | Numpy 1.18+     | For ReplayBuffer. Numpy will be installed along with PyTorch.
    | gym 0.17.0      | For env. Gym provides tutorial env for DRL training. (env.render() bug in gym==0.18 pyglet==1.6. Change to gym==0.17.0, pyglet==1.5)
    | pybullet 2.7+   | For env. We use PyBullet (free) as an alternative of MuJoCo (not free).
    | box2d-py 2.3.8  | For gym. Use pip install Box2D (instead of box2d-py)
    | matplotlib 3.2  | For plots.

    pip3 install gym==0.17.0 pybullet Box2D matplotlib # or pip install -r requirements.txt
    
    To install StarCraftII env,
    bash ./elegantrl/envs/installsc2.sh
    pip install -r sc2_requirements.txt

## Citation:

To cite this repository:

```
@misc{erl,
  author = {Liu, Xiao-Yang and Li, Zechu and Zhu, Ming and Wang, Zhaoran and Zheng, Jiahao},
  title = {{ElegantRL}: Massively Parallel Framework for Cloud-native Deep Reinforcement Learning},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/AI4Finance-Foundation/ElegantRL}},
}
```

```
@article{liu2021elegantrl,
  title={ElegantRL-Podracer: Scalable and elastic library for cloud-native deep reinforcement learning},
  author={Liu, Xiao-Yang and Li, Zechu and Yang, Zhuoran and Zheng, Jiahao and Wang, Zhaoran and Walid, Anwar and Guo, Jian and Jordan, Michael I},
  journal={NeurIPS, Workshop on Deep Reinforcement Learning},
  year={2021}
}
```

---

## File: docs/source/api/config.rst

Configuration: *config.py*
==========================


``Arguments``
---------------------

The ``Arguments`` class contains all parameters of the training process, including environment setup, model training, model evaluation, and resource allocation. It provides users an unified interface to customize the training process. 

The class should be initialized at the start of the training process. For example,

.. code-block:: python

   from elegantrl.train.config import Arguments
   from elegantrl.agents.AgentPPO import AgentPPO
   from elegantrl.train.config import build_env
   import gym
   
   args = Arguments(build_env('Pendulum-v1'), AgentPPO())

The full list of parameters in ``Arguments``:

.. autoclass:: elegantrl.train.config.Arguments
   :members:


Environment registration
---------------------

.. autofunction:: elegantrl.train.config.build_env

.. autofunction:: elegantrl.train.config.check_env


Utils
---------------------

.. autofunction:: elegantrl.train.config.kwargs_filter

---

## File: docs/source/api/evaluator.rst

Evaluator: *evaluator.py*
===============================

In the course of training, ElegantRL provide an ``evaluator`` to periodically evaluate agent's performance and save models.

For agent evaluation, the evaluator runs agent's actor (policy) network on the testing environment and outputs corresponding scores. Commonly used performance metrics are mean and variance of episodic rewards. The score is useful in following two cases:
  - Case 1: the score serves as a goal signal. When the score reaches the target score, it means that the goal of the task is achieved. 
  - Case 2: the score serves as a criterion to determine overfitting of models. When the score continuously drops, we can terminate the training process early to mitigate the performance collapse and the waste of computing power brought by overfitting.

.. note::
  ElegantRL supports a tournament-based ensemble training scheme to empower the population-based training (PBT). We maintain a leaderboard to keep track of agents with high scores and then perform a tournament-based evolution among these agents. In this case, the score from the evaluator serves as a metric for leaderboard.

For model saving, the evaluator saves following three types of files:
  - actor.pth: actor (policy) network of the agent.
  - plot_learning_curve.jpg: learning curve of the agent. 
  - recorder.npy: log file, including total training steps, reward average, reward standard deviation, reward exp, actor loss, and critic loss.  
  
We implement the ``evaluator`` as a microservice, which can be ran as an independent process. When an evaluator is running, it can automatically monitors parallel agents, and provide evaluation when any agent needs, and communicate agent information with the leaderboard.

Implementations
---------------------

.. autoclass:: elegantrl.train.evaluator.Evaluator
   :members:
   

Utils
---------------------

.. autofunction:: elegantrl.train.evaluator.get_episode_return_and_step

.. autofunction:: elegantrl.train.evaluator.save_learning_curve

---

## File: docs/source/api/replay.rst

Replay Buffer: *replay_buffer.py*
=================================

ElegantRL provides ``ReplayBuffer`` to store sampled transitions. 

In ElegantRL, we utilize ``Worker`` for exploration (data sampling) and ``Learner`` for exploitation (model learning), and we view such a relationship as a "producer-consumer" model, where a worker produces transitions and a learner consumes, and a learner updates the actor net at worker to produce new transitions. In this case, the ``ReplayBuffer`` is the storage buffer that connects the worker and learner.

Each transition is in a format (state, (reward, done, action)).

.. note::
  We allocate the ``ReplayBuffer`` on continuous RAM for high performance training. Since the collected transitions are packed in sequence, the addressing speed increases dramatically when a learner randomly samples a batch of transitions.
  
Implementations
---------------------

.. autoclass:: elegantrl.train.replay_buffer.ReplayBuffer
   :members:
   
Multiprocessing
---------------------

.. autoclass:: elegantrl.train.replay_buffer.ReplayBufferMP
   :members:
   
Initialization
---------------------

.. autofunction:: elegantrl.train.replay_buffer.init_replay_buffer

Utils
---------------------

.. autoclass:: elegantrl.train.replay_buffer.BinarySearchTree

---

## File: docs/source/api/run.rst

Run: *run.py*
=================================

In *run.py*, we provide functions to wrap the training (and evaluation) process. 

In ElegantRL, users follow a **two-step procedure** to train an agent in a lightweight and automatic way. 

1. Initializing the agent and environment, and setting hyper-parameters up in ``Arguments``.
2. Passing the ``Arguments`` to functions for the training process, e.g., ``train_and_evaluate`` for single-process training and ``train_and_evaluate_mp`` for multi-process training.

Let's look at a demo for the simple two-step procedure.

.. code-block:: python
   
   from elegantrl.train.config import Arguments
   from elegantrl.train.run import train_and_evaluate, train_and_evaluate_mp
   from elegantrl.envs.Chasing import ChasingEnv
   from elegantrl.agents.AgentPPO import AgentPPO
   
   # Step 1
   args = Arguments(agent=AgentPPO(), env_func=ChasingEnv)
   
   # Step 2
   train_and_evaluate_mp(args)

Single-process
---------------------

.. autofunction:: elegantrl.train.run.train_and_evaluate
   
Multi-process
---------------------

.. autofunction:: elegantrl.train.run.train_and_evaluate_mp
   
Utils
---------------------

.. autoclass:: elegantrl.train.run.safely_terminate_process

.. autoclass:: elegantrl.train.run.check_subprocess

---

## File: docs/source/api/worker.rst

Worker: *worker.py*
=================================

Deep reinforcement learning (DRL) employs a trial-and-error manner to collect training data (transitions) from agent-environment interactions, along with the learning procedure. ElegantRL utilizes ``Worker`` to generate transitions and achieves worker parallelism, thus greatly speeding up the data collection.

Implementations
---------------------

.. autoclass:: elegantrl.train.worker.PipeWorker
   :members:

---

## File: docs/source/algorithms/a2c.rst

.. _a2c:


A2C
==========

`Advantage Actor-Critic (A2C) <https://arxiv.org/abs/1602.01783>`_ is a synchronous and deterministic version of Asynchronous Advantage Actor-Critic (A3C). It combines value optimization and policy optimization approaches. This implementation of the A2C algorithm is built on PPO algorithm for simplicity, and it supports the following extensions:

-  Target network: ✔️
-  Gradient clipping: ✔️
-  Reward clipping: ❌
-  Generalized Advantage Estimation (GAE): ✔️
-  Discrete version: ✔️

.. warning::
    The implementation of A2C serves as a pedagogical goal. For practitioners, we recommend using the PPO algorithm for training agents. Without the trust-region and clipped ratio, hyper-parameters in A2C, e.g., ``repeat_times``, need to be fine-tuned to avoid performance collapse.
    

Code Snippet
------------

.. code-block:: python

    import torch
    from elegantrl.run import train_and_evaluate
    from elegantrl.config import Arguments
    from elegantrl.train.config import build_env
    from elegantrl.agents.AgentA2C import AgentA2C
    
    # train and save
    args = Arguments(env=build_env('Pendulum-v0'), agent=AgentA2C())
    args.cwd = 'demo_Pendulum_A2C'
    args.env.target_return = -200
    args.reward_scale = 2 ** -2
    train_and_evaluate(args) 
    
    # test
    agent = AgentA2C()
    agent.init(args.net_dim, args.state_dim, args.action_dim)
    agent.save_or_load_agent(cwd=args.cwd, if_save=False)
    
    env = build_env('Pendulum-v0')
    state = env.reset()
    episode_reward = 0
    for i in range(2 ** 10):
        action = agent.select_action(state)
        next_state, reward, done, _ = env.step(action)
        
        episode_reward += reward
        if done:
            print(f'Step {i:>6}, Episode return {episode_reward:8.3f}')
            break
        else:
            state = next_state
        env.render()
              
              
              
Parameters
---------------------

.. autoclass:: elegantrl.agents.AgentA2C.AgentA2C
   :members:
   
.. autoclass:: elegantrl.agents.AgentA2C.AgentDiscreteA2C
   :members:
   
.. _a2c_networks:
   
Networks
-------------

.. autoclass:: elegantrl.agents.net.ActorPPO
   :members:
   
.. autoclass:: elegantrl.agents.net.ActorDiscretePPO
   :members:
   
.. autoclass:: elegantrl.agents.net.CriticPPO
   :members:

---

## File: docs/source/algorithms/ddpg.rst

.. _ddpg:


DDPG
==========

`Deep Deterministic Policy Gradient (DDPG) <https://arxiv.org/abs/1509.02971>`_ is an off-policy Actor-Critic algorithm for continuous action space. Since computing the maximum over actions in the target is a challenge in continuous action space, DDPG deals with this using a policy network to compute an action. This implementation provides DDPG and supports the following extensions:

-  Experience replay: ✔️
-  Target network: ✔️
-  Gradient clipping: ✔️
-  Reward clipping: ❌
-  Prioritized Experience Replay (PER): ✔️
-  Ornstein–Uhlenbeck noise: ✔️


.. warning::
    In the DDPG paper, the authors use time-correlated Ornstein-Uhlenbeck Process to add noise to the action output. However, as shown in the later works, the Ornstein-Uhlenbeck Process is an overcomplication that does not have a noticeable effect on performance when compared to uncorrelated Gaussian noise.

Code Snippet
------------

.. code-block:: python

    import torch
    from elegantrl.run import train_and_evaluate
    from elegantrl.config import Arguments
    from elegantrl.train.config import build_env
    from elegantrl.agents.AgentDDPG import AgentDDPG
    
    # train and save
    args = Arguments(env=build_env('Pendulum-v0'), agent=AgentDDPG())
    args.cwd = 'demo_Pendulum_DDPG'
    args.env.target_return = -200
    args.reward_scale = 2 ** -2
    train_and_evaluate(args) 
    
    # test
    agent = AgentDDPG()
    agent.init(args.net_dim, args.state_dim, args.action_dim)
    agent.save_or_load_agent(cwd=args.cwd, if_save=False)
    
    env = build_env('Pendulum-v0')
    state = env.reset()
    episode_reward = 0
    for i in range(2 ** 10):
        action = agent.select_action(state)
        next_state, reward, done, _ = env.step(action)
        
        episode_reward += reward
        if done:
            print(f'Step {i:>6}, Episode return {episode_reward:8.3f}')
            break
        else:
            state = next_state
        env.render()
              
              
              
Parameters
---------------------

.. autoclass:: elegantrl.agents.AgentDDPG.AgentDDPG
   :members:
   
.. _ddpg_networks:
   
Networks
-------------

.. autoclass:: elegantrl.agents.net.Actor
   :members:
   
.. autoclass:: elegantrl.agents.net.Critic
   :members: