## File: README.md

# FinRL: Financial Reinforcement Learning → FinRL-X



[](https://pepy.tech/project/finrl)
[](https://pepy.tech/project/finrl)
[](https://discord.gg/trsr8SXpW5)
[](https://www.python.org/downloads/release/python-360/)
[](https://pypi.org/project/finrl/)
[](https://finrl.readthedocs.io/en/latest/?badge=latest)





[](https://twitter.com/intent/tweet?text=FinRL-Financial-Deep-Reinforcement-Learning%20&url=https://github.com/AI4Finance-Foundation/FinRL&hashtags=DRL,AI) [](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fgithub.com%2FAI4Finance-Foundation%2FFinRL)


> [!IMPORTANT]
> **FinRL-X** is the next-generation evolution of FinRL, designed for AI-native, modular, and production-oriented quantitative trading.
>
> - **This repository (`FinRL`)** preserves the original end-to-end educational and research framework.
> - **For the latest architecture, live trading deployment, and production-focused development, please use [`FinRL-X / FinRL-Trading`](https://github.com/AI4Finance-Foundation/FinRL-Trading).**

**FinRL®** is widely recognized as the first open-source framework for financial reinforcement learning.
This repository contains the original FinRL library for education, benchmarking, and research prototyping.

For the next-generation AI-native and production-oriented trading stack, please visit **[FinRL-X / FinRL-Trading](https://github.com/AI4Finance-Foundation/FinRL-Trading)**.

## FinRL Ecosystem Roadmap

| Generation | Positioning | Target Users | Repository | Description |
|----|----|----|----|----|
| FinRL-Meta | Market Environments | Practitioners | [FinRL-Meta](https://github.com/AI4Finance-Foundation/FinRL-Meta) | Gym-style financial market environments and benchmarks |
| FinRL | Classic End-to-End Framework | Learners, Developers, Researchers | [FinRL](https://github.com/AI4Finance-Foundation/FinRL) | Original train-test-trade pipeline for financial reinforcement learning |
| ElegantRL | Algorithm Layer | Researchers and Experts | [ElegantRL](https://github.com/AI4Finance-Foundation/ElegantRL) | Lightweight and elegant DRL algorithms |
| **FinRL-X** | **Next Generation / Production** | **Professional traders, institutions, hedge funds** | [**FinRL-Trading**](https://github.com/AI4Finance-Foundation/FinRL-Trading) | **AI-native modular infrastructure for deployment-aware quantitative trading** |
> **Recommended for new users:** Start with **[FinRL-X / FinRL-Trading](https://github.com/AI4Finance-Foundation/FinRL-Trading)** if you are building modern or production-oriented trading systems.

### 🔄 FinRL-X vs. FinRL: What Changed

| Capability | FinRL (Stage 1.0) | FinRL-X (Stage 3.0) |
|---|---|---|
| **Paradigm** | Deep Reinforcement Learning | AI-Native (ML + DRL + LLM-ready) |
| **Architecture** | Three-layer coupled monolith | Fully decoupled modular layers |
| **Strategies** | DRL agents (A2C, DDPG, PPO, SAC, TD3) | ML selection + DRL timing + extensible base |
| **Data Layer** | 14 manually-wired processors | Auto-select: Yahoo Finance → FMP → WRDS |
| **Backtesting** | Custom hand-rolled evaluation loops | Professional `bt` library engine |
| **Live Trading** | Basic Alpaca support | Full multi-account integration + risk controls |
| **Configuration** | `config.py` + `config_tickers.py` | Type-safe Pydantic + `.env` multi-env |
| **Risk Management** | Gym environment constraints only | Order · portfolio · strategy-level controls |
| **Target Users** | Researchers & students | Quants, institutions, production deployments |
| **Paper** | [arXiv:2011.09607](https://arxiv.org/abs/2011.09607) | [arXiv:2603.21330](https://arxiv.org/abs/2603.21330) |

[FinGPT](https://github.com/AI4Finance-Foundation/FinGPT): an open-source project for financial large language models, designed for research and real-world FinTech applications.


[](https://discord.gg/trsr8SXpW5)

## Outline

  - [Overview](#overview)
  - [File Structure](#file-structure)
  - [Supported Data Sources](#supported-data-sources)
  - [Installation](#installation)
  - [Status Update](#status-update)
  - [Tutorials](#tutorials)
  - [Publications](#publications)
  - [News](#news)
  - [Citing FinRL](#citing-finrl)
  - [Join and Contribute](#join-and-contribute)
    - [Contributors](#contributors)
    - [Sponsorship](#sponsorship)
  - [LICENSE](#license)

## Project Contributors

FinRL® is an open-source financial reinforcement learning framework developed by contributors from the AI4Finance community and maintained by the AI4Finance Foundation.

Key contributors include:

- [**Hongyang (Bruce) Yang**](https://www.linkedin.com/in/brucehy/) – research and development on financial reinforcement learning frameworks, market environments, and quantitative trading applications
- [other contributors…]

## Overview

FinRL is the original open-source framework for financial reinforcement learning, organized around three core layers:

- **Market Environments**
- **DRL Agents**
- **Financial Applications**

For a trading task, an agent interacts with a market environment and learns sequential decision-making policies.

This repository focuses on the **classic FinRL workflow** for education, experimentation, and research prototyping.

For the **next-generation production-oriented stack**, including modular deployment and AI-native trading infrastructure, please visit **[FinRL-X / FinRL-Trading](https://github.com/AI4Finance-Foundation/FinRL-Trading)**.





Videos [FinRL](http://www.youtube.com/watch?v=ZSGJjtM-5jA) at [AI4Finance Youtube Channel](https://www.youtube.com/channel/UCrVri6k3KPBa3NhapVV4K5g).

## FinRL Stock Trading 2026 Tutorial
This tutorial demonstrates the original FinRL workflow for educational and research purposes.
For the latest production-oriented pipeline, please use **[FinRL-X / FinRL-Trading](https://github.com/AI4Finance-Foundation/FinRL-Trading)**.
### Step 1: Clone the Repository

```bash
git clone https://github.com/AI4Finance-Foundation/FinRL.git
cd FinRL
```

### Step 2: Create and Activate Virtual Environment

```bash
python3 -m venv venv
source venv/bin/activate
```

### Step 3: Install FinRL

```bash
pip install -e .
```

### Step 4: Run the Scripts

**1. Data Download & Preprocessing**

```bash
python examples/FinRL_StockTrading_2026_1_data.py
```

This script downloads DOW 30 stock data from Yahoo Finance, adds technical indicators (MACD, RSI, etc.), VIX, and turbulence index, then splits the data into training set (2014–2025) and trading set (2026-01-01 to 2026-03-20), saving them as `train_data.csv` and `trade_data.csv`.

**2. Train DRL Agents**

```bash
python examples/FinRL_StockTrading_2026_2_train.py
```

This script trains 5 DRL agents (A2C, DDPG, PPO, TD3, SAC) using Stable Baselines 3 on the training data. Trained models are saved to the `trained_models/` directory.

**3. Backtest**

```bash
python examples/FinRL_StockTrading_2026_3_Backtest.py
```

This script loads the trained agents, runs them on the trading data, and compares their performance against two baselines: Mean Variance Optimization (MVO) and the DJIA index. Results are printed to the console and a plot is saved as `backtest_result.png`.


## File Structure

The main folder **finrl** has three subfolders **applications, agents, meta**. We employ a **train-test-trade** pipeline with three files: train.py, test.py, and trade.py.

```
FinRL
├── finrl (main folder)
│   ├── applications
│   	├── Stock_NeurIPS2018
│   	├── imitation_learning
│   	├── cryptocurrency_trading
│   	├── high_frequency_trading
│   	├── portfolio_allocation
│   	└── stock_trading
│   ├── agents
│   	├── elegantrl
│   	├── rllib
│   	└── stablebaseline3
│   ├── meta
│   	├── data_processors
│   	├── env_cryptocurrency_trading
│   	├── env_portfolio_allocation
│   	├── env_stock_trading
│   	├── preprocessor
│   	├── data_processor.py
│       ├── meta_config_tickers.py
│   	└── meta_config.py
│   ├── config.py
│   ├── config_tickers.py
│   ├── main.py
│   ├── plot.py
│   ├── train.py
│   ├── test.py
│   └── trade.py
│
├── examples
├── unit_tests (unit tests to verify codes on env & data)
│   ├── environments
│   	└── test_env_cashpenalty.py
│   └── downloaders
│   	├── test_yahoodownload.py
│   	└── test_alpaca_downloader.py
├── setup.py
├── requirements.txt
└── README.md
```

## Supported Data Sources

|Data Source |Type |Range and Frequency |Request Limits|Raw Data|Preprocessed Data|
|  ----  |  ----  |  ----  |  ----  |  ----  |  ----  |
|[Akshare](https://alpaca.markets/docs/introduction/)| CN Securities| 2015-now, 1day| Account-specific| OHLCV| Prices&Indicators|
|[Alpaca](https://docs.alpaca.markets/docs/getting-started)| US Stocks, ETFs| 2015-now, 1min| Account-specific| OHLCV| Prices&Indicators|
|[Baostock](http://baostock.com/baostock/index.php/Python_API%E6%96%87%E6%A1%A3)| CN Securities| 1990-12-19-now, 5min| Account-specific| OHLCV| Prices&Indicators|
|[Binance](https://binance-docs.github.io/apidocs/spot/en/#public-api-definitions)| Cryptocurrency| API-specific, 1s, 1min| API-specific| Tick-level daily aggregated trades, OHLCV| Prices&Indicators|
|[CCXT](https://docs.ccxt.com/en/latest/manual.html)| Cryptocurrency| API-specific, 1min| API-specific| OHLCV| Prices&Indicators|
|[EODhistoricaldata](https://eodhistoricaldata.com/financial-apis/)| US Securities| Frequency-specific, 1min| API-specific | OHLCV | Prices&Indicators|
|[IEXCloud](https://iexcloud.io/docs/api/)| NMS US securities|1970-now, 1 day|100 per second per IP|OHLCV| Prices&Indicators|
|[JoinQuant](https://www.joinquant.com/)| CN Securities| 2005-now, 1min| 3 requests each time| OHLCV| Prices&Indicators|
|[QuantConnect](https://www.quantconnect.com/docs/v2)| US Securities| 1998-now, 1s| NA| OHLCV| Prices&Indicators|
|[RiceQuant](https://www.ricequant.com/doc/rqdata/python/)| CN Securities| 2005-now, 1ms| Account-specific| OHLCV| Prices&Indicators|
[Sinopac](https://sinotrade.github.io/zh_TW/tutor/prepare/terms/) | Taiwan securities | 2023-04-13~now, 1min | Account-specific | OHLCV | Prices&Indicators|
|[Tushare](https://tushare.pro/document/1?doc_id=131)| CN Securities, A-share| -now, 1 min| Account-specific| OHLCV| Prices&Indicators|
|[WRDS](https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/nyse-trade-and-quote-taq/)| US Securities| 2003-now, 1ms| 5 requests each time| Intraday Trades|Prices&Indicators|
|[YahooFinance](https://pypi.org/project/yfinance/)| US Securities| Frequency-specific, 1min| 2,000/hour| OHLCV | Prices&Indicators|






OHLCV: open, high, low, and close prices; volume. adjusted_close: adjusted close price

Technical indicators: 'macd', 'boll_ub', 'boll_lb', 'rsi_30', 'dx_30', 'close_30_sma', 'close_60_sma'. Users also can add new features.


## Installation
+ [Install description for all operating systems (MAC OS, Ubuntu, Windows 10)](./docs/source/start/installation.rst)
+ [FinRL for Quantitative Finance: Install and Setup Tutorial for Beginners](https://ai4finance.medium.com/finrl-for-quantitative-finance-install-and-setup-tutorial-for-beginners-1db80ad39159)

## Status Update
**Version History** *[click to expand]*


* 2022-06-25
	0.3.5: Formal release of FinRL, neo_finrl is changed to FinRL-Meta with related files in directory: *meta*.
* 2021-08-25
	0.3.1: pytorch version with a three-layer architecture, apps (financial tasks), drl_agents (drl algorithms), neo_finrl (gym env)
* 2020-12-14
  	Upgraded to **Pytorch** with stable-baselines3; Removed TensorFlow 1.x support; TensorFlow 2.0 support was under development at the time.
* 2020-11-27
  	0.1: Beta version with tensorflow 1.5




## Tutorials

+ [Towards Data Science] [Deep Reinforcement Learning for Automated Stock Trading](https://medium.com/data-science/deep-reinforcement-learning-for-automated-stock-trading-f1dad0126a02)


## Publications

|Title |Conference/Journal |Link|Citations|Year|
|  ----  |  ----  |  ----  |  ----  |  ----  |
|Dynamic Datasets and Market Environments for Financial Reinforcement Learning| Machine Learning - Springer Nature| [paper](https://arxiv.org/abs/2304.13174) [code](https://github.com/AI4Finance-Foundation/FinRL-Meta) | 51 | 2024 |
|**FinRL-Meta**: FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning| NeurIPS 2022| [paper](https://arxiv.org/abs/2211.03107) [code](https://github.com/AI4Finance-Foundation/FinRL-Meta) | 136 | 2022 |
|**FinRL**: Deep reinforcement learning framework to automate trading in quantitative finance| ACM International Conference on AI in Finance (ICAIF) | [paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3955949) | 212 | 2021 |
|**FinRL**: A deep reinforcement learning library for automated stock trading in quantitative finance| NeurIPS 2020 Deep RL Workshop  | [paper](https://arxiv.org/abs/2011.09607) | 275 | 2020 |
|Deep reinforcement learning for automated stock trading: An ensemble strategy| ACM International Conference on AI in Finance (ICAIF) | [paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3690996) [code](https://github.com/AI4Finance-Foundation/FinRL-Meta/blob/master/tutorials/2-Advance/FinRL_Ensemble_StockTrading_ICAIF_2020/FinRL_Ensemble_StockTrading_ICAIF_2020.ipynb) | 426 | 2020 |
|Practical deep reinforcement learning approach for stock trading | NeurIPS 2018 Workshop on Challenges and Opportunities for AI in Financial Services| [paper](https://arxiv.org/abs/1811.07522) [code](https://github.com/AI4Finance-Foundation/FinRL](https://github.com/AI4Finance-Foundation/FinRL))| 303 | 2018 |


## News
+ [央广网] [2021 IDEA大会于福田圆满落幕：群英荟萃论道AI 多项目发布亮点纷呈](http://tech.cnr.cn/techph/20211123/t20211123_525669092.shtml)
+ [央广网] [2021 IDEA大会开启AI思想盛宴 沈向洋理事长发布六大前沿产品](https://baijiahao.baidu.com/s?id=1717101783873523790&wfr=spider&for=pc)
+ [IDEA新闻] [2021 IDEA大会发布产品FinRL-Meta——基于数据驱动的强化学习金融风险模拟系统](https://idea.edu.cn/news/20211213143128.html)
+ [知乎] [FinRL-Meta基于数据驱动的强化学习金融元宇宙](https://zhuanlan.zhihu.com/p/437804814)
+ [量化投资与机器学习] [基于深度强化学习的股票交易策略框架（代码+文档)](https://www.mdeditor.tw/pl/p5Gg)
+ [运筹OR帷幄] [领读计划NO.10 | 基于深度增强学习的量化交易机器人：从AlphaGo到FinRL的演变过程](https://zhuanlan.zhihu.com/p/353557417)
+ [深度强化实验室] [【重磅推荐】哥大开源“FinRL”: 一个用于量化金融自动交易的深度强化学习库](https://blog.csdn.net/deeprl/article/details/114828024)
+ [商业新知] [金融科技讲座回顾|AI4Finance: 从AlphaGo到FinRL](https://www.shangyexinzhi.com/article/4170766.html)
+ [Kaggle] [Jane Street Market Prediction](https://www.kaggle.com/c/jane-street-market-prediction/discussion/199313)
+ [矩池云Matpool] [在矩池云上如何运行FinRL股票交易策略框架](http://www.python88.com/topic/111918)
+ [财智无界] [金融学会常务理事陈学彬: 深度强化学习在金融资产管理中的应用](https://www.sohu.com/a/486837028_120929319)
+ [Neurohive] [FinRL: глубокое обучение с подкреплением для трейдинга](https://neurohive.io/ru/gotovye-prilozhenija/finrl-glubokoe-obuchenie-s-podkrepleniem-dlya-trejdinga/)
+ [ICHI.PRO] [양적 금융을위한 FinRL: 단일 주식 거래를위한 튜토리얼](https://ichi.pro/ko/yangjeog-geum-yung-eul-wihan-finrl-dan-il-jusig-geolaeleul-wihan-tyutolieol-61395882412716)
+ [知乎] [基于深度强化学习的金融交易策略（FinRL+Stable baselines3，以道琼斯30股票为例）](https://zhuanlan.zhihu.com/p/563238735)
+ [知乎] [动态数据驱动的金融强化学习](https://zhuanlan.zhihu.com/p/616799055)
+ [知乎] [FinRL的W&B化+超参数搜索和模型优化(基于Stable Baselines 3）](https://zhuanlan.zhihu.com/p/498115373)
+ [知乎] [FinRL-Meta: 未来金融强化学习的元宇宙](https://zhuanlan.zhihu.com/p/544621882)
+
## Citing FinRL

For the next-generation AI-native modular trading infrastructure, see **[FinRL-X / FinRL-Trading](https://github.com/AI4Finance-Foundation/FinRL-Trading)**.

```
@inproceedings{yang2026finrlx,
  title     = {FinRL-X: An AI-Native Modular Infrastructure for Quantitative Trading},
  author    = {Yang, Hongyang and Zhang, Boyu and She, Yang and Liao, Xinyu and Zhang, Xiaoli},
  booktitle = {Proceedings of the 2nd International Workshop on Decision Making and Optimization in Financial Technologies (DMO-FinTech)},
  year      = {2026},
  note      = {Workshop at PAKDD 2026}
}
```

If you use the original FinRL framework, please cite the FinRL papers:

```
@article{finrl2020,
    author  = {Liu, Xiao-Yang and Yang, Hongyang and Chen, Qian and Zhang, Runjia and Yang, Liuqing and Xiao, Bowen and Wang, Christina Dan},
    title   = {{FinRL}: A deep reinforcement learning library for automated stock trading in quantitative finance},
    journal = {Deep RL Workshop, NeurIPS 2020},
    year    = {2020}
}
```

## Join and Contribute

Welcome to **AI4Finance** community!

Please check [Contributing Guidelines](https://github.com/AI4Finance-Foundation/FinRL-Tutorials/blob/master/Contributing.md).

### Contributors

Thanks to all contributors who have helped build FinRL.

<a href="https://github.com/AI4Finance-Foundation/FinRL/graphs/contributors">
  
</a>


## LICENSE

MIT License
```
Trademark Notice

FinRL and the FinRL logo are trademarks of FinRL LLC. Use of these marks by the AI4Finance Foundation is permitted under license. The open-source license for this repository does not grant any right to use the FinRL name, logo, or related trademarks without prior written permission from FinRL LLC, except as permitted by applicable law.

```

**Disclaimer: We are sharing codes for academic purposes under the MIT license. Nothing herein constitutes financial advice or a recommendation to trade real money. Users are solely responsible for any financial decisions made using this software. Consult a qualified professional before deploying capital.**

---

## File: docs/source/developer_guide/contributing.rst

:github_url: https://github.com/AI4Finance-Foundation/FinRL

============================
Contributing Guidelines
============================



This project aims to bring a reinforcement learning environment to the trading community.
There are always competing priorities among the community, and we want to make sure that we are able to achieve together a project that is reliable, sustainable, and maintainable.

Guiding Principles
=======

* We should have reliable codes in this project
    * reliable code with tests
    * reliable code that works
    * reliable code runs without consuming excessive resources
* We should help each other to achieve SOTA results together
* We should write clear codes
    * Code should not be redundant
    * Code should have documentation inline (standard pep format)
    * Code should be organized into classes and functions
* We should leverage outside tools as it makes sense
* We work together, and are kind, patient, and clear in our communication. Jerks are not welcome.

## If you see something, say something!
* Filing an [issue](https://guides.github.com/features/issues/) is a great way to help improve the project


Accepting PRs
=======

* You found a bug and a way to fix it
* You have contributed to an issue that was prioritized by the coordinators of this project
* You have new functionality that you're adding that you've written issues for and has documentation + Tests

PR Guidelines
=======

* Please tag @bruceyang, @spencerromo, or @xiaoyang in every PR. (P.S. we're looking for more collaborators with software experience!)
* Please reference or write and reference an [issue](https://guides.github.com/features/issues/)
* Please have clear commit messages
* Please write detailed documentation and tests for every added piece of functionality
* Please try to not break existing functionality, or if you need to, please plan to justify this necessity and coordinate with the collaborators
* Please be patient and respectful with feedback
* Please use pre-commit hooks


Others
=======

-Using pre-commit
```
pip install pre-commit
pre-commit install
```

-Running Tests
```
-Locally
python3 -m unittest discover

-Docker
./docker/bin/build_container.sh
./docker/bin/test.sh
```

---

## File: docs/source/developer_guide/development_setup.rst

:github_url: https://github.com/AI4Finance-Foundation/FinRL

============================
Development Guide
============================

Git is a commonly used tool in software engineering. PyCharm is a popular IDE for Python, and developers can also choose other IDEs as they like. Now, we take PyCharm as an example. This setup with PyCharm makes it easy to work on all of AI4Finance-Foundation's repositories simultaneously, while allowing easy debugging, committing to the respective repo and creating PRs/MRs.

Step 1: Download Software
=======

-Download and install `Anaconda <https://www.anaconda.com/>`_.

-Download and install `PyCharm <https://www.jetbrains.com/pycharm/>`_.  The Community Edition (free version) offers everything you need except running Jupyter notebooks. The Full-fledged Professional Edition offers everything. A workaround to run existing notebooks in the Community edition is to copy all notebook cells into .py files.
For notebook support, you can consider PyCharm Professional Edition.

-On GitHub, fork `FinRL <https://github.com/AI4Finance-Foundation/FinRL>`_ to your private Github repo.

-On GitHub, fork `ElegantRL <https://github.com/AI4Finance-Foundation/ElegantRL>`_ to your private Github repo.

-On GitHub, fork `FinRL-Meta <https://github.com/AI4Finance-Foundation/FinRL-Meta>`_ to your private Github repo.

-All next steps happen on your local computer.

Step 2: Git Clone
=======

.. code-block:: bash

    mkdir ~/ai4finance
    cd ~/ai4finance
    git clone https://github.com/[your_github_username]/FinRL.git
    git clone https://github.com/[your_github_username]/ElegantRL.git
    git clone https://github.com/[your_github_username]/FinRL-Meta.git


Step 3: Create a Conda Environment
======

.. code-block:: bash

    cd ~/ai4finance
    conda create --name ai4finance python=3.8
    conda activate ai4finance

    cd FinRL
    pip install -r requirements.txt

Install ElegantRL using requirements.txt, or open ElegantRL/setup.py in a text editor and pip install anything you can find: gym, matplotlib, numpy, pybullet, torch, opencv-python, and box2d-py.


Step 4: Configure a PyCharm Project
======

-Launch PyCharm

-File > Open > [ai4finance project folder]

.. image:: ../image/pycharm_status_bar.png

-At the bottom right of the status bar, change or add the interpreter to the ai4finance conda environment. Make sure when you click the "terminal" bar at the bottom left, it shows ai4finance.

.. image:: ../image/pycharm_MarkDirectoryAsSourcesRoot.png

-At the left of the screen, in the project file tree:

    - Right-click on the FinRL folder > Mark Directory as > Sources Root
    - Right-click on the ElegantRL folder > Mark Directory as > Sources Root
    - Right-click on the FinRL-Meta folder > Mark Directory as > Sources Root

-Once you run a .py file, you will notice that you may still have some missing packages.  In that case, simply pip install them.

For example, we revise FinRL.

.. code-block:: bash

    cd ~/ai4finance
    cd ./FinRL
    git checkout -b branch_xxx

where branch_xxx is a new branch name. In this branch, we revise config.py.

Step 5: New a Branch
=======

Please new a new branch based on branch "staging" (NOT "master"), which is for all developers. DO NOT directly push codes to the branch "staging" or "master".


Step 6: Creating Commits and PRs/MRs
=======

-Create commits as you usually do through PyCharm.

-Make sure that each commit covers only 1 of the 3 repo's. Don't create a commit that spans more than one repo, e.g., FinRL and ElegantRL.

.. image:: ../image/pycharm_push_PR.png

-When you do a Git Push, PyCharm will ask you to which of the 3 repos you want to push. Just like the above figure, we select the repo "FinRL".


With respect to creating a pull request (PR) or merge quest (MR), please refer to `Create a PR <https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request>`_ or `Opensource Create a PR <https://opensource.com/article/19/7/create-pull-request-github>`_.

Step 7: Submit PRs/MRs
=======

When submiting PRs/MRs, please choose the branch "staging", NOT "master".

Step 8: Merge "staging" to "master"
=======

This step is for managers. If the branch "staging" is stable and works successfully after a series of tests, the managers of this repo will merge it to the branch "master" every 2-4 weeks. To avoid any risk, we hope managers download the "master" branch locally before merging.

---

## File: docs/source/developer_guide/file_architecture.rst

:github_url: https://github.com/AI4Finance-Foundation/FinRL

=================
File Architecture
=================

FinRL's file architecture strictly follow the :ref:`three-layer architecture`.

.. code:: bash

    FinRL
    ├── finrl (the main folder)
    │   ├── applications
    │   	├── cryptocurrency_trading
    │   	├── high_frequency_trading
    │   	├── portfolio_allocation
    │   	└── stock_trading
    │   ├── agents
    │   	├── elegantrl
    │   	├── rllib
    │   	└── stablebaseline3
    │   ├── meta
    │   	├── data_processors
    │   	├── env_cryptocurrency_trading
    │   	├── env_portfolio_allocation
    │   	├── env_stock_trading
    │   	├── preprocessor
    │   	├── data_processor.py
    │   	└── finrl_meta_config.py
    │   ├── config.py
    │   ├── config_tickers.py
    │   ├── main.py
    │   ├── train.py
    │   ├── test.py
    │   ├── trade.py
    └───└── plot.py

---

## File: docs/source/finrl_meta/background.rst

:github_url: https://github.com/AI4Finance-Foundation/FinRL

=============================
Background
=============================

Why FinRL-Meta?
================

Finance is a particularly difficult playground for deep reinforcement learning (DRL). Some existing works already showed great potential of DRL in financial applications. However, establishing high-quality market environments and benchmarks on financial reinforcement learning are challenging and highly demanded. Thus, we proposed and started FinRL-Meta.


Envrionments and Benchmarks
===========================

MuJoCo and OpenAI’s XLand are famous libraries in the RL area, they built environments for deep reinforcement learning in robotics, games, and common tasks that are widely used in RL academia and industry. However, they barely provide any high quality environments for financial tasks. FinRL-Meta, previously called Neo-FinRL (near real market environments for data driven financial RL), are working to provide hundreds of market environments and tens of benchmarks for financial reinforcement learning.


Metaverse for financial RL
============================

Achieving the goal of hundreds of market environments and benchmarks discribed above, we are aiming to build a metaverse for financial reinforcement learning. Like XLand, we would provide an open-ended market world with different tasks e.g. stock, cryptocurrency, etc. for agents to explore and learn.

Contribute to finance
======================

We believe in the potential of deep reinforcement learning. And we hope that after we build the metaverse for financial reinforcement learning, our agents have chance to be a market simulator, or to explore risk assessment or market fragility.

---

## File: docs/source/finrl_meta/Benchmark.rst

:github_url: https://github.com/AI4Finance-Foundation/FinRL

=============================
Benchmark
=============================

Performance Metrics
====================

FinRL-Meta provides the following unified metrics to measure the trading performance:

- **Cumulative return:** :math:`R = \frac{V - V_0}{V_0}`, where V is final portfolio value, and :math:`V_0` is original capital.
- **Annualized return:** :math:`r = (1+R)^\frac{365}{t}-1`, where t is the number of trading days.
- **Annualized volatility:** :math:`{\sigma}_a = \sqrt{\frac{\sum_{i=1}^{n}{(r_i-\bar{r})^2}}{n-1}}`, where :math:`r_i` is the annualized return in year i, :math:`\bar{r}` is the average annualized return, and n is the number of years.
- **Sharpe ratio:** :math:`S = \frac{r - r_f}{{\sigma}_a}`, where :math:`r_f` is the risk-free rate.
- **Max. drawdown** The maximal percentage loss in portfolio value.

The following baseline trading strategies are provided for comparisons:

• **Passive trading strategy**, a well-known long-term strategy. The investors just buy and hold selected stocks or indexes without further activities.
• ****Mean-variance and min-variance strategy**, both strategies look for a balance between risks and profits. It selects a diversified portfolio to achieve higher profits at lower risk.
• **Equally weighted strategy**, a portfolio allocation strategy that gives equal weights to different assets, avoiding allocating overly high weights on particular stocks.

Tutorials in Jupyter Notebooks
===========================================

For educational purposes, we provide Jupyter notebooks as tutorials to help newcomers get familiar with the whole pipeline. Notebooks can be found `here <https://github.com/AI4Finance-Foundation/FinRL-Tutorials>`_

• Stock trading: We apply popular DRL algorithms to trade multiple stocks.
• Portfolio allocation: We use DRL agents to optimize asset allocation in a set of stocks.
• Cryptocurrency trading: We reproduce the experiment on 10 popular cryptocurrencies.
• Multi-agent RL for liquidation strategy analysis: We reproduce the experiment in [7]. The multi-agent optimizes the shortfalls in the liquidation task, which is to sell given shares of one stock sequentially within a given period, considering the costs arising from the market impact and the risk aversion.
• Ensemble strategy for stock trading: We reproduce the experiment in that employed an ensemble strategy of several DRL algorithms on the stock trading task.
• Paper trading demo: We provide a demo for paper trading. Users could combine their own strategies or trained agents in paper trading.
• China A-share demo: We provide a demo based on the China A-share market data.
• Hyperparameter tuning: We provide several demos for hyperparameter tuning using Optuna or Ray Tune, since hyperparameter tuning is critical for better performance.

---

## File: docs/source/finrl_meta/Data_layer.rst

:github_url: https://github.com/AI4Finance-Foundation/FinRL

Data Layer
==========

In the data layer, we use a unified data processor to access data, clean data, and extract features.

.. image:: ../image/finrl-meta_data_layer.png
    :width: 60%
    :align: center


Data Accessing
--------------

We connect data APIs of different platforms and unify them using a FinRL-Meta data processor. Users can access data from various sources given the start date, end date, stock list, time interval, and kwargs.

.. image:: ../image/FinRL-Meta-Data-layer.png
    :width: 80%
    :align: center

Data Cleaning
--------------

Raw data retrieved from different data sources are usually of various formats and have erroneous or NaN data (missing data) to different extents, making data cleaning highly time-consuming. In FinRL-Meta, we automate the data cleaning process.

The cleaning processes of NaN data are usually different for various time frequencies. For Low-frequency data, except few stocks with extremely low liquidity, the few NaN values usually mean suspension during that time interval. While for high-frequency data, NaN values are pervasive, which usually means no transaction during that time interval. To reduce the simulation-to-reality gap considering of data efficiency, we provide different solutions for these two cases.

In the low-frequency case, we directly delete the rows with NaN values, reflecting suspension in simulated trading environments. However, it is not suitable to directly delete rows with NaN values in high-frequency cases.

In our test of downloading 1-min OHLCV data of DJIA 30 companies from Alpaca during 2021–01–01~2021–05–31, there were 39736 rows for the raw data. However, after dropping rows with NaN values, only 3361 rows are left.

The low data efficiency of the dropping method is unacceptable. Instead, we take an improved forward filling method. We fill the open, high, low, close columns with the last valid value of close price and the volume column with 0, which is a standard method in practice.

Although this filling method sacrifices the authenticity of the simulated environments, it is acceptable compared to significantly improved data efficiency, especially under tickers with high liquidity. Moreover, this filling method can be further improved using bid, ask prices to reduce the simulation-to-reality gap.

Feature Engineering
-------------------

Feature engineering is the last part of the data layer. We automate the calculation of technical indicators by connecting the Stockstats or TAlib library in our data processor. Common technical indicators including Moving Average Convergence Divergence (MACD), Relative Strength Index (RSI), Average Directional Index (ADX), and Commodity Channel Index (CCI), and so on, are supported. Users can also quickly add indicators from other libraries, or add the user-defined features directly.

Users can add their features by two ways: 1) Write user-defined feature extraction functions directly. The returned features will be added to a feature array. 2) Store the features in a file, and move it to a specified folder. Then, these features will be obtained by reading from the specified file.

---

## File: docs/source/finrl_meta/Environment_layer.rst

:github_url: https://github.com/AI4Finance-Foundation/FinRL

Environment Layer
=================

FinRL-Meta follows the OpenAI gym-style to create market environments using the cleaned data from the data layer. It provides hundreds of environments with a common interface. Users can build their environments based on FinRL-Meta environments easily, share their results and compare the strategies’ performance. We will add more environments for convenience in the future.

Incorporating trading constraints to model market frictions
--------------------------------------------------------------------

To better simulate real-world markets, we incorporate common market frictions (e.g., transaction costs and investor risk aversion) and portfolio restrictions (e.g., non-negative balance).

- **Flexible account settings**: Users can choose whether to allow buying on margin or short-selling.
- **Transaction cost**: We incorporate the transaction cost to reflect market friction, e.g., 0.1% of each buy or sell trade.
- **Risk-control for market crash**: In FinRL, a financial turbulence index is used to control risk during market crash situations. However, calculating the turbulence index is time-consuming. It may take minutes, which is not suitable for paper trading and live trading. We replace the financial turbulence index with the volatility index (VIX) that can be accessed immediately.

Multiprocessing training via vector environment
---------------------------------------------------

We utilize GPUs for multiprocessing training, namely, the vector environment technique of Isaac Gym, which significantly accelerates the training process. In each CUDA core, a trading agent interacts with a market environment to produce transitions in the form of {state, action, reward, next state}. Then, all the transitions are stored in a replay buffer to update a learner. By adopting this technique, we successfully achieve the multiprocessing simulation of hundreds of market environments to improve the performance of DRL trading agents on large datasets.

---

## File: docs/source/finrl_meta/overview.rst

:github_url: https://github.com/AI4Finance-Foundation/FinRL

=============================
Overview
=============================

Following the *de facto* standard of OpenAI Gym, we build a universe of market environments for data-driven financial reinforcement learning, namely, FinRL-Meta. We keep the following design principles.

1. Layered structure
======================================
.. image:: ../image/finrl-meta_overview.png
    :width: 80%
    :align: center

We adopt a layered structure for RL in finance, which consists of three layers: data layer, environment layer, and agent layer. Each layer executes its functions and is relatively independent. There are two main advantages:

1. Transparency: layers interact through end-to-end interfaces to implement the complete workflow of algorithm trading, achieving high extensibility.
2. Modularity: Following the APIs between layers, users can easily customize their own functions to substitute default functions in any layer.

2. DataOps Paradigm
=====================

.. image:: ../image/finrl_meta_dataops.png
    :width: 80%
    :align: center

DataOps paradigm is a set of practices, processes and technologies that combined: automated data engineering & agile development. It helps reduce the cycle time of data engineering and improves data quality. To deal with financial big data, we follow the DataOps paradigm and implement an automatic pipeline:

1. Task planning, such as stock trading, portfolio allocation, cryptocurrency trading, etc
2. Data processing, including data accessing and cleaning, and feature engineering.
3. Training-testing-trading, where DRL agent takes part in.
4. Performance monitoring, compare the performance of DRL agent with some baseline trading strategies.

With this pipeline, we are able to continuously produce dynamic market datasets.

3. Training-testing-trading pipeline:
=====================================
.. image:: ../image/timeline.png
    :width: 80%
    :align: center

We employ a training-testing-trading pipeline that the DRL approach follows a standard end-to-end pipeline. The DRL agent is first trained in a training dataset and fined-tuned (adjusting hyperparameters) in a testing dataset. Then, backtest the agent (on historical dataset), or deploy in a paper/live trading market.

This pipeline address the information leakage problem by separating the training/testing-trading periods the agent never see the data in backtesting or paper/live trading stage.

And such a unified pipeline allows fair comparison among different algorithms.

4. Plug-and-play
================

In the development pipeline, we separate market environments from the data layer and the agent layer. Any DRL agent can be directly plugged into our environments, then will be trained and tested. Different agents can run on the same benchmark environment for fair comparisons. Several popular DRL libraries are supported, including ElegantRL, RLlib, and SB3.

---

## File: docs/source/reference/publication.md

# Publications

Papers by the Columbia research team can be found at [Google Scholar](https://scholar.google.com/citations?view_op=list_works&hl=en&hl=en&user=XsdPXocAAAAJ).

|Title |Conference |Link|Citations|Year|
|  ----  |  ----  |  ----  |  ----  |  ----  |
|**FinRL-Meta**: A Universe of Near-Real Market Environments for Data-Driven Deep Reinforcement Learning in Quantitative Finance| NeurIPS 2021 Data-Centric AI Workshop| [paper](https://arxiv.org/abs/2112.06753) ;[code](https://github.com/AI4Finance-Foundation/FinRL-Meta)| 2| 2021|
|Explainable deep reinforcement learning for portfolio management: An empirical approach| ICAIF 2021 : ACM International Conference on AI in Finance | [paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3958005);[code](https://github.com/AI4Finance-Foundation/FinRL)| 1| 2021|
|**FinRL-Podracer**: High performance and scalable deep reinforcement learning for quantitative finance| ICAIF 2021 : ACM International Conference on AI in Finance | [paper](https://arxiv.org/abs/2111.05188);[code](https://github.com/AI4Finance-Foundation/FinRL_Podracer)| 2 | 2021|
|**FinRL**: Deep reinforcement learning framework to automate trading in quantitative finance| ICAIF 2021 : ACM International Conference on AI in Finance | [paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3955949);[code](https://github.com/AI4Finance-Foundation/FinRL)| 4| 2021|
|**FinRL**: A deep reinforcement learning library for automated stock trading in quantitative finance| NeurIPS 2020 Deep RL Workshop  | [paper](https://arxiv.org/abs/2011.09607);[code](https://github.com/AI4Finance-Foundation/FinRL)| 20| 2020|
|Deep reinforcement learning for automated stock trading: An ensemble strategy| ICAIF 2020 : ACM International Conference on AI in Finance | [paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3690996);[code](https://github.com/AI4Finance-Foundation/Deep-Reinforcement-Learning-for-Automated-Stock-Trading-Ensemble-Strategy-ICAIF-2020)| 46 | 2020|
|Multi-agent reinforcement learning for liquidation strategy analysis| ICML 2019 Workshop on AI in Finance: Applications and Infrastructure for Multi-Agent Learning| [paper](https://arxiv.org/abs/1906.11046); [code](https://github.com/AI4Finance-Foundation/Liquidation-Analysis-using-Multi-Agent-Reinforcement-Learning-ICML-2019)| 19 | 2019|
|Practical deep reinforcement learning approach for stock trading| NeurIPS 2018 Workshop on Challenges and Opportunities for AI in Financial Services| [paper](https://arxiv.org/abs/1811.07522); [code](https://github.com/AI4Finance-Foundation/DQN-DDPG_Stock_Trading)| 87| 2018 |