Repository: microsoft/qlib

Stars: 40856

README.md Copy Chapter


:newspaper: What's NEW!   :sparkling_heart: Copy Chapter

Recent released features

Introducing <a href="https://github.com/microsoft/RD-Agent"><img src="docs/_static/img/rdagent_logo.png" alt="RD_Agent" style="height: 2em"></a>: LLM-Based Autonomous Evolving Agents for Industrial Data-Driven R&D Copy Section

We are excited to announce the release of RD-Agent📢, a powerful tool that supports automated factor mining and model optimization in quant investment R&D.

RD-Agent is now available on GitHub, and we welcome your star🌟!

To learn more, please visit our ♾️Demo page. Here, you will find demo videos in both English and Chinese to help you better understand the scenario and usage of RD-Agent.

We have prepared several demo videos for you:
| Scenario | Demo video (English) | Demo video (中文) |
| -- | ------ | ------ |
| Quant Factor Mining | Link | Link |
| Quant Factor Mining from reports | Link | Link |
| Quant Model Optimization | Link | Link |

- 📃Paper: R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization
- 👾Code: https://github.com/microsoft/RD-Agent/

@misc{li2025rdagentquant,
    title={R\&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization},
    author={Yuante Li and Xu Yang and Xiao Yang and Minrui Xu and Xisen Wang and Weiqing Liu and Jiang Bian},
    year={2025},
    eprint={2505.15155},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

*

| Feature | Status |
| -- | ------ |
| R&D-Agent-Quant Published | Apply R&D-Agent to Qlib for quant trading |
| BPQP for End-to-end learning | 📈Coming soon!(Under review) |
| 🔥LLM-driven Auto Quant Factory🔥 | 🚀 Released in ♾️RD-Agent on Aug 8, 2024 |
| KRNN and Sandwich models | :chart_with_upwards_trend: Released on May 26, 2023 |
| Release Qlib v0.9.0 | :octocat: Released on Dec 9, 2022 |
| RL Learning Framework | :hammer: :chart_with_upwards_trend: Released on Nov 10, 2022. #1332, #1322, #1316,#1299,#1263, #1244, #1169, #1125, #1076|
| HIST and IGMTF models | :chart_with_upwards_trend: Released on Apr 10, 2022 |
| Qlib notebook tutorial | 📖 Released on Apr 7, 2022 |
| Ibovespa index data | :rice: Released on Apr 6, 2022 |
| Point-in-Time database | :hammer: Released on Mar 10, 2022 |
| Arctic Provider Backend & Orderbook data example | :hammer: Released on Jan 17, 2022 |
| Meta-Learning-based framework & DDG-DA | :chart_with_upwards_trend: :hammer: Released on Jan 10, 2022 |
| Planning-based portfolio optimization | :hammer: Released on Dec 28, 2021 |
| Release Qlib v0.8.0 | :octocat: Released on Dec 8, 2021 |
| ADD model | :chart_with_upwards_trend: Released on Nov 22, 2021 |
| ADARNN model | :chart_with_upwards_trend: Released on Nov 14, 2021 |
| TCN model | :chart_with_upwards_trend: Released on Nov 4, 2021 |
| Nested Decision Framework | :hammer: Released on Oct 1, 2021. Example and Doc |
| Temporal Routing Adaptor (TRA) | :chart_with_upwards_trend: Released on July 30, 2021 |
| Transformer & Localformer | :chart_with_upwards_trend: Released on July 22, 2021 |
| Release Qlib v0.7.0 | :octocat: Released on July 12, 2021 |
| TCTS Model | :chart_with_upwards_trend: Released on July 1, 2021 |
| Online serving and automatic model rolling | :hammer: Released on May 17, 2021 |
| DoubleEnsemble Model | :chart_with_upwards_trend: Released on Mar 2, 2021 |
| High-frequency data processing example | :hammer: Released on Feb 5, 2021 |
| High-frequency trading example | :chart_with_upwards_trend: Part of code released on Jan 28, 2021 |
| High-frequency data(1min) | :rice: Released on Jan 27, 2021 |
| Tabnet Model | :chart_with_upwards_trend: Released on Jan 22, 2021 |

Features released before 2021 are not listed here.

<p align="center">
<img src="docs/_static/img/logo/1.png" />
</p>

Qlib is an open-source, AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning.

An increasing number of SOTA Quant research works/papers in diverse paradigms are being released in Qlib to collaboratively solve key challenges in quantitative investment. For example, 1) using supervised learning to mine the market's complex non-linear patterns from rich and heterogeneous financial data, 2) modeling the dynamic nature of the financial market using adaptive concept drift technology, and 3) using reinforcement learning to model continuous investment decisions and assist investors in optimizing their trading strategies.

It contains the full ML pipeline of data processing, model training, back-testing; and covers the entire chain of quantitative investment: alpha seeking, risk modeling, portfolio optimization, and order execution.
For more details, please refer to our paper "Qlib: An AI-oriented Quantitative Investment Platform".

<table>
<tbody>
<tr>
<th>Frameworks, Tutorial, Data & DevOps</th>
<th>Main Challenges & Solutions in Quant Research</th>
</tr>
<tr>
<td>
<li><a href="#plans"><strong>Plans</strong></a></li>
<li><a href="#framework-of-qlib">Framework of Qlib</a></li>
<li><a href="#quick-start">Quick Start</a></li>
<ul dir="auto">
<li type="circle"><a href="#installation">Installation</a> </li>
<li type="circle"><a href="#data-preparation">Data Preparation</a></li>
<li type="circle"><a href="#auto-quant-research-workflow">Auto Quant Research Workflow</a></li>
<li type="circle"><a href="#building-customized-quant-research-workflow-by-code">Building Customized Quant Research Workflow by Code</a></li></ul>
<li><a href="#quant-dataset-zoo"><strong>Quant Dataset Zoo</strong></a></li>
<li><a href="#learning-framework">Learning Framework</a></li>
<li><a href="#more-about-qlib">More About Qlib</a></li>
<li><a href="#offline-mode-and-online-mode">Offline Mode and Online Mode</a>
<ul>
<li type="circle"><a href="#performance-of-qlib-data-server">Performance of Qlib Data Server</a></li></ul>
<li><a href="#related-reports">Related Reports</a></li>
<li><a href="#contact-us">Contact Us</a></li>
<li><a href="#contributing">Contributing</a></li>
</td>
<td valign="baseline">
<li><a href="#main-challenges--solutions-in-quant-research">Main Challenges &amp; Solutions in Quant Research</a>
<ul>
<li type="circle"><a href="#forecasting-finding-valuable-signalspatterns">Forecasting: Finding Valuable Signals/Patterns</a>
<ul>
<li type="disc"><a href="#quant-model-paper-zoo"><strong>Quant Model (Paper) Zoo</strong></a>
<ul>
<li type="circle"><a href="#run-a-single-model">Run a Single Model</a></li>
<li type="circle"><a href="#run-multiple-models">Run Multiple Models</a></li>
</ul>
</li>
</ul>
</li>
<li type="circle"><a href="#adapting-to-market-dynamics">Adapting to Market Dynamics</a></li>
<li type="circle"><a href="#reinforcement-learning-modeling-continuous-decisions">Reinforcement Learning: modeling continuous decisions</a></li>
</ul>
</li>
</td>
</tr>
</tbody>
</table>

Plans

Framework of Qlib

<div style="align: center">
<img src="docs/_static/img/framework-abstract.jpg" />
</div>

The high-level framework of Qlib can be found above(users can find the detailed framework of Qlib's design when getting into nitty gritty).
The components are designed as loose-coupled modules, and each component could be used stand-alone.

Qlib provides a strong infrastructure to support Quant research. Data is always an important part.
A strong learning framework is designed to support diverse learning paradigms (e.g. reinforcement learning, supervised learning) and patterns at different levels(e.g. market dynamic modeling).
By modeling the market, trading strategies will generate trade decisions that will be executed. Multiple trading strategies and executors in different levels or granularities can be nested to be optimized and run together.
At last, a comprehensive analysis will be provided and the model can be served online in a low cost.

Quick Start

This quick start guide tries to demonstrate
1. It's very easy to build a complete Quant research workflow and try your ideas with _Qlib_.
2. Though with public data and simple models, machine learning technologies work very well in practical Quant investment.

Here is a quick demo shows how to install `Qlib, and run LightGBM with qrun. But, please make sure you have already prepared the data following the instruction.

Installation Copy Chapter

This table demonstrates the supported Python version of Qlib:
| | install with pip | install from source | plot |
| ------------- |:---------------------:|:--------------------:|:------------------:|
| Python 3.8 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Python 3.9 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Python 3.10 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Python 3.11 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Python 3.12 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |

Note:
1. Conda is suggested for managing your Python environment. In some cases, using Python outside of a conda environment may result in missing header files, causing the installation failure of certain packages.
2. Please pay attention that installing cython in Python 3.6 will raise some error when installing Qlib from source. If users use Python 3.6 on their machines, it is recommended to upgrade Python to version 3.8 or higher, or use conda's Python to install Qlib from source.

Install with pip Copy Section

pip install pyqlib

* Before installing Qlib from source, users need to install some dependencies:

pip install numpy
  pip install --upgrade cython

* Clone the repository and install Qlib as follows.


git clone https://github.com/microsoft/qlib.git && cd qlib
    pip install .  # pip install -e .[dev] is recommended for development. check details in docs/developer/code_standard_and_dev_guide.rst

Tips: If you fail to install Qlib or run the examples in your environment, comparing your steps and the CI workflow may help you find the problem.

Tips for Mac: If you are using Mac with M1, you might encounter issues in building the wheel for LightGBM, which is due to missing dependencies from OpenMP. To solve the problem, install openmp first with brew install libomp and then run pip install . to build it successfully.

Data Preparation Copy Chapter

wget https://github.com/chenditc/investment_data/releases/latest/download/qlib_bin.tar.gz
mkdir -p ~/.qlib/qlib_data/cn_data
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1
rm -f qlib_bin.tar.gz

# get 1d data
  python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn
  # get 1min data
  python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min

# get 1d data
  python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn
  # get 1min data
  python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min

* Automatic update of data to the "qlib" directory each trading day(Linux)
use crontab*: crontab -e
* set up timed tasks:

    1-5 python <script path> update_data_to_bin --qlib_data_1d_dir <user data dir>

python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <user data dir> --trading_date <start date> --end_date <end date>

python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data

python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data --missing_data_num 30055 --large_step_threshold_volume 94485 --large_step_threshold_price 20

import qlib
  from qlib.data import D
  from qlib.constant import REG_CN
  # Initialization
  mount_path = "~/.qlib/qlib_data/cn_data"  # target_dir
  qlib.init(mount_path=mount_path, region=REG_CN)
  # Get stock data by Qlib
  # Load trading calendar with the given time range and frequency
  print(D.calendar(start_time='2010-01-01', end_time='2017-12-31', freq='day')[:2])
  # Parse a given market name into a stockpool config
  instruments = D.instruments('csi500')
  print(D.list_instruments(instruments=instruments, start_time='2010-01-01', end_time='2017-12-31', as_list=True)[:6])
  # Load features of certain instruments in given time range
  instruments = ['SH600000']
  fields = ['$close', '$volume', 'Ref($close, 1)', 'Mean($close, 3)', '$high-$low']
  print(D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day').head())

docker pull pyqlib/qlib_image_stable:stable

docker run -it --name <container name> -v <Mounted local directory>:/app pyqlib/qlib_image_stable:stable

>>> python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn
    >>> python qlib/cli/run.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

>>> exit

docker start -i -a <container name>

docker stop <container name>

docker rm <container name>

1. Quant Research Workflow: Run qrun with lightgbm workflow config (workflow_config_lightgbm_Alpha158.yaml as following.


cd examples  # Avoid running program under the directory contains qlib
      qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

python -m pdb qlib/cli/run.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

'The following are analysis results of the excess return without cost.'
                           risk
    mean               0.000708
    std                0.005626
    annualized_return  0.178316
    information_ratio  1.996555
    max_drawdown      -0.081806
    'The following are analysis results of the excess return with cost.'
                           risk
    mean               0.000512
    std                0.005626
    annualized_return  0.128982
    information_ratio  1.444287
    max_drawdown      -0.091078

2. Graphical Reports Analysis: First, run python -m pip install .[analysis] to install the required dependencies. Then run examples/workflow_by_code.ipynb with jupyter notebook to get graphical reports.
- Forecasting signal (model prediction) analysis
- Cumulative Return of groups
!Cumulative Return
- Return distribution
!long_short
- Information Coefficient (IC)
!Information Coefficient
!Monthly IC
!IC
- Auto Correlation of forecasting signal (model prediction)
!Auto Correlation

- Portfolio analysis
- Backtest return
!Report
<!--
- Score IC
!Score IC
- Cumulative Return
!Cumulative Return
- Risk Analysis
!Risk Analysis
- Rank Label
!Rank Label
-->
- Explanation of above results

Building Customized Quant Research Workflow by Code Copy Chapter

An increasing number of SOTA Quant research works/papers, which focus on building forecasting models to mine valuable signals/patterns in complex financial data, are released in Qlib

Quant Model (Paper) Zoo Quant Model (Paper) Zoo')" title="Copy section prompt for LLMs"> Copy Section

Here is a list of models built on Qlib.
- GBDT based on XGBoost (Tianqi Chen, et al. KDD 2016)
- GBDT based on LightGBM (Guolin Ke, et al. NIPS 2017)
- GBDT based on Catboost (Liudmila Prokhorenkova, et al. NIPS 2018)
- MLP based on pytorch
- LSTM based on pytorch (Sepp Hochreiter, et al. Neural computation 1997)
- GRU based on pytorch (Kyunghyun Cho, et al. 2014)
- ALSTM based on pytorch (Yao Qin, et al. IJCAI 2017)
- GATs based on pytorch (Petar Velickovic, et al. 2017)
- SFM based on pytorch (Liheng Zhang, et al. KDD 2017)
- TFT based on tensorflow (Bryan Lim, et al. International Journal of Forecasting 2019)
- TabNet based on pytorch (Sercan O. Arik, et al. AAAI 2019)
- DoubleEnsemble based on LightGBM (Chuheng Zhang, et al. ICDM 2020)
- TCTS based on pytorch (Xueqing Wu, et al. ICML 2021)
- Transformer based on pytorch (Ashish Vaswani, et al. NeurIPS 2017)
- Localformer based on pytorch (Juyong Jiang, et al.)
- TRA based on pytorch (Hengxu, Dong, et al. KDD 2021)
- TCN based on pytorch (Shaojie Bai, et al. 2018)
- ADARNN based on pytorch (YunTao Du, et al. 2021)
- ADD based on pytorch (Hongshun Tang, et al.2020)
- IGMTF based on pytorch (Wentao Xu, et al.2021)
- HIST based on pytorch (Wentao Xu, et al.2021)
- KRNN based on pytorch
- Sandwich based on pytorch

Your PR of new Quant models is highly welcomed.

The performance of each model on the Alpha158 and Alpha360 datasets can be found here.

Run a single model Copy Section

Qlib provides three different ways to run a single model, users can pick the one that fits their cases best:
- Users can use the tool qrun mentioned above to run a model's workflow based from a config file.
- Users can create a workflow_by_code python script based on the one listed in the examples folder.

- Users can use the script run_all_model.py listed in the examples folder to run a model. Here is an example of the specific shell command to be used: python run_all_model.py run --models=lightgbm, where the --models arguments can take any number of models listed above(the available models can be found in benchmarks). For more use cases, please refer to the file's docstrings.
- NOTE: Each baseline has different environment dependencies, please make sure that your python version aligns with the requirements(e.g. TFT only supports Python 3.6~3.7 due to the limitation of tensorflow==1.15.0)

Run multiple models Copy Section

The script will create a unique virtual environment for each model, and delete the environments after training. Thus, only experiment results such as IC and backtest results will be generated and stored.

Here is an example of running all the models for 10 iterations:


python run_all_model.py run 10

It also provides the API to run specific models at once. For more use cases, please refer to the file's docstrings.

Break change Copy Section

Here is a list of solutions built on Qlib.
- Rolling Retraining
- DDG-DA on pytorch (Wendi, et al. AAAI 2022)

Reinforcement Learning: modeling continuous decisions Copy Chapter

Here is a list of solutions built on Qlib categorized by scenarios.

RL for order execution RL for order execution')" title="Copy section prompt for LLMs"> Copy Section

Here is a tutorial to build dataset with Qlib.
Your PR to build new Quant dataset is highly welcomed.

Learning Framework

Based on learning paradigms, they can be categorized into reinforcement learning and supervised learning.
- For supervised learning, the detailed docs can be found here.
- For reinforcement learning, the detailed docs can be found here. Qlib's RL learning framework leverages Execution Env in Workflow layer to create environments. It's worth noting that NestedExecutor is supported as well. This empowers users to optimize different level of strategies/models/agents together (e.g. optimizing an order execution strategy for a specific portfolio management strategy).

More About Qlib

cd docs/
conda install sphinx sphinx_rtd_theme -y
Otherwise, you can install them with pip

pip install sphinx sphinx_rtd_theme

make html

Under Offline mode, the data will be deployed locally.

Under Online mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too. The documents of the online mode can be found in Qlib-Server. The online mode can be deployed automatically with Azure CLI based scripts. The source code of online data server can be found in Qlib-Server repository.

Performance of Qlib Data Server Copy Chapter

| | HDF5 | MySQL | MongoDB | InfluxDB | Qlib -E -D | Qlib +E -D | Qlib +E +D |
| -- | ------ | ------ | -------- | --------- | ----------- | ------------ | ----------- |
| Total (1CPU) (seconds) | 184.4±3.7 | 365.3±7.5 | 253.6±6.7 | 368.2±3.6 | 147.0±8.8 | 47.6±1.0 | 7.4±0.3 |
| Total (64CPU) (seconds) | | | | | 8.8±0.6 | 4.2±0.2 | |
* +(-)E indicates with (out) ExpressionCache
* +(-)D indicates with (out) DatasetCache

Most general-purpose databases take too much time to load data. After looking into the underlying implementation, we find that data go through too many layers of interfaces and unnecessary format transformations in general-purpose database solutions.
Such overheads greatly slow down the data loading process.
Qlib data are stored in a compact format, which is efficient to be combined into arrays for scientific computation.

Related Reports

You can find some impefect implementation in Qlib by rg 'TODO|FIXME' qlib`

If you would like to become one of Qlib's maintainers to contribute more (e.g. help merge PR, triage issues), please contact us by email([email protected]). We are glad to help to upgrade your permission.

License Copy Chapter

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or
contact [email protected] with any additional questions or comments.