qlib Documentation

1. Project Overview & Quickstart (microsoft/qlib) Copy Chapter

File: README.md Copy Chapter

:newspaper: **What's NEW!**   :sparkling_heart: Copy Chapter

Recent released features

Introducing [](https://github.com/microsoft/RD-Agent): LLM-Based Autonomous Evolving Agents for Industrial Data-Driven R&D Copy Section

We are excited to announce the release of RD-Agent📢, a powerful tool that supports automated factor mining and model optimization in quant investment R&D.

RD-Agent is now available on GitHub, and we welcome your star🌟!

To learn more, please visit the RD-Agent repository. We have prepared several public demo videos for you:

• 📃Paper: R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization
• 👾Code: https://github.com/microsoft/RD-Agent/

@misc{li2025rdagentquant,
    title={R\&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization},
    author={Yuante Li and Xu Yang and Xiao Yang and Minrui Xu and Xisen Wang and Weiqing Liu and Jiang Bian},
    year={2025},
    eprint={2505.15155},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

Features released before 2021 are not listed here.

Qlib is an open-source, AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning.

An increasing number of SOTA Quant research works/papers in diverse paradigms are being released in Qlib to collaboratively solve key challenges in quantitative investment. For example, 1) using supervised learning to mine the market's complex non-linear patterns from rich and heterogeneous financial data, 2) modeling the dynamic nature of the financial market using adaptive concept drift technology, and 3) using reinforcement learning to model continuous investment decisions and assist investors in optimizing their trading strategies.

It contains the full ML pipeline of data processing, model training, back-testing; and covers the entire chain of quantitative investment: alpha seeking, risk modeling, portfolio optimization, and order execution.
For more details, please refer to our paper "Qlib: An AI-oriented Quantitative Investment Platform".

[Framework of Qlib](#framework-of-qlib)
    [Quick Start](#quick-start)
      
        [Installation](#installation) 
        [Data Preparation](#data-preparation)
        [Auto Quant Research Workflow](#auto-quant-research-workflow)
        [Building Customized Quant Research Workflow by Code](#building-customized-quant-research-workflow-by-code)
    [**Quant Dataset Zoo**](#quant-dataset-zoo)
    [Learning Framework](#learning-framework)
    [More About Qlib](#more-about-qlib)
    [Offline Mode and Online Mode](#offline-mode-and-online-mode)
    
      [Performance of Qlib Data Server](#performance-of-qlib-data-server)
    [Related Reports](#related-reports)
    [Contact Us](#contact-us)
    [Contributing](#contributing) | [Main Challenges & Solutions in Quant Research](#main-challenges--solutions-in-quant-research)
      
        [Forecasting: Finding Valuable Signals/Patterns](#forecasting-finding-valuable-signalspatterns)
          
            [**Quant Model (Paper) Zoo**](#quant-model-paper-zoo)
              
                [Run a Single Model](#run-a-single-model)
                [Run Multiple Models](#run-multiple-models)
              
            
          
        
      [Adapting to Market Dynamics](#adapting-to-market-dynamics)
      [Reinforcement Learning: modeling continuous decisions](#reinforcement-learning-modeling-continuous-decisions) |

Plans

New features under development(order by estimated release time).
Your feedbacks about the features are very important.

Framework of Qlib

The high-level framework of Qlib can be found above(users can find the detailed framework of Qlib's design when getting into nitty gritty).
The components are designed as loose-coupled modules, and each component could be used stand-alone.

Qlib provides a strong infrastructure to support Quant research. Data is always an important part.
A strong learning framework is designed to support diverse learning paradigms (e.g. reinforcement learning, supervised learning) and patterns at different levels(e.g. market dynamic modeling).
By modeling the market, trading strategies will generate trade decisions that will be executed. Multiple trading strategies and executors in different levels or granularities can be nested to be optimized and run together.
At last, a comprehensive analysis will be provided and the model can be served online in a low cost.

Quick Start

This quick start guide tries to demonstrate

1. It's very easy to build a complete Quant research workflow and try your ideas with Qlib.
2. Though with public data and simple models, machine learning technologies work very well in practical Quant investment.

Here is a quick demo shows how to install Qlib, and run LightGBM with qrun. But, please make sure you have already prepared the data following the instruction.

Installation Copy Chapter

This table demonstrates the supported Python version of Qlib:

Note:

1. Conda is suggested for managing your Python environment. In some cases, using Python outside of a conda environment may result in missing header files, causing the installation failure of certain packages.
2. Please pay attention that installing cython in Python 3.6 will raise some error when installing Qlib from source. If users use Python 3.6 on their machines, it is recommended to upgrade Python to version 3.8 or higher, or use conda's Python to install Qlib from source.

Install with pip Copy Section

Users can easily install Qlib by pip according to the following command.

pip install pyqlib

Note: pip will install the latest stable qlib. However, the main branch of qlib is in active development. If you want to test the latest scripts or functions in the main branch. Please install qlib with the methods below.

Install from source Copy Section

Also, users can install the latest dev version Qlib by the source code according to the following steps:

• Before installing Qlib from source, users need to install some dependencies:

  bash Copy

  pip install numpy
  pip install --upgrade cython

• Clone the repository and install Qlib as follows.

  bash Copy

  git clone https://github.com/microsoft/qlib.git && cd qlib
  pip install .  # `pip install -e .[dev]` is recommended for development. check details in docs/developer/code_standard_and_dev_guide.rst

Tips: If you fail to install Qlib or run the examples in your environment, comparing your steps and the CI workflow may help you find the problem.

Tips for Mac: If you are using Mac with M1, you might encounter issues in building the wheel for LightGBM, which is due to missing dependencies from OpenMP. To solve the problem, install openmp first with brew install libomp and then run pip install . to build it successfully.

Data Preparation Copy Chapter

❗ Due to more restrict data security policy. The official dataset is disabled temporarily. You can try this data source contributed by the community.
Here is an example to download the latest data.

wget https://github.com/chenditc/investment_data/releases/latest/download/qlib_bin.tar.gz
mkdir -p ~/.qlib/qlib_data/cn_data
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1
rm -f qlib_bin.tar.gz

The official dataset below will resume in short future.

Load and prepare data by running the following code:

Get with module Copy Section

# get 1d data
python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

# get 1min data
python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min

Get from source Copy Section

# get 1d data
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

# get 1min data
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min

This dataset is created by public data collected by crawler scripts, which have been released in
the same repository.
Users could create the same dataset with it. Description of dataset

Please pay ATTENTION that the data is collected from Yahoo Finance, and the data might not be perfect.
We recommend users to prepare their own data if they have a high-quality dataset. For more information, users can refer to the related document.

Automatic update of daily frequency data (from yahoo finance) Copy Section

This step is Optional if users only want to try their models and strategies on history data.

It is recommended that users update the data manually once (--trading_date 2021-05-25) and then set it to update automatically.

NOTE: Users can't incrementally update data based on the offline data provided by Qlib(some fields are removed to reduce the data size). Users should use yahoo collector to download Yahoo data from scratch and then incrementally update it.

For more information, please refer to: yahoo collector

• Automatic update of data to the "qlib" directory each trading day(Linux)

  • use crontab: crontab -e

  • set up timed tasks:

    text Copy

    * * * * 1-5 python <script path> update_data_to_bin --qlib_data_1d_dir

    • script path: scripts/data_collector/yahoo/collector.py

• Manual update of data

  text Copy

  python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir  --trading_date <start date> --end_date <end date>

  • trading_date: start of trading day
  • end_date: end of trading day(not included)

Checking the health of the data Copy Section

• We provide a script to check the health of the data, you can run the following commands to check whether the data is healthy or not.
  text Copy

  python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data

• Of course, you can also add some parameters to adjust the test results, such as this.
  text Copy

  python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data --missing_data_num 30055 --large_step_threshold_volume 94485 --large_step_threshold_price 20

• If you want more information about check_data_health, please refer to the documentation.

Docker images Copy Chapter

1. Pulling a docker image from a docker hub repository
   bash Copy

   docker pull pyqlib/qlib_image_stable:stable

2. Start a new Docker container
   bash Copy

   docker run -it --name <container name> -v <Mounted local directory>:/app pyqlib/qlib_image_stable:stable

3. At this point you are in the docker environment and can run the qlib scripts. An example:
   bash Copy

   >>> python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn
   >>> python qlib/cli/run.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

4. Exit the container
   bash Copy

   >>> exit

5. Restart the container
   bash Copy

   docker start -i -a <container name>

6. Stop the container
   bash Copy

   docker stop <container name>

7. Delete the container
   bash Copy

   docker rm <container name>

8. If you want to know more information, please refer to the documentation.

Auto Quant Research Workflow Copy Chapter

Qlib provides a tool named qrun to run the whole workflow automatically (including building dataset, training models, backtest and evaluation). You can start an auto quant research workflow and have a graphical reports analysis according to the following steps:

1. Quant Research Workflow: Run qrun with lightgbm workflow config (workflow_config_lightgbm_Alpha158.yaml as following.

   bash Copy

   cd examples  # Avoid running program under the directory contains `qlib`
     qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

   If users want to use qrun under debug mode, please use the following command:

   bash Copy

   python -m pdb qlib/cli/run.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

   The result of qrun is as follows, please refer to docs for more explanations about the result.

   bash Copy

   'The following are analysis results of the excess return without cost.'
                          risk
   mean               0.000708
   std                0.005626
   annualized_return  0.178316
   information_ratio  1.996555
   max_drawdown      -0.081806
   'The following are analysis results of the excess return with cost.'
                          risk
   mean               0.000512
   std                0.005626
   annualized_return  0.128982
   information_ratio  1.444287
   max_drawdown      -0.091078

   Here are detailed documents for qrun and workflow.

2. Graphical Reports Analysis: First, run python -m pip install .[analysis] to install the required dependencies. Then run examples/workflow_by_code.ipynb with jupyter notebook to get graphical reports.

   • Forecasting signal (model prediction) analysis

     • Cumulative Return of groups

     • Return distribution

     • Information Coefficient (IC)

     • Auto Correlation of forecasting signal (model prediction)

   • Portfolio analysis

     • Backtest return

   • Explanation of above results

Building Customized Quant Research Workflow by Code Copy Chapter

The automatic workflow may not suit the research workflow of all Quant researchers. To support a flexible Quant research workflow, Qlib also provides a modularized interface to allow researchers to build their own workflow by code. Here is a demo for customized Quant research workflow by code.

Main Challenges & Solutions in Quant Research

Quant investment is a very unique scenario with lots of key challenges to be solved.
Currently, Qlib provides some solutions for several of them.

Forecasting: Finding Valuable Signals/Patterns Copy Chapter

Accurate forecasting of the stock price trend is a very important part to construct profitable portfolios.
However, huge amount of data with various formats in the financial market which make it challenging to build forecasting models.

An increasing number of SOTA Quant research works/papers, which focus on building forecasting models to mine valuable signals/patterns in complex financial data, are released in Qlib

[Quant Model (Paper) Zoo](examples/benchmarks) Copy Section

Here is a list of models built on Qlib.

• GBDT based on XGBoost (Tianqi Chen, et al. KDD 2016)
• GBDT based on LightGBM (Guolin Ke, et al. NIPS 2017)
• GBDT based on Catboost (Liudmila Prokhorenkova, et al. NIPS 2018)
• MLP based on pytorch
• LSTM based on pytorch (Sepp Hochreiter, et al. Neural computation 1997)
• GRU based on pytorch (Kyunghyun Cho, et al. 2014)
• ALSTM based on pytorch (Yao Qin, et al. IJCAI 2017)
• GATs based on pytorch (Petar Velickovic, et al. 2017)
• SFM based on pytorch (Liheng Zhang, et al. KDD 2017)
• TFT based on tensorflow (Bryan Lim, et al. International Journal of Forecasting 2019)
• TabNet based on pytorch (Sercan O. Arik, et al. AAAI 2019)
• DoubleEnsemble based on LightGBM (Chuheng Zhang, et al. ICDM 2020)
• TCTS based on pytorch (Xueqing Wu, et al. ICML 2021)
• Transformer based on pytorch (Ashish Vaswani, et al. NeurIPS 2017)
• Localformer based on pytorch (Juyong Jiang, et al.)
• TRA based on pytorch (Hengxu, Dong, et al. KDD 2021)
• TCN based on pytorch (Shaojie Bai, et al. 2018)
• ADARNN based on pytorch (YunTao Du, et al. 2021)
• ADD based on pytorch (Hongshun Tang, et al.2020)
• IGMTF based on pytorch (Wentao Xu, et al.2021)
• HIST based on pytorch (Wentao Xu, et al.2021)
• KRNN based on pytorch
• Sandwich based on pytorch

Your PR of new Quant models is highly welcomed.

The performance of each model on the Alpha158 and Alpha360 datasets can be found here.

Run a single model Copy Section

All the models listed above are runnable with Qlib. Users can find the config files we provide and some details about the model through the benchmarks folder. More information can be retrieved at the model files listed above.

Qlib provides three different ways to run a single model, users can pick the one that fits their cases best:

• Users can use the tool qrun mentioned above to run a model's workflow based from a config file.

• Users can create a workflow_by_code python script based on the one listed in the examples folder.

• Users can use the script run_all_model.py listed in the examples folder to run a model. Here is an example of the specific shell command to be used: python run_all_model.py run --models=lightgbm, where the --models arguments can take any number of models listed above(the available models can be found in benchmarks). For more use cases, please refer to the file's docstrings.

  • NOTE: Each baseline has different environment dependencies, please make sure that your python version aligns with the requirements(e.g. TFT only supports Python 3.6~3.7 due to the limitation of tensorflow==1.15.0)

Run multiple models Copy Section

Qlib also provides a script run_all_model.py which can run multiple models for several iterations. (Note: the script only support Linux for now. Other OS will be supported in the future. Besides, it doesn't support parallel running the same model for multiple times as well, and this will be fixed in the future development too.)

The script will create a unique virtual environment for each model, and delete the environments after training. Thus, only experiment results such as IC and backtest results will be generated and stored.

Here is an example of running all the models for 10 iterations:

python run_all_model.py run 10

It also provides the API to run specific models at once. For more use cases, please refer to the file's docstrings.

Break change Copy Section

In pandas, group_key is one of the parameters of the groupby method. From version 1.5 to 2.0 of pandas, the default value of group_key has been changed from no default to True, which will cause qlib to report an error during operation. So we set group_key=False, but it doesn't guarantee that some programmes will run correctly, including:

• qlib\examples\rl_order_execution\scripts\gen_training_orders.py
• qlib\examples\benchmarks\TRA\src\dataset.MTSDatasetH.py
• qlib\examples\benchmarks\TFT\tft.py

[Adapting to Market Dynamics](examples/benchmarks_dynamic) Copy Chapter

Due to the non-stationary nature of the environment of the financial market, the data distribution may change in different periods, which makes the performance of models build on training data decays in the future test data.
So adapting the forecasting models/strategies to market dynamics is very important to the model/strategies' performance.

Here is a list of solutions built on Qlib.

• Rolling Retraining
• DDG-DA on pytorch (Wendi, et al. AAAI 2022)

Reinforcement Learning: modeling continuous decisions Copy Chapter

Qlib now supports reinforcement learning, a feature designed to model continuous investment decisions. This functionality assists investors in optimizing their trading strategies by learning from interactions with the environment to maximize some notion of cumulative reward.

Here is a list of solutions built on Qlib categorized by scenarios.

[RL for order execution](examples/rl_order_execution) Copy Section

Here is the introduction of this scenario. All the methods below are compared here.

• TWAP
• PPO: "An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization", IJCAL 2020
• OPDS: "Universal Trading for Order Execution with Oracle Policy Distillation", AAAI 2021

Quant Dataset Zoo

Dataset plays a very important role in Quant. Here is a list of the datasets built on Qlib:

Here is a tutorial to build dataset with Qlib.
Your PR to build new Quant dataset is highly welcomed.

Learning Framework

Qlib is high customizable and a lot of its components are learnable.
The learnable components are instances of Forecast Model and Trading Agent. They are learned based on the Learning Framework layer and then applied to multiple scenarios in Workflow layer.
The learning framework leverages the Workflow layer as well(e.g. sharing Information Extractor, creating environments based on Execution Env).

Based on learning paradigms, they can be categorized into reinforcement learning and supervised learning.

• For supervised learning, the detailed docs can be found here.
• For reinforcement learning, the detailed docs can be found here. Qlib's RL learning framework leverages Execution Env in Workflow layer to create environments. It's worth noting that NestedExecutor is supported as well. This empowers users to optimize different level of strategies/models/agents together (e.g. optimizing an order execution strategy for a specific portfolio management strategy).

More About Qlib

If you want to have a quick glance at the most frequently used components of qlib, you can try notebooks here.

The detailed documents are organized in docs.
Sphinx and the readthedocs theme is required to build the documentation in html formats.

cd docs/
conda install sphinx sphinx_rtd_theme -y
# Otherwise, you can install them with pip
# pip install sphinx sphinx_rtd_theme
make html

You can also view the latest document online directly.

Qlib is in active and continuing development. Our plan is in the roadmap, which is managed as a github project.

Offline Mode and Online Mode

The data server of Qlib can either deployed as Offline mode or Online mode. The default mode is offline mode.

Under Offline mode, the data will be deployed locally.

Under Online mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too. The documents of the online mode can be found in Qlib-Server. The online mode can be deployed automatically with Azure CLI based scripts. The source code of online data server can be found in Qlib-Server repository.

Performance of Qlib Data Server Copy Chapter

The performance of data processing is important to data-driven methods like AI technologies. As an AI-oriented platform, Qlib provides a solution for data storage and data processing. To demonstrate the performance of Qlib data server, we
compare it with several other data storage solutions.

We evaluate the performance of several storage solutions by finishing the same task,
which creates a dataset (14 features/factors) from the basic OHLCV daily data of a stock market (800 stocks each day from 2007 to 2020). The task involves data queries and processing.

• +(-)E indicates with (out) ExpressionCache
• +(-)D indicates with (out) DatasetCache

Most general-purpose databases take too much time to load data. After looking into the underlying implementation, we find that data go through too many layers of interfaces and unnecessary format transformations in general-purpose database solutions.
Such overheads greatly slow down the data loading process.
Qlib data are stored in a compact format, which is efficient to be combined into arrays for scientific computation.

Related Reports

• Guide To Qlib: Microsoft’s AI Investment Platform
• 微软也搞AI量化平台？还是开源的！
• 微矿Qlib：业内首个AI量化投资开源平台

Contact Us

• If you have any issues, please create issue here or send messages in gitter.
• If you want to make contributions to Qlib, please create pull requests.
• For other reasons, you are welcome to contact us by email([email protected]).
  • We are recruiting new members(both FTEs and interns), your resumes are welcome!

Join IM discussion groups:

Contributing

We appreciate all contributions and thank all the contributors!

Before we released Qlib as an open-source project on Github in Sep 2020, Qlib is an internal project in our group. Unfortunately, the internal commit history is not kept. A lot of members in our group have also contributed a lot to Qlib, which includes Ruihua Wang, Yinda Zhang, Haisu Yu, Shuyu Wang, Bochen Pang, and Dong Zhou. Especially thanks to Dong Zhou due to his initial version of Qlib.

Guidance Copy Chapter

This project welcomes contributions and suggestions.
Here are some
code standards and development guidance for submiting a pull request.

Making contributions is not a hard thing. Solving an issue(maybe just answering a question raised in issues list or gitter), fixing/issuing a bug, improving the documents and even fixing a typo are important contributions to Qlib.

For example, if you want to contribute to Qlib's document/code, you can follow the steps in the figure below.

If you don't know how to start to contribute, you can refer to the following examples.

Good first issues are labelled to indicate that they are easy to start your contributions.

You can find some impefect implementation in Qlib by rg 'TODO|FIXME' qlib

If you would like to become one of Qlib's maintainers to contribute more (e.g. help merge PR, triage issues), please contact us by email([email protected]). We are glad to help to upgrade your permission.

License Copy Chapter

Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the right to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or
contact [email protected] with any additional questions or comments.

File: docs/component/rl/framework.rst Copy Chapter

The Framework of QlibRL

QlibRL contains a full set of components that cover the entire lifecycle of an RL pipeline, including building the simulator of the market, shaping states & actions, training policies (strategies), and backtesting strategies in the simulated environment.

QlibRL is basically implemented with the support of Tianshou and Gym frameworks. The high-level structure of QlibRL is demonstrated below:

.. image:: ../../_static/img/QlibRL_framework.png
:width: 600
:align: center

Here, we briefly introduce each component in the figure.

EnvWrapper Copy Chapter

EnvWrapper is the complete capsulation of the simulated environment. It receives actions from outside (policy/strategy/agent), simulates the changes in the market, and then replies rewards and updated states, thus forming an interaction loop.

In QlibRL, EnvWrapper is a subclass of gym.Env, so it implements all necessary interfaces of gym.Env. Any classes or pipelines that accept gym.Env should also accept EnvWrapper. Developers do not need to implement their own EnvWrapper to build their own environment. Instead, they only need to implement 4 components of the EnvWrapper:

• Simulator
  The simulator is the core component responsible for the environment simulation. Developers could implement all the logic that is directly related to the environment simulation in the Simulator in any way they like. In QlibRL, there are already two implementations of Simulator for single asset trading: 1) SingleAssetOrderExecution, which is built based on Qlib's backtest toolkits and hence considers a lot of practical trading details but is slow. 2) SimpleSingleAssetOrderExecution, which is built based on a simplified trading simulator, which ignores a lot of details (e.g. trading limitations, rounding) but is quite fast.
• State interpreter
  The state interpreter is responsible for "interpret" states in the original format (format provided by the simulator) into states in a format that the policy could understand. For example, transform unstructured raw features into numerical tensors.
• Action interpreter
  The action interpreter is similar to the state interpreter. But instead of states, it interprets actions generated by the policy, from the format provided by the policy to the format that is acceptable to the simulator.
• Reward function
  The reward function returns a numerical reward to the policy after each time the policy takes an action.

EnvWrapper will organically organize these components. Such decomposition allows for better flexibility in development. For example, if the developers want to train multiple types of policies in the same environment, they only need to design one simulator and design different state interpreters/action interpreters/reward functions for different types of policies.

QlibRL has well-defined base classes for all these 4 components. All the developers need to do is define their own components by inheriting the base classes and then implementing all interfaces required by the base classes. The API for the above base components can be found here <../../reference/api.html#module-qlib.rl>__.

Policy Copy Chapter

QlibRL directly uses Tianshou's policy. Developers could use policies provided by Tianshou off the shelf, or implement their own policies by inheriting Tianshou's policies.

Training Vessel & Trainer Copy Chapter

As stated by their names, training vessels and trainers are helper classes used in training. A training vessel is a ship that contains a simulator/interpreters/reward function/policy, and it controls algorithm-related parts of training. Correspondingly, the trainer is responsible for controlling the runtime parts of training.

As you may have noticed, a training vessel itself holds all the required components to build an EnvWrapper rather than holding an instance of EnvWrapper directly. This allows the training vessel to create duplicates of EnvWrapper dynamically when necessary (for example, under parallel training).

With a training vessel, the trainer could finally launch the training pipeline by simple, Scikit-learn-like interfaces (i.e., trainer.fit()).

The API for Trainer and TrainingVessel and can be found here <../../reference/api.html#module-qlib.rl.trainer>__.

The RL module is designed in a loosely-coupled way. Currently, RL examples are integrated with concrete business logic.
But the core part of RL is much simpler than what you see.
To demonstrate the simple core of RL, a dedicated notebook <https://github.com/microsoft/qlib/tree/main/examples/rl/simple_example.ipynb>__ for RL without business loss is created.

File: docs/component/rl/guidance.rst Copy Chapter

======== Guidance

.. currentmodule:: qlib

QlibRL can help users quickly get started and conveniently implement quantitative strategies based on reinforcement learning(RL) algorithms. For different user groups, we recommend the following guidance to use QlibRL.

Beginners to Reinforcement Learning Algorithms

Whether you are a quantitative researcher who wants to understand what RL can do in trading or a learner who wants to get started with RL algorithms in trading scenarios, if you have limited knowledge of RL and want to shield various detailed settings to quickly get started with RL algorithms, we recommend the following sequence to learn qlibrl:

• Learn the fundamentals of RL in part1 <https://qlib.readthedocs.io/en/latest/component/rl/overall.html#reinforcement-learning>_.
• Understand the trading scenarios where RL methods can be applied in part2 <https://qlib.readthedocs.io/en/latest/component/rl/overall.html#potential-application-scenarios-in-quantitative-trading>_.
• Run the examples in part3 <https://qlib.readthedocs.io/en/latest/component/rl/quickstart.html>_ to solve trading problems using RL.
• If you want to further explore QlibRL and make some customizations, you need to first understand the framework of QlibRL in part4 <https://qlib.readthedocs.io/en/latest/component/rl/framework.html>_ and rewrite specific components according to your needs.

Reinforcement Learning Algorithm Researcher

If you are already familiar with existing RL algorithms and dedicated to researching RL algorithms but lack domain knowledge in the financial field, and you want to validate the effectiveness of your algorithms in financial trading scenarios, we recommend the following steps to get started with QlibRL:

• Understand the trading scenarios where RL methods can be applied in part2 <https://qlib.readthedocs.io/en/latest/component/rl/overall.html#potential-application-scenarios-in-quantitative-trading>_.
• Choose an RL application scenario (currently, QlibRL has implemented two scenario examples: order execution and algorithmic trading). Run the example in part3 <https://qlib.readthedocs.io/en/latest/component/rl/quickstart.html>_ to get it working.
• Modify the policy <https://github.com/microsoft/qlib/blob/main/qlib/rl/order_execution/policy.py>_ part to incorporate your own RL algorithm.

Quantitative Researcher

If you have a certain level of financial domain knowledge and coding skills, and you want to explore the application of RL algorithms in the investment field, we recommend the following steps to explore QlibRL:

• Learn the fundamentals of RL in part1 <https://qlib.readthedocs.io/en/latest/component/rl/overall.html#reinforcement-learning>_.
• Understand the trading scenarios where RL methods can be applied in part2 <https://qlib.readthedocs.io/en/latest/component/rl/overall.html#potential-application-scenarios-in-quantitative-trading>_.
• Run the examples in part3 <https://qlib.readthedocs.io/en/latest/component/rl/quickstart.html>_ to solve trading problems using RL.
• Understand the framework of QlibRL in part4 <https://qlib.readthedocs.io/en/latest/component/rl/framework.html>_.
• Choose a suitable RL algorithm based on the characteristics of the problem you want to solve (currently, QlibRL supports PPO and DQN algorithms based on tianshou).
• Design the MDP (Markov Decision Process) process based on market trading rules and the problem you want to solve. Refer to the example in order execution and make corresponding modifications to the following modules: State <https://github.com/microsoft/qlib/blob/main/qlib/rl/order_execution/state.py#L70>, Metrics <https://github.com/microsoft/qlib/blob/main/qlib/rl/order_execution/state.py#L18>, ActionInterpreter <https://github.com/microsoft/qlib/blob/main/qlib/rl/order_execution/interpreter.py#L199>, StateInterpreter <https://github.com/microsoft/qlib/blob/main/qlib/rl/order_execution/interpreter.py#L68>, Reward <https://github.com/microsoft/qlib/blob/main/qlib/rl/order_execution/reward.py>, Observation <https://github.com/microsoft/qlib/blob/main/qlib/rl/order_execution/interpreter.py#L44>, Simulator <https://github.com/microsoft/qlib/blob/main/qlib/rl/order_execution/simulator_simple.py>_.

File: docs/component/rl/overall.rst Copy Chapter

===================================================== Reinforcement Learning in Quantitative Trading

Reinforcement Learning

Different from supervised learning tasks such as classification tasks and regression tasks. Another important paradigm in machine learning is Reinforcement Learning(RL),
which attempts to optimize an accumulative numerical reward signal by directly interacting with the environment under a few assumptions such as Markov Decision Process(MDP).

As demonstrated in the following figure, an RL system consists of four elements, 1)the agent 2) the environment the agent interacts with 3) the policy that the agent follows to take actions on the environment and 4)the reward signal from the environment to the agent.
In general, the agent can perceive and interpret its environment, take actions and learn through reward, to seek long-term and maximum overall reward to achieve an optimal solution.

.. image:: ../../_static/img/RL_framework.png
:width: 300
:align: center

RL attempts to learn to produce actions by trial and error.
By sampling actions and then observing which one leads to our desired outcome, a policy is obtained to generate optimal actions.
In contrast to supervised learning, RL learns this not from a label but from a time-delayed label called a reward.
This scalar value lets us know whether the current outcome is good or bad.
In a word, the target of RL is to take actions to maximize reward.

The Qlib Reinforcement Learning toolkit (QlibRL) is an RL platform for quantitative investment, which provides support to implement the RL algorithms in Qlib.

Potential Application Scenarios in Quantitative Trading

RL methods have demonstrated remarkable achievements in various applications, including game playing, resource allocation, recommendation systems, marketing, and advertising.
In the context of investment, which involves continuous decision-making, let's consider the example of the stock market. Investors strive to optimize their investment returns by effectively managing their positions and stock holdings through various buying and selling behaviors.
Furthermore, investors carefully evaluate market conditions and stock-specific information before making each buying or selling decision. From an investor's perspective, this process can be viewed as a continuous decision-making process driven by interactions with the market. RL algorithms offer a promising approach to tackle such challenges.
Here are several scenarios where RL holds potential for application in quantitative investment.

Order Execution Copy Chapter

The order execution task is to execute orders efficiently while considering multiple factors, including optimal prices, minimizing trading costs, reducing market impact, maximizing order fullfill rates, and achieving execution within a specified time frame. RL can be applied to such tasks by incorporating these objectives into the reward function and action selection process. Specifically, the RL agent interacts with the market environment, observes the state from market information, and makes decisions on next step execution. The RL algorithm learns an optimal execution strategy through trial and error, aiming to maximize the expected cumulative reward, which incorporates the desired objectives.

• General Setting

  • Environment: The environment represents the financial market where order execution takes place. It encompasses variables such as the order book dynamics, liquidity, price movements, and market conditions.

  • State: The state refers to the information available to the RL agent at a given time step. It typically includes features such as the current order book state (bid-ask spread, order depth), historical price data, historical trading volume, market volatility, and any other relevant information that can aid in decision-making.

  • Action: The action is the decision made by the RL agent based on the observed state. In order execution, actions can include selecting the order size, price, and timing of execution.

  • Reward: The reward is a scalar signal that indicates the performance of the RL agent's action in the environment. The reward function is designed to encourage actions that lead to efficient and cost-effective order execution. It typically considers multiple objectives, such as maximizing price advantages, minimizing trading costs (including transaction fees and slippage), reducing market impact (the effect of the order on the market price) and maximizing order fullfill rates.

• Scenarios

  • Single-asset order execution: Single-asset order execution focuses on the task of executing a single order for a specific asset, such as a stock or a cryptocurrency. The primary objective is to execute the order efficiently while considering factors such as maximizing price advantages, minimizing trading costs, reducing market impact, and achieving a high fullfill rate. The RL agent interacts with the market environment and makes decisions on order size, price, and timing of execution for that particular asset. The goal is to learn an optimal execution strategy for the single asset, maximizing the expected cumulative reward while considering the specific dynamics and characteristics of that asset.

  • Multi-asset order execution: Multi-asset order execution expands the order execution task to involve multiple assets or securities. It typically involves executing a portfolio of orders across different assets simultaneously or sequentially. Unlike single-asset order execution, the focus is not only on the execution of individual orders but also on managing the interactions and dependencies between different assets within the portfolio. The RL agent needs to make decisions on the order sizes, prices, and timings for each asset in the portfolio, considering their interdependencies, cash constraints, market conditions, and transaction costs. The goal is to learn an optimal execution strategy that balances the execution efficiency for each asset while considering the overall performance and objectives of the portfolio as a whole.

The choice of settings and RL algorithm depends on the specific requirements of the task, available data, and desired performance objectives.

Portfolio Construction Copy Chapter

Portfolio construction is a process of selecting and allocating assets in an investment portfolio. RL provides a framework to optimize portfolio management decisions by learning from interactions with the market environment and maximizing long-term returns while considering risk management.

• General Setting

  • State: The state represents the current information about the market and the portfolio. It typically includes historical prices and volumes, technical indicators, and other relevant data.

  • Action: The action corresponds to the decision of allocating capital to different assets in the portfolio. It determines the weights or proportions of investments in each asset.

  • Reward: The reward is a metric that evaluates the performance of the portfolio. It can be defined in various ways, such as total return, risk-adjusted return, or other objectives like maximizing Sharpe ratio or minimizing drawdown.

• Scenarios

  • Stock market: RL can be used to construct portfolios of stocks, where the agent learns to allocate capital among different stocks.

  • Cryptocurrency market: RL can be applied to construct portfolios of cryptocurrencies, where the agent learns to make allocation decisions.

  • Foreign exchange (Forex) market: RL can be used to construct portfolios of currency pairs, where the agent learns to allocate capital across different currencies based on exchange rate data, economic indicators, and other factors.

Similarly, the choice of basic setting and algorithm depends on the specific requirements of the problem and the characteristics of the market.

File: docs/component/rl/quickstart.rst Copy Chapter

Quick Start

.. currentmodule:: qlib

QlibRL provides an example of an implementation of a single asset order execution task and the following is an example of the config file to train with QlibRL.

.. code-block:: yaml

simulator:
    # Each step contains 30mins
    time_per_step: 30
    # Upper bound of volume, should be null or a float between 0 and 1, if it is a float, represent upper bound is calculated by the percentage of the market volume
    vol_limit: null
env:
    # Concurrent environment workers.
    concurrency: 1
    # dummy or subproc or shmem. Corresponding to `parallelism in tianshou <https://tianshou.readthedocs.io/en/master/api/tianshou.env.html#vectorenv>`_.
    parallel_mode: dummy
action_interpreter:
    class: CategoricalActionInterpreter
    kwargs:
        # Candidate actions, it can be a list with length L: [a_1, a_2,..., a_L] or an integer n, in which case the list of length n+1 is auto-generated, i.e., [0, 1/n, 2/n,..., n/n].
        values: 14
        # Total number of steps (an upper-bound estimation)
        max_step: 8
    module_path: qlib.rl.order_execution.interpreter
state_interpreter:
    class: FullHistoryStateInterpreter
    kwargs:
        # Number of dimensions in data.
        data_dim: 6
        # Equal to the total number of records. For example, in SAOE per minute, data_ticks is the length of the day in minutes.
        data_ticks: 240
        # The total number of steps (an upper-bound estimation). For example, 390min / 30min-per-step = 13 steps.
        max_step: 8
        # Provider of the processed data.
        processed_data_provider:
            class: PickleProcessedDataProvider
            module_path: qlib.rl.data.pickle_styled
            kwargs:
                data_dir: ./data/pickle_dataframe/feature
    module_path: qlib.rl.order_execution.interpreter
reward:
    class: PAPenaltyReward
    kwargs:
        # The penalty for a large volume in a short time.
        penalty: 100.0
    module_path: qlib.rl.order_execution.reward
data:
    source:
        order_dir: ./data/training_order_split
        data_dir: ./data/pickle_dataframe/backtest
        # number of time indexes
        total_time: 240
        # start time index
        default_start_time: 0
        # end time index
        default_end_time: 240
        proc_data_dim: 6
    num_workers: 0
    queue_size: 20
network:
    class: Recurrent
    module_path: qlib.rl.order_execution.network
policy:
    class: PPO
    kwargs:
        lr: 0.0001
    module_path: qlib.rl.order_execution.policy
runtime:
    seed: 42
    use_cuda: false
trainer:
    max_epoch: 2
    # Number of episodes collected in each training iteration
    repeat_per_collect: 5
    earlystop_patience: 2
    # Episodes per collect at training.
    episode_per_collect: 20
    batch_size: 16
    # Perform validation every n iterations
    val_every_n_epoch: 1
    checkpoint_path: ./checkpoints
    checkpoint_every_n_iters: 1

And the config file for backtesting:

.. code-block:: yaml

order_file: ./data/backtest_orders.csv
start_time: "9:45"
end_time: "14:44"
qlib:
    provider_uri_1min: ./data/bin
    feature_root_dir: ./data/pickle
    # feature generated by today's information
    feature_columns_today: [
        "$open", "$high", "$low", "$close", "$vwap", "$volume",
    ]
    # feature generated by yesterday's information
    feature_columns_yesterday: [
        "$open_v1", "$high_v1", "$low_v1", "$close_v1", "$vwap_v1", "$volume_v1",
    ]
exchange:
    # the expression for buying and selling stock limitation
    limit_threshold: ['$close == 0', '$close == 0']
    # deal price for buying and selling
    deal_price: ["If($close == 0, $vwap, $close)", "If($close == 0, $vwap, $close)"]
volume_threshold:
    # volume limits are both buying and selling, "cum" means that this is a cumulative value over time
    all: ["cum", "0.2 * DayCumsum($volume, '9:45', '14:44')"]
    # the volume limits of buying
    buy: ["current", "$close"]
    # the volume limits of selling, "current" means that this is a real-time value and will not accumulate over time
    sell: ["current", "$close"]
strategies: 
    30min: 
        class: TWAPStrategy
        module_path: qlib.contrib.strategy.rule_strategy
        kwargs: {}
    1day: 
        class: SAOEIntStrategy
        module_path: qlib.rl.order_execution.strategy
        kwargs:
        state_interpreter:
            class: FullHistoryStateInterpreter
            module_path: qlib.rl.order_execution.interpreter
            kwargs:
            max_step: 8
            data_ticks: 240
            data_dim: 6
            processed_data_provider:
                class: PickleProcessedDataProvider
                module_path: qlib.rl.data.pickle_styled
                kwargs:
                data_dir: ./data/pickle_dataframe/feature
        action_interpreter: 
            class: CategoricalActionInterpreter
            module_path: qlib.rl.order_execution.interpreter
            kwargs: 
            values: 14
            max_step: 8
        network: 
            class: Recurrent
            module_path: qlib.rl.order_execution.network
            kwargs: {}
        policy: 
            class: PPO
            module_path: qlib.rl.order_execution.policy
            kwargs: 
                lr: 1.0e-4
                # Local path to the latest model. The model is generated during training, so please run training first if you want to run backtest with a trained policy. You could also remove this parameter file to run backtest with a randomly initialized policy.
                weight_file: ./checkpoints/latest.pth
# Concurrent environment workers.
concurrency: 5

With the above config files, you can start training the agent by the following command:

.. code-block:: console

$ python -m qlib.rl.contrib.train_onpolicy.py --config_path train_config.yml

After the training, you can backtest with the following command:

.. code-block:: console

$ python -m qlib.rl.contrib.backtest.py --config_path backtest_config.yml

In that case, :class:~qlib.rl.order_execution.simulator_qlib.SingleAssetOrderExecution and :class:~qlib.rl.order_execution.simulator_simple.SingleAssetOrderExecutionSimple as examples for simulator, :class:qlib.rl.order_execution.interpreter.FullHistoryStateInterpreter and :class:qlib.rl.order_execution.interpreter.CategoricalActionInterpreter as examples for interpreter, :class:qlib.rl.order_execution.policy.PPO as an example for policy, and :class:qlib.rl.order_execution.reward.PAPenaltyReward as an example for reward.
For the single asset order execution task, if developers have already defined their simulator/interpreters/reward function/policy, they could launch the training and backtest pipeline by simply modifying the corresponding settings in the config files.
The details about the example can be found here <https://github.com/microsoft/qlib/blob/main/examples/rl/README.md>_.

In the future, we will provide more examples for different scenarios such as RL-based portfolio construction.

File: docs/component/rl/toctree.rst Copy Chapter

.. _rl:

======================================================================== Reinforcement Learning in Quantitative Trading

.. toctree::
Guidance
Overall
Quick Start
Framework

File: docs/component/data.rst Copy Chapter

.. _data:

================================== Data Layer: Data Framework & Usage

Introduction

Data Layer provides user-friendly APIs to manage and retrieve data. It provides high-performance data infrastructure.

It is designed for quantitative investment. For example, users could build formulaic alphas with Data Layer easily. Please refer to Building Formulaic Alphas <../advanced/alpha.html>_ for more details.

The introduction of Data Layer includes the following parts.

• Data Preparation
• Data API
• Data Loader
• Data Handler
• Dataset
• Cache
• Data and Cache File Structure

Here is a typical example of Qlib data workflow

• Users download data and converting data into Qlib format(with filename suffix .bin). In this step, typically only some basic data are stored on disk(such as OHLCV).
• Creating some basic features based on Qlib's expression Engine(e.g. "Ref($close, 60) / $close", the return of last 60 trading days). Supported operators in the expression engine can be found here <https://github.com/microsoft/qlib/blob/main/qlib/data/ops.py>. This step is typically implemented in Qlib's Data Loader <https://qlib.readthedocs.io/en/latest/component/data.html#data-loader> which is a component of Data Handler <https://qlib.readthedocs.io/en/latest/component/data.html#data-handler> .
• If users require more complicated data processing (e.g. data normalization), Data Handler <https://qlib.readthedocs.io/en/latest/component/data.html#data-handler>_ support user-customized processors to process data(some predefined processors can be found here <https://github.com/microsoft/qlib/blob/main/qlib/data/dataset/processor.py>__). The processors are different from operators in expression engine. It is designed for some complicated data processing methods which is hard to supported in operators in expression engine.
• At last, Dataset <https://qlib.readthedocs.io/en/latest/component/data.html#dataset>_ is responsible to prepare model-specific dataset from the processed data of Data Handler

Data Preparation

Qlib Format Data Copy Chapter

We've specially designed a data structure to manage financial data, please refer to the File storage design section in Qlib paper <https://arxiv.org/abs/2009.11189>_ for detailed information.
Such data will be stored with filename suffix .bin (We'll call them .bin file, .bin format, or qlib format). .bin file is designed for scientific computing on finance data.

Qlib provides two different off-the-shelf datasets, which can be accessed through this link <https://github.com/microsoft/qlib/blob/main/qlib/contrib/data/handler.py>__:

======================== ================= ================
Dataset US Market China Market
======================== ================= ================
Alpha360 √ √

Alpha158 √ √
======================== ================= ================

Also, Qlib provides a high-frequency dataset. Users can run a high-frequency dataset example through this link <https://github.com/microsoft/qlib/tree/main/examples/highfreq>__.

Qlib Format Dataset Copy Chapter

Qlib has provided an off-the-shelf dataset in .bin format, users could use the script scripts/get_data.py to download the China-Stock dataset as follows. User can also use numpy to load .bin file to validate data.
The price volume data look different from the actual dealing price because of they are adjusted (adjusted price <https://www.investopedia.com/terms/a/adjusted_closing_price.asp>_). And then you may find that the adjusted price may be different from different data sources. This is because different data sources may vary in the way of adjusting prices. Qlib normalize the price on first trading day of each stock to 1 when adjusting them.
Users can leverage $factor to get the original trading price (e.g. $close / $factor to get the original close price).

Here are some discussions about the price adjusting of Qlib.

• https://github.com/microsoft/qlib/issues/991#issuecomment-1075252402

.. code-block:: bash

# download 1d
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

# download 1min
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/qlib_cn_1min --region cn --interval 1min

In addition to China-Stock data, Qlib also includes a US-Stock dataset, which can be downloaded with the following command:

.. code-block:: bash

python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/us_data --region us

After running the above command, users can find china-stock and us-stock data in Qlib format in the ~/.qlib/qlib_data/cn_data directory and ~/.qlib/qlib_data/us_data directory respectively.

Qlib also provides the scripts in scripts/data_collector to help users crawl the latest data on the Internet and convert it to qlib format.

When Qlib is initialized with this dataset, users could build and evaluate their own models with it. Please refer to Initialization <../start/initialization.html>_ for more details.

Automatic update of daily frequency data Copy Chapter

It is recommended that users update the data manually once (--trading_date 2021-05-25) and then set it to update automatically.

For more information refer to: yahoo collector <https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo#Automatic-update-of-daily-frequency-data>_

• Automatic update of data to the "qlib" directory each trading day(Linux)

  • use crontab: crontab -e

  • set up timed tasks:

    .. code-block:: bash

    text Copy

    * * * * 1-5 python <script path> update_data_to_bin --qlib_data_1d_dir

    • script path: scripts/data_collector/yahoo/collector.py

• Manual update of data

  .. code-block:: bash

  text Copy

  python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir  --trading_date <start date> --end_date <end date>

  • trading_date: start of trading day
  • end_date: end of trading day(not included)

Converting CSV and Parquet Format into Qlib Format Copy Chapter

Qlib has provided the script scripts/dump_bin.py to convert any data in CSV or Parquet format into .bin files (Qlib format) as long as they are in the correct format.

Besides downloading the prepared demo data, users could download demo data directly from the Collector as follows for reference to the CSV format.
Here are some example:

for daily data:
.. code-block:: bash

python scripts/get_data.py download_data --file_name csv_data_cn.zip --target_dir ~/.qlib/csv_data/cn_data

for 1min data:
.. code-block:: bash

python scripts/data_collector/yahoo/collector.py download_data --source_dir ~/.qlib/stock_data/source/cn_1min --region CN --start 2021-05-20 --end 2021-05-23 --delay 0.1 --interval 1min --limit_nums 10

Users can also provide their own data in CSV or Parquet format. However, the data must satisfies following criterions:

• CSV or Parquet file is named after a specific stock or the CSV or Parquet file includes a column of the stock name

  • Name the CSV or Parquet file after a stock: SH600000.csv, AAPL.csv or SH600000.parquet, AAPL.parquet (not case sensitive).

  • CSV or Parquet file includes a column of the stock name. User must specify the column name when dumping the data. Here is an example:

    .. code-block:: bash

    text Copy

    python scripts/dump_bin.py dump_all ... --symbol_field_name symbol --file_suffix <.csv or .parquet>

    where the data are in the following format:

    text Copy

    +-----------+-------+
      | symbol    | close |
      +===========+=======+
      | SH600000  | 120   |
      +-----------+-------+

• CSV or Parquet file must include a column for the date, and when dumping the data, user must specify the date column name. Here is an example:

  .. code-block:: bash

  text Copy

  python scripts/dump_bin.py dump_all ... --date_field_name date --file_suffix <.csv or .parquet>

  where the data are in the following format:

  text Copy

  +---------+------------+-------+------+----------+
    | symbol  | date       | close | open | volume   |
    +=========+============+=======+======+==========+
    | SH600000| 2020-11-01 | 120   | 121  | 12300000 |
    +---------+------------+-------+------+----------+
    | SH600000| 2020-11-02 | 123   | 120  | 12300000 |
    +---------+------------+-------+------+----------+

Supposed that users prepare their CSV or Parquet format data in the directory ~/.qlib/my_data, they can run the following command to start the conversion.

.. code-block:: bash

python scripts/dump_bin.py dump_all --data_path  ~/.qlib/my_data --qlib_dir ~/.qlib/qlib_data/ --include_fields open,close,high,low,volume,factor --file_suffix <.csv or .parquet>

For other supported parameters when dumping the data into .bin file, users can refer to the information by running the following commands:

.. code-block:: bash

python scripts/dump_bin.py dump_all --help

After conversion, users can find their Qlib format data in the directory ~/.qlib/qlib_data/.

.. note::

The arguments of `--include_fields` should correspond with the column names of CSV or Parquet files. The columns names of dataset provided by ``Qlib`` should include open, close, high, low, volume and factor at least.

- `open`
    The adjusted opening price
- `close`
    The adjusted closing price
- `high`
    The adjusted highest price
- `low`
    The adjusted lowest price
- `volume`
    The adjusted trading volume
- `factor`
    The Restoration factor. Normally, ``factor = adjusted_price / original_price``, `adjusted price` reference: `split adjusted <https://www.investopedia.com/terms/s/splitadjusted.asp>`_

In the convention of `Qlib` data processing, `open, close, high, low, volume, money and factor` will be set to NaN if the stock is suspended.
If you want to use your own alpha-factor which can't be calculate by OCHLV, like PE, EPS and so on, you could add it to the CSV or Parquet files with OHCLV together and then dump it to the Qlib format data.

Checking the health of the data Copy Chapter

Qlib provides a script to check the health of the data.

• The main points to check are as follows

  • Check if any data is missing in the DataFrame.

  • Check if there are any large step changes above the threshold in the OHLCV columns.

  • Check if any of the required columns (OLHCV) are missing in the DataFrame.

  • Check if the 'factor' column is missing in the DataFrame.

• You can run the following commands to check whether the data is healthy or not.

  for daily data:
  .. code-block:: bash
  
  python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data

  for 1min data:
  .. code-block:: bash
  
  python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data_1min --freq 1min

• Of course, you can also add some parameters to adjust the test results.

  • The available parameters are these.

    • freq: Frequency of data.

    • large_step_threshold_price: Maximum permitted price change

    • large_step_threshold_volume: Maximum permitted volume change.

    • missing_data_num: Maximum value for which data is allowed to be null.

• You can run the following commands to check whether the data is healthy or not.

  for daily data:
  .. code-block:: bash
  
  python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data --missing_data_num 30055 --large_step_threshold_volume 94485 --large_step_threshold_price 20

  for 1min data:
  .. code-block:: bash
  
  python scripts/check_data_health.py check_data --qlib_dir ~/.qlib/qlib_data/cn_data --freq 1min --missing_data_num 35806 --large_step_threshold_volume 3205452000000 --large_step_threshold_price 0.91

Stock Pool (Market) Copy Chapter

Qlib defines stock pool <https://github.com/microsoft/qlib/blob/main/examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml#L4>_ as stock list and their date ranges. Predefined stock pools (e.g. csi300) may be imported as follows.

.. code-block:: bash

python collector.py --index_name CSI300 --qlib_dir  --method parse_instruments

Multiple Stock Modes Copy Chapter

Qlib now provides two different stock modes for users: China-Stock Mode & US-Stock Mode. Here are some different settings of these two modes:

============== ================= ================
Region Trade Unit Limit Threshold
============== ================= ================
China 100 0.099

US 1 None
============== ================= ================

The trade unit defines the unit number of stocks can be used in a trade, and the limit threshold defines the bound set to the percentage of ups and downs of a stock.

• If users use Qlib in china-stock mode, china-stock data is required. Users can use Qlib in china-stock mode according to the following steps:

  • Download china-stock in qlib format, please refer to section Qlib Format Dataset <#qlib-format-dataset>_.

  • Initialize Qlib in china-stock mode
    Supposed that users download their Qlib format data in the directory ~/.qlib/qlib_data/cn_data. Users only need to initialize Qlib as follows.

    .. code-block:: python

    text Copy

    from qlib.constant import REG_CN
      qlib.init(provider_uri='~/.qlib/qlib_data/cn_data', region=REG_CN)

• If users use Qlib in US-stock mode, US-stock data is required. Qlib also provides a script to download US-stock data. Users can use Qlib in US-stock mode according to the following steps:

  • Download us-stock in qlib format, please refer to section Qlib Format Dataset <#qlib-format-dataset>_.

  • Initialize Qlib in US-stock mode
    Supposed that users prepare their Qlib format data in the directory ~/.qlib/qlib_data/us_data. Users only need to initialize Qlib as follows.

    .. code-block:: python

    text Copy

    from qlib.config import REG_US
      qlib.init(provider_uri='~/.qlib/qlib_data/us_data', region=REG_US)

.. note::

PRs for new data source are highly welcome! Users could commit the code to crawl data as a PR like `the examples here  <https://github.com/microsoft/qlib/tree/main/scripts>`_. And then we will use the code to create data cache on our server which other users could use directly.

Data API

Data Retrieval Copy Chapter

Users can use APIs in qlib.data to retrieve data, please refer to Data Retrieval <../start/getdata.html>_.

Feature Copy Chapter

Qlib provides Feature and ExpressionOps to fetch the features according to users' needs.

• Feature
  Load data from the data provider. User can get the features like $high, $low, $open, $close, .etc, which should correspond with the arguments of --include_fields, please refer to section Converting CSV Format into Qlib Format <#converting-csv-format-into-qlib-format>_.

• ExpressionOps
ExpressionOps will use operator for feature construction.
  To know more about Operator, please refer to Operator API <../reference/api.html#module-qlib.data.ops>_.
  Also, Qlib supports users to define their own custom Operator, an example has been given in tests/test_register_ops.py.

To know more about Feature, please refer to Feature API <../reference/api.html#module-qlib.data.base>_.

Filter Copy Chapter

Qlib provides NameDFilter and ExpressionDFilter to filter the instruments according to users' needs.

• NameDFilter
  Name dynamic instrument filter. Filter the instruments based on a regulated name format. A name rule regular expression is required.

• ExpressionDFilter
  Expression dynamic instrument filter. Filter the instruments based on a certain expression. An expression rule indicating a certain feature field is required.

  • basic features filter: rule_expression = '$close/$open>5'
  • cross-sectional features filter : rule_expression = '$rank($close)<10'
  • time-sequence features filter: rule_expression = '$Ref($close, 3)>100'

Here is a simple example showing how to use filter in a basic Qlib workflow configuration file:

.. code-block:: yaml

filter: &filter
    filter_type: ExpressionDFilter
    rule_expression: "Ref($close, -2) / Ref($close, -1) > 1"
    filter_start_time: 2010-01-01
    filter_end_time: 2010-01-07
    keep: False

data_handler_config: &data_handler_config
    start_time: 2010-01-01
    end_time: 2021-01-22
    fit_start_time: 2010-01-01
    fit_end_time: 2015-12-31
    instruments: *market
    filter_pipe: [*filter]

To know more about Filter, please refer to Filter API <../reference/api.html#module-qlib.data.filter>_.

Reference Copy Chapter

To know more about Data API, please refer to Data API <../reference/api.html#data>_.

Data Loader

Data Loader in Qlib is designed to load raw data from the original data source. It will be loaded and used in the Data Handler module.

QlibDataLoader Copy Chapter

The QlibDataLoader class in Qlib is such an interface that allows users to load raw data from the Qlib data source.

StaticDataLoader Copy Chapter

The StaticDataLoader class in Qlib is such an interface that allows users to load raw data from file or as provided.

Interface Copy Chapter

Here are some interfaces of the QlibDataLoader class:

.. autoclass:: qlib.data.dataset.loader.DataLoader
:members:
:noindex:

API Copy Chapter

To know more about Data Loader, please refer to Data Loader API <../reference/api.html#module-qlib.data.dataset.loader>_.

Data Handler

The Data Handler module in Qlib is designed to handler those common data processing methods which will be used by most of the models.

Users can use Data Handler in an automatic workflow by qrun, refer to Workflow: Workflow Management <workflow.html>_ for more details.

DataHandlerLP Copy Chapter

In addition to use Data Handler in an automatic workflow with qrun, Data Handler can be used as an independent module, by which users can easily preprocess data (standardization, remove NaN, etc.) and build datasets.

In order to achieve so, Qlib provides a base class qlib.data.dataset.DataHandlerLP <../reference/api.html#qlib.data.dataset.handler.DataHandlerLP>_. The core idea of this class is that: we will have some learnable Processors which can learn the parameters of data processing(e.g., parameters for zscore normalization). When new data comes in, these trained Processors can then process the new data and thus processing real-time data in an efficient way becomes possible. More information about Processors will be listed in the next subsection.

Interface Copy Chapter

Here are some important interfaces that DataHandlerLP provides:

.. autoclass:: qlib.data.dataset.handler.DataHandlerLP
:members: init, fetch, get_cols
:noindex:

If users want to load features and labels by config, users can define a new handler and call the static method parse_config_to_fields of qlib.contrib.data.handler.Alpha158.

Also, users can pass qlib.contrib.data.processor.ConfigSectionProcessor that provides some preprocess methods for features defined by config into the new handler.

Processor Copy Chapter

The Processor module in Qlib is designed to be learnable and it is responsible for handling data processing such as normalization and drop none/nan features/labels.

Qlib provides the following Processors:

• DropnaProcessor: processor that drops N/A features.
• DropnaLabel: processor that drops N/A labels.
• TanhProcess: processor that uses tanh to process noise data.
• ProcessInf: processor that handles infinity values, it will be replaces by the mean of the column.
• Fillna: processor that handles N/A values, which will fill the N/A value by 0 or other given number.
• MinMaxNorm: processor that applies min-max normalization.
• ZscoreNorm: processor that applies z-score normalization.
• RobustZScoreNorm: processor that applies robust z-score normalization.
• CSZScoreNorm: processor that applies cross sectional z-score normalization.
• CSRankNorm: processor that applies cross sectional rank normalization.
• CSZFillna: processor that fills N/A values in a cross sectional way by the mean of the column.

Users can also create their own processor by inheriting the base class of Processor. Please refer to the implementation of all the processors for more information (Processor Link <https://github.com/microsoft/qlib/blob/main/qlib/data/dataset/processor.py>_).

To know more about Processor, please refer to Processor API <../reference/api.html#module-qlib.data.dataset.processor>_.

Example Copy Chapter

Data Handler can be run with qrun by modifying the configuration file, and can also be used as a single module.

Know more about how to run Data Handler with qrun, please refer to Workflow: Workflow Management <workflow.html>_

Qlib provides implemented data handler Alpha158. The following example shows how to run Alpha158 as a single module.

.. note:: Users need to initialize Qlib with qlib.init first, please refer to initialization <../start/initialization.html>_.

.. code-block:: Python

import qlib
from qlib.contrib.data.handler import Alpha158

data_handler_config = {
    "start_time": "2008-01-01",
    "end_time": "2020-08-01",
    "fit_start_time": "2008-01-01",
    "fit_end_time": "2014-12-31",
    "instruments": "csi300",
}

if __name__ == "__main__":
    qlib.init()
    h = Alpha158(**data_handler_config)

    # get all the columns of the data
    print(h.get_cols())

    # fetch all the labels
    print(h.fetch(col_set="label"))

    # fetch all the features
    print(h.fetch(col_set="feature"))

.. note:: In the Alpha158, Qlib uses the label Ref($close, -2)/Ref($close, -1) - 1 that means the change from T+1 to T+2, rather than Ref($close, -1)/$close - 1, of which the reason is that when getting the T day close price of a china stock, the stock can be bought on T+1 day and sold on T+2 day.

API Copy Chapter

To know more about Data Handler, please refer to Data Handler API <../reference/api.html#module-qlib.data.dataset.handler>_.

Dataset

The Dataset module in Qlib aims to prepare data for model training and inferencing.

The motivation of this module is that we want to maximize the flexibility of different models to handle data that are suitable for themselves. This module gives the model the flexibility to process their data in an unique way. For instance, models such as GBDT may work well on data that contains nan or None value, while neural networks such as MLP will break down on such data.

If user's model need process its data in a different way, user could implement his own Dataset class. If the model's
data processing is not special, DatasetH can be used directly.

The DatasetH class is the dataset with Data Handler. Here is the most important interface of the class:

.. autoclass:: qlib.data.dataset.init.DatasetH
:members:
:noindex:

API Copy Chapter

To know more about Dataset, please refer to Dataset API <../reference/api.html#dataset>_.

Cache

Cache is an optional module that helps accelerate providing data by saving some frequently-used data as cache file. Qlib provides a Memcache class to cache the most-frequently-used data in memory, an inheritable ExpressionCache class, and an inheritable DatasetCache class.

Global Memory Cache Copy Chapter

Memcache is a global memory cache mechanism that composes of three MemCacheUnit instances to cache Calendar, Instruments, and Features. The MemCache is defined globally in cache.py as H. Users can use H['c'], H['i'], H['f'] to get/set memcache.

.. autoclass:: qlib.data.cache.MemCacheUnit
:members:
:noindex:

.. autoclass:: qlib.data.cache.MemCache
:members:
:noindex:

ExpressionCache Copy Chapter

ExpressionCache is a cache mechanism that saves expressions such as Mean($close, 5). Users can inherit this base class to define their own cache mechanism that saves expressions according to the following steps.

• Override self._uri method to define how the cache file path is generated
• Override self._expression method to define what data will be cached and how to cache it.

The following shows the details about the interfaces:

.. autoclass:: qlib.data.cache.ExpressionCache
:members:
:noindex:

Qlib has currently provided implemented disk cache DiskExpressionCache which inherits from ExpressionCache . The expressions data will be stored in the disk.

DatasetCache Copy Chapter

DatasetCache is a cache mechanism that saves datasets. A certain dataset is regulated by a stock pool configuration (or a series of instruments, though not recommended), a list of expressions or static feature fields, the start time, and end time for the collected features and the frequency. Users can inherit this base class to define their own cache mechanism that saves datasets according to the following steps.

• Override self._uri method to define how their cache file path is generated
• Override self._expression method to define what data will be cached and how to cache it.

The following shows the details about the interfaces:

.. autoclass:: qlib.data.cache.DatasetCache
:members:
:noindex:

Qlib has currently provided implemented disk cache DiskDatasetCache which inherits from DatasetCache . The datasets' data will be stored in the disk.

Data and Cache File Structure

We've specially designed a file structure to manage data and cache, please refer to the File storage design section in Qlib paper <https://arxiv.org/abs/2009.11189>_ for detailed information. The file structure of data and cache is listed as follows.

.. code-block::

- data/
    [raw data] updated by data providers
    - calendars/
        - day.txt
    - instruments/
        - all.txt
        - csi500.txt
        - ...
    - features/
        - sh600000/
            - open.day.bin
            - close.day.bin
            - ...
        - ...
    [cached data] updated when raw data is updated
    - calculated features/
        - sh600000/
            - [hash(instrtument, field_expression, freq)]
                - all-time expression -cache data file
                - .meta : an assorted meta file recording the instrument name, field name, freq, and visit times
        - ...
    - cache/
        - [hash(stockpool_config, field_expression_list, freq)]
            - all-time Dataset-cache data file
            - .meta : an assorted meta file recording the stockpool config, field names and visit times
            - .index : an assorted index file recording the line index of all calendars
        - ...

File: docs/component/highfreq.rst Copy Chapter

.. _highfreq:

======================================================================== Design of Nested Decision Execution Framework for High-Frequency Trading

.. currentmodule:: qlib

Introduction

Daily trading (e.g. portfolio management) and intraday trading (e.g. orders execution) are two hot topics in Quant investment and are usually studied separately.

To get the join trading performance of daily and intraday trading, they must interact with each other and run backtest jointly.
In order to support the joint backtest strategies at multiple levels, a corresponding framework is required. None of the publicly available high-frequency trading frameworks considers multi-level joint trading, which makes the backtesting aforementioned inaccurate.

Besides backtesting, the optimization of strategies from different levels is not standalone and can be affected by each other.
For example, the best portfolio management strategy may change with the performance of order executions(e.g. a portfolio with higher turnover may become a better choice when we improve the order execution strategies).
To achieve overall good performance, it is necessary to consider the interaction of strategies at a different levels.

Therefore, building a new framework for trading on multiple levels becomes necessary to solve the various problems mentioned above, for which we designed a nested decision execution framework that considers the interaction of strategies.

.. image:: ../_static/img/framework.svg

The design of the framework is shown in the yellow part in the middle of the figure above. Each level consists of Trading Agent and Execution Env. Trading Agent has its own data processing module (Information Extractor), forecasting module (Forecast Model) and decision generator (Decision Generator). The trading algorithm generates the decisions by the Decision Generator based on the forecast signals output by the Forecast Module, and the decisions generated by the trading algorithm are passed to the Execution Env, which returns the execution results.

The frequency of the trading algorithm, decision content and execution environment can be customized by users (e.g. intraday trading, daily-frequency trading, weekly-frequency trading), and the execution environment can be nested with finer-grained trading algorithm and execution environment inside (i.e. sub-workflow in the figure, e.g. daily-frequency orders can be turned into finer-grained decisions by splitting orders within the day). The flexibility of the nested decision execution framework makes it easy for users to explore the effects of combining different levels of trading strategies and break down the optimization barriers between different levels of the trading algorithm.

The optimization for the nested decision execution framework can be implemented with the support of QlibRL <./rl/overall.html>. To know more about how to use the QlibRL, go to API Reference: RL API <../reference/api.html#rl>.

Example

An example of a nested decision execution framework for high-frequency can be found here <https://github.com/microsoft/qlib/blob/main/examples/nested_decision_execution/workflow.py>_.

Besides, the above examples, here are some other related works about high-frequency trading in Qlib.

• Prediction with high-frequency data <https://github.com/microsoft/qlib/tree/main/examples/highfreq#benchmarks-performance-predicting-the-price-trend-in-high-frequency-data>_
• Examples <https://github.com/microsoft/qlib/blob/main/examples/orderbook_data/>_ to extract features from high-frequency data without fixed frequency.
• A paper <https://github.com/microsoft/qlib/tree/high-freq-execution#high-frequency-execution>_ for high-frequency trading.

File: docs/component/meta.rst Copy Chapter

.. _meta:

====================================================== Meta Controller: Meta-Task & Meta-Dataset & Meta-Model

.. currentmodule:: qlib

Introduction

Meta Controller provides guidance to Forecast Model, which aims to learn regular patterns among a series of forecasting tasks and use learned patterns to guide forthcoming forecasting tasks. Users can implement their own meta-model instance based on Meta Controller module.

Meta Task

A Meta Task instance is the basic element in the meta-learning framework. It saves the data that can be used for the Meta Model. Multiple Meta Task instances may share the same Data Handler, controlled by Meta Dataset. Users should use prepare_task_data() to obtain the data that can be directly fed into the Meta Model.

.. autoclass:: qlib.model.meta.task.MetaTask
:members:

Meta Dataset

Meta Dataset controls the meta-information generating process. It is on the duty of providing data for training the Meta Model. Users should use prepare_tasks to retrieve a list of Meta Task instances.

.. autoclass:: qlib.model.meta.dataset.MetaTaskDataset
:members:

Meta Model

General Meta Model Copy Chapter

Meta Model instance is the part that controls the workflow. The usage of the Meta Model includes:

1. Users train their Meta Model with the fit function.
2. The Meta Model instance guides the workflow by giving useful information via the inference function.

.. autoclass:: qlib.model.meta.model.MetaModel
:members:

Meta Task Model Copy Chapter

This type of meta-model may interact with task definitions directly. Then, the Meta Task Model is the class for them to inherit from. They guide the base tasks by modifying the base task definitions. The function prepare_tasks can be used to obtain the modified base task definitions.

.. autoclass:: qlib.model.meta.model.MetaTaskModel
:members:

Meta Guide Model Copy Chapter

This type of meta-model participates in the training process of the base forecasting model. The meta-model may guide the base forecasting models during their training to improve their performances.

.. autoclass:: qlib.model.meta.model.MetaGuideModel
:members:

Example

Qlib provides an implementation of Meta Model module, DDG-DA,
which adapts to the market dynamics.

DDG-DA includes four steps:

1. Calculate meta-information and encapsulate it into Meta Task instances. All the meta-tasks form a Meta Dataset instance.
2. Train DDG-DA based on the training data of the meta-dataset.
3. Do the inference of the DDG-DA to get guide information.
4. Apply guide information to the forecasting models to improve their performances.

The above example <https://github.com/microsoft/qlib/tree/main/examples/benchmarks_dynamic/DDG-DA>_ can be found in examples/benchmarks_dynamic/DDG-DA/workflow.py.

File: docs/component/model.rst Copy Chapter

.. _model:

=========================================== Forecast Model: Model Training & Prediction

Introduction

Forecast Model is designed to make the prediction score about stocks. Users can use the Forecast Model in an automatic workflow by qrun, please refer to Workflow: Workflow Management <workflow.html>_.

Because the components in Qlib are designed in a loosely-coupled way, Forecast Model can be used as an independent module also.

Base Class & Interface

Qlib provides a base class qlib.model.base.Model <../reference/api.html#module-qlib.model.base>_ from which all models should inherit.

The base class provides the following interfaces:

.. autoclass:: qlib.model.base.Model
:members:
:noindex:

Qlib also provides a base class qlib.model.base.ModelFT <../reference/api.html#qlib.model.base.ModelFT>_, which includes the method for finetuning the model.

For other interfaces such as finetune, please refer to Model API <../reference/api.html#module-qlib.model.base>_.

Example

Qlib's Model Zoo includes models such as LightGBM, MLP, LSTM, etc.. These models are treated as the baselines of Forecast Model. The following steps show how to run LightGBM as an independent module.

• Initialize Qlib with qlib.init first, please refer to Initialization <../start/initialization.html>_.

• Run the following code to get the prediction score pred_score
  .. code-block:: Python

  text Copy

  from qlib.contrib.model.gbdt import LGBModel
    from qlib.contrib.data.handler import Alpha158
    from qlib.utils import init_instance_by_config, flatten_dict
    from qlib.workflow import R
    from qlib.workflow.record_temp import SignalRecord, PortAnaRecord
  
    market = "csi300"
    benchmark = "SH000300"
  
    data_handler_config = {
        "start_time": "2008-01-01",
        "end_time": "2020-08-01",
        "fit_start_time": "2008-01-01",
        "fit_end_time": "2014-12-31",
        "instruments": market,
    }
  
    task = {
        "model": {
            "class": "LGBModel",
            "module_path": "qlib.contrib.model.gbdt",
            "kwargs": {
                "loss": "mse",
                "colsample_bytree": 0.8879,
                "learning_rate": 0.0421,
                "subsample": 0.8789,
                "lambda_l1": 205.6999,
                "lambda_l2": 580.9768,
                "max_depth": 8,
                "num_leaves": 210,
                "num_threads": 20,
            },
        },
        "dataset": {
            "class": "DatasetH",
            "module_path": "qlib.data.dataset",
            "kwargs": {
                "handler": {
                    "class": "Alpha158",
                    "module_path": "qlib.contrib.data.handler",
                    "kwargs": data_handler_config,
                },
                "segments": {
                    "train": ("2008-01-01", "2014-12-31"),
                    "valid": ("2015-01-01", "2016-12-31"),
                    "test": ("2017-01-01", "2020-08-01"),
                },
            },
        },
    }
  
    # model initialization
    model = init_instance_by_config(task["model"])
    dataset = init_instance_by_config(task["dataset"])
  
    # start exp
    with R.start(experiment_name="workflow"):
        # train
        R.log_params(**flatten_dict(task))
        model.fit(dataset)
  
        # prediction
        recorder = R.get_recorder()
        sr = SignalRecord(model, dataset, recorder)
        sr.generate()

  .. note::

  text Copy

  `Alpha158` is the data handler provided by ``Qlib``, please refer to `Data Handler <data.html#data-handler>`_.
    `SignalRecord` is the `Record Template` in ``Qlib``, please refer to `Workflow <recorder.html#record-template>`_.

Also, the above example has been given in examples/train_backtest_analyze.ipynb.
Technically, the meaning of the model prediction depends on the label setting designed by user.
By default, the meaning of the score is normally the rating of the instruments by the forecasting model. The higher the score, the more profit the instruments.

Custom Model

Qlib supports custom models. If users are interested in customizing their own models and integrating the models into Qlib, please refer to Custom Model Integration <../start/integration.html>_.

API

Please refer to Model API <../reference/api.html#module-qlib.model.base>_.

2. Official Technical Reference & Guides (microsoft/microsoft.github.io) Copy Chapter

[microsoft.github.io](https://microsoft.github.io)

Many open source projects released by Microsoft products, services and teams also have
deployments powered by GitHub Pages on this domain.

To discover Microsoft-released open source projects, learn about Microsoft's open source
approach and other information, check us out online at opensource.microsoft.com.

If we can help answer questions about the open source program at Microsoft, you can
contact opensource at microsoft com - although we are not able to provide support
for projects through this mechanism.

Microsoft GitHub resources Copy Chapter

• Default repo templates including SECURITY.MD and code of conduct files: microsoft/repo-templates
• Code of Conduct (FAQ)

Microsoft open source resources Copy Chapter

• Open Source Blog
• The Open Source Show on Channel 9
• https://opensource.microsoft.com/

Resources for Microsoft employees Copy Chapter

• docs.opensource.microsoft.com
• Link your GitHub Account and join GitHub organizations automatically

How to identify an official Microsoft organization on GitHub Copy Chapter

• Official Microsoft GitHub orgs have a professional avatar and include a web address at Microsoft.com or an associated official domain
• Official Microsoft orgs are Verified on GitHub - look for the domain verification indicator on GitHub