## 1. Project Overview & Quickstart (microsoft/RD-Agent) ## File: docs/research/benchmark.rst ============================== Benchmark ============================== Introduction ============= Benchmarking the capabilities of R&D is a crucial research problem in this area. We are continuously exploring methods to benchmark these capabilities. The current benchmarks are listed on this page. Development Capability Benchmarking =================================== Benchmarking is used to evaluate the effectiveness of factors with fixed data. It mainly includes the following steps: 1. :ref:`read and prepare the eval_data ` 2. :ref:`declare the method to be tested and pass the arguments ` 3. :ref:`declare the eval method and pass the arguments ` 4. :ref:`run the eval ` 5. :ref:`save and show the result ` Configuration ------------- .. _config: .. autopydantic_settings:: rdagent.components.benchmark.conf.BenchmarkSettings Example +++++++ .. _example: The default value for ``bench_test_round`` is 10, which takes about 2 hours to run. To modify it from ``10`` to ``2``, adjust the environment variables in the .env file as shown below. .. code-block:: Properties BENCHMARK_BENCH_TEST_ROUND=2 Data Format ------------- .. _data: The sample data in ``bench_data_path`` is a dictionary where each key represents a factor name. The value associated with each key is factor data containing the following information: - **description**: A textual description of the factor. - **formulation**: A LaTeX formula representing the model's formulation. - **variables**: A dictionary of variables involved in the factor. - **Category**: The category or classification of the factor. - **Difficulty**: The difficulty level of implementing or understanding the factor. - **gt_code**: A piece of code associated with the factor. Here is an example of this data format: .. literalinclude:: ../../rdagent/components/benchmark/example.json :language: json Ensure the data is placed in the ``FACTOR_COSTEER_SETTINGS.data_folder_debug``. The data files should be in ``.h5`` or ``.md`` format and must not be stored in any subfolders. LLM-Agents will review the file content and implement the tasks. .. TODO: Add a script to automatically generate the data in the `rdagent/app/quant_factor_benchmark/data` folder. Run Benchmark ------------- .. _run: Start the benchmark after completing the :doc:`../installation_and_configuration`. .. code-block:: Properties dotenv run -- python rdagent/app/benchmark/factor/eval.py Once completed, a pkl file will be generated, and its path will be printed on the last line of the console. Show Result ------------- .. _show: The ``analysis.py`` script reads data from the pkl file and converts it to an image. Modify the Python code in ``rdagent/app/quant_factor_benchmark/analysis.py`` to specify the path to the pkl file and the output path for the png file. .. code-block:: Properties dotenv run -- python rdagent/app/benchmark/factor/analysis.py A png file will be saved to the designated path as shown below. .. image:: ../_static/benchmark.png Related Paper ------------- - `Towards Data-Centric Automatic R&D `_: We have developed a comprehensive benchmark called RD2Bench to assess data and model R&D capabilities. This benchmark includes a series of tasks that outline the features or structures of models. These tasks are used to evaluate the ability of LLM-Agents to implement them. .. code-block:: bibtex @misc{chen2024datacentric, title={Towards Data-Centric Automatic R&D}, author={Haotian Chen and Xinjie Shen and Zeqi Ye and Wenjun Feng and Haoxue Wang and Xiao Yang and Xu Yang and Weiqing Liu and Jiang Bian}, year={2024}, eprint={2404.11276}, archivePrefix={arXiv}, primaryClass={cs.AI} } .. image:: https://github.com/user-attachments/assets/494f55d3-de9e-4e73-ba3d-a787e8f9e841 To replicate the benchmark detailed in the paper, please consult the factors listed in the following file: `RD2bench.json <../_static/RD2bench.json>`_. Please note use ``only_correct_format=False`` when evaluating the results. --- ## File: docs/research/catalog.rst =========== Research =========== To achieve the good effects and improve R&D capabilities, we face multiple challenges, the most important of which is the continuous evolution capability. Existing large language models (LLMs) find it difficult to continue growing their capabilities after training is completed. Moreover, the training process of LLMs focuses more on general knowledge, and the lack of depth in more specialized knowledge becomes an obstacle to solving professional R&D problems within the industry. This specialized knowledge needs to be learned and acquired from in-depth industry practice. Our RD-Agent, on the other hand, can continuously acquire in-depth domain knowledge through deep exploration during the R&D phase, allowing its R&D capabilities to keep growing. To address these key challenges and achieve industrial value, a series of research work needs to be completed. .. list-table:: Research Areas and Descriptions :header-rows: 1 * - Research Area - Description * - :doc:`Benchmark ` - Benchmark the R&D abilities * - Research - Idea proposal: Explore new ideas or refine existing ones * - :doc:`Development ` - Ability to realize ideas: Implement and execute ideas .. toctree:: :maxdepth: 1 :caption: Doctree: :hidden: benchmark dev --- ## File: docs/research/dev.rst ============================== Development ============================== Related Paper ------------- - `Collaborative Evolving Strategy for Automatic Data-Centric Development `_ Co-STEER is a method to tackle data-centric development (AD2) tasks and highlight its main challenges, which need expert-like implementation (i.e., learning domain knowledge from practice) and task scheduling capability (e.g., starting with easier tasks for better overall efficiency), areas that previous work has largely overlooked. Our Co-STEER agent enhances its domain knowledge through our evolving strategy and improves both its scheduling and implementation skills by gathering and using domain-specific practical experience. With a better schedule, implementation becomes faster. At the same time, as implementation feedback becomes more detailed, scheduling accuracy improves. These two capabilities grow together through practical feedback, enabling a collaborative evolution process. .. code-block:: bibtex @misc{yang2024collaborative, title={Collaborative Evolving Strategy for Automatic Data-Centric Development}, author={Xu Yang and Haotian Chen and Wenjun Feng and Haoxue Wang and Zeqi Ye and Xinjie Shen and Xiao Yang and Shizhao Sun and Weiqing Liu and Jiang Bian}, year={2024}, eprint={2407.18690}, archivePrefix={arXiv}, primaryClass={cs.AI} } .. image:: https://github.com/user-attachments/assets/75d9769b-0edd-4caf-9d45-57d1e577054b :alt: Collaborative Evolving Strategy for Automatic Data-Centric Development --- ## File: docs/scens/catalog.rst ========================= Scenarios ========================= Scenario lists ========================= In the two key areas of data-driven scenarios, model implementation and data building, our system aims to serve two main roles: 🦾copilot and πŸ€–agent. - The 🦾copilot follows human instructions to automate repetitive tasks. - The πŸ€–agent, being more autonomous, actively proposes ideas for better results in the future. The supported scenarios are listed below: .. list-table:: :header-rows: 1 * - Scenario/Target - Model Implementation - Data Building * - πŸ’Ή Finance - :ref:`πŸ₯‡The First Data-Centric Quant Multi-Agent Framework ` - :ref:`πŸ€–Iteratively Proposing Ideas & Evolving ` :ref:`🦾Auto reports reading & implementation ` :ref:`πŸ€–Iteratively Proposing Ideas & Evolving ` * - 🏭 General - :ref:`🦾Auto paper reading & implementation ` :ref:`πŸ§ͺFT-Agent for LLM fine-tuning ` - :ref:`πŸ€– Data Science ` .. toctree:: :maxdepth: 1 :caption: Doctree: :hidden: quant_agent_fin data_agent_fin data_copilot_fin model_agent_fin model_copilot_general data_science finetune --- ## File: docs/scens/data_agent_fin.rst .. _data_agent_fin: ===================== Finance Data Agent ===================== **πŸ€– Automated Quantitative Trading & Iterative Factors Evolution** ------------------------------------------------------------------- πŸ“– Background ~~~~~~~~~~~~~~ In the dynamic world of quantitative trading, **factors** serve as the strategic tools that enable traders to exploit market inefficiencies. These factorsβ€”ranging from simple metrics like price-to-earnings ratios to complex models like discounted cash flowsβ€”are the key to predicting stock prices with a high degree of accuracy. By leveraging these factors, quantitative traders can develop sophisticated strategies that not only identify market patterns but also significantly enhance trading efficiency and precision. The ability to systematically analyze and apply these factors is what separates ordinary trading from truly strategic market outmaneuvering. And this is where the **Finance Model Agent** comes into play. πŸŽ₯ `Demo `_ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. raw:: html 🌟 Introduction ~~~~~~~~~~~~~~~~ In this scenario, our agent illustrates the iterative process of hypothesis generation, knowledge construction, and decision-making. It highlights how financial factors evolve through continuous feedback and refinement. Here's an enhanced outline of the steps: **Step 1 : Hypothesis Generation πŸ”** - Generate and propose initial hypotheses based on previous experiment analysis and domain expertise, with thorough reasoning and financial justification. **Step 2 : Factor Creation ✨** - Based on the hypothesis, divide the tasks. - Each task involves developing, defining, and implementing a new financial factor, including its name, description, formulation, and variables. **Step 3 : Factor Implementation πŸ‘¨β€πŸ’»** - Implement the factor code based on the description, evolving it as a developer would. - Quantitatively validate the newly created factors. **Step 4 : Backtesting with Qlib πŸ“‰** - Integrate the full dataset into the factor implementation code and prepare the factor library. - Conduct backtesting using the Alpha158 plus newly developed factors and LGBModel in Qlib to evaluate the new factors' effectiveness and performance. +----------------+------------+----------------+----------------------------------------------------+ | Dataset | Model | Factors | Data Split | +================+============+================+====================================================+ | CSI300 | LGBModel | Alpha158 Plus | +-----------+--------------------------+ | | | | | | Train | 2008-01-01 to 2014-12-31 | | | | | | +-----------+--------------------------+ | | | | | | Valid | 2015-01-01 to 2016-12-31 | | | | | | +-----------+--------------------------+ | | | | | | Test | 2017-01-01 to 2020-08-01 | | | | | | +-----------+--------------------------+ | +----------------+------------+----------------+----------------------------------------------------+ **Step 5 : Feedback Analysis πŸ”** - Analyze backtest results to assess performance. - Incorporate feedback to refine hypotheses and improve the model. **Step 6 :Hypothesis Refinement ♻️** - Refine hypotheses based on feedback from backtesting. - Repeat the process to continuously improve the model. ⚑ Quick Start ~~~~~~~~~~~~~~~~~ Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency. You can try our demo by running the following command: - 🐍 Create a Conda Environment - Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI): .. code-block:: sh conda create -n rdagent python=3.10 - Activate the environment: .. code-block:: sh conda activate rdagent - πŸ“¦ Install the RDAgent - You can install the RDAgent package from PyPI: .. code-block:: sh pip install rdagent - πŸš€ Run the Application - You can directly run the application by using the following command: .. code-block:: sh rdagent fin_factor πŸ› οΈ Usage of modules ~~~~~~~~~~~~~~~~~~~~~ .. _Env Config: - **Env Config** The following environment variables can be set in the `.env` file to customize the application's behavior: .. autopydantic_settings:: rdagent.app.qlib_rd_loop.conf.FactorBasePropSetting :settings-show-field-summary: False :exclude-members: Config .. autopydantic_settings:: rdagent.components.coder.factor_coder.config.FactorCoSTEERSettings :settings-show-field-summary: False :members: coder_use_cache, data_folder, data_folder_debug, file_based_execution_timeout, select_method, max_loop, knowledge_base_path, new_knowledge_base_path :exclude-members: Config, fail_task_trial_limit, v1_query_former_trace_limit, v1_query_similar_success_limit, v2_query_component_limit, v2_query_error_limit, v2_query_former_trace_limit, v2_error_summary, v2_knowledge_sampler :no-index: --- ## File: docs/scens/data_copilot_fin.rst .. _data_copilot_fin: ===================== Finance Data Copilot ===================== **πŸ€– Automated Quantitative Trading & Factors Extraction from Financial Reports** --------------------------------------------------------------------------------- πŸ“– Background ~~~~~~~~~~~~~~ **Research reports** are treasure troves of insights, often unveiling potential **factors** that can drive successful quantitative trading strategies. Yet, with the sheer volume of reports available, extracting the most valuable insights efficiently becomes a daunting task. Furthermore, rather than hastily replicating factors from a report, it's essential to delve into the underlying logic of their construction. Does the factor capture the essential market dynamics? How unique is it compared to the factors already in your library? Therefore, there is an urgent need for a systematic approach to design a framework that can effectively manage this process. And this is where the **Finance Data Copilot** steps in. πŸŽ₯ `Demo `_ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. raw:: html 🌟 Introduction ~~~~~~~~~~~~~~~~ In this scenario, RDAgent demonstrates the process of extracting factors from financial research reports, implementing these factors, and analyzing their performance through Qlib backtesting. This process continually expands and refines the factor library. Here's an enhanced outline of the steps: **Step 1 : Hypothesis Generation πŸ”** - Generate and propose initial hypotheses based on insights from financial reports with thorough reasoning and financial justification. **Step 2 : Factor Creation ✨** - Based on the hypothesis and financial reports, divide the tasks. - Each task involves developing, defining, and implementing a new financial factor, including its name, description, formulation, and variables. **Step 3 : Factor Implementation πŸ‘¨β€πŸ’»** - Implement the factor code based on the description, evolving it as a developer would. - Quantitatively validate the newly created factors. **Step 4 : Backtesting with Qlib πŸ“‰** - Integrate the full dataset into the factor implementation code and prepare the factor library. - Conduct backtesting using the Alpha158 plus newly developed factors and LGBModel in Qlib to evaluate the new factors' effectiveness and performance. +----------------+------------+----------------+----------------------------------------------------+ | Dataset | Model | Factors | Data Split | +================+============+================+====================================================+ | CSI300 | LGBModel | Alpha158 Plus | +-----------+--------------------------+ | | | | | | Train | 2008-01-01 to 2014-12-31 | | | | | | +-----------+--------------------------+ | | | | | | Valid | 2015-01-01 to 2016-12-31 | | | | | | +-----------+--------------------------+ | | | | | | Test | 2017-01-01 to 2020-08-01 | | | | | | +-----------+--------------------------+ | +----------------+------------+----------------+----------------------------------------------------+ **Step 5 : Feedback Analysis πŸ”** - Analyze backtest results to assess performance. - Incorporate feedback to refine hypotheses and improve the model. **Step 6 :Hypothesis Refinement ♻️** - Refine hypotheses based on feedback from backtesting. - Repeat the process to continuously improve the model. ⚑ Quick Start ~~~~~~~~~~~~~~~~~ Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency. You can try our demo by running the following command: - 🐍 Create a Conda Environment - Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI): .. code-block:: sh conda create -n rdagent python=3.10 - Activate the environment: .. code-block:: sh conda activate rdagent - πŸ“¦ Install the RDAgent - You can install the RDAgent package from PyPI: .. code-block:: sh pip install rdagent - πŸš€ Run the Application - Download the financial reports you wish to extract factors from and store them in your preferred folder. - Specifically, you can follow this example, or use your own method: .. code-block:: sh wget https://github.com/SunsetWolf/rdagent_resource/releases/download/reports/all_reports.zip unzip all_reports.zip -d git_ignore_folder/reports - Run the application with the following command: .. code-block:: sh rdagent fin_factor_report --report-folder=git_ignore_folder/reports - Alternatively, you can store the paths of the reports in `report_result_json_file_path`. The format should be: .. code-block:: json [ "git_ignore_folder/report/fin_report1.pdf", "git_ignore_folder/report/fin_report2.pdf", "git_ignore_folder/report/fin_report3.pdf" ] - Then, run the application using the following command: .. code-block:: sh rdagent fin_factor_report πŸ› οΈ Usage of modules ~~~~~~~~~~~~~~~~~~~~~ .. _Env Config: - **Env Config** The following environment variables can be set in the `.env` file to customize the application's behavior: .. autopydantic_settings:: rdagent.app.qlib_rd_loop.conf.FactorFromReportPropSetting :settings-show-field-summary: False :show-inheritance: :exclude-members: Config .. autopydantic_settings:: rdagent.components.coder.factor_coder.config.FactorCoSTEERSettings :settings-show-field-summary: False :members: coder_use_cache, data_folder, data_folder_debug, file_based_execution_timeout, select_method, max_loop, knowledge_base_path, new_knowledge_base_path :exclude-members: Config, python_bin, fail_task_trial_limit, v1_query_former_trace_limit, v1_query_similar_success_limit, v2_query_component_limit, v2_query_error_limit, v2_query_former_trace_limit, v2_error_summary, v2_knowledge_sampler :no-index: --- ## File: docs/scens/data_science.rst .. _data_science_agent: ======================= Data Science Agent ======================= **πŸ€– Automated Feature Engineering & Model Tuning Evolution** ------------------------------------------------------------------------------------------ The Data Science Agent is an agent that can automatically perform feature engineering and model tuning. It can be used to solve various data science problems, such as image classification, time series forecasting, and text classification. 🌟 Introduction ~~~~~~~~~~~~~~~~~~ In this scenario, our automated system proposes hypothesis, choose action, implements code, conducts validation, and utilizes feedback in a continuous, iterative process. The goal is to automatically optimize performance metrics within the validation set or Kaggle Leaderboard, ultimately discovering the most efficient features and models through autonomous research and development. Here's an enhanced outline of the steps: **Step 1 : Hypothesis Generation πŸ”** - Generate and propose initial hypotheses based on previous experiment analysis and domain expertise, with thorough reasoning and financial justification. **Step 2 : Experiment Creation ✨** - Transform the hypothesis into a task. - Choose a specific action within feature engineering or model tuning. - Develop, define, and implement a new feature or model, including its name, description, and formulation. **Step 3 : Model/Feature Implementation πŸ‘¨β€πŸ’»** - Implement the model code based on the detailed description. - Evolve the model iteratively as a developer would, ensuring accuracy and efficiency. **Step 4 : Validation on Test Set or Kaggle πŸ“‰** - Validate the newly developed model using the test set or Kaggle dataset. - Assess the model's effectiveness and performance based on the validation results. **Step 5: Feedback Analysis πŸ”** - Analyze validation results to assess performance. - Use insights to refine hypotheses and enhance the model. **Step 6: Hypothesis Refinement ♻️** - Adjust hypotheses based on validation feedback. - Iterate the process to continuously improve the model. πŸ“– Data Science Background ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In the evolving landscape of artificial intelligence, **Data Science** represents a powerful paradigm where machines engage in autonomous exploration, hypothesis testing, and model development across diverse domains β€” from healthcare and finance to logistics and research. The **Data Science** Agent stands as a central engine in this transformation, enabling users to automate the entire machine learning workflow: from hypothesis generation to code implementation, validation, and refinement β€” all guided by performance feedback. By leveraging the **Data Science** Agent, researchers and developers can accelerate experimentation cycles. Whether fine-tuning custom models or competing in high-stakes benchmarks like Kaggle, the Data Science Agent unlocks new frontiers in intelligent, self-directed discovery. 🧭 Example Guide - Customized dataset ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ πŸ”§ **Set up RD-Agent Environment** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Before you start, please make sure you have installed RD-Agent and configured the environment for RD-Agent correctly. If you want to know how to install and configure the RD-Agent, please refer to the `documentation <../installation_and_configuration.html>`_. - πŸ”© **Setting the Environment variables at .env file** - Determine the path where the data will be stored and add it to the ``.env`` file. .. code-block:: sh dotenv set DS_LOCAL_DATA_PATH /ds_data dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen πŸ“₯ **Prepare Customized datasets** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - A data science competition dataset usually consists of two parts: ``competition dataset`` and ``evaluation dataset``. (We provide `a sample `_ of a customized dataset named: `arf-12-hours-prediction-task as a reference`.) - The ``competition dataset`` contains **training data**, **test data**, **description files**, **formatted submission files**, **data sampling codes**. - The ``evaluation dataset`` contains **standard answer file**, **data checking codes**, and **Code for calculation of scores**. - We use the ``arf-12-hours-prediction-task`` data as a sample to introduce the preparation workflow for the competition dataset. - Create a ``ds_data/source_data/arf-12-hours-prediction-task`` folder, which will be used to store your raw dataset. - The raw files for the competition ``arf-12-hours-prediction-task`` have two files: ``ARF_12h.csv`` and ``X.npz``. - Create a ``ds_data/source_data/arf-12-hours-prediction-task/prepare.py`` file that splits your raw data into **training data**, **test data**, **formatted submission file**, and **standard answer file**. (You will need to write a script based on your raw data.) - The following shows the preprocessing code for the raw data of ``arf-12-hours-prediction-task``. .. literalinclude:: ../../rdagent/scenarios/data_science/example/source_data/arf-12-hours-prediction-task/prepare.py :language: python :caption: ds_data/source_data/arf-12-hours-prediction-task/prepare.py :linenos: - At the end of program execution, the ``ds_data`` folder structure will look like this: .. code-block:: text ds_data β”œβ”€β”€ arf-12-hours-prediction-task β”‚ β”œβ”€β”€ train β”‚ β”‚ β”œβ”€β”€ ARF_12h.csv β”‚ β”‚ └── X.npz β”‚ β”œβ”€β”€ test β”‚ β”‚ β”œβ”€β”€ ARF_12h.csv β”‚ β”‚ └── X.npz β”‚ └── sample_submission.csv β”œβ”€β”€ eval β”‚ └── arf-12-hours-prediction-task β”‚ └── submission_test.csv └── source_data └── arf-12-hours-prediction-task β”œβ”€β”€ ARF_12h.csv β”œβ”€β”€ prepare.py └── X.npz - Create a ``ds_data/arf-12-hours-prediction-task/description.md`` file to describe your competition, Objective, dataset, and other information. - The following shows the description file for ``arf-12-hours-prediction-task`` .. literalinclude:: ../../rdagent/scenarios/data_science/example/arf-12-hours-prediction-task/description.md :language: markdown :caption: ds_data/arf-12-hours-prediction-task/description.md :linenos: - Create a ``ds_data/arf-12-hours-prediction-task/sample.py`` file to construct the debugging sample data. - The following shows the script for constructing the debugging sample data based on the ``arf-12-hours-prediction-task`` dataset implementation. .. literalinclude:: ../../rdagent/scenarios/data_science/example/arf-12-hours-prediction-task/sample.py :language: markdown :caption: ds_data/arf-12-hours-prediction-task/sample.py :linenos: - Create a ``ds_data/eval/arf-12-hours-prediction-task/valid.py`` file, which is used to check the validity of the submission files to ensure that their formatting is consistent with the reference file. - The following shows a script that checks the validity of a submission based on the ``arf-12-hours-prediction-task`` data. .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/arf-12-hours-prediction-task/valid.py :language: markdown :caption: ds_data/eval/arf-12-hours-prediction-task/valid.py :linenos: - Create a ``ds_data/eval/arf-12-hours-prediction-task/grade.py`` file, which is used to calculate the score based on the submission file and the **standard answer file**, and output the result in JSON format. - The following shows a grading script based on the ``arf-12-hours-prediction-task`` data implementation. .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/arf-12-hours-prediction-task/grade.py :language: markdown :caption: ds_data/eval/arf-12-hours-prediction-task/grade.py :linenos: - At this point, you have created a complete dataset. The correct structure of the dataset should look like this. .. code-block:: text ds_data β”œβ”€β”€ arf-12-hours-prediction-task β”‚ β”œβ”€β”€ train β”‚ β”‚ β”œβ”€β”€ ARF_12h.csv β”‚ β”‚ └── X.npz β”‚ β”œβ”€β”€ test β”‚ β”‚ β”œβ”€β”€ ARF_12h.csv β”‚ β”‚ └── X.npz β”‚ β”œβ”€β”€ description.md β”‚ β”œβ”€β”€ sample_submission.csv β”‚ └── sample.py β”œβ”€β”€ eval β”‚ └── arf-12-hours-prediction-task β”‚ β”œβ”€β”€ grade.py β”‚ β”œβ”€β”€ submission_test.csv β”‚ └── valid.py └── source_data └── arf-12-hours-prediction-task β”œβ”€β”€ ARF_12h.csv β”œβ”€β”€ prepare.py └── X.npz - The above shows the complete dataset creation workflow, some of the files are not required, in practice you can customize the dataset according to your own needs. - If we don't need the test set scores, then we can choose not to generate **formatted submission files** and **standard answer file** in the prepare code, and we don't need to write **data checking codes** and **Code for calculation of scores**. - **Data sampling code** can also be created according to the actual need, if you do not provide **data sampling code**, RD-Agent will be handed over to the LLM sampling at runtime. - In the default sampling method (``create_debug_data``), the default sampling ratio (parameter: ``min_frac``) is 1%, if 1% of the data is less than 5, then 5 data will be sampled (parameter: ``min_num``), you can adjust the sampling ratio by adjusting these two parameters. - If you have customized data sampling code, you need to set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` (default is True) in the ``.env`` file before running, so that the program will use the customized sampling code when running, and you can just execute this line of code in the command line: .. code-block:: sh dotenv set DS_SAMPLE_DATA_BY_LLM False - In addition, we provide a data sampling method in `rdagent.scenarios.data_science.debug.data.create_debug_data `_, in this method, the default sampling ratio (parameter: ``min_frac``) is 1%, if 1% of the data is less than 5, then 5 data will be sampled (parameter: ``min_num``), you can use this method by the following two ways. - You can set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` in the ``.env`` file so that when the program runs, it will use the sampling code provided by RD-Agent. .. code-block:: sh dotenv set DS_SAMPLE_DATA_BY_LLM False - If you think that the parameters in the receipt sampling method provided by RD-Agent are not suitable, you can customize the parameters in the following command and run it, and set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` in the ``.env`` so that the program will use the sampling data you provided when running. .. code-block:: sh python rdagent/app/data_science/debug.py --dataset_path --competition --min_frac --min_num dotenv set DS_SAMPLE_DATA_BY_LLM False - If you don't need the scores from the test set and leave the data sampling to the LLM, or if you use the sampling method provided by the RD-Agent, you only need to prepare a minimal dataset. The structure of the simplest dataset should be as shown below. .. code-block:: text ds_data β”œβ”€β”€ arf-12-hours-prediction-task β”‚ β”œβ”€β”€ train β”‚ β”‚ β”œβ”€β”€ ARF_12h.csv β”‚ β”‚ └── X.npz β”‚ β”œβ”€β”€ test β”‚ β”‚ β”œβ”€β”€ ARF_12h.csv β”‚ β”‚ └── X.npz β”‚ └── description.md └── source_data └── arf-12-hours-prediction-task β”œβ”€β”€ ARF_12h.csv β”œβ”€β”€ prepare.py └── X.npz - We have prepared a dataset based on the above description for your reference. You can download it with the following command. .. code-block:: sh wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip βš™οΈ **Set up Environment for Customized datasets** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: sh dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen dotenv set DS_LOCAL_DATA_PATH /ds_data dotenv set DS_CODER_ON_WHOLE_PIPELINE True - πŸ“˜ More Environment Variables (Optional) - If you want to see all the available environment variables, you can refer to the configuration file for Data Science scenarios: .. literalinclude:: ../../rdagent/app/data_science/conf.py :language: python :linenos: - These variables allow you to have finer-grained control in Data Science scenarios. πŸš€ **Run the Application** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - 🌏 You can directly run the application by using the following command: .. code-block:: sh rdagent data_science --competition - The following shows the command to run based on the ``arf-12-hours-prediction-task`` data .. code-block:: sh rdagent data_science --competition arf-12-hours-prediction-task - More CLI Parameters for `rdagent data_science` command: .. automodule:: rdagent.app.data_science.loop :members: :no-index: - πŸ“ˆ Visualize the R&D Process - We provide a web UI to visualize the log. You just need to run: .. code-block:: sh rdagent ui --port --log-dir --data_science True - Then you can input the log path and visualize the R&D process. - πŸ§ͺ Scoring the test results - Finally, shutdown the program, and get the test set scores with this command. .. code-block:: sh dotenv run -- python rdagent/log/mle_summary.py grade Here, refers to the parent directory of the log folder generated during the run. πŸ•ΉοΈ Kaggle Agent ~~~~~~~~~~~~~~~~ πŸ“– Background ^^^^^^^^^^^^^^ In the landscape of data science competitions, Kaggle serves as the ultimate arena where data enthusiasts harness the power of algorithms to tackle real-world challenges. The Kaggle Agent stands as a pivotal tool, empowering participants to seamlessly integrate cutting-edge models and datasets, transforming raw data into actionable insights. By utilizing the **Kaggle Agent**, data scientists can craft innovative solutions that not only uncover hidden patterns but also drive significant advancements in predictive accuracy and model robustness. 🧭 Example Guide - Kaggle Dataset ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ πŸ› οΈ Preparing For The Competition """""""""""""""""""""""""""""""""" - πŸ”¨ **Configuring the Kaggle API** - Register and login on the `Kaggle `_ website. - Click on the avatar (usually in the top right corner of the page) -> ``Settings`` -> ``Create New Token``, A file called ``kaggle.json`` will be downloaded. - Move ``kaggle.json`` to ``~/.config/kaggle/`` - Modify the permissions of the ``kaggle.json`` file. .. code-block:: sh chmod 600 ~/.config/kaggle/kaggle.json - For more information about Kaggle API Settings, refer to the `Kaggle API `_. - πŸ”© **Setting the Environment variables at .env file** - Determine the path where the data will be stored and add it to the ``.env`` file. .. code-block:: sh mkdir -p /ds_data dotenv set KG_LOCAL_DATA_PATH /ds_data - πŸ“˜ More Environment Variables (Optional) - If you want to see all the available environment variables, you can refer to the configuration file for Data Science scenarios: .. literalinclude:: ../../rdagent/app/data_science/conf.py :language: python :linenos: - These variables allow you to have finer-grained control in Data Science scenarios. - πŸ—³οΈ **Join the competition** - If your Kaggle API account has not joined a competition, you will need to join the competition before running the program. - At the bottom of the competition details page, you can find the ``Join the competition`` button, click on it and select ``I Understand and Accept`` to join the competition. - In the **Competition List Available** below, you can jump to the competition details page. πŸ“₯ Preparing Competition DataDataset && Set up RD-Agent Environment """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - As a subset of data science, kaggle's dataset still follows the data science format. Based on this, the kaggle dataset can be divided into two categories depending on whether or not it is supported by the **MLE-Bench**. - What is **MLE-Bench**? - **MLE-Bench** is a comprehensive benchmark designed to evaluate the **machine learning engineering** capabilities of AI systems using real-world scenarios. The dataset includes multiple Kaggle competitions. Since Kaggle does not provide reserved test sets for these competitions, the benchmark includes preparation scripts for splitting publicly available training data into new training and test sets, and scoring scripts for each competition to accurately evaluate submission scores. - I'm running a competition Is **MLE-Bench** supported? - You can see all the competitions supported by **MLE-Bench** `here `_. - Prepare datasets for **MLE-Bench** supported competitions. - If you agree with the **MLE-Bench** standard, then you don't need to prepare the dataset, you just need to configure your ``.env`` file to automate the download of the dataset. - Configure environment variables, add ``DS_IF_USING_MLE_DATA`` to environment variables, and set it to ``True``. .. code-block:: sh dotenv set DS_IF_USING_MLE_DATA True - Configure environment variables, add ``DS_SAMPLE_DATA_BY_LLM`` to environment variables, and set it to ``True``. .. code-block:: sh dotenv set DS_SAMPLE_DATA_BY_LLM True - Configure environment variables, add ``DS_SCEN`` to environment variables, and set it to ``rdagent.scenarios.data_science.scen.KaggleScen``. .. code-block:: sh dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen - At this point, you are ready to start running your competition, which will automatically download the data, and the LLM will automatically extract the minimum dataset. - After running the program the structure of the ds_data folder should look like this (Using the ``tabular-playground-series-dec-2021`` contest as an example). .. code-block:: text ds_data β”œβ”€β”€ tabular-playground-series-dec-2021 β”‚ β”œβ”€β”€ description.md β”‚ β”œβ”€β”€ sample_submission.csv β”‚ β”œβ”€β”€ test.csv β”‚ └── train.csv └── zip_files └── tabular-playground-series-dec-2021 └── tabular-playground-series-dec-2021.zip - The ``ds_data/zip_files`` folder contains a zip file of the raw competition data downloaded from kaggle website. - At runtime, RD-Agent will automatically build the Docker image specified at `rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile `_. This image is responsible for downloading the required datasets and grading files for MLE-Bench. Note: The first run may take longer than subsequent runs as the Docker image and data are being downloaded and set up for the first time. - Prepare datasets for competitions that are not supported by **MLE-Bench**. - As a subset of data science, we can follow the format and steps of data science dataset to prepare kaggle dataset. Below we will describe the workflow for preparing a kaggle dataset using the competition ``playground-series-s4e9`` as an example. - Create a ``ds_data/source_data/playground-series-s4e9`` folder, which will be used to store your raw dataset. - The raw files for the competition ``playground-series-s4e9`` have two files: ``train.csv``, ``test.csv``, ``sample_submission.csv``, and there are two ways to get the raw data: - You can find the raw data required for the competition on the `official kaggle website `_. - Or you can use the command line to download the raw data for the competition, the download command is as follows. .. code-block:: sh kaggle competitions download -c playground-series-s4e9 - Create a ``ds_data/source_data/playground-series-s4e9/prepare.py`` file that splits your raw data into **training data**, **test data**, **formatted submission file**, and **standard answer file**. (You will need to write a script based on your raw data.) - The following shows the preprocessing code for the raw data of ``playground-series-s4e9``. .. literalinclude:: ../../rdagent/scenarios/data_science/example/source_data/playground-series-s4e9/prepare.py :language: python :caption: ds_data/source_data/playground-series-s4e9/prepare.py :linenos: - At the end of program execution, the ``ds_data`` folder structure will look like this: .. code-block:: text ds_data β”œβ”€β”€ playground-series-s4e9 β”‚ β”œβ”€β”€ train.csv β”‚ β”œβ”€β”€ test.csv β”‚ └── sample_submission.csv β”œβ”€β”€ eval β”‚ └── playground-series-s4e9 β”‚ └── submission_test.csv └── source_data └── playground-series-s4e9 β”œβ”€β”€ prepare.py β”œβ”€β”€ sample_submission.csv β”œβ”€β”€ test.csv └── train.csv - Create a ``ds_data/playground-series-s4e9/description.md`` file to describe your competition, dataset description, and other information. We can find the `competition description information `_ and the `dataset description information `_ from the Kaggle website. - The following shows the description file for ``playground-series-s4e9`` .. literalinclude:: ../../rdagent/scenarios/data_science/example/playground-series-s4e9/description.md :language: markdown :caption: ds_data/playground-series-s4e9/description.md :linenos: - Create a ``ds_data/eval/playground-series-s4e9/valid.py`` file, which is used to check the validity of the submission files to ensure that their formatting is consistent with the reference file. - The following shows a script that checks the validity of a submission based on the ``playground-series-s4e9`` data. .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/playground-series-s4e9/valid.py :language: markdown :caption: ds_data/eval/playground-series-s4e9/valid.py :linenos: - Create a ``ds_data/eval/playground-series-s4e9/grade.py`` file, which is used to calculate the score based on the submission file and the **standard answer file**, and output the result in JSON format. - The following shows a grading script based on the ``playground-series-s4e9`` data implementation. .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/playground-series-s4e9/grade.py :language: markdown :caption: ds_data/eval/playground-series-s4e9/grade.py :linenos: - In this example we don't create a ``ds_data/eval/playground-series-s4e9/sample.py``, we use the sample method provided by RD-Agent by default. - At this point, you have created a complete dataset. The correct structure of the dataset should look like this. .. code-block:: text ds_data β”œβ”€β”€ playground-series-s4e9 β”‚ β”œβ”€β”€ train.csv β”‚ β”œβ”€β”€ test.csv β”‚ β”œβ”€β”€ description.md β”‚ └── sample_submission.csv β”œβ”€β”€ eval β”‚ └── playground-series-s4e9 β”‚ β”œβ”€β”€ grade.py β”‚ β”œβ”€β”€ submission_test.csv β”‚ └── valid.py └── source_data └── playground-series-s4e9 β”œβ”€β”€ prepare.py β”œβ”€β”€ sample_submission.csv β”œβ”€β”€ test.csv └── train.csv - We have prepared a dataset based on the above description for your reference. You can download it with the following command. .. code-block:: sh wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/playground-series-s4e9.zip - Next, we need to configure the environment for the ``playground-series-s4e9`` contest. You can do this by executing the following command at the command line. .. code-block:: sh dotenv set DS_IF_USING_MLE_DATA False dotenv set DS_SAMPLE_DATA_BY_LLM False dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen πŸš€ **Run the Application** """""""""""""""""""""""""""""""""""" - 🌏 You can directly run the application by using the following command: .. code-block:: sh rdagent data_science --competition - The following shows the command to run based on the ``playground-series-s4e9`` data .. code-block:: sh rdagent data_science --competition playground-series-s4e9 - More CLI Parameters for `rdagent data_science` command: .. automodule:: rdagent.app.data_science.loop :members: :no-index: - πŸ“ˆ Visualize the R&D Process - We provide a web UI to visualize the log. You just need to run: .. code-block:: sh rdagent ui --port --log-dir --data_science True - Then you can input the log path and visualize the R&D process. - πŸ§ͺ Scoring the test results - Finally, shutdown the program, and get the test set scores with this command. .. code-block:: sh dotenv run -- python rdagent/log/mle_summary.py grade - If you have configured the full output in ``ds_data/eval/playground-series-s4e9/grade.py``, or if you are running a competition that receives **MLE-Bench** support, you can also summarize the scores by running the following command. .. code-block:: sh rdagent grade_summary --log-folder= Here, refers to the parent directory of the log folder generated during the run. --- ## File: docs/scens/model_agent_fin.rst .. _model_agent_fin: ======================= Finance Model Agent ======================= **πŸ€– Automated Quantitative Trading & Iterative Model Evolution** ------------------------------------------------------------------------------------------ πŸ“– Background ~~~~~~~~~~~~~~ In the realm of quantitative finance, both factor discovery and model development play crucial roles in driving performance. While much attention is often given to the discovery of new financial factors, the **models** that leverage these factors are equally important. The effectiveness of a quantitative strategy depends not only on the factors used but also on how well these factors are integrated into robust, predictive models. However, the process of developing and optimizing these models can be labor-intensive and complex, requiring continuous refinement and adaptation to ever-changing market conditions. And this is where the **Finance Model Agent** steps in. πŸŽ₯ `Demo `_ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. raw:: html
🌟 Introduction ~~~~~~~~~~~~~~~~ In this scenario, our automated system proposes hypothesis, constructs model, implements code, conducts back-testing, and utilizes feedback in a continuous, iterative process. The goal is to automatically optimize performance metrics within the Qlib library, ultimately discovering the most efficient code through autonomous research and development. Here's an enhanced outline of the steps: **Step 1 : Hypothesis Generation πŸ”** - Generate and propose initial hypotheses based on previous experiment analysis and domain expertise, with thorough reasoning and financial justification. **Step 2 : Model Creation ✨** - Transform the hypothesis into a task. - Develop, define, and implement a quantitative model, including its name, description, and formulation. **Step 3 : Model Implementation πŸ‘¨β€πŸ’»** - Implement the model code based on the detailed description. - Evolve the model iteratively as a developer would, ensuring accuracy and efficiency. **Step 4 : Backtesting with Qlib πŸ“‰** - Conduct backtesting using the newly developed model and 20 factors extracted from Alpha158 in Qlib. - Evaluate the model's effectiveness and performance. +----------------+------------+------------------------+----------------------------------------------------+ | Dataset | Model | Factors | Data Split | +================+============+========================+====================================================+ | CSI300 | RDAgent-dev| 20 factors (Alpha158) | +-----------+--------------------------+ | | | | | | Train | 2008-01-01 to 2014-12-31 | | | | | | +-----------+--------------------------+ | | | | | | Valid | 2015-01-01 to 2016-12-31 | | | | | | +-----------+--------------------------+ | | | | | | Test | 2017-01-01 to 2020-08-01 | | | | | | +-----------+--------------------------+ | +----------------+------------+------------------------+----------------------------------------------------+ **Step 5 : Feedback Analysis πŸ”** - Analyze backtest results to assess performance. - Incorporate feedback to refine hypotheses and improve the model. **Step 6 :Hypothesis Refinement ♻️** - Refine hypotheses based on feedback from backtesting. - Repeat the process to continuously improve the model. ⚑ Quick Start ~~~~~~~~~~~~~~~~~ Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency. You can try our demo by running the following command: - 🐍 Create a Conda Environment - Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI): .. code-block:: sh conda create -n rdagent python=3.10 - Activate the environment: .. code-block:: sh conda activate rdagent - πŸ“¦ Install the RDAgent - You can install the RDAgent package from PyPI: .. code-block:: sh pip install rdagent - πŸš€ Run the Application - You can directly run the application by using the following command: .. code-block:: sh rdagent fin_model πŸ› οΈ Usage of modules ~~~~~~~~~~~~~~~~~~~~~ .. _Env Config: - **Env Config** The following environment variables can be set in the `.env` file to customize the application's behavior: .. autopydantic_settings:: rdagent.app.qlib_rd_loop.conf.ModelBasePropSetting :settings-show-field-summary: False :exclude-members: Config - **Qlib Config** - The `config.yaml` file located in the `model_template` folder contains the relevant configurations for running the developed model in Qlib. The default settings include key information such as: - **market**: Specifies the market, which is set to `csi300`. - **fields_group**: Defines the fields group, with the value `feature`. - **col_list**: A list of columns used, including various indicators such as `RESI5`, `WVMA5`, `RSQR5`, and others. - **start_time**: The start date for the data, set to `2008-01-01`. - **end_time**: The end date for the data, set to `2020-08-01`. - **fit_start_time**: The start date for fitting the model, set to `2008-01-01`. - **fit_end_time**: The end date for fitting the model, set to `2014-12-31`. - The default hyperparameters used in the configuration are as follows: - **n_epochs**: The number of epochs, set to `100`. - **lr**: The learning rate, set to `1e-3`. - **early_stop**: The early stopping criterion, set to `10`. - **batch_size**: The batch size, set to `2000`. - **metric**: The evaluation metric, set to `loss`. - **loss**: The loss function, set to `mse`. - **n_jobs**: The number of parallel jobs, set to `20`. --- ## File: docs/scens/model_copilot_general.rst .. _model_copilot_general: ====================== General Model Copilot ====================== **πŸ€– Automated Model Research & Development Co-Pilot** -------------------------------------------------------- πŸ“– Background ~~~~~~~~~~~~~~ In the fast-paced field of artificial intelligence, the number of academic papers published each year is skyrocketing. These papers introduce new models, techniques, and approaches that can significantly advance the state of the art. However, reproducing and implementing these models can be a daunting task, requiring substantial time and expertise. Researchers often face challenges in extracting the essential details from these papers and converting them into functional code. And this is where the **General Model Copilot** steps in. πŸŽ₯ `Demo `_ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. raw:: html 🌟 Introduction ~~~~~~~~~~~~~~~~ In this scenario, our automated system proposes hypotheses, constructs models, implements code, performs back-testing, and uses feedback to iterate continuously. The system aims to automatically optimize performance metrics from the Qlib library, finding the best code through autonomous research and development. Model R&D CoPilot Scenario ~~~~~~~~~~~~~~~~~~~~~~~~~~ **Overview** This demo automates the extraction and iterative development of models from academic papers, ensuring functionality and correctness. This scenario automates the development of PyTorch models by reading academic papers or other sources. It supports various data types, including tabular, time-series, and graph data. The primary workflow involves two main components: the Reader and the Coder. **Workflow Components** 1. **Reader** - Parses and extracts relevant model information from academic papers or sources, including architectures, parameters, and implementation details. - Uses Large Language Models to convert content into a structured format for the Coder. 2. **Evolving Coder** - Translates structured information from the Reader into executable PyTorch code. - Utilizes an evolving coding mechanism to ensure correct tensor shapes, verified with sample input tensors. - Iteratively refines the code to align with source material specifications. **Supported Data Types** - **Tabular Data:** Structured data with rows and columns, such as spreadsheets or databases. - **Time-Series Data:** Sequential data points indexed in time order, useful for forecasting and temporal pattern recognition. - **Graph Data:** Data structured as nodes and edges, suitable for network analysis and relational tasks. ⚑ Quick Start ~~~~~~~~~~~~~~~~~ Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency. You can try our demo by running the following command: - 🐍 Create a Conda Environment - Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI): .. code-block:: sh conda create -n rdagent python=3.10 - Activate the environment: .. code-block:: sh conda activate rdagent - πŸ“¦ Install the RDAgent - You can install the RDAgent package from PyPI: .. code-block:: sh pip install rdagent - πŸš€ Run the Application - Prepare relevant files (in pdf format) by uploading papers to the directory below and copy the path as report_file_path. .. code-block:: sh rdagent/scenarios/general_model - Run the following command in your terminal within the same virtual environment: .. code-block:: sh rdagent general_model --report-file-path= --- ## File: docs/api_reference.rst ============= API Reference ============= Here you can find all ``RDAgent``'s interfaces. RD Loop ======= Research -------- .. automodule:: rdagent.core.proposal :members: ## 2. Official Technical Reference & Guides (microsoft/microsoft.github.io) ## File: README.md [microsoft.github.io](https://microsoft.github.io) =================== Many open source projects released by Microsoft products, services and teams also have deployments powered by GitHub Pages on this domain. To discover Microsoft-released open source projects, learn about Microsoft's open source approach and other information, check us out online at [opensource.microsoft.com](https://opensource.microsoft.com). If we can help answer questions about the open source program at Microsoft, you can contact `opensource at microsoft com` - although we are not able to provide support for projects through this mechanism. ## Microsoft GitHub resources - Default repo templates including `SECURITY.MD` and code of conduct files: [microsoft/repo-templates](https://github.com/microsoft/repo-templates) - [Code of Conduct](https://opensource.microsoft.com/codeofconduct) ([FAQ](https://opensource.microsoft.com/codeofconduct/faq/)) ## Microsoft open source resources - [Open Source Blog](https://cloudblogs.microsoft.com/opensource/) - [The Open Source Show on Channel 9](https://channel9.msdn.com/Shows/The-Open-Source-Show) - [https://opensource.microsoft.com/](https://opensource.microsoft.com/) ## Resources for Microsoft employees - [docs.opensource.microsoft.com](https://docs.opensource.microsoft.com) - [Link your GitHub Account and join GitHub organizations automatically](https://repos.opensource.microsoft.com) ## How to identify an official Microsoft organization on GitHub - Official Microsoft GitHub orgs have a professional avatar and include a web address at Microsoft.com or an associated official domain - Official Microsoft orgs are [Verified on GitHub](https://github.blog/changelog/2018-08-07-domain-verification/) - look for the domain verification indicator on GitHub --- ## File: SECURITY.MD # IMPORTANT: THIS IS NOT THE SOURCE OF TRUTH FOR THIS FILE The `SECURITY.MD` file used by default in new Microsoft GitHub repos can be found in the repo `microsoft/repo-templates` at https://github.com/microsoft/repo-templates. Please contribute any fixes there. ------ ## Security Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below. ## Reporting Security Issues **Please do not report security vulnerabilities through public GitHub issues.** Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report). If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc). You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) * Full paths of source file(s) related to the manifestation of the issue * The location of the affected source code (tag/branch/commit or direct URL) * Any special configuration required to reproduce the issue * Step-by-step instructions to reproduce the issue * Proof-of-concept or exploit code (if possible) * Impact of the issue, including how an attacker might exploit the issue This information will help us triage your report more quickly. If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs. ## Preferred Languages We prefer all communications to be in English. ## Policy Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd). --- ## File: codeofconduct/index.md # Microsoft Open Source Code of Conduct This code of conduct outlines expectations for participation in Microsoft-managed open source communities, as well as steps for reporting unacceptable behavior. We are committed to providing a welcoming and inspiring community for all. People violating this code of conduct may be banned from the community. Our open source communities strive to: * **Be friendly and patient:** Remember you might not be communicating in someone else's primary spoken or programming language, and others may not have your level of understanding. * **Be welcoming:** Our communities welcome and support people of all backgrounds and identities. This includes, but is not limited to members of any race, ethnicity, culture, national origin, color, immigration status, social and economic class, educational level, sex, sexual orientation, gender identity and expression, age, size, family status, political belief, religion, and mental and physical ability. * **Be respectful:** We are a world-wide community of professionals, and we conduct ourselves professionally. Disagreement is no excuse for poor behavior and poor manners. Disrespectful and unacceptable behavior includes, but is not limited to: * Violent threats or language. * Discriminatory or derogatory jokes and language. * Posting sexually explicit or violent material. * Posting, or threatening to post, people's personally identifying information ("doxing"). * Insults, especially those using discriminatory terms or slurs. * Behavior that could be perceived as sexual attention. * Advocating for or encouraging any of the above behaviors. * **Understand disagreements:** Disagreements, both social and technical, are useful learning opportunities. Seek to understand the other viewpoints and resolve differences constructively. * This code is not exhaustive or complete. It serves to capture our common understanding of a productive, collaborative environment. We expect the code to be followed in spirit as much as in the letter. ## Scope This code of conduct applies to all repos and communities for Microsoft-managed open source projects regardless of whether or not the repo explicitly calls out its use of this code. The code also applies in public spaces when an individual is representing a project or its community. Examples include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. Note: Some Microsoft-managed communities have codes of conduct that pre-date this document and issue resolution process. While communities are not required to change their code, they are expected to use the resolution process outlined here. The review team will coordinate with the communities involved to address your concerns. ## Reporting Code of Conduct Issues We encourage all communities to resolve issues on their own whenever possible. This builds a broader and deeper understanding and ultimately a healthier interaction. In the event that an issue cannot be resolved locally, please feel free to report your concerns by contacting [opencode@microsoft.com](mailto:opencode@microsoft.com). Your report will be handled in accordance with the issue resolution process described in the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/). In your report please include: * Your contact information. * Names (real, usernames or pseudonyms) of any individuals involved. If there are additional witnesses, please include them as well. * Your account of what occurred, and if you believe the incident is ongoing. If there is a publicly available record (e.g. a mailing list archive or a public chat log), please include a link or attachment. * Any additional information that may be helpful. All reports will be reviewed by a multi-person team and will result in a response that is deemed necessary and appropriate to the circumstances. Where additional perspectives are needed, the team may seek insight from others with relevant expertise or experience. The confidentiality of the person reporting the incident will be kept at all times. Involved parties are never part of the review team. Anyone asked to stop unacceptable behavior is expected to comply immediately. If an individual engages in unacceptable behavior, the review team may take any action they deem appropriate, including a permanent ban from the community. _This code of conduct is based on the template established by the [TODO Group](http://todogroup.org/) and used by numerous other large communities (e.g., [Facebook](https://code.facebook.com/pages/876921332402685/open-source-code-of-conduct), Twitter, GitHub and the Scope section from the [Contributor Covenant version 1.4](http://contributor-covenant.org/version/1/4/)._ --- ## File: codeofconduct/faq/index.md # Microsoft Open Source Code of Conduct FAQ ## Contents * [Why have a Code of Conduct?](#whycode) * [Why this Code of Conduct?](#whythis) * [Why do this across all repos?](#allrepos) * [How does a project adopt the code and process?](#howadopt) * [What if my Microsoft project already has a Code of Conduct?](#alreadyhave) * [How do I raise a concern?](#howreport) * [What is the process for addressing issues that arise?](#process) * [How does the resolution team work?](#howteamworks) * [Can I use this Code of Conduct in my projects?](#caniuse) ## Why have a Code of Conduct? People and communities are the foundation of open source. Communities thrive through diversity of thought and the safety of its members. Microsoft has a deep-seated commitment to diversity and inclusion. All Microsoft employees enjoy a safe work environment and a culture of mutual respect and responsibility. Our team members and open source partners should enjoy the same environment when collaborating on open source projects. While conduct issues seldom arise, when they do, they are often very public and very passionate. It is best for the community to be prepared with: * A clear and accessible **Code of Conduct** stating the norms under which projects operate * An **Issue Resolution Process** for addressing cases where these norms are not being maintained ## Why this Code of Conduct? The [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) is an instantiation of the [TODO Group](http://todogroup.org/) Code of Conduct template, that captures the Microsoft culture of equality, respect and inclusion. This same template is used by industry colleagues such as Facebook, Twitter, GitHub, Yahoo and others in their open source projects. ## Why do this across all repos? Having one code and process is both efficient and consistent — community members can participate in any Microsoft-driven project and be confident that they know the norms and that they are supported by a robust process. ## How does a project adopt the code and process? All Microsoft projects are automatically covered by the Code of Conduct and the Issue Resolution Process. **However**, it is critical that everyone in the communities be aware of the code and process. For that reason, all projects must link to the Code of Conduct in their README and/or CONTRIBUTING files using the following markdown. ``` This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. ``` ## What if my Microsoft project already has a Code of Conduct? Thanks for being proactive in this important area. You are welcome to continue using your existing Code of Conduct, or you may change to the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). Either way, all Microsoft projects must use the same issue resolution process. Please ensure that your documentation and guidance is updated to point people at the process outlined in this FAQ (see below). Projects in the .NET Foundation are already covered by the [.NET Foundation Contributors Code of Conduct](https://www.dotnetfoundation.org/code-of-conduct). As such, there is no need for .NET Foundation projects to make any changes. ## How do I raise a concern? If you have witnessed or been subjected to a violation of the Code of Conduct, please send an email to [opencode@microsoft.com](mailto:opencode@microsoft.com). Your message will be handled in a secure and confidential manner. This email address is monitored by people who are not active in open source projects or communities — so you can be sure that you are not communicating with a person involved in the issue you are reporting. You will receive a response within one business day acknowledging receipt of your email and describing the process for its resolution. (See below for more details.) ## What is the process for addressing issues that arise? Emails sent to [opencode@microsoft.com](mailto:opencode@microsoft.com) kick off the following process: * Your message will be acknowledged within one business day. * Within the following business day, a small, 2-3, person team will be assembled from a pool of Microsoft employee volunteers to review your concern. This team will be as diverse as possible given its size and will pull in additional people as needed to gain further insight and provide guidance. The team will not include anyone directly involved in the issue that has been raised. * From there team will work with you and the others involved to come to a conclusion. While issue complexity varies, the goal is to resolve issues within five working days. * All communication will be confidential with very limited circulation. ## How does the resolution team work? Beyond the norms and values set out in the Code of Conduct, issue review teams operate under the following principles: * **Less is more** — As much as possible, let the community work it out. It is much better to have communities self-correct than to have outsiders come in and β€œfix” problems. * **Equality** — Contribution value or status in the community are not relevant to the review. Key people do not have more rights (either to abuse or be protected from abuse) than others in the community. * **Independence** — Reviewers must act and be seen to act with independence from the project(s) in question and from Microsoft. ## Can I use this Code of Conduct in my projects? Yes. The document is licenced under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0). You will of course need to update the document to suit the specifics of your projects, but by all means use this however you can to drive great communities.