RD-Agent Documentation

Technical Documentation: microsoft/RD-Agent

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

🖥️ Live Demo |
🎥 Demo Video ▶️YouTube |
📖 Documentation |
📄 Tech Report |
📃 Papers

📰 News

🧪 Agent² RL-Bench

Agent² RL-Bench evaluates whether LLM agents can autonomously engineer end-to-end post-training pipelines, spanning static optimization and stateful online RL.

🏆 The Best Machine Learning Engineering Agent!

MLE-bench is a comprehensive benchmark evaluating the performance of AI agents on machine learning engineering tasks. Utilizing datasets from 75 Kaggle competitions, MLE-bench provides robust assessments of AI systems' capabilities in real-world ML engineering scenarios.

R&D-Agent currently leads as the top-performing machine learning engineering agent on MLE-bench:

Notes:

• O3(R)+GPT-4.1(D): This version is designed to both reduce average time per loop and leverage a cost-effective combination of backend LLMs by seamlessly integrating Research Agent (o3) with Development Agent (GPT-4.1).
• AIDE o1-preview: Represents the previously best public result on MLE-bench as reported in the original MLE-bench paper.
• Average and standard deviation results for R&D-Agent o1-preview is based on a independent of 5 seeds and for R&D-Agent o3(R)+GPT-4.1(D) is based on 6 seeds.
• According to MLE-Bench, the 75 competitions are categorized into three levels of complexity: Low==Lite if we estimate that an experienced ML engineer can produce a sensible solution in under 2 hours, excluding the time taken to train any models; Medium if it takes between 2 and 10 hours; and High if it takes more than 10 hours.

You can inspect the detailed runs of the above results online.

• R&D-Agent o1-preview detailed runs
• R&D-Agent o3(R)+GPT-4.1(D) detailed runs

For running R&D-Agent on MLE-bench, refer to MLE-bench Guide: Running ML Engineering via MLE-bench

🥇 The First Data-Centric Quant Multi-Agent Framework!

R&D-Agent for Quantitative Finance, in short RD-Agent(Q), is the first data-centric, multi-agent framework designed to automate the full-stack research and development of quantitative strategies via coordinated factor-model co-optimization.

Extensive experiments in real stock markets show that, at a cost under $10, RD-Agent(Q) achieves approximately 2× higher ARR than benchmark factor libraries while using over 70% fewer factors. It also surpasses state-of-the-art deep time-series models under smaller resource budgets. Its alternating factor–model optimization further delivers excellent trade-off between predictive accuracy and strategy robustness.

You can learn more details about RD-Agent(Q) through the paper and reproduce it through the documentation.

Data Science Agent Preview

Check out our demo video showcasing the current progress of our Data Science Agent under development:

https://github.com/user-attachments/assets/3eccbecb-34a4-4c81-bce4-d3f8862f7305

🌟 Introduction

R&D-Agent aims to automate the most critical and valuable aspects of the industrial R&D process, and we begin with focusing on the data-driven scenarios to streamline the development of models and data.
Methodologically, we have identified a framework with two key components: 'R' for proposing new ideas and 'D' for implementing them.
We believe that the automatic evolution of R&D will lead to solutions of significant industrial value.

R&D is a very general scenario. The advent of R&D-Agent can be your

• 💰 Automatic Quant Factory (🎥Demo Video|▶️YouTube)
• 🤖 Data Mining Agent: Iteratively proposing data & models (🎥Demo Video 1|▶️YouTube) (🎥Demo Video 2|▶️YouTube) and implementing them by gaining knowledge from data.
• 🦾 Research Copilot: Auto read research papers (🎥Demo Video|▶️YouTube) / financial reports (🎥Demo Video|▶️YouTube) and implement model structures or building datasets.
• 🤖 Kaggle Agent: Auto Model Tuning and Feature Engineering(🎥Demo Video Coming Soon...) and implementing them to achieve more in competitions.
• 🧪 FT-Agent: Autonomous LLM fine-tuning for benchmark-driven domain adaptation. See the LLM fine-tuning guide.
• ...

You can click the links above to view the demo. We're continuously adding more methods and scenarios to the project to enhance your R&D processes and boost productivity.

Additionally, you can take a closer look at the examples in our 🖥️ Live Demo.

⚡ Quick start

RD-Agent currently only supports Linux. Copy Section

You can try above demos by running the following command:

🐳 Docker installation. Copy Section

Users must ensure Docker is installed before attempting most scenarios. Please refer to the official 🐳Docker page for installation instructions.
Ensure the current user can run Docker commands without using sudo. You can verify this by executing docker run hello-world.

🐍 Create a Conda Environment Copy Section

• Create a new conda environment with Python (3.10 and 3.11 are well-tested in our CI):
  sh Copy

  conda create -n rdagent python=3.10

• Activate the environment:
  sh Copy

  conda activate rdagent

🛠️ Install the R&D-Agent Copy Section

For Users

• You can directly install the R&D-Agent package from PyPI:
  sh Copy

  pip install rdagent

For Developers

• If you want to try the latest version or contribute to RD-Agent, you can install it from the source and follow the development setup:
  sh Copy

  git clone https://github.com/microsoft/RD-Agent
  cd RD-Agent
  make dev

More details can be found in the development setup.

💊 Health check Copy Section

• rdagent provides a health check that currently checks two things.
  • whether the docker installation was successful.
  • whether the default port used by the rdagent ui is occupied.
  sh Copy

  rdagent health_check --no-check-env

⚙️ Configuration Copy Section

• The demos requires following ability:

  • ChatCompletion
  • json_mode
  • embedding query

  You can set your Chat Model and Embedding Model in the following ways:

  🔥 Attention: We now provide experimental support for DeepSeek models! You can use DeepSeek's official API for cost-effective and high-performance inference. See the configuration example below for DeepSeek setup.

• Using LiteLLM (Default): We now support LiteLLM as a backend for integration with multiple LLM providers. You can configure in multiple ways:

  Option 1: Unified API base for both models

  Configuration Example: OpenAI Setup :

  bash Copy

  cat << EOF  > .env
  # Set to any model supported by LiteLLM.
  CHAT_MODEL=gpt-4o 
  EMBEDDING_MODEL=text-embedding-3-small
  # Configure unified API base
  OPENAI_API_BASE=<your_unified_api_base>
  OPENAI_API_KEY=<replace_with_your_openai_api_key>

  Configuration Example: Azure OpenAI Setup :

  Before using this configuration, please confirm in advance that your Azure OpenAI API key supports embedded models.

  bash Copy

  cat << EOF  > .env
  EMBEDDING_MODEL=azure/<Model deployment supporting embedding>
  CHAT_MODEL=azure/<your deployment name>
  AZURE_API_KEY=<replace_with_your_openai_api_key>
  AZURE_API_BASE=<your_unified_api_base>
  AZURE_API_VERSION=<azure api version>

  Option 2: Separate API bases for Chat and Embedding models

  bash Copy

  cat << EOF  > .env
  # Set to any model supported by LiteLLM.
  # Configure separate API bases for chat and embedding
  
  # CHAT MODEL:
  CHAT_MODEL=gpt-4o 
  OPENAI_API_BASE=<your_chat_api_base>
  OPENAI_API_KEY=<replace_with_your_openai_api_key>
  
  # EMBEDDING MODEL:
  # TAKE siliconflow as an example, you can use other providers.
  # Note: embedding requires litellm_proxy prefix
  EMBEDDING_MODEL=litellm_proxy/BAAI/bge-large-en-v1.5
  LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
  LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1

  Configuration Example: DeepSeek Setup :

  Since many users encounter configuration errors when setting up DeepSeek. Here's a complete working example for DeepSeek Setup:

  bash Copy

  cat << EOF  > .env
  # CHAT MODEL: Using DeepSeek Official API
  CHAT_MODEL=deepseek/deepseek-chat 
  DEEPSEEK_API_KEY=<replace_with_your_deepseek_api_key>
  
  # EMBEDDING MODEL: Using SiliconFlow for embedding since deepseek has no embedding model.
  # Note: embedding requires litellm_proxy prefix
  EMBEDDING_MODEL=litellm_proxy/BAAI/bge-m3
  LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
  LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1

  Notice: If you are using reasoning models that include thought processes in their responses (such as \ tags), you need to set the following environment variable:

  bash Copy

  REASONING_THINK_RM=True

  You can also use a deprecated backend if you only use OpenAI API or Azure OpenAI directly. For this deprecated setting and more configuration information, please refer to the documentation.

• If your environment configuration is complete, please execute the following commands to check if your configuration is valid. This step is necessary.

  bash Copy

  rdagent health_check

🚀 Run the Application Copy Section

The 🖥️ Live Demo is implemented by the following commands(each item represents one demo, you can select the one you prefer):

• Run the Automated Quantitative Trading & Iterative Factors Model Joint Evolution: Qlib self-loop factor & model proposal and implementation application

  sh Copy

  rdagent fin_quant

• Run the Automated Quantitative Trading & Iterative Factors Evolution: Qlib self-loop factor proposal and implementation application

  sh Copy

  rdagent fin_factor

• Run the Automated Quantitative Trading & Iterative Model Evolution: Qlib self-loop model proposal and implementation application

  sh Copy

  rdagent fin_model

• Run the Automated Quantitative Trading & Factors Extraction from Financial Reports: Run the Qlib factor extraction and implementation application based on financial reports

  sh Copy

  # 1. Generally, you can run this scenario using the following command:
  rdagent fin_factor_report --report-folder=<Your financial reports folder path>
  
  # 2. Specifically, you need to prepare some financial reports first. You can follow this concrete example:
  wget https://github.com/SunsetWolf/rdagent_resource/releases/download/reports/all_reports.zip
  unzip all_reports.zip -d git_ignore_folder/reports
  rdagent fin_factor_report --report-folder=git_ignore_folder/reports

• Run the Automated Model Research & Development Copilot: model extraction and implementation application

  sh Copy

  # 1. Generally, you can run your own papers/reports with the following command:
  rdagent general_model <Your paper URL>
  
  # 2. Specifically, you can do it like this. For more details and additional paper examples, use `rdagent general_model -h`:
  rdagent general_model  "https://arxiv.org/pdf/2210.09789"

• Run the Automated Medical Prediction Model Evolution: Medical self-loop model proposal and implementation application

  bash Copy

  # Generally, you can run the data science program with the following command:
  rdagent data_science --competition <your competition name>
  
  # Specifically, you need to create a folder for storing competition files (e.g., competition description file, competition datasets, etc.), and configure the path to the folder in your environment. In addition, you need to use chromedriver when you download the competition descriptors, which you can follow for this specific example:
  
  # 1. Download the dataset, extract it to the target folder.
  wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip
  unzip arf-12-hours-prediction-task.zip -d ./git_ignore_folder/ds_data/
  
  # 2. Configure environment variables in the `.env` file
  dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/ds_data"
  dotenv set DS_CODER_ON_WHOLE_PIPELINE True
  dotenv set DS_IF_USING_MLE_DATA False
  dotenv set DS_SAMPLE_DATA_BY_LLM False
  dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
  
  # 3. run the application
  rdagent data_science --competition arf-12-hours-prediction-task

  NOTE: For more information about the dataset, please refer to the documentation.

• Run the Automated Kaggle Model Tuning & Feature Engineering: self-loop model proposal and feature engineering implementation application

  Using tabular-playground-series-dec-2021 as an example.

  1. Register and login on the Kaggle website.
  2. Configuring the Kaggle API.
     (1) Click on the avatar (usually in the top right corner of the page) -> Settings -> Create New Token, A file called kaggle.json will be downloaded.
     (2) Move kaggle.json to ~/.config/kaggle/
     (3) Modify the permissions of the kaggle.json file. Reference command: chmod 600 ~/.config/kaggle/kaggle.json
  3. Join the competition: Click Join the competition -> I Understand and Accept at the bottom of the competition details page.

  bash Copy

  # Generally, you can run the Kaggle competition program with the following command:
  rdagent data_science --competition <your competition name>
  
  # 1. Configure environment variables in the `.env` file
  mkdir -p ./git_ignore_folder/ds_data
  dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/ds_data"
  dotenv set DS_CODER_ON_WHOLE_PIPELINE True
  dotenv set DS_IF_USING_MLE_DATA True
  dotenv set DS_SAMPLE_DATA_BY_LLM True
  dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen
  
  # 2. run the application
  rdagent data_science --competition tabular-playground-series-dec-2021

• Run FT-Agent for Autonomous LLM Fine-Tuning: an ICML 2026 LLM fine-tuning scenario for benchmark-driven data processing, training, evaluation, and feedback-guided refinement.

  bash Copy

  # See the full setup, benchmark descriptions, dataset notes, and examples:
  # rdagent/app/finetune/llm/README.md
  # Configure FT_TARGET_BENCHMARK and FT_BENCHMARK_DESCRIPTION before running.
  rdagent llm_finetune --base-model Qwen/Qwen2.5-7B-Instruct

🖥️ Monitor the Application Results Copy Section

Streamlit UI

Use the Streamlit UI to view run logs, especially for the data_science scenario.

rdagent ui --port 19899 --log-dir <your log folder like "log/"> --data-science

About the data_science parameter: If you want to see the logs of the data science scenario, set the data_science parameter to True; otherwise set it to False.

Web UI

We also provide a separate web frontend in web/ for the Flask backend started by server_ui.

NOTE: This web UI is different from rdagent ui. The current web UI does not support the data_science scenario yet. For the data_science scenario, please continue to use rdagent ui --data-science.

cd web
npm install

To build the frontend for the Flask backend, generate the static assets into the default directory used by server_ui:

cd web
npm run build:flask

By default, server_ui serves static files from ./git_ignore_folder/static. If you need a different location, set the UI_STATIC_PATH environment variable before starting the backend.

Start the Flask backend and serve the built frontend together with the real-time APIs:

export UI_SERVER_AUTH_TOKEN='<a-long-random-token>'
rdagent server_ui --port 19899

After that, open http://127.0.0.1:19899/?token=<a-long-random-token> once in your browser.

Web UI security and remote access

The Flask backend listens on 127.0.0.1 by default. All API access requires authentication, including localhost: a malicious website can send requests to a browser's local services. Configure a long, random UI_SERVER_AUTH_TOKEN before starting the server (for example, generate one with python -c 'import secrets; print(secrets.token_urlsafe(32))').

To access the Web UI from another machine, explicitly bind it to a non-local address and configure an authentication token:

export UI_SERVER_AUTH_TOKEN='<a-long-random-token>'
rdagent server_ui --port 19899 --host 0.0.0.0

Then open the following URL once to establish an authenticated browser session:

http://<server-host>:19899/?token=<a-long-random-token>

The server removes the token from the address bar by redirecting to / and stores it in an HTTP-only, same-site cookie. API clients can instead send it in the request header:

Authorization: Bearer <a-long-random-token>

The server refuses to start unless UI_SERVER_AUTH_TOKEN is set, even on localhost. When exposing it outside a trusted development machine, put it behind an HTTPS reverse proxy and avoid recording token-bearing query strings in proxy logs. The --host option controls the address when the server is started through the CLI; UI_SERVER_HOST is the corresponding default for direct use of the backend entry point.

Cross-origin browser access is disabled by default. If the frontend and backend are served from different origins, configure an explicit JSON allowlist rather than enabling every origin:

export UI_CORS_ALLOWED_ORIGINS='["https://ui.example.com"]'

Origins must be exact HTTP(S) origins, including any non-default port; wildcards and URL paths are rejected. The server validates request origins before executing API handlers. Cookie-authenticated POSTs require a same-origin or allowlisted Origin (or Referer when Origin is absent). Requests with missing provenance are rejected. Non-browser clients and internal trace publishers must send a Bearer token. Cross-origin clients must authenticate as well; CORS permission does not grant API access. When a reverse proxy changes the backend's host or scheme, add the exact browser-facing origin to the allowlist; arbitrary forwarded headers are not trusted.

The Web UI starts tasks that generate and execute code. Treat authenticated access as trusted operator access, and run workers in an isolated, least-privileged environment with restricted network egress and no host credentials or Docker socket exposed to generated code.

Web UI storage and compatibility settings

The Flask backend uses the following environment variables. Uploaded input files are deliberately kept outside the trace directory so that they cannot be discovered and deserialized as persisted traces.

Uploads whose filenames end in .dill, .pickle, .pkl, .py, .pyc, or .pyo are rejected. Existing workflows that use these formats as uploaded inputs must convert them to a non-executable data format or provide them through another trusted mechanism.

The /upload API accepts actual file uploads only; URL and server-path text inputs are rejected before task creation. General Model Implementation requires exactly one uploaded report. Download any report through a trusted process first, then upload it. This removes the API's arbitrary report-fetch entry point; it does not sandbox code generated by an authorized task.

Legacy pickle trace loading is disabled by default because pickle deserialization can execute code. Consequently, after a server restart, an existing trace may still appear in the history list but its saved messages will not be loaded into the Web UI. If compatibility with trusted historical traces is required, opt in explicitly:

export UI_LOAD_LEGACY_PICKLE_TRACES=true
rdagent server_ui --port 19899

Only enable this setting when every file under UI_TRACE_FOLDER is trusted and the directory is not writable by untrusted users or services.

Data-science trace share links no longer accept a URL-controlled log_folder. A link can preserve the selected trace, but the recipient must have the corresponding log folder configured or select it in the UI.

Common Notes

Port 19899 is used in the examples above. Before starting either UI, check whether this port is already occupied. If it is, please change it to another available port.

You can check whether the port is occupied by running:

rdagent health_check --no-check-env --no-check-docker

🏭 Scenarios

We have applied R&D-Agent to multiple valuable data-driven industrial scenarios.

🎯 Goal: Agent for Data-driven R&D Copy Chapter

In this project, we are aiming to build an Agent to automate Data-Driven R&D that can

• 📄 Read real-world material (reports, papers, etc.) and extract key formulas, descriptions of interested features and models, which are the key components of data-driven R&D .
• 🛠️ Implement the extracted formulas (e.g., features, factors, and models) in runnable codes.
  • Due to the limited ability of LLM in implementing at once, build an evolving process for the agent to improve performance by learning from feedback and knowledge.
• 💡 Propose new ideas based on current knowledge and observations.

📈 Scenarios/Demos Copy Chapter

In the two key areas of data-driven scenarios, model implementation and data building, our system aims to serve two main roles: 🦾Copilot and 🤖Agent.

• The 🦾Copilot follows human instructions to automate repetitive tasks.
• The 🤖Agent, being more autonomous, actively proposes ideas for better results in the future.

The supported scenarios are listed below:

• RoadMap: Currently, we are working hard to add new features to the Kaggle scenario.

Different scenarios vary in entrance and configuration. Please check the detailed setup tutorial in the scenarios documents.

Here is a gallery of successful explorations (5 traces showed in 🖥️ Live Demo). You can download and view the execution trace using this command from the documentation.

Please refer to 📖readthedocs_scen for more details of the scenarios.

⚙️ Framework

Automating the R&D process in data science is a highly valuable yet underexplored area in industry. We propose a framework to push the boundaries of this important research field.

The research questions within this framework can be divided into three main categories:

We believe that the key to delivering high-quality solutions lies in the ability to evolve R&D capabilities. Agents should learn like human experts, continuously improving their R&D skills.

More documents can be found in the 📖 readthedocs.

📃 Paper/Work list

Overall Technical Report Copy Chapter

• R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science

@misc{yang2025rdagentllmagentframeworkautonomous,
      title={R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science}, 
      author={Xu Yang and Xiao Yang and Shikai Fang and Yifei Zhang and Jian Wang and Bowen Xian and Qizheng Li and Jingyuan Li and Minrui Xu and Yuante Li and Haoran Pan and Yuge Zhang and Weiqing Liu and Yelong Shen and Weizhu Chen and Jiang Bian},
      year={2025},
      eprint={2505.14738},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2505.14738}, 
}

📊 Benchmark Copy Chapter

• Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training? (Project Page · Code)

@misc{chen2026agent2rlbench,
      title={Agent$^2$ RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?},
      author={Wanyi Chen and Xiao Yang and Xu Yang and Tianming Sha and Qizheng Li and Zhuo Wang and Bowen Xian and Fang Kong and Weiqing Liu and Jiang Bian},
      year={2026},
      eprint={2604.10547},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2604.10547}
}

• Towards Data-Centric Automatic R&D

@misc{chen2024datacentric,
    title={Towards Data-Centric Automatic R&D},
    author={Haotian Chen and Xinjie Shen and Zeqi Ye and Wenjun Feng and Haoxue Wang and Xiao Yang and Xu Yang and Weiqing Liu and Jiang Bian},
    year={2024},
    eprint={2404.11276},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

🔍 Research Copy Chapter

In a data mining expert's daily research and development process, they propose a hypothesis (e.g., a model structure like RNN can capture patterns in time-series data), design experiments (e.g., finance data contains time-series and we can verify the hypothesis in this scenario), implement the experiment as code (e.g., Pytorch model structure), and then execute the code to get feedback (e.g., metrics, loss curve, etc.). The experts learn from the feedback and improve in the next iteration.

Based on the principles above, we have established a basic method framework that continuously proposes hypotheses, verifies them, and gets feedback from the real-world practice. This is the first scientific research automation framework that supports linking with real-world verification.

For more detail, please refer to our 🖥️ Live Demo page.

🛠️ Development Copy Chapter

• Collaborative Evolving Strategy for Automatic Data-Centric Development

@misc{yang2024collaborative,
    title={Collaborative Evolving Strategy for Automatic Data-Centric Development},
    author={Xu Yang and Haotian Chen and Wenjun Feng and Haoxue Wang and Zeqi Ye and Xinjie Shen and Xiao Yang and Shizhao Sun and Weiqing Liu and Jiang Bian},
    year={2024},
    eprint={2407.18690},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

Deep Application in Diverse Scenarios Copy Chapter

• FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents

@misc{li2026ftdojo,
  title={FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents},
  author={Qizheng Li and Yifei Zhang and Xiao Yang and Xu Yang and Zhuo Wang and Weiqing Liu and Jiang Bian},
  year={2026},
  eprint={2603.01712},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2603.01712}
}

FT-Agent, the autonomous LLM fine-tuning scenario from this paper, is available through the LLM fine-tuning guide.

• R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization

@misc{li2025rdagentquantmultiagentframeworkdatacentric,
      title={R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization}, 
      author={Yuante Li and Xu Yang and Xiao Yang and Minrui Xu and Xisen Wang and Weiqing Liu and Jiang Bian},
      year={2025},
      eprint={2505.15155},
      archivePrefix={arXiv},
      primaryClass={q-fin.CP},
      url={https://arxiv.org/abs/2505.15155}, 
}

• Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search

@article{zhang2026reasoning,
  title={Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search},
  author={Zhang, Yifei and Yang, Xu and Yang, Xiao and Xian, Bowen and Li, Qizheng and Fang, Shikai and Li, Jingyuan and Wang, Jian and Xu, Mingrui and Liu, Weiqing and others},
  journal={arXiv preprint arXiv:2603.01692},
  year={2026}
}

You can check the detailed execution traces online at Gome GPT-5 Traces.

🤝 Contributing

We welcome contributions and suggestions to improve R&D-Agent. Please refer to the Contributing Guide for more details on how to contribute.

Before submitting a pull request, ensure that your code passes the automatic CI checks.

📝 Guidelines Copy Chapter

This project welcomes contributions and suggestions.
Contributing to this project is straightforward and rewarding. Whether it's solving an issue, addressing a bug, enhancing documentation, or even correcting a typo, every contribution is valuable and helps improve R&D-Agent.

To get started, you can explore the issues list, or search for TODO: comments in the codebase by running the command grep -r "TODO:".

Before we released R&D-Agent as an open-source project on GitHub, it was an internal project within our group. Unfortunately, the internal commit history was not preserved when we removed some confidential code. As a result, some contributions from our group members, including Haotian Chen, Wenjun Feng, Haoxue Wang, Zeqi Ye, Xinjie Shen, and Jinhui Li, were not included in the public commits.

⚖️ Legal disclaimer

The RD-agent is provided “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. The RD-agent is aimed to facilitate research and development process in the financial industry and not ready-to-use for any financial investment or advice. Users shall independently assess and test the risks of the RD-agent in a specific use scenario, ensure the responsible use of AI technology, including but not limited to developing and integrating risk mitigation measures, and comply with all applicable laws and regulations in all applicable jurisdictions. The RD-agent does not provide financial opinions or reflect the opinions of Microsoft, nor is it designed to replace the role of qualified financial professionals in formulating, assessing, and approving finance products. The inputs and outputs of the RD-agent belong to the users and users shall assume all liability under any theory of liability, whether in contract, torts, regulatory, negligence, products liability, or otherwise, associated with use of the RD-agent and any inputs and outputs thereof.

Overview (`.devcontainer/README.md`) Copy Section

Introduction

!!!!!This dev container is not for public development!!!!!!
!!!!!Please don't use it if you are just a public open-source user.!!!!!!

Steps to run the dev container (for internal use only)

Prerequisites(this is the reason why this dev container is not for public use):

• Make sure you have the rdagentappregistry.azurecr.io/rd-agent-mle:20250623 image locally & DevContainer is installed in your IDE
• The kaggle dataset is located at /home/shared/RD-Agent/kaggle

1. Open the project and select "Open In DevContainer"
2. Set up your Kaggle Key (do not share this; other internal URLs are hardcoded in the config files)

export KAGGLE_USERNAME=
export KAGGLE_KEY=

3. Run: python rdagent/app/data_science/loop.py --competition nomad2018-predict-transparent-conductors

Additional Notes

• Please install and use this Dev Container in VS Code.
• You must open VS Code remotely and enter the RD-Agent directory before running the DevContainer configuration (.devcontainer/devcontainer.json). Otherwise, the workspace and path mappings will not work as expected.
• To open the DevContainer correctly in VS Code:
  1. Remotely connect to the machine and open the RD-Agent folder in VS Code.
  2. Press Ctrl+Shift+P (or Cmd+Shift+P on Mac), type and select "Dev Containers: Reopen in Container".

How to grade your submission in the DevContainer

1. save your submission file in ./sumission.csv

2. Run evaluation
   DS_COMPETITION=
   conda run -n mlebench mlebench grade-sample submission.csv $DS_COMPETITION --data-dir /tmp/kaggle/zip_files/

Overview (`web/README.md`) Copy Section

R&D-Agent

Project setup Copy Chapter

npm install

Compiles and hot-reloads for development Copy Section

npm run dev

Compiles and minifies for production Copy Section

npm run build

API URL behavior after build Copy Section

This project uses the current page origin as the API base URL.

If the built frontend is served by the same Flask server that also exposes /upload, /trace, /control and other APIs, no extra frontend configuration is needed. The frontend will automatically call the same host and port that served the page.

This template should help get you started developing with Vue 3 and TypeScript in Vite. The template uses Vue 3 <script setup> SFCs, check out the script setup docs to learn more.

Recommended Setup Copy Chapter

• VS Code + Vue - Official (previously Volar) and disable Vetur

• Use vue-tsc for performing the same type checking from the command line, or for generating d.ts files for SFCs.

Overview (`test/utils/README.md`) Copy Section

🐳 Run Docker & Qlib

📄 Description Copy Chapter

This guide explains how to run the Qlib Docker test file located at test/utils/test_env.py in the RD-Agent repository.

🚀 Running Instructions Copy Chapter

1. Install the required Python libraries Copy Section

• Ensure that the docker Python library is installed:
  sh Copy

  pip install docker

2. Run the test script Copy Section

• Execute the test script to verify the Docker environment setup:
  sh Copy

  python test/utils/test_env.py

Troubleshooting Copy Section

• PermissionError: [Errno 13] Permission denied.

  This error occurs when the current user does not have the necessary permissions to access the Docker socket. To resolve this issue, follow these steps:

1. Add the current user to the docker group
   Docker requires root or docker group user permissions to access the Docker socket. Add the current user to the docker group:

   sh Copy

   sudo usermod -aG docker $USER

2. Refresh group changes
   To apply the group changes, log out and log back in, or use the following command:

   sh Copy

   newgrp docker

3. Verify Docker access
   Run the following command to ensure that Docker can be accessed:

   sh Copy

   docker run hello-world

4. Rerun the test script
   After completing these steps, rerun the test script:

   sh Copy

   python test/utils/test_env.py

🛠️ Detailed Qlib Docker Function Framework Copy Chapter

Here, we provide an overview of the specific functions within the Qlib Docker framework, their purposes, and examples of how to call them.

QTDockerEnv Class in `env.py` Copy Section

The QTDockerEnv class is responsible for setting up and running Docker environments for Qlib experiments.

Methods:

1. prepare()

   • Purpose: Prepares the Docker environment for running experiments. This includes building the Docker image if necessary.
   • Example:
     python Copy

     qtde = QTDockerEnv()
     qtde.prepare()

2. run(local_path: str, entry: str) -> str

   • Purpose: Runs a specified entry point (e.g., a configuration file) in the prepared Docker environment.
   • Parameters:
     • local_path: Path to the local directory to mount into the Docker container.
     • entry: Command or entry point to run inside the Docker container.
   • Returns: The stdout output from the Docker container.
   • Example:
     python Copy

     result = qtde.run(local_path="/path/to/env_tpl", entry="qrun conf.yaml")

📊 Expected Output Copy Section

Upon successful execution, the test script will produce analysis results of benchmark returns and various risk metrics. The expected output should be similar to:

'The following are analysis results of benchmark return (1 day).'
risk
mean               0.000477
std                0.012295
annualized_return  0.113561
information_ratio  0.598699
max_drawdown      -0.370479

'The following are analysis results of the excess return without cost (1 day).'
risk
mean               0.000530
std                0.005718
annualized_return  0.126029
information_ratio  1.428574
max_drawdown      -0.072310

'The following are analysis results of the excess return with cost (1 day).'
risk
mean               0.000339
std                0.005717
annualized_return  0.080654
information_ratio  0.914486
max_drawdown      -0.086083

'The following are analysis results of indicators (1 day).'
value
ffr    1.0
pa     0.0
pos    0.0

By following these steps and using the provided functions, you should be able to run the Qlib Docker tests and obtain the expected analysis results.

Overview (`rdagent/app/CI/README.md`) Copy Section

CI 检查

.github/workflows/ci.yml配置了提交时自动运行Makefile: 91~103行的命令，可以在这调整执行的命令

在.env中设置USE_CHAT_CACHE=True可以让第二次修复快一些

Rules

pyproject.toml中配置全局屏蔽的规则

• ruff: [tool.ruff.lint].ignore
• mypy: [tool.mypy]

ruff rules Copy Chapter

ruff rules 比较好修改, 大多可以自动修复

对于一些规则可以在代码中添加注释来局部屏蔽, 例如添加 # noqa E234,ANN001
遇到的不好修改的规则:

• 捕获异常时应该处理每一种异常，不应该统一当作Exception处理
• subprogress() 调用命令应该先判断命令是否安全
• ...

规则列表: ruff rules

mypy rules Copy Chapter

Mypy检查Python中类型标注, 常遇到需要修改结构/同时修改其他文件的情况, 自动修复效果不好

局部屏蔽: # type: ignore

规则列表: mypy rules

Optimization (Maybe)

• 添加指定文件夹检查的功能
• 增加一个修改选项: 调用vim, 用户直接修改此部分代码
• 显示时把Original Code部分去掉, 直接在输出的表示修改的diff部分用^^^^^^在代码行下标注出错误位置，这样能更直观地观察错误修复情况
• 当前为线性执行完所有修复后交给用户检查, 可修改成 后台多线程 / 进程处理修复的任务, 终端实时展示处理完的修复让用户检查
• ...

Overview (`rdagent/log/server/README.md`) Copy Section

API

All API requests require UI_SERVER_AUTH_TOKEN, including on localhost.
API clients must send Authorization: Bearer <token>. Browser cookie sessions
must provide a trusted Origin or Referer on POST requests. Cross-origin
requests are rejected unless their exact origin is explicitly allowed through
UI_CORS_ALLOWED_ORIGINS; an allowed origin still requires authentication.

A. Controls Copy Chapter

1. /upload [POST] Copy Section

Request

• "scenario": one of six values
  1. "Finance Data Building"
  2. "Finance Data Building (Reports)"
  3. "Finance Model Implementation"
  4. "General Model Implementation"
  5. "Medical Model Implementation"
  6. "Data Science"
• "files": 2 scenarios need this
  1. in "Finance Data Building (Reports)" Scenario, one or more pdf files.
  2. in "General Model Implementation" Scenario, exactly one uploaded PDF file. URL and server-path text inputs are rejected.
• "competition": Data Science Scenario need this, one of 75 competitions.
• "loops": Number of loops after which RD-Agent will automatically stop (optional; if not set, it will not stop automatically and must be stopped manually).
• "all_duration": Total duration (in hours) for which the RD-Agent should run before stopping automatically. If not set, the agent will continue running until stopped manually or by the "loops" parameter.

Response

• "id": a unique identifier string, such as /home/rdagent_log/data_science/competition_A/trace_1 or /home/rdagent_log/finance/trace_1, used to mark the series of logs generated by this RD-Agent run.

2. /control [POST] Copy Section

Request

• "id": identifier
• "action": one of three values
  1. "pause"
  2. "resume"
  3. "stop"

Response

• "status": "success" / "error: ..."

3. /trace [POST] Copy Section

Returns the sequence of Messages generated for the current id on the backend that have not yet been returned to the frontend.

Request

• "id": identifier
• "all": True / False. True means all Messages not yet provided to the frontend will be returned; False returns a random 1 to 10 Messages. In most cases, this should be True.
• "reset": True / False. Reset means the pointer for "not yet returned to the frontend" will be set back to the first Message generated for this id, i.e., return from the beginning. In most cases, this should be False.

Response

• a list of Messages

B. Messages Copy Chapter

Research Copy Section

Only 2 Message in one loop

1. hypothesis

{
    "tag": "research.hypothesis",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "content": {
        "hypothesis": "...",
        "reason": "...",
        "component": "...", // only exists in Data Science Scenario
        "concise_reason": "...",
        "concise_justification": "...",
        "concise_observation": "...",
        "concise_knowledge": "...",
    }
}

2. tasks

{
    "tag": "research.tasks",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "content": [    // list of tasks
        {
            "name": "...",
            "description": "...",
            "model_type": "...", // only exists in "Finance Model Implementation", "General Model Implementation", "Medical Model Implementation", or some tasks of "Data Science"
            "architecture": "...", // same as above
            "hyperparameters": "...", // same as above
        },
        {

        }
        //... same as above
    ]
}

evolving Copy Section

• 1 to 10 pairs of Messages (codes & feedbacks), each identified by an "evo_id" indicating the evolving round.
• In the Data Science scenario, each evolving round contains only one task, but the "codes" for that task may include multiple code files.
• In other scenarios, each evolving round may contain multiple tasks, but each task's "codes" will include only one code file.

1. codes

{
    "tag": "evolving.codes",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "evo_id": "0",
    "content": [ // list of task_name & codes
        {
            "evo_id": "0",
            "target_task_name": "task_1",
            "workspace": { // one or more codes
                "a.py": "...<python codes>",
                "b.py": "...<python codes>",
                //...
            }
        },
        {
            "evo_id": "0",
            "target_task_name": "task_2",
            "workspace": {
                "a.py": "...<python codes>",
                //...
            }
        }
        //... same as above
    ]
}

{
    "tag": "evolving.codes",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "evo_id": "1",
    "content": [
        //... same as above
    ]
}

2. feedbacks

{
    "tag": "evolving.feedbacks",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "evo_id": "0",
    "content": [ // list of feedbacks
        {
            "evo_id": "0",
            "final_decision": "True", // True or False
            "execution": "...",
            "code": "...",
            "return_checking": "..."
        },
        //... same as above
    ]
}

{
    "tag": "evolving.codes",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "evo_id": "1",
    "content": [
        //... same as above
    ]
}

feedback Copy Section

Each tag below appears only once per loop.

1. config (only exists in "Finance Data Building"/"Finance Data Building (Reports)"/"Finance Model Implementation")

{
    "tag": "feedback.config",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "content": {
        "config": "a markdown string",
    }
}

2. return_chart (only exists in "Finance Data Building"/"Finance Data Building (Reports)"/"Finance Model Implementation")

{
    "tag": "feedback.return_chart",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "content": {
        "chart_html": "chart html codes string",
    }
}

3. metric

{
    "tag": "feedback.metric",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "content": {
        "result": "{ \"<metric_name>\": <value>, ... }" // A JSON string containing metric names and their corresponding values.
    }
}

4. hypothesis_feedback

{
    "tag": "feedback.hypothesis_feedback",
    "timestamp": "<isoformat>",
    "loop_id": "1",
    "content": {
        "decision": "True",
        "reason": "...",
        "exception": "...",
        "observations": "...", // may not exists
        "hypothesis_evaluation": "...", // may not existsc
        "new_hypothesis": "...", // may not exists
    }
}

TODO

Session Copy Chapter

• How to continue.
• show & copy trace_id(name)?

Page Copy Chapter

1. remove Medical, add Finance Whole Pipeline

Overview (`rdagent/scenarios/kaggle/README.md`) Copy Section

Kaggle Crawler

Install chrome & chromedriver for Linux Copy Chapter

In one folder

# install chrome
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb
google-chrome --version

# install chromedriver
wget https://storage.googleapis.com/chrome-for-testing-public/<chrome-version>/linux64/chromedriver-linux64.zip
unzip chromedriver-linux64.zip
cd chromedriver-linux64
sudo mv chromedriver /usr/local/bin
sudo chmod +x /usr/local/bin/chromedriver

chromedriver --version

config Copy Chapter

1. authentication: ~/.kaggle/kaggle.json
2. Accept Rules in competition website. (Join Competition)

notebook crawler Copy Chapter

1. download_notebooks()
2. convert_notebooks_to_text()

Overview (`rdagent/utils/repo/README.md`) Copy Section

RepoAnalyzer

RepoAnalyzer is a Python utility for analyzing and summarizing the contents of a Python repository. It provides a high-level overview of the repository structure, including a tree-like representation of the directory structure and details about files, classes, and functions.

Features Copy Chapter

• Generate a tree-like structure of the repository
• Summarize an entire repository
• Adjust verbosity levels for summaries
• Extract content from specific files
• Analyze Python files for classes and functions

Usage Copy Chapter

Basic Usage Copy Section

from repo_utils import RepoAnalyzer

# Initialize the RepoAnalyzer with the path to your repository
repo_analyzer = RepoAnalyzer("/path/to/your/repo")

# Generate a summary of the repository
summary = repo_analyzer.summarize_repo()
print(summary)

# Extract content from specific files
highlighted_content = repo_analyzer.highlight(["file1.py", "file2.py"])
print(highlighted_content)

Adjusting Verbosity Levels Copy Section

You can adjust the verbosity of the summary using the following parameters:

• verbose_level: Controls the overall detail level of the summary
  • 0: Minimal (file names only)
  • 1: Default (file info, class names, function names)
  • 2+: Detailed (includes method details within classes)
• doc_str_level: Controls the inclusion of docstrings (0-2)
• sign_level: Controls the inclusion of function signatures (0-2)

Example:

detailed_summary = repo_analyzer.summarize_repo(verbose_level=2, doc_str_level=1, sign_level=1)
print(detailed_summary)

Example Output Copy Chapter

Repository Summary Copy Section

Workspace Summary for my_project
========================================

Repository Structure:
my_project/
├── main.py
├── utils/
│   ├── helper.py
│   └── config.py
├── models/
│   ├── model_a.py
│   └── model_b.py

This workspace contains 5 Python files.

File 1 of 5:
File: main.py
----------------------------------------
This file contains 1 class and 2 top-level functions.

Class: MainApp
  Description: Main application class for the project.
  This class has 3 methods.

Function: setup_logging
  Accepts parameters: log_level
  Purpose: Configure the logging for the application.

Function: main
  Purpose: Entry point of the application.

...

File Highlight Copy Section

highlighted_content = repo_analyzer.highlight(["main.py"])
print(highlighted_content["main.py"])

This will print the entire content of the main.py file.

Key Components Copy Chapter

RepoAnalyzer Class Copy Section

The main class that provides the functionality for analyzing repositories.

Methods:

• summarize_repo(verbose_level=1, doc_str_level=1, sign_level=1): Generates a comprehensive summary of the repository, including a tree-like structure.
• highlight(file_names): Extracts and returns the content of specified files.

Tree-like Structure Copy Section

The summary now includes a visual representation of the repository's directory structure, making it easier to understand the overall organization of the project.

Overview (`rdagent/app/benchmark/model/README.md`) Copy Section

Tasks

Task Extraction Copy Chapter

From paper to task.

# python rdagent/app/model_implementation/task_extraction.py
# It may based on rdagent/document_reader/document_reader.py
python rdagent/components/task_implementation/model_implementation/task_extraction.py ./PaperImpBench/raw_paper/

Complete workflow Copy Chapter

From paper to implementation

# Similar to
# rdagent/app/factor_extraction_and_implementation/factor_extract_and_implement.py

Paper benchmark Copy Chapter

# TODO: it does not work well now.
python rdagent/app/model_implementation/eval.py

TODO:

• Create reasonable benchmark
  • with uniform input
  • manually create task
• Create reasonable evaluation metrics

Evolving Copy Chapter

Overview (`rdagent/app/finetune/llm/README.md`) Copy Section

FT-Agent: Autonomous LLM Fine-Tuning

This directory contains the RD-Agent LLM fine-tuning scenario used by FT-Agent in the ICML 2026 paper FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents.

FT-Agent automates a benchmark-driven fine-tuning loop:

1. inspect the target benchmark and available raw datasets;
2. generate data processing code and LLaMA-Factory training configs;
3. run fail-fast validation before full training;
4. fine-tune the target model;
5. evaluate with OpenCompass and use feedback for the next iteration.

The implementation is research-oriented. A full run can download large datasets, build training/evaluation environments, call LLM APIs for data processing, and consume GPU hours.

Supported Benchmarks Copy Chapter

The scenario currently includes benchmark adapters for the FT-Dojo tasks and several related extensions.

Dataset registration lives in rdagent/scenarios/finetune/datasets/__init__.py. Benchmark adapters live in rdagent/scenarios/finetune/benchmark/data/adaptor.py.

Current dataset behavior: scenario startup prepares the registered dataset resources under FT_FILE_PATH. The first run can therefore be large and slow. Reusing the same FT_FILE_PATH lets later runs reuse already downloaded assets. The --dataset option constrains the agent's selected dataset after preparation; it does not change the current preparation step.

Prerequisites Copy Chapter

• Linux.
• Docker available to the current user without sudo, or a compatible conda setup.
• NVIDIA GPU for realistic fine-tuning runs.
• LLM API access for the RD-Agent planner and optional data processing calls.
• Hugging Face access for target models and datasets. Some datasets may require accepting upstream licenses or setting HF_TOKEN.

Install RD-Agent from source when using this scenario:

git clone https://github.com/microsoft/RD-Agent
cd RD-Agent
make dev

Minimal `.env` Copy Chapter

Create .env in the repository root. Adjust model names and API settings to your provider.

# LLM backend
BACKEND=rdagent.oai.backend.LiteLLMAPIBackend
CHAT_MODEL=gpt-4o
CHAT_TEMPERATURE=1
CHAT_STREAM=True
OPENAI_API_KEY=<your_api_key>
# OPENAI_API_BASE=<your_api_base_if_needed>

# Embedding model used by RD-Agent infrastructure
EMBEDDING_MODEL=text-embedding-3-small

# Fine-tuning workspace. Keep this stable to reuse downloaded models/datasets.
FT_FILE_PATH=/absolute/path/to/finetune_files

# Runtime environment: docker is the default path; conda is available for local setups.
FT_Coder_CoSTEER_env_type=docker

# Target task. You may also pass these through CLI arguments.
FT_TARGET_BENCHMARK=aime25
FT_BENCHMARK_DESCRIPTION="AIME 2025 math competition problems. Each answer is an integer from 0 to 999. Expected Output Format: put the final answer within \\boxed{}, for example \\boxed{42}."

# Target model and data-processing settings
FT_BASE_MODEL=Qwen/Qwen2.5-7B-Instruct
FT_UPPER_DATA_SIZE_LIMIT=2000
FT_API_MAX_WORKERS=8
FT_STRONG_MODELS='["gpt-4o"]'
FT_WEAK_MODELS='["gpt-4o-mini"]'

# Hugging Face token, if required by a model or dataset.
# HF_TOKEN=<your_hf_token>

FT_API_MAX_WORKERS defaults to 8 to avoid surprising rate-limit and cost spikes for public users. If you have a high-throughput internal endpoint, increase it explicitly in .env.

Single-Task Run Copy Chapter

Using the CLI entry point:

rdagent llm_finetune \
  --benchmark aime25 \
  --benchmark-description "AIME 2025 math competition problems. Each answer is an integer from 0 to 999. Expected Output Format: put the final answer within \\boxed{}, for example \\boxed{42}." \
  --base-model Qwen/Qwen2.5-7B-Instruct \
  --loop-n 3 \
  --timeout 12h

Equivalent direct Python entry point:

dotenv run -- python rdagent/app/finetune/llm/loop.py \
  --benchmark aime25 \
  --benchmark-description "AIME 2025 math competition problems. Each answer is an integer from 0 to 999. Expected Output Format: put the final answer within \\boxed{}, for example \\boxed{42}." \
  --base-model Qwen/Qwen2.5-7B-Instruct \
  --loop-n 3 \
  --timeout 12h

Useful arguments:

Batch Runs Copy Chapter

For multiple benchmark/model runs, use the job helper in this directory:

cp rdagent/app/finetune/llm/job/tasks.json.example rdagent/app/finetune/llm/job/tasks.json
cp .env rdagent/app/finetune/llm/job/.env
bash rdagent/app/finetune/llm/job/run_ft_job.sh rdagent/app/finetune/llm/job/tasks.json

The job runner reads benchmark descriptions from rdagent/app/finetune/llm/job/scenarios.json when a task does not provide benchmark_description directly.

Logs and UI Copy Chapter

Runs write RD-Agent traces under the configured log directory. For the Streamlit FT UI:

streamlit run rdagent/app/finetune/llm/ui/app.py

For the generic RD-Agent log UI:

rdagent ui --port 19899 --log-dir <your_log_folder>

Notes Copy Chapter

• The first run is expected to be slow because it may download models, datasets, LLaMA-Factory assets, and OpenCompass assets.
• Docker mode mounts models and datasets from FT_FILE_PATH into the training/evaluation environments.
• Generated training data must use Alpaca-style instruction, input, and output fields; the validator checks this before full training.
• Evaluation uses validation feedback for agent iteration and keeps the test split for final reporting/front-end display.

Overview (`rdagent/scenarios/data_science/example/README.md`) Copy Section

Detailed Explanation for Customized Data in R\&D-Agent Data Science Pipeline

R&D-Agent Data Science Pipeline supports automated R&D optimization for competitions hosted on the Kaggle platform, as well as custom user-defined datasets.

Specifically, you need to prepare files in a structure similar to the provided example. Here, we use the arf-12-hours-prediction-task dataset as an illustration.

arf-12-hours-prediction-task Introduction Copy Chapter

Acute Respiratory Failure (ARF) is a life-threatening condition that often develops rapidly in critically ill patients. Accurate early prediction of ARF is essential in Intensive Care Units (ICUs) to enable timely clinical interventions and effective resource allocation. In this task, you are required to build a machine learning model that predicts whether a patient will develop ARF within the next 12 hours, using multivariate clinical time-series data.

The dataset has been extracted from electronic health records (EHRs) and preprocessed through the FIDDLE pipeline, generating structured temporal features for each patient.

Example Folder Structure Copy Chapter

• source_data (required)

  • arf-12-hours-prediction-task (Task Name, required)

    • prepare.py Used for data preprocessing to split the raw data into: training data, test data, formatted submission file, and standard answer file.

  • playground-series-s4e9 (Task Name, required)

    • prepare.py (required): Used for data preprocessing to split the raw data into: training data, test data, formatted submission file, and standard answer file.

  NOTE: Due to the large size of the raw data, we do not show the raw data in this project, if you want to see the raw data, you can download the full dataset through the link at the bottom.

• arf-12-hours-prediction-task (Task Name)

  • description.md (required): A detailed description of the task, including sections such as Task Description, Objective, Data Description, Data usage Notes, Modeling, Evaluation and Submission Format.

  • sample.py (optional): A Python script to sample the dataset for debugging purposes. If not provided, a default sampling logic in R&D-Agent will be used. Refer to the create_debug_data function in rdagent/scenarios/data_science/debug/data.py.

• playground-series-s4e9 (Task Name)

  • description.md (required): A detailed description of the task, including sections such as Task Description, Goal, Evaluation, Data Description, and Submission Format.

• eval (optional)

  • arf-12-hours-prediction-task (Task Name, optional)

    • grade.py: Calculates the task score on the test dataset.
    • valid.py: Checks the validity of the generated submission.csv file.

  • playground-series-s4e9 (Task Name, optional)

    • grade.py: Calculates the task score on the test dataset.
    • valid.py: Checks the validity of the generated submission.csv file.

  NOTE: You don't need to create the eval folder if you are ignoring test set scores.

The complete dataset folder for arf-12-hours-prediction-task can be downloaded from here.

The raw dataset for arf-12-hours-prediction-task comes from PhysioNet. You can apply for an account at PhysioNet and then request access to the FIDDLE preprocessed data: FIDDLE Dataset.

The complete dataset folder for playground-series-s4e9 can be downloaded from here.

The raw dataset for playground-series-s4e9 comes from Kaggle. You can apply for an account at Kaggle and then request access to the competition dataset.

NOTE: For more information about the dataset, please refer to the documentation.

Overview (`rdagent/scenarios/finetune/datasets/README.md`) Copy Section

数据集管理模块

本模块管理 LLM Finetune 场景的数据集，通过 snapshot_download 下载完整的 HuggingFace 仓库。

设计目标 Copy Chapter

1. 简洁性: 下载完整的 HF 仓库，保留原始文件结构
2. 可扩展性: 支持可选的 post_download_fn 进行自定义处理（如删除测试集）

使用方法 Copy Chapter

from rdagent.scenarios.finetune.datasets import prepare, prepare_all, DATASETS

# 1. 查看已注册的数据集
print(DATASETS.keys())
# ['chemcot', 'panorama', 'deepscaler', 'financeiq']

# 2. 准备单个数据集（下载到本地）
path = prepare("chemcot")
# 下载至: datasets/chemcot/

# 3. 准备所有数据集
prepare_all()

数据集配置 Copy Chapter

每个数据集通过 DatasetConfig 配置：

@dataclass
class DatasetConfig:
    repo_id: str                                          # HuggingFace 仓库 ID
    post_download_fn: Optional[Callable[[str], None]]     # 下载后处理函数

已注册数据集 Copy Chapter

添加新数据集 Copy Chapter

在 __init__.py 的 DATASETS 字典中添加配置：

DATASETS["my-dataset"] = DatasetConfig(
    repo_id="organization/dataset-name",
    post_download_fn=my_cleanup_function,  # 可选
)

README 替换机制 Copy Chapter

重要: 下载数据集时，本地 README 会覆盖 HuggingFace 原始 README。

工作原理 Copy Section

# __init__.py 中的逻辑
custom_readme = Path(__file__).parent / name / "README.md"
if custom_readme.exists():
    shutil.copy(custom_readme, out_dir / "README.md")

1. 数据集下载完成后，检查 datasets/{name}/README.md 是否存在
2. 如果存在，用本地版本覆盖下载目录中的 README
3. 这样可以为每个数据集提供定制化的文档

目录结构 Copy Section

rdagent/scenarios/finetune/datasets/
├── __init__.py          # 主模块: prepare(), prepare_all(), DATASETS
├── README.md            # 本文档
├── chemcot/
│   └── README.md        # ChemCoT 数据集文档（会覆盖 HF 原版）
├── panorama/
│   └── README.md        # PANORAMA 数据集文档（会覆盖 HF 原版）
├── deepscaler/
│   └── README.md        # DeepScaleR 数据集文档（会覆盖 HF 原版）
└── financeiq/
    └── README.md        # FinanceIQ 数据集文档（会覆盖 HF 原版）

README 编写规范 Copy Chapter

为每个数据集编写 README 时，建议包含以下内容：

1. 基础信息（必需） Copy Section

# 数据集名称

简要描述 + 论文链接

**Repository**: [HuggingFace 链接]

## Overview

数据集规模、来源、用途的概述

2. 数据集规模（必需） Copy Section

## Dataset Scale

| 类别 | 子任务 | 样本数 |
|------|--------|--------|
| xxx | xxx | 1,234 |
| **Total** | **N subtasks** | **总数** |

3. 数据字段说明（必需） Copy Section

## Data Fields

| 字段 | 类型 | 描述 |
|------|------|------|
| `id` | string | 唯一标识符 |
| `query` | string | 问题/指令 |
| `answer` | string | 答案 |
| ... | ... | ... |

4. CoT 质量评估（关键） Copy Section

这是最重要的部分，直接告诉使用者数据是否可用、如何处理：

## CoT Quality Assessment

**IMPORTANT**: [数据质量的核心警告]

| Dimension | Value |
|-----------|-------|
| baseline_quality | low / medium / high / N/A |
| task_type | math / chemistry / legal / ... |
| polish_difficulty | low / medium / high |

**Baseline**: [详细说明]
- 如果有 CoT: 说明来源、验证方式、质量问题
- 如果没有 CoT: 明确标注 "NO CoT"，说明必须生成

5. Baseline 性能（推荐） Copy Section

## Baseline Performance

| Task | Best Model | Score |
|------|-----------|-------|
| xxx | GPT-4o | 85.2% |

6. 许可证（必需） Copy Section

## License

MIT / CC-BY-NC-4.0 / ...

示例参考 Copy Chapter

• DeepScaleR: deepscaler/README.md - 标杆示例，CoT Quality Assessment 写得最清晰
• ChemCoT: chemcot/README.md - 有 CoT 但需要精化的情况
• PANORAMA: panorama/README.md - 没有 CoT 的情况

注意事项 Copy Chapter

1. Token: 私有数据集需要设置 HF_TOKEN 环境变量
2. 缓存: HuggingFace hub 会自动缓存下载内容
3. 强制刷新: 使用 prepare(name, force=True) 重新下载
4. README 优先级: 本地 README 会覆盖 HuggingFace 原版，确保文档一致性

Overview (`rdagent/scenarios/kaggle/experiment/README.md`) Copy Section

Meta template

It is an example of how we organize the workspace of a competition.
We expect all the competitions to align with it so the knowledge in modules (model, feature) can transfer.

The generation process of the initial template is hoped to be conducted by LLM (however, it is based on human efforts currently).

Overview (`rdagent/scenarios/kaggle/knowledge_management/README.md`) Copy Section

Usage Copy Chapter

This folder implements a knowledge base using RAG based on Kaggle competitions.
It allows you to store Kaggle competition experiences into the knowledge base, as well as store experimental experiences from RD-Agent.

1. First, generate a knowledge base (in JSON format) by running the main function in extract_knowledge.py.
2. Then, create a vector base in vector_base.py and save it.
3. Finally, add the field KG_RAG_PATH="xxx.pkl" (the path to the saved vector base) in your .env file.

Overview (`rdagent/scenarios/rl/autorl_bench/README.md`) Copy Section

Agent² RL-Bench / AutoRL-Bench

Paper: Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

Project page: agent2-rlbench

Status: arXiv preprint; this directory contains the released benchmark implementation.

让大模型（如 GPT-5.2）自主驱动 RL 训练流程，提升小模型（如 Qwen2.5-7B）在各类 Benchmark 上的表现，并评测"大模型驱动 RL"的增益效果。

核心问题：给定一个 Benchmark 及其 baseline，大模型通过 Workflow 对小模型进行 RL 训练后，小模型的分数能否超过 baseline？

快速开始 Copy Chapter

1. 环境安装 Copy Section

# --- 1a. Clone 代码 ---
git clone [email protected]:microsoft/RD-Agent.git ~/RD-Agent
cd ~/RD-Agent

# --- 1b. 基础 conda 环境 ---
conda create -n autorl python=3.10 -y
conda activate autorl
pip install -e .

# 全局依赖（trl, vllm, torch, opencompass 等）
pip install -r rdagent/scenarios/rl/autorl_bench/requirements.txt

# --- 1c. 按需安装 benchmark 额外依赖 ---

# ALFWorld
pip install -r rdagent/scenarios/rl/autorl_bench/benchmarks/alfworld/requirements.txt

# GSM8K：无额外依赖

# HumanEval
git clone https://github.com/XianBW/human-eval.git ~/human-eval
cd ~/human-eval && pip install -e .
cd ~/RD-Agent

# WebShop（需要 Java 11+）
conda install -c conda-forge openjdk=11 faiss-cpu -y
pip install -r rdagent/scenarios/rl/autorl_bench/benchmarks/webshop/requirements.txt
python -c "from spacy.cli.download import download; download('en_core_web_sm')"

# AlpacaEval 2.0
pip install -r rdagent/scenarios/rl/autorl_bench/benchmarks/alpacaeval/requirements.txt

# --- 1d. OpenHands Agent（如需使用）---
git clone [email protected]:couragec/openhands-rl.git ~/openhands-rl
# OpenHands 需要独立 conda 环境（Python 3.12）
conda create -n openhands python=3.12 -y
conda run -n openhands pip install -r ~/openhands-rl/requirements.txt

2. 配置 `.env` Copy Section

cp .env.example .env  # 或手动创建

.env 中需要配置的关键项：

# LLM API（OpenHands Agent 必需）
OPENAI_API_KEY=your_api_key
OPENAI_API_BASE=https://your-api-endpoint/v1
CHAT_MODEL=gpt-5.2

# OpenHands 环境（可选，有默认值）
# CONDA_ENV_OPENHANDS=openhands      # 默认 openhands
# OPENHANDS_RL_ROOT=$HOME/openhands-rl  # 默认 ~/openhands-rl

# rl-smith benchmark 路径（可选，默认 ../rl-smith/benchmarks/）
# SMITH_BENCH_DIR=/path/to/rl-smith/benchmarks

3. 运行 Copy Section

cd /path/to/RD-Agent
conda activate autorl

# Example Agent（简单 GRPO 训练，验证流程）
python -m rdagent.scenarios.rl.autorl_bench.run \
    --agent example_agent --task gsm8k --model Qwen/Qwen2.5-1.5B --timeout 7200

# OpenHands Agent + GSM8K
python -m rdagent.scenarios.rl.autorl_bench.run \
    --agent openhands --task gsm8k --model Qwen/Qwen2.5-1.5B --timeout 41600

# OpenHands Agent + ALFWorld（首次运行自动下载 ~2GB 游戏数据）
python -m rdagent.scenarios.rl.autorl_bench.run \
    --agent openhands --task alfworld --model Qwen/Qwen2.5-1.5B-Instruct --timeout 41600

# Smith benchmark（来自 rl-smith，自动发现，无需手动注册）
python -m rdagent.scenarios.rl.autorl_bench.run \
    --agent openhands --task smith-bbh --model Qwen/Qwen2.5-1.5B --timeout 7200

# 后台运行（推荐）
nohup python -m rdagent.scenarios.rl.autorl_bench.run \
    --agent openhands --task alfworld --model Qwen/Qwen2.5-1.5B-Instruct \
    --timeout 41600 > /dev/null 2>&1 &

数据自动下载：首次运行某个 benchmark 时，run.py 会自动调用对应 data.py 下载训练数据，无需手动操作。

• GSM8K：从 HuggingFace 下载 (~5MB)
• HumanEval：从 HuggingFace 下载 (~164 条样本)
• ALFWorld：调用 alfworld-download 从 GitHub Releases 下载 (~2GB，含 json/pddl/tw-pddl/logic)

4. 查看结果 Copy Section

# 实时查看运行日志
tail -f workspace/alfworld/20260228T100000_openhands/agent.log

# 查看评分记录
cat workspace/alfworld/20260228T100000_openhands/scores.json

# 查看全局实验汇总
cat rdagent/scenarios/rl/autorl_bench/results.csv

# Web UI（Streamlit 面板）
streamlit run rdagent/scenarios/rl/autorl_bench/core/ui.py --server.port 8511

命令行参数 Copy Section

核心流程 Copy Chapter

run.py 启动
 │
 ├─ 1. 准备资源：下载模型（HuggingFace）+ 下载训练数据（各 benchmark 的 data.py）
 ├─ 2. 构建 workspace：创建隔离目录、软链接模型和数据
 ├─ 3. 挂载文件：description.md + instructions.md + benchmark 特有文件
 ├─ 4. 启动 Grading Server（Flask 评测服务）
 ├─ 5. 评测 baseline：用原始模型跑一次基准分（有缓存）
 ├─ 6. 运行 Agent：Agent 在 workspace 内训练 + 多次提交评测
 ├─ 7. 收集结果：从 Grading Server 获取所有提交记录
 └─ 8. 保存结果：追加到 results.json，更新全局 best

资源存储 Copy Section

模型和数据下载后统一存储在 git_ignore_folder/rl_files/（可通过 AUTORL_FILE_PATH 覆盖）：

git_ignore_folder/rl_files/
├── models/Qwen/Qwen2.5-1.5B/    # 模型权重（snapshot_download）
├── datasets/
│   ├── gsm8k/train.jsonl         # 训练数据（agent 可见）
│   └── alfworld/train → ...      # 训练游戏数据（agent 可见，评估数据不在这）
└── baseline_workspace/           # baseline 分数缓存
    └── gsm8k_Qwen_Qwen2.5-1.5B.json

Workspace（每次运行隔离） Copy Section

每次运行创建独立的 workspace 目录（workspace/<task>/<run_id>/），通过软链接挂载资源：

workspace/gsm8k/
├── 20260211T143000_openhands/        # 一次独立实验（agent 在时限内的完整生命周期）
│   ├── code/                         # Agent 代码区（所有自行编写的代码）
│   │   ├── train.py                  # 训练脚本
│   │   └── ...                       # 分析、处理等其他脚本
│   ├── output/                       # 模型输出（$OUTPUT_DIR）
│   │   ├── v1/                       # 第一版模型
│   │   └── v2/                       # 第二版模型（迭代优化）
│   ├── models/Qwen/Qwen2.5-1.5B →   # 软链接 → rl_files/models/...（只读）
│   ├── data →                        # 软链接 → rl_files/datasets/gsm8k/（只读）
│   ├── description.md →              # 软链接 → benchmarks/gsm8k/description.md
│   ├── instructions.md →             # 软链接 → core/instructions.md
│   ├── scores.json                   # 本次实验内所有提交的评分记录
│   └── grading_server.log            # Grading Server 日志
└── 20260211T160000_rdagent/          # 另一次独立实验
    └── ...

评测原则：每次实验（一次 run.py 调用）是一个独立的评测单元。
Agent 在 --timeout 时限内可以多次训练、多次提交，最终取本次实验内的最高分。
不同实验之间完全隔离，不存在跨实验的"全局最优"。

results.csv（实验日志） Copy Section

autorl_bench/results.csv 是纯日志记录，用于论文实验汇总，不参与评测逻辑：

run_id,timestamp,task,agent,base_model,baseline,best_score,improvement,submissions,duration_s,success,workspace
20260211T143000,2026-02-11 14:30:00,gsm8k,openhands,Qwen/Qwen2.5-1.5B,21.61,22.37,0.76,3,3600,True,workspace/gsm8k/...
20260211T160000,2026-02-11 16:00:00,gsm8k,rdagent,Qwen/Qwen2.5-1.5B,21.61,23.12,1.51,7,3600,True,workspace/gsm8k/...

每行记录一次独立实验的结果，方便对比不同 agent 在相同条件下的表现。

Agent 环境变量 Copy Chapter

Agent 启动时（start.sh）可用的环境变量：

Grading Server API Copy Section

/submit 响应：

{
  "submission_id": 3,
  "score": 65.0,
  "baseline_score": 45.0,
  "improvement": 20.0,
  "best": {"submission_id": 2, "score": 68.0},
  "total_submissions": 3
}

代码结构 Copy Chapter

autorl_bench/
├── run.py                    # 入口脚本
├── conf.py                   # 路径配置
│
├── core/                     # 【主干代码】
│   ├── evaluator.py          # BaseEvaluator 基类
│   ├── opencompass.py        # OpenCompassEvaluator（通用评测器）
│   ├── server.py             # Grading Server（Flask）
│   ├── utils.py              # 工具函数（下载、软链接、baseline）
│   └── instructions.md       # Agent 通用指导说明
│
├── benchmarks/               # 【Benchmark 扩展】
│   ├── __init__.py           # 注册表 BENCHMARKS（含 smith 自动发现）
│   ├── smith/                # rl-smith 自动发现适配
│   │   ├── __init__.py       #   discover_smith_benchmarks()
│   │   └── per_sample_eval.py#   逐条评测器
│   ├── gsm8k/
│   │   ├── data.py           # 数据下载（train split）
│   │   └── description.md
│   └── alfworld/
│       ├── data.py           # 数据下载（训练游戏数据）
│       ├── eval.py           # 自定义评测器
│       ├── requirements.txt  # 额外依赖（alfworld, textworld）
│       ├── description.md
│       └── react_prompts.json
│
├── agents/                   # 【Agent 扩展】
│   ├── registry.py           # 注册表（读 config.yaml）
│   ├── example_agent/        # 简单 GRPO 训练
│   ├── openhands/            # OpenHands SDK
│   └── rdagent/              # RD-Agent
│
└── workspace/                # [运行时] 工作区 + 结果

扩展指南 Copy Chapter

添加新 Benchmark Copy Section

方式一：通过 rl-smith 自动生成（推荐）

将 benchmark 定义放在 rl-smith/benchmarks/<name>/ 下，RD-Agent 启动时自动发现并注册为 smith-<name> 任务。

自动生成（给一个 GitHub URL，AI agent 自动生成全部文件）：

cd /path/to/rl-smith
python generate_benchmark.py https://github.com/suzgunmirac/BIG-Bench-Hard --name my_bbh

# 生成后直接可用
cd /path/to/RD-Agent
python -m rdagent.scenarios.rl.autorl_bench.run --task smith-my_bbh --agent openhands --model Qwen/Qwen2.5-1.5B

手动创建 rl-smith/benchmarks/<name>/ 目录，需要：

发现机制：discover_smith_benchmarks() 扫描 $SMITH_BENCH_DIR/*/config.yaml（默认 ../rl-smith/benchmarks/），按 eval_mode 选择评测器（per_sample → PerSampleEvaluator，opencompass → OpenCompassEvaluator）。

详见 rl-smith README。

方式二：在 RD-Agent 内手动注册

新建 benchmarks/new_task/ 目录，需要 3 个文件：

1. data.py — 数据下载（只给 agent 训练数据，评估数据自己管）

# benchmarks/new_task/data.py
from pathlib import Path
from loguru import logger

def download_train_data(target_dir: Path) -> None:
    """下载训练数据到 target_dir，agent 只能看到这里的内容"""
    # target_dir 会被软链接到 workspace/data
    ...

2. description.md — 任务描述（agent 可见）

3. 注册到 benchmarks/__init__.py

BENCHMARKS["new_task"] = BenchmarkConfig(
    id="new_task",
    evaluator_class="rdagent.scenarios.rl.autorl_bench.core.opencompass.OpenCompassEvaluator",
    data_module="rdagent.scenarios.rl.autorl_bench.benchmarks.new_task.data",
    description="新任务描述",
    eval_config={"dataset": "opencompass.configs.datasets.xxx"},
)

如果需要自定义评测逻辑（不用 OpenCompass），再加一个 eval.py：

# benchmarks/new_task/eval.py
from rdagent.scenarios.rl.autorl_bench.core import BaseEvaluator

class NewTaskEvaluator(BaseEvaluator):
    def __init__(self, config):
        self.config = config

    def run_eval(self, model_path: str, workspace_path: str, **kwargs) -> dict:
        return {"score": 85.0, "accuracy_summary": {...}}

添加新 Agent Copy Section

# agents/my_agent/config.yaml
name: "My Agent"
start: "start.sh"
env_vars:
  MY_PARAM: "value"

# agents/my_agent/start.sh
#!/bin/bash
# 在 code/ 下编写训练脚本，模型输出到 output/
python $WORKSPACE/code/train.py --model $MODEL_PATH --data $DATA_PATH --output $OUTPUT_DIR/v1
curl -X POST $GRADING_SERVER_URL/submit \
    -H "Content-Type: application/json" \
    -d '{"model_path": "'$OUTPUT_DIR'/v1"}'

Agent 通过 config.yaml 自动注册，无需修改代码。

Overview (`rdagent/app/finetune/llm/job/README.md`) Copy Section

FT Job Runner

run_ft_job.sh launches multiple FT-Agent runs in parallel under one job directory. It is a convenience helper for multi-GPU machines; for a single run, prefer the command in ../README.md.

Quick Start Copy Chapter

From the RD-Agent repository root:

# 1. Prepare the normal FT-Agent config first.
# See rdagent/app/finetune/llm/README.md for required values.
cp .env rdagent/app/finetune/llm/job/.env

# 2. Prepare task definitions.
cp rdagent/app/finetune/llm/job/tasks.json.example rdagent/app/finetune/llm/job/tasks.json

# 3. Run the job.
bash rdagent/app/finetune/llm/job/run_ft_job.sh rdagent/app/finetune/llm/job/tasks.json

The script opens one tmux window per task and writes logs under log/<job_id>/.

Requirements Copy Chapter

• jq
• tmux
• conda
• an RD-Agent conda environment, named rdagent by default

If your RD-Agent environment has a different name, set CONDA_ENV_NAME before running the script or add it to job/.env:

CONDA_ENV_NAME=my-rdagent-env bash rdagent/app/finetune/llm/job/run_ft_job.sh

The FT training/evaluation backend is still controlled by FT_Coder_CoSTEER_env_type in job/.env. In docker mode, the job runner skips conda readiness checks for the training and OpenCompass environments. In conda mode, it waits for the llm_finetune and opencompass environments after the first task initializes them.

Task Config Copy Chapter

tasks.json contains a list of independent runs:

{
  "tasks": [
    {
      "model": "Qwen/Qwen2.5-7B-Instruct",
      "benchmark": "aime25",
      "gpus": "0,1",
      "timeout": "12h"
    },
    {
      "model": "Qwen/Qwen2.5-7B-Instruct",
      "benchmark": "chemcotbench_mol_edit",
      "gpus": "2,3",
      "benchmark_description": "Molecule Editing - perform valid SMILES edits and return JSON: {\"output\": \"<valid SMILES>\"}."
    }
  ]
}

If benchmark_description is omitted, the runner looks it up in scenarios.json.

Monitoring Copy Chapter

tmux attach -t rdagent
tail -f log/<job_id>/*.log
streamlit run rdagent/app/finetune/llm/ui/app.py

In the Streamlit UI, select the generated job folder as the log source.

Overview (`rdagent/components/coder/data_science/raw_data_loader/README.md`) Copy Section

CoSTEER

• subworkspace使用主experiment_workspace RD-Agent/rdagent/scenarios/data_science/experiment/experiment.py

evolving_strategy ( implement_one_task() ) Copy Chapter

1. xxxTask (in exp.py)
   • spec
   • description

evaluator Copy Chapter

1. queried_knowledge部分 共用
2. eval_test脚本

Overview (`rdagent/scenarios/data_science/proposal/exp_gen/README.md`) Copy Section

Folder structure design

Concepts Copy Chapter

When we are optimizing solutions, we have the following strategies.

• draft: create a new solution
• idea: propose an idea to improve solutions
• merge: merge different solutions

Optimization is a long journey; we may switch between different strategies. These strategies are called routers.

• router: a meta strategy to route between different strategies. Router may have different implementations.

Tools:

• select: It provides features to be used by other strategies or steps.
  1. submit: before we end the optimization, we have to select the only one solution to submit.
  2. expand: we may have select one point to start the next expansion.

Suggest folder structure Copy Chapter

So the suggested folder structure is:

- router/
- idea/
  - samll_step(or refine?).py
  - normal.py
- draft/
- merge/
- select/
  - expand.py
  - submit.py

Overview (`rdagent/scenarios/finetune/datasets/bioprobench/README.md`) Copy Section

license: cc-by-4.0
configs:

• config_name: PQA
  data_files:
  - split: train
  path: PQA.json
  - split: test
  path: PQA_test.json
• config_name: ERR
  data_files:
  - split: train
  path: ERR.json
  - split: test
  path: ERR_test.json
• config_name: ORD
  data_files:
  - split: train
  path: ORD.json
  - split: test
  path: ORD_test.json
• config_name: GEN
  data_files:
  - split: train
  path: GEN.json
  - split: test
  path: GEN_test.json

BioProBench Dataset for LLM Fine-Tuning

BioProBench is a large-scale, multi-task benchmark focused on biological protocol understanding and reasoning for large language models (LLMs). It spans four fine-tuning tasks provided here: Protocol Question Answering (PQA), Step Ordering (ORD), Error Correction (ERR), and Protocol Generation (GEN).

This dataset is built on a raw corpus of ~27K biological protocols and provides over 550K structured instances across tasks, with a held-out test set of 1,000 examples per task. See the original benchmark for full details:

• Code: https://github.com/YuyangSunshine/bioprotocolbench/
• Dataset hub: https://huggingface.co/BioProBench
• License: CC BY 4.0

Data Files Copy Chapter

The JSON files for each task (train/test) are organized per task. If your fine-tuning pipeline expects local files, place them alongside this README or update paths accordingly.

• PQA: bioprobench/PQA.json, bioprobench/PQA_test.json
• ERR: bioprobench/ERR.json, bioprobench/ERR_test.json
• ORD: bioprobench/ORD.json, bioprobench/ORD_test.json
• GEN: bioprobench/GEN.json, bioprobench/GEN_test.json

Task Definitions and Fields Copy Chapter

PQA — Protocol Question Answering Copy Section

Multiple-choice QA over protocol content.

• Fields:
  • question: the question string
  • choices: list of candidate answers
  • answer: the correct answer
  • type: category of the question (e.g., parameter, reagent, operation)
  • id: unique identifier

ORD — Step Ordering Copy Section

Order protocol steps correctly (top-level or sub-step sequences).

• Fields:
  • question: prompt describing the step list and context/title
  • wrong_steps: list of steps in a shuffled or incorrect order
  • correct_steps: steps in the correct chronological order
  • type: sequence granularity (e.g., top, child)
  • id: unique identifier

ERR — Error Correction Copy Section

Detect and correct errors in protocol text with local context.

• Fields:
  • context: object with purpose, prior_step, next_step
  • corrupted_text: the erroneous text (may be null for correct cases)
  • corrected_text: corrected version of the text
  • is_correct: boolean indicating whether the provided text was already correct
  • type: category (e.g., parameter, reagent, operation, or correct)
  • error_description: brief rationale for the correction
  • id: unique identifier

GEN — Protocol Generation Copy Section

Generate concise, single-level, numbered protocol steps from prompts.

• Fields:
  • system_prompt: role/system instruction
  • instruction: formatting and style constraints
  • input: task description or query
  • output: list of numbered steps (flat 1., 2., 3. ...)
  • id: unique identifier
  • type: difficulty tag (e.g., easy)

Splits Copy Chapter

• Train: use the non-_test.json files per task.
• Test: each task provides a held-out set of 1,000 examples.

Training Data Guidelines (CRITICAL for Fine-tuning) Copy Chapter

ERR — Error Correction Copy Section

CRITICAL: Answer Semantics

The benchmark prompt says: "If you find anything wrong, answer False."

Important: The training data generation script MUST use this logic:

def gold_answer_from_is_correct(is_correct: bool) -> str:
    # True = step is correct, False = step has errors
    return ANSWER_TRUE if is_correct else ANSWER_FALSE

Do NOT invert this logic - the benchmark evaluator compares model output directly with is_correct field.

ORD — Step Ordering Copy Section

Output Format (CRITICAL)

• Answer MUST be a valid Python list: [0, 2, 1, 3]
• NOT space-separated: 0 2 1 3 (WRONG)
• NOT comma-separated without brackets: 0, 2, 1, 3 (WRONG)

Training Data Format

• Can include brief reasoning (1-2 sentences)
• Final answer MUST be in format: [ANSWER_START][list][ANSWER_END]
• Example: [ANSWER_START][2, 0, 1, 3][ANSWER_END]

GEN — Protocol Generation Copy Section

Output Format

• Step-by-step protocol wrapped in [ANSWER_START]...[ANSWER_END]
• CoT (Chain-of-Thought) format is acceptable for this task

Common Notes Copy Section

• All tasks support ... tags for CoT reasoning (evaluator will strip them)
• Answer MUST be wrapped in [ANSWER_START] and [ANSWER_END] tags

License Copy Chapter

• CC BY 4.0 — see https://creativecommons.org/licenses/by/4.0/

Notes Copy Chapter

• Tasks cover protocol QA, ordering, correction, and generation (REA is part of the broader benchmark but not included in the files above).
• Data spans diverse biological domains and repositories; see the original benchmark for details.

Overview (`rdagent/scenarios/finetune/datasets/chemcot/README.md`) Copy Section

language:

• en
  license: mit
  tags:
• chemistry
• chain-of-thought
• molecular-reasoning
  size_categories:
• 10K<n<100K
  task_categories:
• text-generation
• question-answering

ChemCoT Dataset

Chemical reasoning dataset with Chain-of-Thought annotations from ChemCoTBench.

Repository: OpenMol/ChemCoTDataset

Overview Copy Chapter

The ChemCoTDataset provides ~23K high-quality chain-of-thought samples for training chemical reasoning models. CoT annotations were distilled from state-of-the-art reasoning models (Gemini-2.5-pro, DeepSeek-R1, Claude-3.7-sonnet-thinking) and validated by 13 chemistry PhD candidates with >90% accuracy.

Dataset Scale Copy Section

Tasks Copy Chapter

1. Molecular Understanding (mol_und) Copy Section

Metrics: MAE for counting, Tanimoto similarity for scaffold extraction

2. Molecular Editing (mol_edit) Copy Section

Metrics: Pass@1 (validity and instruction matching)

3. Molecular Optimization (mol_opt) Copy Section

Metrics: Mean improvement rate, Success rate

4. Reaction Prediction (rxn) Copy Section

Metrics: Top-1 accuracy, Fingerprint similarity

Data Format Copy Chapter

CoT Quality Assessment Copy Chapter

IMPORTANT: Distilled CoT may require domain refinement.

Baseline: CoT distilled from Gemini-2.5-pro/DeepSeek-R1/Claude, validated by 13 chemistry PhD candidates (>90% accuracy). Paper notes: "distillation strategy falters in chemistry" - consider expert refinement for optimal results.

License Copy Chapter

MIT License

Overview (`rdagent/scenarios/finetune/datasets/deepscaler/README.md`) Copy Section

language:

• en
  size_categories:
• 10K<n<100K
  license: mit
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
    splits:
  • name: train
    num_examples: 40315

DeepScaleR Mathematical Reasoning Dataset

Dataset for DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL.

DeepScaleR-1.5B-Preview achieves 43.1% Pass@1 accuracy on AIME 2024, representing a 15% improvement over the base model (28.8%) and surpassing OpenAI's O1-Preview performance with just 1.5B parameters through distributed reinforcement learning.

Overview Copy Chapter

The DeepScaleR dataset is a carefully curated collection of approximately 40,000 unique mathematics problem-answer pairs designed for training mathematical reasoning models through reinforcement learning. This dataset covers a wide range of competition-level mathematics problems from high school to olympiad level, providing a robust foundation for scaling RL algorithms on reasoning tasks.

DeepScaleR demonstrates that sophisticated mathematical reasoning can be achieved through strategic data curation combined with iterative context length scaling (8K→16K→24K) using Group Relative Policy Optimization (GRPO).

Data Sources Copy Section

Our training dataset consists of problems compiled from prestigious mathematics competitions and curated datasets:

• AIME (American Invitational Mathematics Examination) problems (1984-2023)
• AMC (American Mathematics Competition) problems (prior to 2023)
• Omni-MATH dataset
• Still dataset

Data Fields Copy Section

The dataset contains three key fields:

• problem: The mathematical problem statement, formatted with LaTeX notation
• solution: Official solution to the problem, including LaTeX formatting and boxed final answers. If there is no solution, the solution field is an empty string
• answer: The final mathematical result/answer, usually extracted from the solution

CoT Quality Assessment Copy Chapter

IMPORTANT: Raw data must be polished before training.

Baseline: 82% empty solution, 18% too short (p50=373 tokens, summary-style). Need to generate exploratory CoT (For your reference, the length of a well-structured CoT is usually longer than 1/4 * the model max_position_embeddings tokens) for all samples.

License Copy Chapter

This dataset is released under the MIT License.

Overview (`rdagent/scenarios/finetune/datasets/financeiq/README.md`) Copy Section

language:

• zh
  license: cc-by-nc-sa-4.0
  tags:
• finance
• chinese
• multiple-choice
• professional-certification
  size_categories:
• 1K<n<10K
  task_categories:
• question-answering
• text-generation

FinanceIQ Dataset

Chinese financial professional certification exam questions covering 10 major financial domains.

Repository: LlamaFactory/FinanceIQ

Overview Copy Chapter

The FinanceIQ dataset is a comprehensive collection of approximately 6,179 multiple-choice questions from Chinese financial professional certification exams. It covers 10 distinct financial domains, providing a robust benchmark for evaluating financial reasoning capabilities in Chinese language models.

Dataset Scale Copy Section

Tasks Copy Chapter

Single-Choice Question Answering Copy Section

Task: Select the correct answer (A/B/C/D) from four options for each financial question.

Evaluation: LLM Judge (comparing model's answer selection with ground truth)

Metrics: Accuracy per category, Average accuracy

Data Format (CSV Fields) Copy Chapter

Example Data Copy Section

Question,A,B,C,D,Answer
关于生命价值理论的理解，以下哪一项表述是不正确的？,补偿生命经济价值可能受到的损失...,个人预期收入的货币价值...,任何触及个人收入能力的事件...,早逝、残疾、退休或失业可能导致...,B

Data Split Strategy Copy Chapter

The dataset uses an end-based split strategy:

• Test set: Takes from the END of each category (up to 50 samples per category, or 50% if fewer)
• Train set: Takes the remaining samples from the START

This ensures consistent train/test separation across all categories.

Category Distribution Analysis Copy Chapter

Important: Sample distribution is highly imbalanced:

Recommendation: When generating training data, ensure balanced sampling across categories, especially for Actuary (精算师) which has only 44 samples.

CoT Quality Assessment Copy Chapter

IMPORTANT: Raw data contains only Q&A pairs, no reasoning chains.

Baseline: Questions are multiple-choice format without explanations. You MUST generate CoT (chain-of-thought reasoning) for training samples to achieve good results.

Baseline Performance Copy Chapter

Note: The Actuary (精算师-金融数学) category is particularly challenging, with baseline accuracy around 27-36%.

License Copy Chapter

CC-BY-NC-SA-4.0 License

Overview (`rdagent/scenarios/finetune/datasets/panorama/README.md`) Copy Section

language:

• en
  license: cc-by-nc-4.0
  tags:
• patent
• legal
• retrieval
• classification
  size_categories:
• 100K<n<1M
  task_categories:
• text-classification
• question-answering

PANORAMA Dataset

Patent examination benchmark capturing decision trails and rationales from PANORAMA.

Repository: LG-AI-Research/PANORAMA

Tasks Copy Chapter

1. PAR4PC: Prior-Art Retrieval for Patent Claims Copy Section

Task: Multi-label classification - select relevant prior-art documents from 8 candidates.

Train samples: 54,028

Metrics: Exact Match Accuracy, Custom Score (partial credit)

2. PI4PC: Paragraph Identification for Patent Claims Copy Section

Task: Single-choice - identify the most relevant paragraph in a prior-art document.

Train samples: 64,210

Metrics: Exact Match Accuracy

3. NOC4PC: Novelty and Non-Obviousness Classification Copy Section

Task: Ternary classification - determine if a claim should be ALLOW, 102 rejection, or 103 rejection.

Train samples: 136,211

Metrics: Macro F1-score, Per-class Accuracy

Legal Background Copy Chapter

• 35 U.S.C. §102 (Novelty): Claim rejected if anticipated by a single prior art reference
• 35 U.S.C. §103 (Non-Obviousness): Claim rejected if obvious from combining prior art

Data Format (Parquet Fields) Copy Chapter

PAR4PC / PI4PC Format Copy Section

Note: PI4PC has an additional prior_art_specification field containing the relevant prior-art document text.

NOC4PC Format Copy Section

Important: Array fields (gold_answers, silver_answers, negative_answers) are numpy.ndarray type.
Use .tolist() to convert to Python list before processing.

Example Data Copy Section

{
    "application_number": 14281639,
    "claim_number": 1,
    "context": {
        "abstract": "In an endodontic procedure...",
        "claims": ["claim 1 text", "claim 2 text", ...],
        "title": "Method for irrigating root canals"
    },
    "options": {
        "A": {"abstract": "...", "claims": [...], "patent_id": "US1234567", "title": "..."},
        "B": {"abstract": "...", "claims": [...], "patent_id": "US2345678", "title": "..."},
        # ... G, H
    },
    "gold_answers": array(['G'], dtype=object),  # numpy.ndarray, use .tolist() -> ['G']
    "negative_answers": array(['A', 'B', 'C', 'D', 'E', 'F', 'H'], dtype=object)
}

CoT Quality Assessment Copy Chapter

IMPORTANT: This dataset does NOT contain CoT annotations.

Baseline: Raw data contains rejection reasons but NO step-by-step reasoning chains. Paper explicitly states "lacked ground-truth CoTs". You MUST generate CoT for all samples before training.

Baseline Performance (CoT Prompting) Copy Chapter

Citation Copy Chapter

@article{panorama2024,
  title={PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination},
  author={LG AI Research and KAIST},
  year={2024},
  url={https://huggingface.co/datasets/LG-AI-Research/PANORAMA}
}

License Copy Chapter

CC-BY-NC-4.0 License

Overview (`rdagent/scenarios/finetune/datasets/tableinstruct/README.md`) Copy Section

language:

• en
  size_categories:
• 1K<n<10K
  license: mit
  configs:
• config_name: test
  data_files:
  • split: test
    path: data/test-*
    splits:
  • name: test
    num_examples: 886
• config_name: train
  data_files:
  • split: train
    path: data/train-*
    splits:
  • name: train
    num_examples: ~10K

TableBench: Table Question Answering Dataset

Dataset for TableBench: A Comprehensive and Complex Benchmark for Table Question Answering.

TableBench is a comprehensive and complex benchmark designed to evaluate Table Question Answering (TableQA) capabilities, covering 18 question categories across 4 major categories with 886 carefully curated test cases.

Overview Copy Chapter

The TableBench dataset consists of two main components:

1. TableBench (Test): 886 high-quality test cases for evaluation across 4 major reasoning categories
2. TableInstruct (Train): Large-scale training dataset with diverse table QA examples

TableBench substantially pushes the boundaries of large language models in complex TableQA scenarios, aligning closely with the "Reasoning Complexity of Questions" dimension in real-world Table QA applications.

Task Categories Copy Section

The benchmark covers 4 major categories with 18 sub-tasks:

1. Fact Checking: Verify factual statements against table data

   • Simple fact verification, cross-table validation, temporal consistency

2. Numerical Reasoning: Mathematical computations and comparisons

   • Arithmetic operations, aggregations, comparative analysis

3. Data Analysis: Complex analytical reasoning

   • Impact analysis, correlation analysis, trend forecasting, statistical analysis

4. Visualization: Chart generation and interpretation

   • Bar charts, line charts, pie charts, scatter plots

Data Sources Copy Section

Test Data (TableBench):

• Repository: Multilingual-Multimodal-NLP/TableBench
• 886 carefully curated and verified test cases
• Enhanced version released April 2025 with error corrections

Train Data (TableInstruct):

• Repository: Multilingual-Multimodal-NLP/TableInstruct
• Large-scale instruction tuning dataset for table QA
• Diverse question types and reasoning patterns

Data Fields Copy Section

The dataset contains the following key fields:

The TableInstruct dataset contains the following fields:

• id (string): Unique identifier for each sample
• qtype (string): Major task category (4 values)
  • FactChecking, NumericalReasoning, DataAnalysis, Visualization
• qsubtype (string): Specific sub-task type (18 values)
  • Examples: Counting, Aggregation, Comparison, CorrelationAnalysis, etc.
• instruction (string): Complete instruction template with task guidelines
  • Contains the full prompt template defining how to approach the task
  • Includes role definition, guidelines, code format requirements
  • Typically 800-15,000 characters depending on instruction type
• instruction_type (string): Reasoning strategy type (4 values)
  • DP (Direct Prompting), TCoT (Textual Chain-of-Thought)
  • PoT (Program-of-Thought), SCoT (Structured Chain-of-Thought)
• table (string): Table data in JSON format
  • Structure: {"columns": [...], "data": [[...], [...], ...]}
• question (string): Specific question about the table
• response (string): Model's answer including reasoning process

TableBench Test Dataset Fields:

• question: The table question or task description
• table: The table data (JSON format)
• answer: The ground truth answer
• category: Major category
• subcategory: Specific sub-task type

Instruction Types and Reasoning Strategies Copy Section

Tablebench training data (TableInstruct) supports multiple instruction types content that define how the model approaches reasoning and generates answers. Understanding these types is crucial for dataset filtering and fine-tuning strategy selection.

Available Instruction Type Copy Section

1. Direct Prompting(DP)
Characteristics:

• Provides solutions directly without intermediate reasoning steps
• Simplest instruction format focused on immediate answer generation
• Best for straightforward fact-checking and simple queries
  Instruction Template Pattern：
  You are a table analyst. Your task is to answer questions based on the table content.
  Read the table below in JSON format: [TABLE]
  Question: [QUESTION]
  Answer directly.
  Response Format:
  [Direct Answer]

2. Textual Chain-of-Thought (TCoT)
Characteristics:

• LLMs incrementally derive intermediate steps through textual reasoning
• Natural language explanations for each reasoning step
• Suitable for complex reasoning requiring logical deduction

Instruction Template Pattern:
You are a table analyst. Your task is to answer questions based on the table content.
[Guidelines for step-by-step reasoning]
Think step by step
Show your reasoning process
Provide the final answer
*Response Format:
Let's analyze this step by step:
[First reasoning step]
[Second reasoning step]
...
Final Answer: [Answer]

3. Program-of-Thought (PoT)

Characteristics:

• Decomposes problems into executable Python code
• Separates computation from reasoning using programming
• Ideal for numerical reasoning and computational tasks
• Most common type in TableInstruct for analytical tasks

Instruction Template Pattern (actual from dataset):
You are a data analyst proficient in Python. Your task is to write executable Python
code to analyze the table and then answer questions.
[Guidelines]

1. Based on the question, write out your analytical approach, then write Python code
2. The code needs to be concise and easy to understand
3. Code blocks need to strictly start with
   '''
   import pandas as pd
   df = pd.read_csv('table.csv')
   ...
   print(f'Final Answer: {answer}')
   '''
   4.Your analysis must be based entirely on the above data
   5.Generate executable code with results using print function
   6.Ensure to load the table with: df = pd.read_csv('table.csv')

4. Symbolic Chain-of-Thought (SCoT)

Characteristics:

• A methodology that utilizes Python-based instructions to facilitate logical reasoning
• Combines symbolic reasoning with executable code verification
• Three primary steps repeated until a definitive conclusion is derived
• Distinguishes itself from PoT by emphasizing iterative analysis-generation-simulation cycles

Three-Step Process:

• STEP-1: Analyzing the available information to determine the next move
• STEP-2: Generating instructions using Python programming language commands
• STEP-3: Simulating the outcomes by executing the instructions and analyzing the results

Instruction Template Pattern:
You are a table analyst. Use symbolic reasoning with iterative Python commands.
Process:
STEP-1: Analyze available information to determine the next move
STEP-2: Generate Python programming language commands
STEP-3: Simulate outcomes by executing instructions and analyzing results
Repeat these three steps until reaching a definitive conclusion

Evaluation Metrics Copy Section

Different metrics are used based on task type:

CoT Quality Assessment Copy Chapter

IMPORTANT: Consider enhancing reasoning chains during training preparation.

Baseline: Training data (TableInstruct) contains reasoning examples, but test data focuses on final answers. For complex reasoning tasks (Data Analysis, Numerical Reasoning), generating detailed step-by-step CoT can significantly improve model performance.

Recommendation: For Data Analysis and Numerical Reasoning categories, expand reasoning chains to include:

• Table understanding and schema identification
• Step-by-step computation or logical reasoning
• Intermediate results and verification
• Final answer with confidence indicators

Example Copy Chapter

Fact Checking Copy Section

{
  "question": "Based on the table, verify if the statement is true: 'Company A had higher revenue than Company B in Q4 2023'",
  "table": "| Company | Q4 2023 Revenue |\n|---------|----------------|\n| A       | $2.5M          |\n| B       | $3.1M          |",
  "answer": "False",
  "category": "Fact Checking",
  "subcategory": "simple_fact_verification"
}

Numerical Reasoning Copy Section

{
  "question": "What is the total revenue across all quarters for Product X?",
  "table": "| Quarter | Product X Revenue |\n|---------|------------------|\n| Q1      | 150              |\n| Q2      | 200              |\n| Q3      | 175              |\n| Q4      | 225              |",
  "answer": "750",
  "category": "Numerical Reasoning",
  "subcategory": "aggregation"
}

Data Analysis Copy Section

{
  "question": "Analyze the correlation between marketing spend and sales growth. What is the correlation coefficient?",
  "table": "| Month | Marketing ($K) | Sales Growth (%) |\n|-------|----------------|------------------|\n| Jan   | 50             | 12               |\n| Feb   | 75             | 18               |\n| Mar   | 60             | 15               |",
  "answer": "0.95",
  "category": "Data Analysis",
  "subcategory": "correlation_analysis"
}

License Copy Chapter

This dataset is released under the MIT License.

Overview (`rdagent/scenarios/kaggle/tpl_ex/aerial-cactus-identification/README.md`) Copy Section

Motivation of the example

We use a runnable concrete example to demonstrate what the project should be like after being generated by a large language model.

Content example and the workflow

NOTE: the README.md itself is note generated by LLM. the content remains are generated by LLM.

Extra input information beyond the competition information Copy Chapter

[[../meta/spec.md]]

• TODO

Step0: Specification generation Copy Chapter

• Generate specification
  [[spec.md]]

  • TODO: perfect

• Generate loading data
  [[load_data.py]]

• Why do we merge this step together.

  • Successfully run load_data.py is a kind of verification of spec.md

Step1: write the feature engineering code Copy Chapter

• We can generate some file like [[feature.py]] that match the pattern feat.*\.py

Step2: Model training Copy Chapter

Step3: ensemble and decision Copy Chapter

• generate ens_and_decsion
  • why we generate score on ensemble phase
  • ensemble has following tasks which has great overlap
    • ensemble usually check the performance before ensemble
    • A additional step to record performance is easier.

Step4: Build workflow Copy Chapter

[[main.py]]

Overview (`rdagent/scenarios/qlib/experiment/factor_data_template/README.md`) Copy Section

How to read files.

For example, if you want to read filename.h5

import pandas as pd
df = pd.read_hdf("filename.h5", key="data")

NOTE: **key is always "data" for all hdf5 files **.

Here is a short description about the data

For different data, We have some basic knowledge for them

Daily price and volume data Copy Chapter

$open: open price of the stock on that day.
$close: close price of the stock on that day.
$high: high price of the stock on that day.
$low: low price of the stock on that day.
$volume: volume of the stock on that day.
$factor: factor value of the stock on that day.

Overview (`rdagent/scenarios/qlib/experiment/factor_template/README.md`) Copy Section

Overview (`rdagent/scenarios/qlib/experiment/model_template/README.md`) Copy Section

This folder is a template to be copied from for each model implementation & running process. Copy Chapter

Components: Dummy model.py, versatile conf.yaml, and a result reader.

Api Reference (`docs/api_reference.rst`) Copy Section

============= API Reference

Here you can find all RDAgent's interfaces.

RD Loop

Research Copy Chapter

.. automodule:: rdagent.core.proposal
:members:

Development (`docs/development.rst`) Copy Section

========================= For Development

If you want to try the latest version or contribute to RD-Agent. You can install it from the source and follow the commands in this page.

.. code-block:: bash

git clone https://github.com/microsoft/RD-Agent

🔧Prepare for development

• Set up the development environment.

  .. code-block:: bash

  text Copy

  make dev

• Run linting and checking.

  .. code-block:: bash

  text Copy

  make lint

• Some linting issues can be fixed automatically. We have added a command in the Makefile for easy use.

  .. code-block:: bash

  text Copy

  make auto-lint

Code Structure

.. code-block:: text

📂 src
➥ 📂 : avoid namespace conflict
  ➥ 📁 core
  ➥ 📁 components/A
  ➥ 📁 components/B
  ➥ 📁 components/C
  ➥ 📁 scenarios/X
  ➥ 📁 scenarios/Y
  ➥ 📂 app
➥ 📁 scripts

.. list-table::
:header-rows: 1

• • Folder Name
  • Description
• • 📁 core
  • The core framework of the system. All classes should be abstract and usually can't be used directly.
• • 📁 component/A
  • Useful components that can be used by others (e.g., scenarios). Many subclasses of core classes are located here.
• • 📁 scenarios/X
  • Concrete features for specific scenarios (usually built based on components or core). These modules are often unreusable across scenarios.
• • 📁 app
  • Applications for specific scenarios (usually built based on components or scenarios). Removing any of them does not affect the system's completeness or other scenarios.
• • 📁 scripts
  • Quick and dirty things. These are candidates for core, components, scenarios, and apps.

Conventions

File Naming Convention Copy Chapter

.. list-table::
:header-rows: 1

• • Name
  • Description
• • conf.py
  • The configuration for the module, app, and project.

..

Security boundaries for generated code and artifacts

RD-Agent executes generated code and processes files produced by external
tools. Treat filenames, archive members, subprocess output, competition names,
and environment names as untrusted input even when the surrounding workflow is
started by a trusted operator.

Workspace file operations Copy Chapter

Use :meth:FBWorkspace.inject_files <rdagent.core.experiment.FBWorkspace.inject_files>
and :meth:FBWorkspace.remove_files <rdagent.core.experiment.FBWorkspace.remove_files>
for files owned by an experiment workspace. These methods accept relative paths
inside workspace_path. Parent traversal, absolute paths, and paths that
escape through a symbolic link are rejected before a file is written or
deleted.

Do not construct an unchecked path from a generated filename and then call
write_text(), unlink(), or another filesystem operation directly. New
workspace APIs should apply the same resolved-path containment rule to every
write and delete branch.

Archive extraction Copy Chapter

Use rdagent.utils.archive.safe_extract_zip and
rdagent.utils.archive.safe_extract_tar for archives that may contain
externally supplied entries. The helpers reject:

• absolute and parent-traversal member paths;
• symbolic links and hard links;
• device nodes and other special file types; and
• archives containing more than 10,000 members by default.

Avoid ZipFile.extractall() and TarFile.extractall() in these paths. If a
workflow legitimately needs links, special files, or a larger archive, handle
that input in a separately reviewed trusted-data path rather than weakening the
shared helpers.

Commands and parsed process output Copy Chapter

Pass subprocess arguments as a list with shell=False whenever any argument
can vary. Conda environment names are limited to letters, digits, _, -,
and ., and Python versions must be numeric dotted versions. Kaggle
competition identifiers use the corresponding competition-slug validator.

Never use eval() to parse subprocess or container output. Score output must
be finite numeric JSON. Legacy Python dictionary-shaped training metrics may be
parsed with ast.literal_eval() only when JSON cannot be used.

Index (`docs/index.rst`) Copy Section

.. RDAgent documentation master file, created by
sphinx-quickstart on Mon Jul 15 04:27:50 2024.
You can adapt this file completely to your liking, but it should at least
contain the root toctree directive.

Welcome to RDAgent's documentation!

.. image:: _static/logo.png
:alt: RD-Agent Logo

.. toctree::
:maxdepth: 3
:caption: Doctree:

introduction
installation_and_configuration
scens/catalog
project_framework_introduction
ui
research/catalog
development
api_reference
policy

GitHub https://github.com/microsoft/RD-Agent

Indices and tables

• :ref:genindex
• :ref:modindex
• :ref:search

Installation And Configuration (`docs/installation_and_configuration.rst`) Copy Section

============================== Installation and Configuration

Installation

Install RDAgent: For different scenarios

• for purely users: please use pip install rdagent to install RDAgent
• for dev users: See development <development.html>_

Install Docker: RDAgent is designed for research and development, acting like a human researcher and developer. It can write and run code in various environments, primarily using Docker for code execution. This keeps the remaining dependencies simple. Users must ensure Docker is installed before attempting most scenarios. Please refer to the official 🐳Docker page <https://docs.docker.com/engine/install/>_ for installation instructions.
Ensure the current user can run Docker commands without using sudo. You can verify this by executing docker run hello-world.

LiteLLM Backend Configuration (Default)

.. note::
🔥 Attention: We now provide experimental support for DeepSeek models! You can use DeepSeek's official API for cost-effective and high-performance inference. See the configuration example below for DeepSeek setup.

Option 1: Unified API base for both models Copy Chapter

.. code-block:: Properties

# Set to any model supported by LiteLLM.
  CHAT_MODEL=gpt-4o 
  EMBEDDING_MODEL=text-embedding-3-small
  # Configure unified API base
  # The backend api_key fully follows the convention of litellm.
  OPENAI_API_BASE=<your_unified_api_base>
  OPENAI_API_KEY=<replace_with_your_openai_api_key>

Option 2: Separate API bases for Chat and Embedding models Copy Chapter

.. code-block:: Properties

# Set to any model supported by LiteLLM.
  
  # CHAT MODEL:
  CHAT_MODEL=gpt-4o 
  OPENAI_API_BASE=<your_chat_api_base>
  OPENAI_API_KEY=<replace_with_your_openai_api_key>

  # EMBEDDING MODEL:
  # TAKE siliconflow as an example, you can use other providers.
  # Note: embedding requires litellm_proxy prefix
  EMBEDDING_MODEL=litellm_proxy/BAAI/bge-large-en-v1.5
  LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
  LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1

Configuration Example: DeepSeek Setup Copy Chapter

Many users encounter configuration errors when setting up DeepSeek. Here's a complete working example:

.. code-block:: Properties

# CHAT MODEL: Using DeepSeek Official API
  CHAT_MODEL=deepseek/deepseek-chat 
  DEEPSEEK_API_KEY=<replace_with_your_deepseek_api_key>

  # EMBEDDING MODEL: Using SiliconFlow for embedding since DeepSeek has no embedding model.
  # Note: embedding requires litellm_proxy prefix
  EMBEDDING_MODEL=litellm_proxy/BAAI/bge-m3
  LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
  LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1

Necessary parameters include:

• CHAT_MODEL: The model name of the chat model.

• EMBEDDING_MODEL: The model name of the embedding model.

• OPENAI_API_BASE: The base URL of the API. If EMBEDDING_MODEL does not start with litellm_proxy/, this is used for both chat and embedding models; otherwise, it is used for CHAT_MODEL only.

Optional parameters (required if your embedding model is provided by a different provider than CHAT_MODEL):

• LITELLM_PROXY_API_KEY: The API key for the embedding model, required if EMBEDDING_MODEL starts with litellm_proxy/.

• LITELLM_PROXY_API_BASE: The base URL for the embedding model, required if EMBEDDING_MODEL starts with litellm_proxy/.

Note: If you are using an embedding model from a provider different from the chat model, remember to add the litellm_proxy/ prefix to the EMBEDDING_MODEL name.

The CHAT_MODEL and EMBEDDING_MODEL parameters will be passed into LiteLLM's completion function.

Therefore, when utilizing models provided by different providers, first review the interface configuration of LiteLLM. The model names must match those allowed by LiteLLM.

Additionally, you need to set up the the additional parameters for the respective model provider, and the parameter names must align with those required by LiteLLM.

For example, if you are using a DeepSeek model, you need to set as follows:

.. code-block:: Properties

# For some models LiteLLM requires a prefix to the model name.
  CHAT_MODEL=deepseek/deepseek-chat
  DEEPSEEK_API_KEY=<replace_with_your_deepseek_api_key>

Besides, when you are using reasoning models, the response might include the thought process. For this case, you need to set the following environment variable:

.. code-block:: Properties

REASONING_THINK_RM=True

For more details on LiteLLM requirements, refer to the official LiteLLM documentation <https://docs.litellm.ai/docs>_.

Configuration Example 2: Azure OpenAI Setup Copy Chapter

Here’s a sample configuration specifically for Azure OpenAI, based on the official LiteLLM documentation <https://docs.litellm.ai/docs>_:

If you're using Azure OpenAI, below is a working example using the Python SDK, following the LiteLLM Azure OpenAI documentation <https://docs.litellm.ai/docs/providers/azure/>_:

.. code-block:: Properties

from litellm import completion
  import os
  
  # Set Azure OpenAI environment variables
  os.environ["AZURE_API_KEY"] = "<your_azure_api_key>"
  os.environ["AZURE_API_BASE"] = "<your_azure_api_base>"
  os.environ["AZURE_API_VERSION"] = "<version>"
  
  # Make a request to your Azure deployment
  response = completion(
    "azure/<your_deployment_name>",
    messages = [{ "content": "Hello, how are you?", "role": "user" }]
  )

To align with the Python SDK example above, you can configure the CHAT_MODEL based on the response model setting and use the corresponding os.environ variables by writing them into your local .env file as follows:

.. code-block:: Properties

cat << EOF > .env
  # CHAT MODEL: Azure OpenAI via LiteLLM
  CHAT_MODEL=azure/<your_deployment_name>
  AZURE_API_BASE=https://<your_azure_base>.openai.azure.com/
  AZURE_API_KEY=<your_azure_api_key>
  AZURE_API_VERSION=<version>
  
  # EMBEDDING MODEL: Using SiliconFlow via litellm_proxy
  EMBEDDING_MODEL=litellm_proxy/BAAI/bge-large-en-v1.5
  LITELLM_PROXY_API_KEY=<your_siliconflow_api_key>
  LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1
  EOF

This configuration allows you to call Azure OpenAI through LiteLLM while using an external provider (e.g., SiliconFlow) for embeddings.

If your Azure OpenAI API Key`` supports embedding model`, you can refer to the following configuration example.

.. code-block:: Properties

cat << EOF  > .env
  EMBEDDING_MODEL=azure/<Model deployment supporting embedding>
  CHAT_MODEL=azure/<your deployment name>
  AZURE_API_KEY=<replace_with_your_openai_api_key>
  AZURE_API_BASE=<your_unified_api_base>
  AZURE_API_VERSION=<azure api version>

Execution Environment Configuration

Coder Environment Configuration (Docker vs. Conda)

RD-Agent's coders can execute code in different environments. You can control this behavior by setting environment variables in your .env file. This is useful for switching between a local Conda environment and an isolated Docker container.

To configure the environment, add the corresponding line to your .env file based on the scenario you are running.

For the Model (Quant) Scenario:

The execution environment is determined by the MODEL_COSTEER_ENV_TYPE variable, which is read from rdagent/components/coder/model_coder/conf.py.

• To use Docker (recommended for isolated execution):

  .. code-block:: properties

  MODEL_COSTEER_ENV_TYPE=docker

• To use Conda (for running in a local Conda environment):

  .. code-block:: properties

  MODEL_COSTEER_ENV_TYPE=conda

For the Data Science Scenario:

The execution environment is determined by the DS_CODER_COSTEER_ENV_TYPE variable, which is read from rdagent/components/coder/data_science/conf.py.

• To use Docker (recommended for isolated execution):

  .. code-block:: properties

  DS_CODER_COSTEER_ENV_TYPE=docker

• To use Conda (for running in a local Conda environment):

  .. code-block:: properties

  DS_CODER_COSTEER_ENV_TYPE=conda

Persisted Artifact Security

RD-Agent persists Python objects for sessions, logs, caches, summaries, and
knowledge bases. These objects use a signed serialization envelope. RD-Agent
verifies the signature before invoking Python pickle or dill deserialization,
and rejects unsigned or modified artifacts by default.

The signing key is created automatically on first use at
~/.rdagent/artifact_signing.key with owner-only permissions. Back up this
file if existing sessions and knowledge bases must remain readable. Losing or
rotating the key makes artifacts signed by the previous key unavailable.

The following environment variables control artifact verification:

.. list-table::
:header-rows: 1
:widths: 32 28 70

• • Environment variable
  • Default
  • Description
• • ARTIFACT_SIGNING_KEY
  • empty
  • Optional signing secret containing at least 32 bytes. Use the same value
    on trusted RD-Agent processes that share persisted artifacts. Prefer a
    secret manager instead of committing it to a repository or .env
    file.
• • ARTIFACT_SIGNING_KEY_PATH
  • ~/.rdagent/artifact_signing.key
  • Path used for the automatically generated local signing key when
    ARTIFACT_SIGNING_KEY is not configured.
• • ALLOW_UNSAFE_LEGACY_PICKLE
  • false
  • Allows unsigned legacy pickle and dill artifacts. Enable temporarily
    only when importing artifacts whose complete provenance is trusted.

Keep the signing key outside log, cache, upload, workspace, and other shared
artifact directories. Do not mount it into containers that execute generated
or otherwise untrusted code. File access to the signing key is equivalent to
permission to create artifacts trusted by the host process.

Shared filesystems and multiple hosts Copy Chapter

The automatically generated key is installation-local. If several trusted
hosts read and write the same sessions, logs, caches, or knowledge bases,
configure the same ARTIFACT_SIGNING_KEY through the deployment secret
manager on every host. Artifacts written with a different key fail signature
verification instead of being deserialized.

Legacy artifact migration Copy Chapter

Unsigned artifacts created by an earlier RD-Agent version are rejected by
default. Legacy cache entries are treated as cache misses and rebuilt without
executing their contents. For sessions, logs, summaries, and knowledge bases:

#. Verify the artifact source and restrict write access to its directory.
#. Back up both the artifact and the new signing key.
#. Temporarily set ALLOW_UNSAFE_LEGACY_PICKLE=true in an isolated trusted
environment.
#. Load the artifact and let the corresponding workflow save a new snapshot,
or export the needed data to a non-executable format.
#. Disable ALLOW_UNSAFE_LEGACY_PICKLE immediately after migration.

Never enable legacy loading for downloaded archives, shared directories, or
paths writable by another user or service. HMAC verification establishes that
an artifact was written by a holder of the signing key; it does not make an
arbitrary pickle intrinsically safe.

Container result formats Copy Chapter

Results crossing from generated/container code into the host process no longer
use pickle. Supported values are transported with JSON manifests and typed
files: Parquet for pandas tables, NPY with allow_pickle=False for numeric
arrays, and JSON or text for simple values. Unsupported Python object types are
rejected rather than deserialized on the host. Qlib tabular results and Kaggle
validation cache data are stored as Parquet.

Custom Time Segment Configuration (Train / Valid / Test)

RD-Agent now supports user-defined time segments for training, validation,
and testing (backtesting). Users can customize these segments via environment
variables in the .env file, depending on the scenario being executed.

This feature allows greater flexibility when running experiments on different
time ranges without modifying code or YAML configurations.

Fin-Factor Scenario Copy Chapter

When running the fin_factor scenario, you can configure the time segments
using the following environment variables. These variables are read by the
Factor-related PropSettings and directly affect the execution process.

Add the following entries to your .env file as needed:

.. code-block:: properties

QLIB_FACTOR_TRAIN_START=
QLIB_FACTOR_TRAIN_END=
QLIB_FACTOR_VALID_START=<valid start date, default is 2015-01-01>
QLIB_FACTOR_VALID_END=<valid end date, default is 2016-12-31>
QLIB_FACTOR_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_FACTOR_TEST_END=<test / backtest end date, default is 2020-12-31>

Fin-Model Scenario Copy Chapter

When running the fin_model scenario, the model training, validation, and
testing time segments can be configured independently via the following
environment variables:

.. code-block:: properties

QLIB_MODEL_TRAIN_START=
QLIB_MODEL_TRAIN_END=
QLIB_MODEL_VALID_START=<valid start date, default is 2015-01-01>
QLIB_MODEL_VALID_END=<valid end date, default is 2016-12-31>
QLIB_MODEL_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_MODEL_TEST_END=<test / backtest end date, default is 2020-12-31>

These settings are used during model training and evaluation and directly
impact the execution workflow.

Fin-Quant Scenario Copy Chapter

When running the fin_quant scenario, RD-Agent supports configuring time
segments for factor, model, and quant stages simultaneously.

Note: The QLIB_QUANT_* variables are only used for front-end UI display
purposes and do not affect the actual execution process.

You may configure the following variables in your .env file:

.. code-block:: properties

QLIB_FACTOR_TRAIN_START=
QLIB_FACTOR_TRAIN_END=
QLIB_FACTOR_VALID_START=<valid start date, default is 2015-01-01>
QLIB_FACTOR_VALID_END=<valid end date, default is 2016-12-31>
QLIB_FACTOR_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_FACTOR_TEST_END=<test / backtest end date, default is 2020-12-31>

QLIB_MODEL_TRAIN_START=
QLIB_MODEL_TRAIN_END=
QLIB_MODEL_VALID_START=<valid start date, default is 2015-01-01>
QLIB_MODEL_VALID_END=<valid end date, default is 2016-12-31>
QLIB_MODEL_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_MODEL_TEST_END=<test / backtest end date, default is 2020-12-31>

QLIB_QUANT_TRAIN_START=
QLIB_QUANT_TRAIN_END=
QLIB_QUANT_VALID_START=<valid start date, default is 2015-01-01>
QLIB_QUANT_VALID_END=<valid end date, default is 2016-12-31>
QLIB_QUANT_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_QUANT_TEST_END=<test / backtest end date, default is 2020-12-31>

This setup allows the front-end to display consistent segment information
across different stages while keeping execution logic unchanged.

Configuration(deprecated)

To run the application, please create a .env file in the root directory of the project and add environment variables according to your requirements.

If you are using this deprecated version, you should set BACKEND to rdagent.oai.backend.DeprecBackend.

.. code-block:: Properties

BACKEND=rdagent.oai.backend.DeprecBackend

Here are some other configuration options that you can use:

OpenAI API Copy Chapter

Here is a standard configuration for the user using the OpenAI API.

.. code-block:: Properties

OPENAI_API_KEY=<your_api_key>
  EMBEDDING_MODEL=text-embedding-3-small
  CHAT_MODEL=gpt-4-turbo

Azure OpenAI Copy Chapter

The following environment variables are standard configuration options for the user using the OpenAI API.

.. code-block:: Properties

USE_AZURE=True

  EMBEDDING_OPENAI_API_KEY=<replace_with_your_azure_openai_api_key>
  EMBEDDING_AZURE_API_BASE=  # The endpoint for the Azure OpenAI API.
  EMBEDDING_AZURE_API_VERSION=  # The version of the Azure OpenAI API.
  EMBEDDING_MODEL=text-embedding-3-small

  CHAT_OPENAI_API_KEY=<replace_with_your_azure_openai_api_key>
  CHAT_AZURE_API_BASE=  # The endpoint for the Azure OpenAI API.
  CHAT_AZURE_API_VERSION=  # The version of the Azure OpenAI API.
  CHAT_MODEL=  # The model name of the Azure OpenAI API.

Use Azure Token Provider Copy Chapter

If you are using the Azure token provider, you need to set the CHAT_USE_AZURE_TOKEN_PROVIDER and EMBEDDING_USE_AZURE_TOKEN_PROVIDER environment variable to True. then
use the environment variables provided in the Azure Configuration section _.

☁️ Azure Configuration

• Install Azure CLI:

  sh Copy

  curl -L https://aka.ms/InstallAzureCli | bash

• Log in to Azure:

  sh Copy

  az login --use-device-code

• exit and re-login to your environment (this step may not be necessary).

Configuration List Copy Chapter

.. TODO: use autodoc-pydantic .

• OpenAI API Setting

+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| Configuration Option | Meaning | Default Value |
+===================================+=================================================================+=========================+
| OPENAI_API_KEY | API key for both chat and embedding models | None |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| EMBEDDING_OPENAI_API_KEY | Use a different API key for embedding model | None |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| CHAT_OPENAI_API_KEY | Set to use a different API key for chat model | None |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| EMBEDDING_MODEL | Name of the embedding model | text-embedding-3-small |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| CHAT_MODEL | Name of the chat model | gpt-4-turbo |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| EMBEDDING_AZURE_API_BASE | Base URL for the Azure OpenAI API | None |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| EMBEDDING_AZURE_API_VERSION | Version of the Azure OpenAI API | None |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| CHAT_AZURE_API_BASE | Base URL for the Azure OpenAI API | None |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| CHAT_AZURE_API_VERSION | Version of the Azure OpenAI API | None |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| USE_AZURE | True if you are using Azure OpenAI | False |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| CHAT_USE_AZURE_TOKEN_PROVIDER | True if you are using an Azure Token Provider in chat model | False |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+
| EMBEDDING_USE_AZURE_TOKEN_PROVIDER| True if you are using an Azure Token Provider in embedding model| False |
+-----------------------------------+-----------------------------------------------------------------+-------------------------+

• Globol Setting

+-----------------------------+--------------------------------------------------+-------------------------+
| Configuration Option | Meaning | Default Value |
+=============================+==================================================+=========================+
| max_retry | Maximum number of times to retry | 10 |
+-----------------------------+--------------------------------------------------+-------------------------+
| retry_wait_seconds | Number of seconds to wait before retrying | 1 |
+-----------------------------+--------------------------------------------------+-------------------------+

• log_trace_path | Path to log trace file | None |
  +-----------------------------+--------------------------------------------------+-------------------------+
• log_llm_chat_content | Flag to indicate if chat content is logged | True |
  +-----------------------------+--------------------------------------------------+-------------------------+

• Cache Setting

.. TODO: update Meaning for caches

+------------------------------+--------------------------------------------------+-------------------------+
| Configuration Option | Meaning | Default Value |
+==============================+==================================================+=========================+
| dump_chat_cache | Flag to indicate if chat cache is dumped | False |
+------------------------------+--------------------------------------------------+-------------------------+
| dump_embedding_cache | Flag to indicate if embedding cache is dumped | False |
+------------------------------+--------------------------------------------------+-------------------------+
| use_chat_cache | Flag to indicate if chat cache is used | False |
+------------------------------+--------------------------------------------------+-------------------------+
| use_embedding_cache | Flag to indicate if embedding cache is used | False |
+------------------------------+--------------------------------------------------+-------------------------+
| prompt_cache_path | Path to prompt cache | ./prompt_cache.db |
+------------------------------+--------------------------------------------------+-------------------------+
| max_past_message_include | Maximum number of past messages to include | 10 |
+------------------------------+--------------------------------------------------+-------------------------+

Loading Configuration Copy Chapter

For users' convenience, we provide a CLI interface called rdagent, which automatically runs load_dotenv() to load environment variables from the .env file.
However, this feature is not enabled by default for other scripts. We recommend users load the environment with the following steps:

• ⚙️ Environment Configuration

  • Place the .env file in the same directory as the .env.example file.

    • The .env.example file contains the environment variables required for users using the OpenAI API (Please note that .env.example is an example file. .env is the one that will be finally used.)

  • Export each variable in the .env file:

    .. code-block:: sh

    text Copy

    export $(grep -v '^#' .env | xargs)

  • If you want to change the default environment variables, you can refer to the above configuration and edith the .env file.

Introduction (`docs/introduction.rst`) Copy Section

========================= Introduction

In modern industry, research and development (R&D) is crucial for the enhancement of industrial productivity, especially in the AI era, where the core aspects of R&D are mainly focused on data and models. We are committed to automate these high-value generic R&D processes through our open source R&D automation tool RDAgent, which let AI drive data-driven AI.

.. image:: _static/scen.png
:alt: Our focused scenario

Our RDAgent is designed to automate the most critical industrial R&D processes, focusing first on data-driven scenarios, to greatly boost the development productivity of models and data.

Methodologically, we propose an autonomous agent framework that consists of two key parts: (R)esearch stands for actively exploring by proposing new ideas, and (D)evelopment stands for realizing these ideas. The effectiveness of these two components will ultimately get feedbacks through practice, and both research and development capabilities can continuously learn and grow in the process.

For a quick start, visit our GitHub home page <https://github.com/microsoft/RD-Agent>_ ⚡. If you've already checked it out and want more details, please keep reading.

Policy (`docs/policy.rst`) Copy Section

====== Policy

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or
contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
Microsoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.

Project Framework Introduction (`docs/project_framework_introduction.rst`) Copy Section

=============================== Framework Design & Components

Framework & Components

.. NOTE: This depends on the correctness of c-v of github.

.. image:: _static/Framework-RDAgent.png
:alt: Components & Feature Level

The image above shows the overall framework of RDAgent.

In a data mining expert's daily research and development process, they propose a hypothesis (e.g., a model structure like RNN can capture patterns in time-series data), design experiments (e.g., finance data contains time-series and we can verify the hypothesis in this scenario), implement the experiment as code (e.g., Pytorch model structure), and then execute the code to get feedback (e.g., metrics, loss curve, etc.). The experts learn from the feedback and improve in the next iteration.

We have established a basic method framework that continuously proposes hypotheses, verifies them, and gets feedback from the real world. This is the first scientific research automation framework that supports linking with real-world verification.

.. image:: https://github.com/user-attachments/assets/60cc2712-c32a-4492-a137-8aec59cdc66e
:alt: Class Level Figure

The figure above shows the main classes and how they fit into the workflow for those interested in the detailed code.

.. Detailed Design
.. ===============

Ui (`docs/ui.rst`) Copy Section

============== User Interface

Introduction

RD-Agent will generate some logs during the R&D process. These logs are very useful for debugging and understanding the R&D process. However, just viewing the terminal log is not intuitive enough. RD-Agent provides a web app as UI to visualize the R&D process. You can easily view the R&D process and understand the R&D process better.

Streamlit UI

Start Web App Copy Chapter

In RD-Agent/ folder, run:

.. code-block:: bash

rdagent ui --port  --log-dir <log_dir like "log/"> [--debug]

This will start a web app on http://localhost:.

NOTE: The log_dir parameter is not required. You can manually enter the log_path in the web app. If you set the log_dir parameter, you can easily select a different log_path in the web app.

--debug is optional, it will show a "Single Step Run" button in sidebar and saved objects info in the web app.

Use Web App Copy Chapter

1. Open the sidebar.

.. TODO: update these

2. Select the scenario you want to show. There are some pre-defined scenarios:

   • Qlib Model
   • Qlib Factor
   • Data Mining
   • Model from Paper
   • Kaggle

3. Click the Config⚙️ button and input the log path (if you set the log_dir parameter, you can select a log_path in the dropdown list).

4. Click the buttons below Config⚙️ to show the scenario execution process. Buttons are:

   • All Loops: Show complete scenario execution process.
   • Next Loop: Show one success R&D Loop.
   • One Evolving: Show one evolving step of development part.
   • refresh logs: clear shown logs.

Flask Web UI

RD-Agent also provides a separate frontend in web/ backed by the Flask log
server started with rdagent server_ui. This UI provides real-time trace,
upload, process-control, and user-interaction APIs.

Build and start Copy Chapter

Install the frontend dependencies and build the static assets:

.. code-block:: bash

cd web
npm install
npm run build:flask
cd ..

The generated assets are served from ./git_ignore_folder/static by default.
Set UI_STATIC_PATH before starting the server to use another directory.

Start the server locally:

.. code-block:: bash

export UI_SERVER_AUTH_TOKEN='<a-long-random-token>'
rdagent server_ui --port 19899

Then open http://127.0.0.1:19899/?token=<a-long-random-token> once.
The server listens on localhost by default. Authentication is required even
on localhost because malicious websites can send requests to local services.
Generate a token with python -c 'import secrets; print(secrets.token_urlsafe(32))'.

Remote access and authentication Copy Chapter

To access the Flask Web UI remotely, explicitly select a non-local address and
configure a long, random authentication token:

.. code-block:: bash

export UI_SERVER_AUTH_TOKEN='<a-long-random-token>'
rdagent server_ui --port 19899 --host 0.0.0.0

The server refuses to start on any address, including localhost, unless
UI_SERVER_AUTH_TOKEN is set. Open the following URL once to establish an
authenticated browser session:

.. code-block:: text

http://<server-host>:19899/?token=<a-long-random-token>

The server redirects to / after storing the token in an HTTP-only,
same-site cookie. API clients can supply the same token without using a cookie:

.. code-block:: text

Authorization: Bearer <a-long-random-token>

Put remotely accessible deployments behind an HTTPS reverse proxy. Avoid
recording the initial token-bearing URL in proxy logs or sharing it through an
untrusted channel. When the server is started through the CLI, --host
controls the listening address; UI_SERVER_HOST is the corresponding default
when invoking the backend entry point directly.

CORS is disabled by default. If a browser frontend is hosted on another origin,
configure an explicit JSON allowlist:

.. code-block:: bash

export UI_CORS_ALLOWED_ORIGINS='["https://ui.example.com"]'

Origins must be exact HTTP(S) origins, including non-default ports. Wildcards
and URL paths are rejected. API requests with untrusted origins are rejected
before handlers run. Cookie-authenticated POSTs require a same-origin or
allowlisted Origin, or a trusted Referer if Origin is absent.
Missing provenance is rejected. Non-browser clients and internal trace
publishers must use Bearer authentication. Allowed cross-origin clients still
need authentication. If a reverse proxy changes the host or scheme, allowlist
the exact browser-facing origin; arbitrary forwarded headers are not trusted.

Authenticated operators can start tasks that generate and execute code. Run
workers in an isolated environment with least privilege, restricted network
egress, and no host credentials or Docker socket exposed to generated code.

Configuration Copy Chapter

The Flask Web UI supports the following environment variables:

.. list-table::
:header-rows: 1
:widths: 30 25 70

• • Environment variable
  • Default
  • Description
• • UI_STATIC_PATH
  • ./git_ignore_folder/static
  • Directory containing the built Web UI assets.
• • UI_TRACE_FOLDER
  • ./git_ignore_folder/traces
  • Directory containing generated trace data and process logs.
• • UI_UPLOAD_FOLDER
  • ./git_ignore_folder/uploads
  • Isolated directory for uploaded inputs. Mount, back up, and clean it
    separately from the trace directory.
• • UI_SERVER_HOST
  • 127.0.0.1
  • Default host used by the backend entry point. Use server_ui --host
    when starting the server through the CLI.
• • UI_SERVER_AUTH_TOKEN
  • empty
  • Bearer/cookie authentication token. Required for all bindings, including localhost.
• • UI_CORS_ALLOWED_ORIGINS
  • []
  • JSON list of allowed browser origins. CORS is disabled when empty.
• • UI_MAX_UPLOAD_MB
  • 20
  • Maximum size in MiB of the complete HTTP request, including all files and
    form data.
• • UI_LOAD_LEGACY_PICKLE_TRACES
  • false
  • Whether to deserialize persisted pickle traces at startup. Enable only
    for a fully trusted trace directory.

Upload and trace safety Copy Chapter

The /upload API rejects URL and server-path text inputs before creating
tasks. General Model Implementation requires exactly one uploaded report.
Download reports through a trusted process first, then upload the actual file.
This removes the arbitrary report-fetch API entry point; it does not sandbox
code generated by an authorized task.

Uploaded input files are stored outside UI_TRACE_FOLDER so they cannot be
discovered and deserialized as persisted traces. Uploads ending in .dill,
.pickle, .pkl, .py, .pyc, or .pyo are rejected. Workflows
using these formats as uploaded inputs must convert them to a non-executable
data format or provide them through another trusted mechanism.

Legacy pickle trace loading is disabled by default because deserializing a
pickle can execute code. After a restart, a historical trace may still appear
in the history list while its saved messages remain unloaded. To browse trusted
historical traces, opt in explicitly:

.. code-block:: bash

export UI_LOAD_LEGACY_PICKLE_TRACES=true
rdagent server_ui --port 19899

Only enable this setting when every file under UI_TRACE_FOLDER is trusted
and the directory is not writable by untrusted users or services.

Data-science trace share links no longer accept a URL-controlled log_folder.
A link can preserve the selected trace, but the recipient must configure or
select the corresponding log folder in the UI.

Static RD2bench.Json (`docs/_static/RD2bench.json`) Copy Section

{
"alpha053_15": {
"description": "Reversal class factor, negative delta of a ratio involving close, low, and high prices over 15 days.",
"formulation": "-1 times Deltaleft(frac{(text{close} - text{low}) - (text{high} - text{close})}{text{close} - text{low}}, 15right)",
"variables": {
"Delta(x, d)": "Change in 'x' over 'd' days.",
"text{close}": "Closing price of the stock.",
"text{low}": "Lowest price of the stock for the day.",
"text{high}": "Highest price of the stock for the day."
},
"Category": "Volume&Price",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha053\nnew_df['ratio'] = (new_df['$close'] - new_df['$low'] - (new_df['$high'] - new_df['$close'])) / (new_df['$close'] - new_df['$low'])\n# the change of ratio in new_df over the 15 days\nnew_df['result']=-new_df['ratio'].diff(15)\n# transfer the result to series\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
},
"liquidity_imbalance": {
"description": "liquidity_imbalance=std(minute trading liquidity_imbalance)/mean(minute trading liquidity_imbalance).",
"formulation": "liquidity_imbalance = frac{text{std}(text{minute trading liquidity_imbalance})}{text{mean}(text{minute liquidity_imbalance})}",
"variables": {
"std(minute liquidity_imbalance)": "Standard deviation of trading liquidity_imbalance for each minute of the trading day.",
"mean(minute liquidity_imbalance)": "Mean of trading liquidity_imbalance for each minute of the trading day.",
"liquidity_imbalance": "(bid_size-ask_size)/(bid_size+ask_size), we use something like bidV for the size"
},
"Category": "High-Frequency",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['liquidity_imbalance']=(sample_df['bidV']-sample_df['askV'])/(sample_df['bidV']+sample_df['askV'])\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['liquidity_imbalance']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\n# Calculate Z value for each instrument per day\nstats['liquidity_imbalance'] = stats['std'] / stats['mean']\n# Display the calculated Z values\nresult=stats['liquidity_imbalance']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"liquidity_imbalance_2": {
"description": "liquidity_imbalance=std(minute trading liquidity_imbalance)/mean(minute trading liquidity_imbalance).",
"formulation": "liquidity_imbalance = frac{text{std}(text{minute trading liquidity_imbalance})}{text{mean}(text{minute liquidity_imbalance})}",
"variables": {
"std(minute liquidity_imbalance)": "Standard deviation of trading liquidity_imbalance for each minute of the trading day.",
"mean(minute liquidity_imbalance)": "Mean of trading liquidity_imbalance for each minute of the trading day.",
"liquidity_imbalance": "(bid_size-ask_size)/2*(bid_size+ask_size), we use something like bidV for the size"
},
"Category": "High-Frequency",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['liquidity_imbalance']=(sample_df['bidV']-sample_df['askV'])/((sample_df['bidV']+sample_df['askV'])2)\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['liquidity_imbalance']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\n# Calculate Z value for each instrument per day\nstats['liquidity_imbalance'] = stats['std'] / stats['mean']\n# Display the calculated Z values\nresult=stats['liquidity_imbalance']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"liquidity_imbalance_3": {
"description": "liquidity_imbalance=std(minute trading liquidity_imbalance)/mean(minute trading liquidity_imbalance).",
"formulation": "liquidity_imbalance = frac{text{std}(text{minute trading liquidity_imbalance})}{text{mean}(text{minute liquidity_imbalance})}",
"variables": {
"std(minute liquidity_imbalance)": "Standard deviation of trading liquidity_imbalance for each minute of the trading day.",
"mean(minute liquidity_imbalance)": "Mean of trading liquidity_imbalance for each minute of the trading day.",
"liquidity_imbalance": "(bid_size-ask_size)/3(bid_size+ask_size), we use something like bidV for the size"
},
"Category": "High-Frequency",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['liquidity_imbalance']=(sample_df['bidV']-sample_df['askV'])/((sample_df['bidV']+sample_df['askV'])*3)\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['liquidity_imbalance']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\n# Calculate Z value for each instrument per day\nstats['liquidity_imbalance'] = stats['std'] / stats['mean']\n# Display the calculated Z values\nresult=stats['liquidity_imbalance']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"micro_price": {
"description": "micro_price=std(minute trading micro_price)/mean(minute trading micro_price).",
"formulation": "micro_price = frac{text{std}(text{minute trading micro_price})}{text{mean}(text{minute micro_price})}",
"variables": {
"std(minute micro_price)": "Standard deviation of trading micro_price for each minute of the trading day.",
"mean(minute micro_price)": "Mean of trading micro_price for each minute of the trading day.",
"micro_price": "((df['bid_price'] * df['ask_size']) + (df['ask_price'] * df['bid_size'])) / (df['bid_size'] + df['ask_size'])"
},
"Category": "High-Frequency",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['micro_price']=(sample_df['bid']*sample_df['askV']+sample_df['ask']sample_df['bidV'])/(sample_df['bidV']+sample_df['askV'])\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['micro_price']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\n# Calculate Z value for each instrument per day\nstats['micro_price'] = stats['std'] / stats['mean']\n# Display the calculated Z values\nresult=stats['micro_price']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"micro_price_2": {
"description": "micro_price_2=std(minute trading micro_price)/mean(minute trading micro_price).",
"formulation": "micro_price_2 = frac{text{std}(text{minute trading micro_price})}{text{mean}(text{minute micro_price})}",
"variables": {
"std(minute micro_price)": "Standard deviation of trading micro_price for each minute of the trading day.",
"mean(minute micro_price)": "Mean of trading micro_price for each minute of the trading day.",
"micro_price": "((df['bid_price'] * df['ask_size']) + (df['ask_price'] * df['bid_size'])) / 2(df['bid_size'] + df['ask_size']), we use something like bidV for the size"
},
"Category": "High-Frequency",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['micro_price']=(sample_df['bid']*sample_df['askV']+sample_df['ask']*sample_df['bidV'])/((sample_df['bidV']+sample_df['askV'])2)\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['micro_price']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\n# Calculate Z value for each instrument per day\nstats['micro_price'] = stats['std'] / stats['mean']\n# Display the calculated Z values\nresult=stats['micro_price']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"micro_price_3": {
"description": "micro_price_3=std(minute trading micro_price)/mean(minute trading micro_price).",
"formulation": "micro_price_3 = frac{text{std}(text{minute trading micro_price})}{text{mean}(text{minute micro_price})}",
"variables": {
"std(minute micro_price)": "Standard deviation of trading micro_price for each minute of the trading day.",
"mean(minute micro_price)": "Mean of trading micro_price for each minute of the trading day.",
"micro_price": "((df['bid_price'] * df['ask_size']) + (df['ask_price'] * df['bid_size'])) / 3(df['bid_size'] + df['ask_size']), we use something like bidV for the size"
},
"Category": "High-Frequency",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['micro_price']=(sample_df['bid']*sample_df['askV']+sample_df['ask']*sample_df['bidV'])/((sample_df['bidV']+sample_df['askV'])*3)\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['micro_price']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\n# Calculate Z value for each instrument per day\nstats['micro_price'] = stats['std'] / stats['mean']\n# Display the calculated Z values\nresult=stats['micro_price']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"mid_price": {
"description": "mid_price=std(minute trading mid_price)/mean(minute trading mid_price).",
"formulation": "mid_price = frac{text{std}(text{minute trading mid price})}{text{mean}(text{minute mid price})}",
"variables": {
"std(minute mid_price)": "Standard deviation of trading mid_price for each minute of the trading day.",
"mean(minute mid_price)": "Mean of trading mid_price for each minute of the trading day.",
"mid_price": "The average of the bid and ask prices."
},
"Category": "High-Frequency",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['mid_price']=(sample_df['bid']+sample_df['ask'])/2\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['mid_price']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\nstats['mid_price'] = stats['std'] / stats['mean']\nresult=stats['mid_price']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"mid_price_2": {
"description": "mid_price=std(minute trading mid_price)/mean(minute trading mid_price).",
"formulation": "mid_price = frac{text{std}(text{minute trading mid price})}{text{mean}(text{minute mid price})}",
"variables": {
"std(minute mid_price)": "Standard deviation of trading mid_price for each minute of the trading day.",
"mean(minute mid_price)": "Mean of trading mid_price for each minute of the trading day.",
"mid_price_2": "the average of the bid and ask prices plus the the average of the bid and ask size (bidV and askV)."
},
"Category": "High-Frequency",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['mid_price']=(sample_df['bid']+sample_df['ask'])/2+(sample_df['bidV']+sample_df['askV'])/2\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['mid_price']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\nstats['mid_price'] = stats['std'] / stats['mean']\nresult=stats['mid_price']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"mid_price_3": {
"description": "mid_price=std(minute trading mid_price)/mean(minute trading mid_price).",
"formulation": "mid_price = frac{text{std}(text{minute trading mid price})}{text{mean}(text{minute mid price})}",
"variables": {
"std(minute mid_price)": "Standard deviation of trading mid_price for each minute of the trading day.",
"mean(minute mid_price)": "Mean of trading mid_price for each minute of the trading day.",
"mid_price_3": "The coefficient of variation (CV) of the mid-price for each minute of the trading day, calculated as the standard deviation of the mid-price divided by the mean mid-price."
},
"Category": "High-Frequency",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_hf = pd.read_hdf('high_freq.h5')\nsample_df= data_hf.reset_index()\n# Convert 'datetime' column to datetime and extract date for grouping\nsample_df['date'] = sample_df['datetime'].dt.date\nsample_df['mid_price']=(sample_df['bid']+sample_df['ask'])/3\n# Group by instrument and date\ngrouped = sample_df.groupby(['date','instrument'])['mid_price']\n# Calculate mean and standard deviation of the volume for each group\nstats = grouped.agg(['mean', 'std'])\nstats['mid_price'] = stats['std'] / stats['mean']\nresult=stats['mid_price']\nresult.index.names = ['datetime','instrument']\n# result = result.swaplevel().sort_index()\nresult.to_hdf('result.h5', key='data')"
},
"PB_ROE": {
"description": "Constructed using the ranking difference between PB and ROE, with regression versions of PB and ROE replacing original PB and ROE to obtain reconstructed factor values.",
"formulation": "text{rank}(PB_t) - rank(ROE_t)",
"variables": {
"text{rank}(PB_t)": "Ranking of regression version PB on cross-section at time t.",
"text{rank}(ROE_t)": "Ranking of regression version single-quarter ROE on cross-section at time t."
},
"Category": "Fundamentals",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\ndata = data_f.reset_index()\n# Calculate the rank of PB and ROE\ndata['PB_rank'] = data.groupby('datetime')['B/P'].rank()\ndata['ROE_rank'] = data.groupby('datetime')['ROE'].rank()\n# Calculate the difference between the ranks\ndata['PB_ROE'] = data['PB_rank'] - data['ROE_rank']\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(data['PB_ROE']).set_index(data_f.index)\n# transfer the result to series\nresult=result['PB_ROE']\nresult.to_hdf('result.h5', key='data')"
},
"PB_ROE_2": {
"description": "Constructed using the ranking difference between PB/2 and ROE, with regression versions of PB and ROE replacing original PB and ROE to obtain reconstructed factor values.",
"formulation": "text{rank}(PB_t)/2 - rank(ROE_t)",
"variables": {
"text{rank}(PB_t)": "Ranking of regression version PB on cross-section at time t.",
"text{rank}(ROE_t)": "Ranking of regression version single-quarter ROE on cross-section at time t."
},
"Category": "Fundamentals",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\ndata = data_f.reset_index()\n# Calculate the rank of PB and ROE\ndata['PB_rank'] = data.groupby('datetime')['B/P'].rank()\ndata['ROE_rank'] = data.groupby('datetime')['ROE'].rank()\n# Calculate the difference between the ranks\ndata['PB_ROE'] = data['PB_rank']/2 - data['ROE_rank']\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(data['PB_ROE']).set_index(data_f.index)\n# transfer the result to series\nresult=result['PB_ROE']\nresult.to_hdf('result.h5', key='data')"
},
"PB_ROE_3": {
"description": "Constructed using the ranking difference between PB/3 and ROE, with regression versions of PB and ROE replacing original PB and ROE to obtain reconstructed factor values.",
"formulation": "text{rank}(PB_t)/3 - rank(ROE_t)",
"variables": {
"text{rank}(PB_t)": "Ranking of regression version PB on cross-section at time t.",
"text{rank}(ROE_t)": "Ranking of regression version single-quarter ROE on cross-section at time t."
},
"Category": "Fundamentals",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\ndata = data_f.reset_index()\n# Calculate the rank of PB and ROE\ndata['PB_rank'] = data.groupby('datetime')['B/P'].rank()\ndata['ROE_rank'] = data.groupby('datetime')['ROE'].rank()\n# Calculate the difference between the ranks\ndata['PB_ROE'] = data['PB_rank']/3 - data['ROE_rank']\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(data['PB_ROE']).set_index(data_f.index)\n# transfer the result to series\nresult=result['PB_ROE']\nresult.to_hdf('result.h5', key='data')"
},
"PB_ROE_movement": {
"description": "PB_ROE_movement=five day PB_ROE movement indicator(-1 and 1 or 0).",
"formulation": "PB_ROE_movement = 5_day_movement(PB_ROE), PB_ROE = text{rank}(PB_t) - rank(ROE_t)",
"variables": {
"PB_ROE": "the ranking difference between PB and ROE.",
"5_day_PB_ROE_movement": "1 if PB_ROE is higher than the PB_ROE 5 days ago, -1 if PB_ROE is lower than the PB_ROE 5 days ago, 0 if PB_ROE is the same as the PB_ROE 5 days ago.",
"text{rank}(PB_t)": "Ranking of regression version PB on cross-section at time t.",
"text{rank}(ROE_t)": "Ranking of regression version single-quarter ROE on cross-section at time t."
},
"Category": "Fundamentals",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\nsample_df = data_f.reset_index()\n# Calculate the rank of PB and ROE\nsample_df['PB_rank'] = sample_df.groupby('datetime')['B/P'].rank()\nsample_df['ROE_rank'] = sample_df.groupby('datetime')['ROE'].rank()\nsample_df['PB_ROE'] = sample_df['PB_rank'] - sample_df['ROE_rank']\n# Group by instrument and date\nsample_df['PB_ROE_movement'] = sample_df['PB_ROE'].diff(periods=5).apply(lambda x: 1 if x > 0 else (-1 if x < 0 else 0))\n#calculate the mid_price_movement ratio for each day\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(sample_df['PB_ROE_movement']).set_index(data_f.index)\n# transfer the result to series\nresult=result['PB_ROE_movement']\nresult.to_hdf('result.h5', key='data')"
},
"PB_ROE_movement_10": {
"description": "PB_ROE_movement=10 days PB_ROE movement indicator(-1 and 1 or 0).",
"formulation": "PB_ROE_movement = 10_day_movement(PB_ROE), PB_ROE = text{rank}(PB_t) - rank(ROE_t)",
"variables": {
"PB_ROE": "the ranking difference between PB and ROE.",
"10_day_PB_ROE_movement": "1 if PB_ROE is higher than the PB_ROE 10 days ago, -1 if PB_ROE is lower than the PB_ROE 10 days ago, 0 if PB_ROE is the same as the PB_ROE 10 days ago.",
"text{rank}(PB_t)": "Ranking of regression version PB on cross-section at time t.",
"text{rank}(ROE_t)": "Ranking of regression version single-quarter ROE on cross-section at time t."
},
"Category": "Fundamentals",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\nsample_df = data_f.reset_index()\n# Calculate the rank of PB and ROE\nsample_df['PB_rank'] = sample_df.groupby('datetime')['B/P'].rank()\nsample_df['ROE_rank'] = sample_df.groupby('datetime')['ROE'].rank()\nsample_df['PB_ROE'] = sample_df['PB_rank'] - sample_df['ROE_rank']\n# Group by instrument and date\nsample_df['PB_ROE_movement'] = sample_df['PB_ROE'].diff(periods=10).apply(lambda x: 1 if x > 0 else (-1 if x < 0 else 0))\n#calculate the mid_price_movement ratio for each day\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(sample_df['PB_ROE_movement']).set_index(data_f.index)\n# transfer the result to series\nresult=result['PB_ROE_movement']\nresult.to_hdf('result.h5', key='data')"
},
"PB_ROE_movement_20": {
"description": "PB_ROE_movement=20 days PB_ROE movement indicator(-1 and 1 or 0).",
"formulation": "PB_ROE_movement = 20_day_movement(PB_ROE), PB_ROE = text{rank}(PB_t) - rank(ROE_t)",
"variables": {
"PB_ROE": "the ranking difference between PB and ROE.",
"20_day_PB_ROE_movement": "1 if PB_ROE is higher than the PB_ROE 20 days ago, -1 if PB_ROE is lower than the PB_ROE 20 days ago, 0 if PB_ROE is the same as the PB_ROE 20 days ago.",
"text{rank}(PB_t)": "Ranking of regression version PB on cross-section at time t.",
"text{rank}(ROE_t)": "Ranking of regression version single-quarter ROE on cross-section at time t."
},
"Category": "Fundamentals",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\nsample_df = data_f.reset_index()\n# Calculate the rank of PB and ROE\nsample_df['PB_rank'] = sample_df.groupby('datetime')['B/P'].rank()\nsample_df['ROE_rank'] = sample_df.groupby('datetime')['ROE'].rank()\nsample_df['PB_ROE'] = sample_df['PB_rank'] - sample_df['ROE_rank']\n# Group by instrument and date\nsample_df['PB_ROE_movement'] = sample_df['PB_ROE'].diff(periods=20).apply(lambda x: 1 if x > 0 else (-1 if x < 0 else 0))\n#calculate the mid_price_movement ratio for each day\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(sample_df['PB_ROE_movement']).set_index(data_f.index)\n# transfer the result to series\nresult=result['PB_ROE_movement']\nresult.to_hdf('result.h5', key='data')"
},
"ROE_movement": {
"description": "ROE_movement=five day ROE movement indicator(-1 and 1 or 0).",
"formulation": "ROE_movement = 5_day_movement(ROE)",
"variables": {
"ROE": "ROE in fundamental statistics.",
"5_day_ROE_movement": "1 if ROE is higher than the ROE 5 days ago, -1 if ROE is lower than the ROE 5 days ago, 0 if ROE is the same as the ROE 5 days ago."
},
"Category": "Fundamentals",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\nsample_df = data_f.reset_index()\n# Group by instrument and date\nsample_df['ROE_movement'] = sample_df['ROE'].diff(periods=5).apply(lambda x: 1 if x > 0 else (-1 if x < 0 else 0))\n#calculate the mid_price_movement ratio for each day\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(sample_df['ROE_movement']).set_index(data_f.index)\n# transfer the result to series\nresult=result['ROE_movement']\nresult.to_hdf('result.h5', key='data')"
},
"ROE_movement_10": {
"description": "ROE_movement_10=ten day ROE movement indicator(-1 and 1 or 0).",
"formulation": "ROE_movement = 10_day_movement(ROE)",
"variables": {
"ROE": "ROE in fundamental statistics.",
"10_day_ROE_movement": "1 if ROE is higher than the ROE 10 days ago, -1 if ROE is lower than the ROE 10 days ago, 0 if ROE is the same as the ROE 10 days ago."
},
"Category": "Fundamentals",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\nsample_df = data_f.reset_index()\n# Group by instrument and date\nsample_df['ROE_movement'] = sample_df['ROE'].diff(periods=10).apply(lambda x: 1 if x > 0 else (-1 if x < 0 else 0))\n#calculate the mid_price_movement ratio for each day\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(sample_df['ROE_movement']).set_index(data_f.index)\n# transfer the result to series\nresult=result['ROE_movement']\nresult.to_hdf('result.h5', key='data')"
},
"ROE_movement_20": {
"description": "ROE_movement_20=20 day ROE movement indicator(-1 and 1 or 0).",
"formulation": "ROE_movement_20 = 20_day_movement(ROE)",
"variables": {
"ROE": "ROE in fundamental statistics.",
"20_day_ROE_movement": "1 if ROE is higher than the ROE 20 days ago, -1 if ROE is lower than the ROE 20 days ago, 0 if ROE is the same as the ROE 20 days ago."
},
"Category": "Fundamentals",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_f = pd.read_hdf('daily_f.h5')\nsample_df = data_f.reset_index()\n# Group by instrument and date\nsample_df['ROE_movement'] = sample_df['ROE'].diff(periods=20).apply(lambda x: 1 if x > 0 else (-1 if x < 0 else 0))\n#calculate the mid_price_movement ratio for each day\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(sample_df['ROE_movement']).set_index(data_f.index)\n# transfer the result to series\nresult=result['ROE_movement']\nresult.to_hdf('result.h5', key='data')"
},
"alpha_pv_diff": {
"description": "alpha_pv_diff is defined as the ratio of the difference between close prices 10 days change and open prices 10 days change to the sum of the highest minus lowest prices plus a small constant.",
"formulation": "frac{(text{close_diff10} - text{open_diff10})}{(text{high} - text{low} + 0.001)}",
"variables": {
"close": "Closing price of the stock",
"open": "Opening price of the stock",
"high": "Highest price of the stock during the day",
"low": "Lowest price of the stock during the day"
},
"Category": "Volume&Price",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha101\nnew_df['result'] = (new_df['$close'].diff(10) - new_df['$open'].diff(10)) / (new_df['$high'] - new_df['$low'] + 0.001)\n# keep the index of the original dataframe\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\n# transfer the result to series\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
},
"alpha_pv_diff_15": {
"description": "alpha_pv_diff is defined as the ratio of the difference between close prices 15 days change and open prices 15 days change to the sum of the highest minus lowest prices plus a small constant.",
"formulation": "frac{(text{close_diff15} - text{open_diff15})}{(text{high} - text{low} + 0.001)}",
"variables": {
"close": "Closing price of the stock",
"open": "Opening price of the stock",
"high": "Highest price of the stock during the day",
"low": "Lowest price of the stock during the day"
},
"Category": "Volume&Price",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha101\nnew_df['result'] = (new_df['$close'].diff(15) - new_df['$open'].diff(15)) / (new_df['$high'] - new_df['$low'] + 0.001)\n# keep the index of the original dataframe\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\n# transfer the result to series\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
},
"alpha_pv_diff_20": {
"description": "alpha_pv_diff is defined as the ratio of the difference between close prices 20 days change and open prices 20 days change to the sum of the highest minus lowest prices plus a small constant.",
"formulation": "frac{(text{close_diff20} - text{open_diff20})}{(text{high} - text{low} + 0.001)}",
"variables": {
"close": "Closing price of the stock",
"open": "Opening price of the stock",
"high": "Highest price of the stock during the day",
"low": "Lowest price of the stock during the day"
},
"Category": "Volume&Price",
"Difficulty": "Medium",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha101\nnew_df['result'] = (new_df['$close'].diff(20) - new_df['$open'].diff(20)) / (new_df['$high'] - new_df['$low'] + 0.001)\n# keep the index of the original dataframe\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\n# transfer the result to series\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
},
"alpha_pv_diff_pct": {
"description": "alpha_pv is defined as the ratio of the difference between close prices 10 days change and open prices 10 days change to the sum of the highest prices 10 days change ratio minus lowest prices 10 days change ratio plus a small constant.",
"formulation": "frac{(text{close_diff10} - text{open_diff10})}{(text{high_pct10} - text{low_pct10} + 0.001)}",
"variables": {
"close": "Closing price of the stock",
"open": "Opening price of the stock",
"high": "Highest price of the stock during the day",
"low": "Lowest price of the stock during the day"
},
"Category": "Volume&Price",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha101\nnew_df['result'] = (new_df['$close'].diff(10) - new_df['$open'].diff(10)) / (new_df['$high'].pct_change(10) - new_df['$low'].pct_change(10) + 0.001)\n# keep the index of the original dataframe\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\n# transfer the result to series\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
},
"alpha_pv_diff_pct_15": {
"description": "alpha_pv is defined as the ratio of the difference between close prices 15 days change and open prices 15 days change to the sum of the highest prices 10 days change ratio minus lowest prices 10 days change ratio plus a small constant.",
"formulation": "frac{(text{close_diff15} - text{open_diff15})}{(text{high_pct10} - text{low_pct10} + 0.001)}",
"variables": {
"close": "Closing price of the stock",
"open": "Opening price of the stock",
"high": "Highest price of the stock during the day",
"low": "Lowest price of the stock during the day"
},
"Category": "Volume&Price",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha101\nnew_df['result'] = (new_df['$close'].diff(15) - new_df['$open'].diff(15)) / (new_df['$high'].pct_change(10) - new_df['$low'].pct_change(10) + 0.001)\n# keep the index of the original dataframe\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\n# transfer the result to series\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
},
"alpha_pv_diff_pct_20": {
"description": "alpha_pv is defined as the ratio of the difference between close prices 20 days change and open prices 20 days change to the sum of the highest prices 10 days change ratio minus lowest prices 10 days change ratio plus a small constant.",
"formulation": "frac{(text{close_diff20} - text{open_diff20})}{(text{high_pct10} - text{low_pct10} + 0.001)}",
"variables": {
"close": "Closing price of the stock",
"open": "Opening price of the stock",
"high": "Highest price of the stock during the day",
"low": "Lowest price of the stock during the day"
},
"Category": "Volume&Price",
"Difficulty": "Hard",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha101\nnew_df['result'] = (new_df['$close'].diff(20) - new_df['$open'].diff(20)) / (new_df['$high'].pct_change(10) - new_df['$low'].pct_change(10) + 0.001)\n# keep the index of the original dataframe\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\n# transfer the result to series\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
},
"alpha053": {
"description": "Reversal class factor, negative delta of a ratio involving close, low, and high prices over 9 days.",
"formulation": "-1 times Deltaleft(frac{(text{close} - text{low}) - (text{high} - text{close})}{text{close} - text{low}}, 9right)",
"variables": {
"Delta(x, d)": "Change in 'x' over 'd' days.",
"text{close}": "Closing price of the stock.",
"text{low}": "Lowest price of the stock for the day.",
"text{high}": "Highest price of the stock for the day."
},
"Category": "Volume&Price",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha053\nnew_df['ratio'] = (new_df['$close'] - new_df['$low'] - (new_df['$high'] - new_df['$close'])) / (new_df['$close'] - new_df['$low'])\n# the change of ratio in new_df over the 9 days\nnew_df['result']=-new_df['ratio'].diff(9)\n# transfer the result to series\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
},
"alpha053_5": {
"description": "Reversal class factor, negative delta of a ratio involving close, low, and high prices over 5 days.",
"formulation": "-1 times Deltaleft(frac{(text{close} - text{low}) - (text{high} - text{close})}{text{close} - text{low}}, 5right)",
"variables": {
"Delta(x, d)": "Change in 'x' over 'd' days.",
"text{close}": "Closing price of the stock.",
"text{low}": "Lowest price of the stock for the day.",
"text{high}": "Highest price of the stock for the day."
},
"Category": "Volume&Price",
"Difficulty": "Easy",
"gt_code": "import pandas as pd\ndata_pv = pd.read_hdf('daily_pv.h5')\nnew_df= data_pv.reset_index()\n# Calculate Alpha053\nnew_df['ratio'] = (new_df['$close'] - new_df['$low'] - (new_df['$high'] - new_df['$close'])) / (new_df['$close'] - new_df['$low'])\n# the change of ratio in new_df over the 5 days\nnew_df['result']=-new_df['ratio'].diff(5)\n# transfer the result to series\nresult=pd.DataFrame(new_df['result']).set_index(data_pv.index)\nresult=result['result']\nresult.to_hdf('result.h5', key='data')"
}
}

Research Benchmark (`docs/research/benchmark.rst`) Copy Section

============================== Benchmark

Introduction

Benchmarking the capabilities of R&D is a crucial research problem in this area. We are continuously exploring methods to benchmark these capabilities. The current benchmarks are listed on this page.

Development Capability Benchmarking

Benchmarking is used to evaluate the effectiveness of factors with fixed data. It mainly includes the following steps:

1. :ref:read and prepare the eval_data <data>

2. :ref:declare the method to be tested and pass the arguments <config>

3. :ref:declare the eval method and pass the arguments <config>

4. :ref:run the eval <run>

5. :ref:save and show the result <show>

Configuration Copy Chapter

.. _config:

.. autopydantic_settings:: rdagent.components.benchmark.conf.BenchmarkSettings

Example
+++++++
.. _example:

The default value for bench_test_round is 10, which takes about 2 hours to run. To modify it from 10 to 2, adjust the environment variables in the .env file as shown below.

.. code-block:: Properties

BENCHMARK_BENCH_TEST_ROUND=2

Data Format Copy Chapter

.. _data:

The sample data in bench_data_path is a dictionary where each key represents a factor name. The value associated with each key is factor data containing the following information:

• description: A textual description of the factor.
• formulation: A LaTeX formula representing the model's formulation.
• variables: A dictionary of variables involved in the factor.
• Category: The category or classification of the factor.
• Difficulty: The difficulty level of implementing or understanding the factor.
• gt_code: A piece of code associated with the factor.

Here is an example of this data format:

.. literalinclude:: ../../rdagent/components/benchmark/example.json
:language: json

Ensure the data is placed in the FACTOR_COSTEER_SETTINGS.data_folder_debug. The data files should be in .h5 or .md format and must not be stored in any subfolders. LLM-Agents will review the file content and implement the tasks.

.. TODO: Add a script to automatically generate the data in the rdagent/app/quant_factor_benchmark/data folder.

Run Benchmark Copy Chapter

.. _run:

Start the benchmark after completing the :doc:../installation_and_configuration.

.. code-block:: Properties

dotenv run -- python rdagent/app/benchmark/factor/eval.py

Once completed, a pkl file will be generated, and its path will be printed on the last line of the console.

Show Result Copy Chapter

.. _show:

The analysis.py script reads data from the pkl file and converts it to an image. Modify the Python code in rdagent/app/quant_factor_benchmark/analysis.py to specify the path to the pkl file and the output path for the png file.

.. code-block:: Properties

dotenv run -- python rdagent/app/benchmark/factor/analysis.py <log/path to.pkl>

A png file will be saved to the designated path as shown below.

.. image:: ../_static/benchmark.png

Related Paper Copy Chapter

• Towards Data-Centric Automatic R&D <https://arxiv.org/abs/2404.11276>_:
  We have developed a comprehensive benchmark called RD2Bench to assess data and model R&D capabilities. This benchmark includes a series of tasks that outline the features or structures of models. These tasks are used to evaluate the ability of LLM-Agents to implement them.

.. code-block:: bibtex

@misc{chen2024datacentric,
    title={Towards Data-Centric Automatic R&D},
    author={Haotian Chen and Xinjie Shen and Zeqi Ye and Wenjun Feng and Haoxue Wang and Xiao Yang and Xu Yang and Weiqing Liu and Jiang Bian},
    year={2024},
    eprint={2404.11276},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

.. image:: https://github.com/user-attachments/assets/494f55d3-de9e-4e73-ba3d-a787e8f9e841

To replicate the benchmark detailed in the paper, please consult the factors listed in the following file: RD2bench.json <../_static/RD2bench.json>_.
Please note use only_correct_format=False when evaluating the results.

Research Catalog (`docs/research/catalog.rst`) Copy Section

=========== Research

To achieve the good effects and improve R&D capabilities, we face multiple challenges, the most important of which is the continuous evolution capability. Existing large language models (LLMs) find it difficult to continue growing their capabilities after training is completed. Moreover, the training process of LLMs focuses more on general knowledge, and the lack of depth in more specialized knowledge becomes an obstacle to solving professional R&D problems within the industry. This specialized knowledge needs to be learned and acquired from in-depth industry practice.

Our RD-Agent, on the other hand, can continuously acquire in-depth domain knowledge through deep exploration during the R&D phase, allowing its R&D capabilities to keep growing.

To address these key challenges and achieve industrial value, a series of research work needs to be completed.

.. list-table:: Research Areas and Descriptions
:header-rows: 1

• • Research Area
  • Description
• • :doc:Benchmark
  • Benchmark the R&D abilities
• • Research
  • Idea proposal: Explore new ideas or refine existing ones
• • :doc:Development <dev>
  • Ability to realize ideas: Implement and execute ideas

.. toctree::
:maxdepth: 1
:caption: Doctree:
:hidden:

benchmark
dev

Research Dev (`docs/research/dev.rst`) Copy Section

============================== Development

Related Paper Copy Chapter

• Collaborative Evolving Strategy for Automatic Data-Centric Development <https://arxiv.org/abs/2407.18690>_
  Co-STEER is a method to tackle data-centric development (AD2) tasks and highlight its main challenges, which need expert-like implementation (i.e., learning domain knowledge from practice) and task scheduling capability (e.g., starting with easier tasks for better overall efficiency), areas that previous work has largely overlooked. Our Co-STEER agent enhances its domain knowledge through our evolving strategy and improves both its scheduling and implementation skills by gathering and using domain-specific practical experience. With a better schedule, implementation becomes faster. At the same time, as implementation feedback becomes more detailed, scheduling accuracy improves. These two capabilities grow together through practical feedback, enabling a collaborative evolution process.

.. code-block:: bibtex

@misc{yang2024collaborative,
    title={Collaborative Evolving Strategy for Automatic Data-Centric Development},
    author={Xu Yang and Haotian Chen and Wenjun Feng and Haoxue Wang and Zeqi Ye and Xinjie Shen and Xiao Yang and Shizhao Sun and Weiqing Liu and Jiang Bian},
    year={2024},
    eprint={2407.18690},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

.. image:: https://github.com/user-attachments/assets/75d9769b-0edd-4caf-9d45-57d1e577054b
:alt: Collaborative Evolving Strategy for Automatic Data-Centric Development

Scens Catalog (`docs/scens/catalog.rst`) Copy Section

========================= Scenarios

Scenario lists

In the two key areas of data-driven scenarios, model implementation and data building, our system aims to serve two main roles: 🦾copilot and 🤖agent.

• The 🦾copilot follows human instructions to automate repetitive tasks.
• The 🤖agent, being more autonomous, actively proposes ideas for better results in the future.

The supported scenarios are listed below:

.. list-table::
:header-rows: 1

* - Scenario/Target
  - Model Implementation
  - Data Building
* - 💹 Finance
  - :ref:`🥇The First Data-Centric Quant Multi-Agent Framework <quant_agent_fin>`
  - :ref:`🤖Iteratively Proposing Ideas & Evolving <model_agent_fin>`  
    
    :ref:`🦾Auto reports reading & implementation <data_copilot_fin>`  
    
    :ref:`🤖Iteratively Proposing Ideas & Evolving <data_agent_fin>`
* - 🏭 General
  - :ref:`🦾Auto paper reading & implementation <model_copilot_general>`  
    
    :ref:`🧪FT-Agent for LLM fine-tuning <finetune_agent>`
  - :ref:`🤖 Data Science <data_science_agent>`

.. toctree::
:maxdepth: 1
:caption: Doctree:
:hidden:

quant_agent_fin
data_agent_fin
data_copilot_fin
model_agent_fin
model_copilot_general
data_science
finetune

Scens Data Agent Fin (`docs/scens/data_agent_fin.rst`) Copy Section

.. _data_agent_fin:

===================== Finance Data Agent

**🤖 Automated Quantitative Trading & Iterative Factors Evolution** Copy Chapter

📖 Background

In the dynamic world of quantitative trading, **factors** serve as the strategic tools that enable traders to exploit market inefficiencies. 
These factors—ranging from simple metrics like price-to-earnings ratios to complex models like discounted cash flows—are the key to predicting stock prices with a high degree of accuracy.

By leveraging these factors, quantitative traders can develop sophisticated strategies that not only identify market patterns but also significantly enhance trading efficiency and precision. 
The ability to systematically analyze and apply these factors is what separates ordinary trading from truly strategic market outmaneuvering.
And this is where the **Finance Model Agent** comes into play.

🎥 `Demo <https://rdagent.azurewebsites.net/factor_loop>`_

.. raw:: html

<video width="600" controls>
    
    Your browser does not support the video tag.
  </video>

🌟 Introduction

In this scenario, our agent illustrates the iterative process of hypothesis generation, knowledge construction, and decision-making. 

It highlights how financial factors evolve through continuous feedback and refinement. 

Here's an enhanced outline of the steps:

**Step 1 : Hypothesis Generation 🔍**

- Generate and propose initial hypotheses based on previous experiment analysis and domain expertise, with thorough reasoning and financial justification.

**Step 2 : Factor Creation ✨**

- Based on the hypothesis, divide the tasks.
- Each task involves developing, defining, and implementing a new financial factor, including its name, description, formulation, and variables.

**Step 3 : Factor Implementation 👨‍💻**

- Implement the factor code based on the description, evolving it as a developer would.
- Quantitatively validate the newly created factors.

**Step 4 : Backtesting with Qlib 📉**

- Integrate the full dataset into the factor implementation code and prepare the factor library.
- Conduct backtesting using the Alpha158 plus newly developed factors and LGBModel in Qlib to evaluate the new factors' effectiveness and performance.

+----------------+------------+----------------+----------------------------------------------------+
| Dataset        | Model      | Factors        | Data Split                                         |
+================+============+================+====================================================+
| CSI300         | LGBModel   | Alpha158 Plus  | +-----------+--------------------------+           |
|                |            |                | | Train     | 2008-01-01 to 2014-12-31 |           |
|                |            |                | +-----------+--------------------------+           |
|                |            |                | | Valid     | 2015-01-01 to 2016-12-31 |           |
|                |            |                | +-----------+--------------------------+           |
|                |            |                | | Test      | 2017-01-01 to 2020-08-01 |           |
|                |            |                | +-----------+--------------------------+           |
+----------------+------------+----------------+----------------------------------------------------+


**Step 5 : Feedback Analysis 🔍**

- Analyze backtest results to assess performance.
- Incorporate feedback to refine hypotheses and improve the model.

**Step 6 :Hypothesis Refinement ♻️**

- Refine hypotheses based on feedback from backtesting.
- Repeat the process to continuously improve the model.

⚡ Quick Start

Please refer to the installation part in :doc:../installation_and_configuration to prepare your system dependency.

You can try our demo by running the following command:

• 🐍 Create a Conda Environment

  • Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI):

    .. code-block:: sh

    text Copy

    conda create -n rdagent python=3.10

  • Activate the environment:

    .. code-block:: sh

    text Copy

    conda activate rdagent

• 📦 Install the RDAgent

  • You can install the RDAgent package from PyPI:

    .. code-block:: sh

    text Copy

    pip install rdagent

• 🚀 Run the Application

  • You can directly run the application by using the following command:

    .. code-block:: sh

    text Copy

    rdagent fin_factor

🛠️ Usage of modules

.. _Env Config: 

- **Env Config**

The following environment variables can be set in the `.env` file to customize the application's behavior:

.. autopydantic_settings:: rdagent.app.qlib_rd_loop.conf.FactorBasePropSetting
    :settings-show-field-summary: False
    :exclude-members: Config

.. autopydantic_settings:: rdagent.components.coder.factor_coder.config.FactorCoSTEERSettings
    :settings-show-field-summary: False
    :members: coder_use_cache, data_folder, data_folder_debug, file_based_execution_timeout, select_method, max_loop, knowledge_base_path, new_knowledge_base_path
    :exclude-members: Config, fail_task_trial_limit, v1_query_former_trace_limit, v1_query_similar_success_limit, v2_query_component_limit, v2_query_error_limit, v2_query_former_trace_limit, v2_error_summary, v2_knowledge_sampler
    :no-index:

---

### Scens Data Copilot Fin (`docs/scens/data_copilot_fin.rst`)

.. _data_copilot_fin:

=====================
Finance Data Copilot
=====================


**🤖 Automated Quantitative Trading & Factors Extraction from Financial Reports**
---------------------------------------------------------------------------------

📖 Background
~~~~~~~~~~~~~~
**Research reports** are treasure troves of insights, often unveiling potential **factors** that can drive successful quantitative trading strategies. 
Yet, with the sheer volume of reports available, extracting the most valuable insights efficiently becomes a daunting task.

Furthermore, rather than hastily replicating factors from a report, it's essential to delve into the underlying logic of their construction. 
Does the factor capture the essential market dynamics? How unique is it compared to the factors already in your library?

Therefore, there is an urgent need for a systematic approach to design a framework that can effectively manage this process. 
And this is where the **Finance Data Copilot** steps in.


🎥 `Demo <https://rdagent.azurewebsites.net/report_factor>`_

.. raw:: html

<div style="display: flex; justify-content: center; align-items: center;">
  <video width="600" controls>
    <source src="https://rdagent.azurewebsites.net/media/7b14b2bd3d8771da9cf7eb799b6d96729cec3d35c8d4f68060f3e2fd.mp4" type="video/mp4">
    Your browser does not support the video tag.
  </video>
</div>

🌟 Introduction

In this scenario, RDAgent demonstrates the process of extracting factors from financial research reports, implementing these factors, and analyzing their performance through Qlib backtesting. 
This process continually expands and refines the factor library.

Here's an enhanced outline of the steps:

**Step 1 : Hypothesis Generation 🔍**

- Generate and propose initial hypotheses based on insights from financial reports with thorough reasoning and financial justification.

**Step 2 : Factor Creation ✨**

- Based on the hypothesis and financial reports, divide the tasks. 
- Each task involves developing, defining, and implementing a new financial factor, including its name, description, formulation, and variables.

**Step 3 : Factor Implementation 👨‍💻**

- Implement the factor code based on the description, evolving it as a developer would.
- Quantitatively validate the newly created factors.

**Step 4 : Backtesting with Qlib 📉**

- Integrate the full dataset into the factor implementation code and prepare the factor library.
- Conduct backtesting using the Alpha158 plus newly developed factors and LGBModel in Qlib to evaluate the new factors' effectiveness and performance.

+----------------+------------+----------------+----------------------------------------------------+
| Dataset        | Model      | Factors        | Data Split                                         |
+================+============+================+====================================================+
| CSI300         | LGBModel   | Alpha158 Plus  | +-----------+--------------------------+           |
|                |            |                | | Train     | 2008-01-01 to 2014-12-31 |           |
|                |            |                | +-----------+--------------------------+           |
|                |            |                | | Valid     | 2015-01-01 to 2016-12-31 |           |
|                |            |                | +-----------+--------------------------+           |
|                |            |                | | Test      | 2017-01-01 to 2020-08-01 |           |
|                |            |                | +-----------+--------------------------+           |
+----------------+------------+----------------+----------------------------------------------------+

**Step 5 : Feedback Analysis 🔍**

- Analyze backtest results to assess performance.
- Incorporate feedback to refine hypotheses and improve the model.

**Step 6 :Hypothesis Refinement ♻️**

- Refine hypotheses based on feedback from backtesting.
- Repeat the process to continuously improve the model.

⚡ Quick Start

Please refer to the installation part in :doc:../installation_and_configuration to prepare your system dependency.

You can try our demo by running the following command:

• 🐍 Create a Conda Environment

  • Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI):

    .. code-block:: sh

    text Copy

    conda create -n rdagent python=3.10

  • Activate the environment:

    .. code-block:: sh

    text Copy

    conda activate rdagent

• 📦 Install the RDAgent

  • You can install the RDAgent package from PyPI:

    .. code-block:: sh

    text Copy

    pip install rdagent

• 🚀 Run the Application

  • Download the financial reports you wish to extract factors from and store them in your preferred folder.

  • Specifically, you can follow this example, or use your own method:

    .. code-block:: sh

    text Copy

    wget https://github.com/SunsetWolf/rdagent_resource/releases/download/reports/all_reports.zip
    unzip all_reports.zip -d git_ignore_folder/reports

  • Run the application with the following command:

    .. code-block:: sh

    text Copy

    rdagent fin_factor_report --report-folder=git_ignore_folder/reports

  • Alternatively, you can store the paths of the reports in report_result_json_file_path. The format should be:

    .. code-block:: json

    text Copy

    [
        "git_ignore_folder/report/fin_report1.pdf",
        "git_ignore_folder/report/fin_report2.pdf",
        "git_ignore_folder/report/fin_report3.pdf"
    ]

  • Then, run the application using the following command:

    .. code-block:: sh

    text Copy

    rdagent fin_factor_report

🛠️ Usage of modules

.. _Env Config: 

- **Env Config**

The following environment variables can be set in the `.env` file to customize the application's behavior:

.. autopydantic_settings:: rdagent.app.qlib_rd_loop.conf.FactorFromReportPropSetting
    :settings-show-field-summary: False
    :show-inheritance:
    :exclude-members: Config

.. autopydantic_settings:: rdagent.components.coder.factor_coder.config.FactorCoSTEERSettings
    :settings-show-field-summary: False
    :members: coder_use_cache, data_folder, data_folder_debug, file_based_execution_timeout, select_method, max_loop, knowledge_base_path, new_knowledge_base_path
    :exclude-members: Config, python_bin, fail_task_trial_limit, v1_query_former_trace_limit, v1_query_similar_success_limit, v2_query_component_limit, v2_query_error_limit, v2_query_former_trace_limit, v2_error_summary, v2_knowledge_sampler
    :no-index:

---

### Scens Data Science (`docs/scens/data_science.rst`)

.. _data_science_agent:

=======================
Data Science Agent
=======================

**🤖 Automated Feature Engineering & Model Tuning Evolution**
------------------------------------------------------------------------------------------
The Data Science Agent is an agent that can automatically perform feature engineering and model tuning. It can be used to solve various data science problems, such as image classification, time series forecasting, and text classification.

🌟 Introduction
~~~~~~~~~~~~~~~~~~

In this scenario, our automated system proposes hypothesis, choose action, implements code, conducts validation, and utilizes feedback in a continuous, iterative process.

The goal is to automatically optimize performance metrics within the validation set or Kaggle Leaderboard, ultimately discovering the most efficient features and models through autonomous research and development.

Here's an enhanced outline of the steps:

**Step 1 : Hypothesis Generation 🔍**

- Generate and propose initial hypotheses based on previous experiment analysis and domain expertise, with thorough reasoning and financial justification.

**Step 2 : Experiment Creation ✨**

- Transform the hypothesis into a task.
- Choose a specific action within feature engineering or model tuning.
- Develop, define, and implement a new feature or model, including its name, description, and formulation.

**Step 3 : Model/Feature Implementation 👨‍💻**

- Implement the model code based on the detailed description.
- Evolve the model iteratively as a developer would, ensuring accuracy and efficiency.

**Step 4 : Validation on Test Set or Kaggle 📉**

- Validate the newly developed model using the test set or Kaggle dataset.
- Assess the model's effectiveness and performance based on the validation results.

**Step 5: Feedback Analysis 🔍**

- Analyze validation results to assess performance.
- Use insights to refine hypotheses and enhance the model.

**Step 6: Hypothesis Refinement ♻️**

- Adjust hypotheses based on validation feedback.
- Iterate the process to continuously improve the model.

📖 Data Science Background

In the evolving landscape of artificial intelligence, Data Science represents a powerful paradigm where machines engage in autonomous exploration, hypothesis testing, and model development across diverse domains — from healthcare and finance to logistics and research.

The Data Science Agent stands as a central engine in this transformation, enabling users to automate the entire machine learning workflow: from hypothesis generation to code implementation, validation, and refinement — all guided by performance feedback.

By leveraging the Data Science Agent, researchers and developers can accelerate experimentation cycles. Whether fine-tuning custom models or competing in high-stakes benchmarks like Kaggle, the Data Science Agent unlocks new frontiers in intelligent, self-directed discovery.

🧭 Example Guide - Customized dataset

🔧 **Set up RD-Agent Environment**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Before you start, please make sure you have installed RD-Agent and configured the environment for RD-Agent correctly. If you want to know how to install and configure the RD-Agent, please refer to the `documentation <../installation_and_configuration.html>`_.

- 🔩 **Setting the Environment variables at .env file**

  - Determine the path where the data will be stored and add it to the ``.env`` file.

  .. code-block:: sh

    dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
    dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen

📥 **Prepare Customized datasets**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  - A data science competition dataset usually consists of two parts: ``competition dataset`` and ``evaluation dataset``. (We provide `a sample <https://github.com/microsoft/RD-Agent/tree/main/rdagent/scenarios/data_science/example>`_ of a customized dataset named: `arf-12-hours-prediction-task as a reference`.)
    
    - The ``competition dataset`` contains **training data**, **test data**, **description files**, **formatted submission files**, **data sampling codes**.
    
    - The ``evaluation dataset`` contains **standard answer file**, **data checking codes**, and **Code for calculation of scores**.

  - We use the ``arf-12-hours-prediction-task`` data as a sample to introduce the preparation workflow for the competition dataset.
  
    - Create a ``ds_data/source_data/arf-12-hours-prediction-task`` folder, which will be used to store your raw dataset.

      - The raw files for the competition ``arf-12-hours-prediction-task`` have two files: ``ARF_12h.csv`` and ``X.npz``.
    
    - Create a ``ds_data/source_data/arf-12-hours-prediction-task/prepare.py`` file that splits your raw data into **training data**, **test data**, **formatted submission file**, and **standard answer file**. (You will need to write a script based on your raw data.)
      
      - The following shows the preprocessing code for the raw data of ``arf-12-hours-prediction-task``.

      .. literalinclude:: ../../rdagent/scenarios/data_science/example/source_data/arf-12-hours-prediction-task/prepare.py
        :language: python
        :caption: ds_data/source_data/arf-12-hours-prediction-task/prepare.py
        :linenos:

      - At the end of program execution, the ``ds_data`` folder structure will look like this:

      .. code-block:: text

        ds_data
        ├── arf-12-hours-prediction-task
        │   ├── train
        │   │   ├── ARF_12h.csv
        │   │   └── X.npz
        │   ├── test
        │   │   ├── ARF_12h.csv
        │   │   └── X.npz
        │   └── sample_submission.csv
        ├── eval
        │   └── arf-12-hours-prediction-task
        │       └── submission_test.csv
        └── source_data
            └── arf-12-hours-prediction-task
                ├── ARF_12h.csv
                ├── prepare.py
                └── X.npz

    - Create a ``ds_data/arf-12-hours-prediction-task/description.md`` file to describe your competition, Objective, dataset, and other information.

      - The following shows the description file for ``arf-12-hours-prediction-task``

      .. literalinclude:: ../../rdagent/scenarios/data_science/example/arf-12-hours-prediction-task/description.md
        :language: markdown
        :caption: ds_data/arf-12-hours-prediction-task/description.md
        :linenos:

    - Create a ``ds_data/arf-12-hours-prediction-task/sample.py`` file to construct the debugging sample data.

      - The following shows the script for constructing the debugging sample data based on the ``arf-12-hours-prediction-task`` dataset implementation.

      .. literalinclude:: ../../rdagent/scenarios/data_science/example/arf-12-hours-prediction-task/sample.py
        :language: markdown
        :caption: ds_data/arf-12-hours-prediction-task/sample.py
        :linenos:

    - Create a ``ds_data/eval/arf-12-hours-prediction-task/valid.py`` file, which is used to check the validity of the submission files to ensure that their formatting is consistent with the reference file.

      - The following shows a script that checks the validity of a submission based on the ``arf-12-hours-prediction-task`` data.

      .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/arf-12-hours-prediction-task/valid.py
        :language: markdown
        :caption: ds_data/eval/arf-12-hours-prediction-task/valid.py
        :linenos:

    - Create a ``ds_data/eval/arf-12-hours-prediction-task/grade.py`` file, which is used to calculate the score based on the submission file and the **standard answer file**, and output the result in JSON format.

      - The following shows a grading script based on the ``arf-12-hours-prediction-task`` data implementation.

      .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/arf-12-hours-prediction-task/grade.py
        :language: markdown
        :caption: ds_data/eval/arf-12-hours-prediction-task/grade.py
        :linenos:

  - At this point, you have created a complete dataset. The correct structure of the dataset should look like this.

    .. code-block:: text

        ds_data
        ├── arf-12-hours-prediction-task
        │   ├── train
        │   │   ├── ARF_12h.csv
        │   │   └── X.npz
        │   ├── test
        │   │   ├── ARF_12h.csv
        │   │   └── X.npz
        │   ├── description.md
        │   ├── sample_submission.csv
        │   └── sample.py
        ├── eval
        │   └── arf-12-hours-prediction-task
        │       ├── grade.py
        │       ├── submission_test.csv
        │       └── valid.py
        └── source_data
            └── arf-12-hours-prediction-task
                ├── ARF_12h.csv
                ├── prepare.py
                └── X.npz

  - The above shows the complete dataset creation workflow, some of the files are not required, in practice you can customize the dataset according to your own needs.

    - If we don't need the test set scores, then we can choose not to generate **formatted submission files** and **standard answer file** in the prepare code, and we don't need to write **data checking codes** and **Code for calculation of scores**.

    - **Data sampling code** can also be created according to the actual need, if you do not provide **data sampling code**, RD-Agent will be handed over to the LLM sampling at runtime.

      - In the default sampling method (``create_debug_data``), the default sampling ratio (parameter: ``min_frac``) is 1%, if 1% of the data is less than 5, then 5 data will be sampled (parameter: ``min_num``), you can adjust the sampling ratio by adjusting these two parameters.

        - If you have customized data sampling code, you need to set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` (default is True) in the ``.env`` file before running, so that the program will use the customized sampling code when running, and you can just execute this line of code in the command line:

          .. code-block:: sh

            dotenv set DS_SAMPLE_DATA_BY_LLM False

        - In addition, we provide a data sampling method in `rdagent.scenarios.data_science.debug.data.create_debug_data <https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/data_science/debug/data.py#L605>`_, in this method, the default sampling ratio (parameter: ``min_frac``) is 1%, if 1% of the data is less than 5, then 5 data will be sampled (parameter: ``min_num``), you can use this method by the following two ways.

          - You can set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` in the ``.env`` file so that when the program runs, it will use the sampling code provided by RD-Agent.

            .. code-block:: sh

              dotenv set DS_SAMPLE_DATA_BY_LLM False

          - If you think that the parameters in the receipt sampling method provided by RD-Agent are not suitable, you can customize the parameters in the following command and run it, and set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` in the ``.env`` so that the program will use the sampling data you provided when running.

            .. code-block:: sh

              python rdagent/app/data_science/debug.py --dataset_path <dataset path> --competition <competiton_name> --min_frac <sampling ratio> --min_num <minimum number of sampling>
              dotenv set DS_SAMPLE_DATA_BY_LLM False

  - If you don't need the scores from the test set and leave the data sampling to the LLM, or if you use the sampling method provided by the RD-Agent, you only need to prepare a minimal dataset. The structure of the simplest dataset should be as shown below.

    .. code-block:: text

        ds_data
        ├── arf-12-hours-prediction-task
        │   ├── train
        │   │   ├── ARF_12h.csv
        │   │   └── X.npz
        │   ├── test
        │   │   ├── ARF_12h.csv
        │   │   └── X.npz
        │   └── description.md
        └── source_data
            └── arf-12-hours-prediction-task
                ├── ARF_12h.csv
                ├── prepare.py
                └── X.npz

  - We have prepared a dataset based on the above description for your reference. You can download it with the following command.

    .. code-block:: sh

      wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip

⚙️ **Set up Environment for Customized datasets**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  .. code-block:: sh

      dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
      dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
      dotenv set DS_CODER_ON_WHOLE_PIPELINE True

  - 📘 More Environment Variables (Optional)

    - If you want to see all the available environment variables, you can refer to the configuration file for Data Science scenarios:

    .. literalinclude:: ../../rdagent/app/data_science/conf.py
      :language: python
      :linenos:

    - These variables allow you to have finer-grained control in Data Science scenarios.

🚀 **Run the Application**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  - 🌏 You can directly run the application by using the following command:
    
    .. code-block:: sh

        rdagent data_science --competition <Competition ID>

    - The following shows the command to run based on the ``arf-12-hours-prediction-task`` data

      .. code-block:: sh

          rdagent data_science --competition arf-12-hours-prediction-task

    -  More CLI Parameters for `rdagent data_science` command:

      .. automodule:: rdagent.app.data_science.loop
        :members:
        :no-index:

  - 📈 Visualize the R&D Process

    - We provide a web UI to visualize the log. You just need to run:

      .. code-block:: sh

          rdagent ui --port <custom port> --log-dir <your log folder like "log/"> --data_science True

    - Then you can input the log path and visualize the R&D process.

  - 🧪 Scoring the test results

    - Finally, shutdown the program, and get the test set scores with this command.

    .. code-block:: sh

      dotenv run -- python rdagent/log/mle_summary.py grade 

    Here,  refers to the parent directory of the log folder generated during the run.

🕹️ Kaggle Agent
~~~~~~~~~~~~~~~~

📖 Background
^^^^^^^^^^^^^^

In the landscape of data science competitions, Kaggle serves as the ultimate arena where data enthusiasts harness the power of algorithms to tackle real-world challenges.
The Kaggle Agent stands as a pivotal tool, empowering participants to seamlessly integrate cutting-edge models and datasets, transforming raw data into actionable insights.

By utilizing the **Kaggle Agent**, data scientists can craft innovative solutions that not only uncover hidden patterns but also drive significant advancements in predictive accuracy and model robustness.

🧭 Example Guide - Kaggle Dataset
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

🛠️ Preparing For The Competition
""""""""""""""""""""""""""""""""""

- 🔨 **Configuring the Kaggle API**

  - Register and login on the `Kaggle <https://www.kaggle.com/>`_ website.
  - Click on the avatar (usually in the top right corner of the page) -> ``Settings`` -> ``Create New Token``, A file called ``kaggle.json`` will be downloaded.
  - Move ``kaggle.json`` to ``~/.config/kaggle/``
  - Modify the permissions of the ``kaggle.json`` file.

    .. code-block:: sh

      chmod 600 ~/.config/kaggle/kaggle.json

  - For more information about Kaggle API Settings, refer to the `Kaggle API <https://github.com/Kaggle/kaggle-api>`_.

- 🔩 **Setting the Environment variables at .env file**

  - Determine the path where the data will be stored and add it to the ``.env`` file.

  .. code-block:: sh

    mkdir -p <your local directory>/ds_data
    dotenv set KG_LOCAL_DATA_PATH <your local directory>/ds_data

  - 📘 More Environment Variables (Optional)

    - If you want to see all the available environment variables, you can refer to the configuration file for Data Science scenarios:

    .. literalinclude:: ../../rdagent/app/data_science/conf.py
      :language: python
      :linenos:

    - These variables allow you to have finer-grained control in Data Science scenarios.

- 🗳️ **Join the competition**

  - If your Kaggle API account has not joined a competition, you will need to join the competition before running the program.

    - At the bottom of the competition details page, you can find the ``Join the competition`` button, click on it and select ``I Understand and Accept`` to join the competition.

    - In the **Competition List Available** below, you can jump to the competition details page.

📥 Preparing Competition DataDataset && Set up RD-Agent Environment
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

- As a subset of data science, kaggle's dataset still follows the data science format. Based on this, the kaggle dataset can be divided into two categories depending on whether or not it is supported by the **MLE-Bench**.

  - What is **MLE-Bench**?

    - **MLE-Bench** is a comprehensive benchmark designed to evaluate the **machine learning engineering** capabilities of AI systems using real-world scenarios. The dataset includes multiple Kaggle competitions. Since Kaggle does not provide reserved test sets for these competitions, the benchmark includes preparation scripts for splitting publicly available training data into new training and test sets, and scoring scripts for each competition to accurately evaluate submission scores.

  - I'm running a competition Is **MLE-Bench** supported?

    - You can see all the competitions supported by **MLE-Bench** `here <https://github.com/openai/mle-bench/tree/main/mlebench/competitions>`_.

- Prepare datasets for **MLE-Bench** supported competitions.

  - If you agree with the **MLE-Bench** standard, then you don't need to prepare the dataset, you just need to configure your ``.env`` file to automate the download of the dataset.

    - Configure environment variables, add ``DS_IF_USING_MLE_DATA`` to environment variables, and set it to ``True``.

      .. code-block:: sh

        dotenv set DS_IF_USING_MLE_DATA True

    - Configure environment variables, add ``DS_SAMPLE_DATA_BY_LLM`` to environment variables, and set it to ``True``.

      .. code-block:: sh

        dotenv set DS_SAMPLE_DATA_BY_LLM True

    - Configure environment variables, add ``DS_SCEN`` to environment variables, and set it to ``rdagent.scenarios.data_science.scen.KaggleScen``.

      .. code-block:: sh

        dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen

  - At this point, you are ready to start running your competition, which will automatically download the data, and the LLM will automatically extract the minimum dataset.

    - After running the program the structure of the ds_data folder should look like this (Using the ``tabular-playground-series-dec-2021`` contest as an example).

      .. code-block:: text

        ds_data
        ├── tabular-playground-series-dec-2021
        │   ├── description.md
        │   ├── sample_submission.csv
        │   ├── test.csv
        │   └── train.csv
        └── zip_files
            └── tabular-playground-series-dec-2021
                └── tabular-playground-series-dec-2021.zip

      - The ``ds_data/zip_files`` folder contains a zip file of the raw competition data downloaded from kaggle website.

  - At runtime, RD-Agent will automatically build the Docker image specified at `rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile <https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile>`_. This image is responsible for downloading the required datasets and grading files for MLE-Bench.

  Note: The first run may take longer than subsequent runs as the Docker image and data are being downloaded and set up for the first time.

- Prepare datasets for competitions that are not supported by **MLE-Bench**.

  - As a subset of data science, we can follow the format and steps of data science dataset to prepare kaggle dataset. Below we will describe the workflow for preparing a kaggle dataset using the competition ``playground-series-s4e9`` as an example.
  
    - Create a ``ds_data/source_data/playground-series-s4e9`` folder, which will be used to store your raw dataset.

      - The raw files for the competition ``playground-series-s4e9`` have two files: ``train.csv``, ``test.csv``, ``sample_submission.csv``, and there are two ways to get the raw data:

        - You can find the raw data required for the competition on the `official kaggle website <https://www.kaggle.com/competitions/playground-series-s4e9/data>`_.

        - Or you can use the command line to download the raw data for the competition, the download command is as follows.

          .. code-block:: sh

            kaggle competitions download -c playground-series-s4e9

    - Create a ``ds_data/source_data/playground-series-s4e9/prepare.py`` file that splits your raw data into **training data**, **test data**, **formatted submission file**, and **standard answer file**. (You will need to write a script based on your raw data.)

      - The following shows the preprocessing code for the raw data of ``playground-series-s4e9``.

      .. literalinclude:: ../../rdagent/scenarios/data_science/example/source_data/playground-series-s4e9/prepare.py
        :language: python
        :caption: ds_data/source_data/playground-series-s4e9/prepare.py
        :linenos:

      - At the end of program execution, the ``ds_data`` folder structure will look like this:

      .. code-block:: text

        ds_data
        ├── playground-series-s4e9
        │   ├── train.csv
        │   ├── test.csv
        │   └── sample_submission.csv
        ├── eval
        │   └── playground-series-s4e9
        │       └── submission_test.csv
        └── source_data
            └── playground-series-s4e9
                ├── prepare.py
                ├── sample_submission.csv
                ├── test.csv
                └── train.csv

    - Create a ``ds_data/playground-series-s4e9/description.md`` file to describe your competition, dataset description, and other information. We can find the `competition description information <https://www.kaggle.com/competitions/playground-series-s4e9/overview>`_ and the `dataset description information <https://www.kaggle.com/competitions/playground-series-s4e9/data>`_ from the Kaggle website.

      - The following shows the description file for ``playground-series-s4e9``

        .. literalinclude:: ../../rdagent/scenarios/data_science/example/playground-series-s4e9/description.md
          :language: markdown
          :caption: ds_data/playground-series-s4e9/description.md
          :linenos:

    - Create a ``ds_data/eval/playground-series-s4e9/valid.py`` file, which is used to check the validity of the submission files to ensure that their formatting is consistent with the reference file.

      - The following shows a script that checks the validity of a submission based on the ``playground-series-s4e9`` data.

      .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/playground-series-s4e9/valid.py
        :language: markdown
        :caption: ds_data/eval/playground-series-s4e9/valid.py
        :linenos:

    - Create a ``ds_data/eval/playground-series-s4e9/grade.py`` file, which is used to calculate the score based on the submission file and the **standard answer file**, and output the result in JSON format.

      - The following shows a grading script based on the ``playground-series-s4e9`` data implementation.

      .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/playground-series-s4e9/grade.py
        :language: markdown
        :caption: ds_data/eval/playground-series-s4e9/grade.py
        :linenos:

  - In this example we don't create a ``ds_data/eval/playground-series-s4e9/sample.py``, we use the sample method provided by RD-Agent by default.

  - At this point, you have created a complete dataset. The correct structure of the dataset should look like this.

    .. code-block:: text

        ds_data
        ├── playground-series-s4e9
        │   ├── train.csv
        │   ├── test.csv
        │   ├── description.md
        │   └── sample_submission.csv
        ├── eval
        │   └── playground-series-s4e9
        │       ├── grade.py
        │       ├── submission_test.csv
        │       └── valid.py
        └── source_data
            └── playground-series-s4e9
                ├── prepare.py
                ├── sample_submission.csv
                ├── test.csv
                └── train.csv

  - We have prepared a dataset based on the above description for your reference. You can download it with the following command.

    .. code-block:: sh

      wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/playground-series-s4e9.zip

  - Next, we need to configure the environment for the ``playground-series-s4e9`` contest. You can do this by executing the following command at the command line.

    .. code-block:: sh

      dotenv set DS_IF_USING_MLE_DATA False
      dotenv set DS_SAMPLE_DATA_BY_LLM False
      dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen

🚀 **Run the Application**
""""""""""""""""""""""""""""""""""""

  - 🌏 You can directly run the application by using the following command:

    .. code-block:: sh

        rdagent data_science --competition <Competition ID>

    - The following shows the command to run based on the ``playground-series-s4e9`` data

      .. code-block:: sh

          rdagent data_science --competition playground-series-s4e9

    -  More CLI Parameters for `rdagent data_science` command:

      .. automodule:: rdagent.app.data_science.loop
        :members:
        :no-index:

  - 📈 Visualize the R&D Process

    - We provide a web UI to visualize the log. You just need to run:

      .. code-block:: sh

          rdagent ui --port <custom port> --log-dir <your log folder like "log/"> --data_science True

    - Then you can input the log path and visualize the R&D process.

  - 🧪 Scoring the test results

    - Finally, shutdown the program, and get the test set scores with this command.

    .. code-block:: sh

      dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>

    - If you have configured the full output in ``ds_data/eval/playground-series-s4e9/grade.py``, or if you are running a competition that receives **MLE-Bench** support, you can also summarize the scores by running the following command.

    .. code-block:: sh

      rdagent grade_summary --log-folder=<url_to_log>

    Here, <url_to_log> refers to the parent directory of the log folder generated during the run.

---

### Scens Finetune (`docs/scens/finetune.rst`)

.. _finetune_agent:

================================
FT-Agent for LLM Fine-Tuning
================================

FT-Agent is the RD-Agent scenario for autonomous LLM fine-tuning, introduced in
the ICML 2026 paper `FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language
Agents <https://arxiv.org/abs/2603.01712>`_.

The scenario automates benchmark-driven data processing, LLaMA-Factory training
configuration, fail-fast validation, OpenCompass evaluation, and feedback-based
iteration.

The full user guide is maintained in the repository:

`rdagent/app/finetune/llm/README.md <https://github.com/microsoft/RD-Agent/blob/main/rdagent/app/finetune/llm/README.md>`_

Minimal command after configuring the required ``FT_*`` settings:

.. code-block:: sh

   rdagent llm_finetune --base-model Qwen/Qwen2.5-7B-Instruct

Please read the full guide before running this scenario. A first run can download
large dataset/model assets and consume LLM API calls and GPU hours.

---

### Scens Model Agent Fin (`docs/scens/model_agent_fin.rst`)

.. _model_agent_fin:

=======================
Finance Model Agent
=======================

**🤖 Automated Quantitative Trading & Iterative Model Evolution**
------------------------------------------------------------------------------------------

📖 Background
~~~~~~~~~~~~~~
In the realm of quantitative finance, both factor discovery and model development play crucial roles in driving performance. 
While much attention is often given to the discovery of new financial factors, the **models** that leverage these factors are equally important. 
The effectiveness of a quantitative strategy depends not only on the factors used but also on how well these factors are integrated into robust, predictive models.

However, the process of developing and optimizing these models can be labor-intensive and complex, requiring continuous refinement and adaptation to ever-changing market conditions. 
And this is where the **Finance Model Agent** steps in.


🎥 `Demo <https://rdagent.azurewebsites.net/model_loop>`_

.. raw:: html

<video width="600" controls>
    
    Your browser does not support the video tag.
  </video>

🌟 Introduction

In this scenario, our automated system proposes hypothesis, constructs model, implements code, conducts back-testing, and utilizes feedback in a continuous, iterative process.

The goal is to automatically optimize performance metrics within the Qlib library, ultimately discovering the most efficient code through autonomous research and development.

Here's an enhanced outline of the steps:

**Step 1 : Hypothesis Generation 🔍**

- Generate and propose initial hypotheses based on previous experiment analysis and domain expertise, with thorough reasoning and financial justification.

**Step 2 : Model Creation ✨**

- Transform the hypothesis into a task.
- Develop, define, and implement a quantitative model, including its name, description, and formulation.

**Step 3 : Model Implementation 👨‍💻**

- Implement the model code based on the detailed description.
- Evolve the model iteratively as a developer would, ensuring accuracy and efficiency.

**Step 4 : Backtesting with Qlib 📉**

- Conduct backtesting using the newly developed model and 20 factors extracted from Alpha158 in Qlib.
- Evaluate the model's effectiveness and performance.

+----------------+------------+------------------------+----------------------------------------------------+
| Dataset        | Model      | Factors                | Data Split                                         |
+================+============+========================+====================================================+
| CSI300         | RDAgent-dev| 20 factors (Alpha158)  | +-----------+--------------------------+           |
|                |            |                        | | Train     | 2008-01-01 to 2014-12-31 |           |
|                |            |                        | +-----------+--------------------------+           |
|                |            |                        | | Valid     | 2015-01-01 to 2016-12-31 |           |
|                |            |                        | +-----------+--------------------------+           |
|                |            |                        | | Test      | 2017-01-01 to 2020-08-01 |           |
|                |            |                        | +-----------+--------------------------+           |
+----------------+------------+------------------------+----------------------------------------------------+

**Step 5 : Feedback Analysis 🔍**

- Analyze backtest results to assess performance.
- Incorporate feedback to refine hypotheses and improve the model.

**Step 6 :Hypothesis Refinement ♻️**

- Refine hypotheses based on feedback from backtesting.
- Repeat the process to continuously improve the model.

⚡ Quick Start

Please refer to the installation part in :doc:../installation_and_configuration to prepare your system dependency.

You can try our demo by running the following command:

• 🐍 Create a Conda Environment

  • Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI):

    .. code-block:: sh

    text Copy

    conda create -n rdagent python=3.10

  • Activate the environment:

    .. code-block:: sh

    text Copy

    conda activate rdagent

• 📦 Install the RDAgent

  • You can install the RDAgent package from PyPI:

    .. code-block:: sh

    text Copy

    pip install rdagent

• 🚀 Run the Application

  • You can directly run the application by using the following command:

    .. code-block:: sh

    text Copy

    rdagent fin_model

🛠️ Usage of modules

.. _Env Config: 

- **Env Config**

The following environment variables can be set in the `.env` file to customize the application's behavior:

.. autopydantic_settings:: rdagent.app.qlib_rd_loop.conf.ModelBasePropSetting
    :settings-show-field-summary: False
    :exclude-members: Config

- **Qlib Config**
    - The `config.yaml` file located in the `model_template` folder contains the relevant configurations for running the developed model in Qlib. The default settings include key information such as:
        - **market**: Specifies the market, which is set to `csi300`.
        - **fields_group**: Defines the fields group, with the value `feature`.
        - **col_list**: A list of columns used, including various indicators such as `RESI5`, `WVMA5`, `RSQR5`, and others.
        - **start_time**: The start date for the data, set to `2008-01-01`.
        - **end_time**: The end date for the data, set to `2020-08-01`.
        - **fit_start_time**: The start date for fitting the model, set to `2008-01-01`.
        - **fit_end_time**: The end date for fitting the model, set to `2014-12-31`.

    - The default hyperparameters used in the configuration are as follows:
        - **n_epochs**: The number of epochs, set to `100`.
        - **lr**: The learning rate, set to `1e-3`.
        - **early_stop**: The early stopping criterion, set to `10`.
        - **batch_size**: The batch size, set to `2000`.
        - **metric**: The evaluation metric, set to `loss`.
        - **loss**: The loss function, set to `mse`.
        - **n_jobs**: The number of parallel jobs, set to `20`.

---

### Scens Model Copilot General (`docs/scens/model_copilot_general.rst`)

.. _model_copilot_general:

======================
General Model Copilot
======================

**🤖 Automated Model Research & Development Co-Pilot**
--------------------------------------------------------

📖 Background
~~~~~~~~~~~~~~
In the fast-paced field of artificial intelligence, the number of academic papers published each year is skyrocketing. 
These papers introduce new models, techniques, and approaches that can significantly advance the state of the art. 
However, reproducing and implementing these models can be a daunting task, requiring substantial time and expertise. 
Researchers often face challenges in extracting the essential details from these papers and converting them into functional code.
And this is where the **General Model Copilot** steps in.

🎥 `Demo <https://rdagent.azurewebsites.net/report_model>`_

.. raw:: html

<div style="display: flex; justify-content: center; align-items: center;">
  <video width="600" controls>
    <source src="https://rdagent.azurewebsites.net/media/b35f904765b05099b0fcddbebe041a04f4d7bde239657e5fc24bf0cc.mp4" type="video/mp4">
    Your browser does not support the video tag.
  </video>
</div>

🌟 Introduction

In this scenario, our automated system proposes hypotheses, constructs models, implements code, performs back-testing, and uses feedback to iterate continuously. The system aims to automatically optimize performance metrics from the Qlib library, finding the best code through autonomous research and development.

Model R&D CoPilot Scenario

Overview

This demo automates the extraction and iterative development of models from academic papers, ensuring functionality and correctness. This scenario automates the development of PyTorch models by reading academic papers or other sources. It supports various data types, including tabular, time-series, and graph data. The primary workflow involves two main components: the Reader and the Coder.

Workflow Components

1. Reader

   • Parses and extracts relevant model information from academic papers or sources, including architectures, parameters, and implementation details.
   • Uses Large Language Models to convert content into a structured format for the Coder.

2. Evolving Coder

   • Translates structured information from the Reader into executable PyTorch code.
   • Utilizes an evolving coding mechanism to ensure correct tensor shapes, verified with sample input tensors.
   • Iteratively refines the code to align with source material specifications.

Supported Data Types

• Tabular Data: Structured data with rows and columns, such as spreadsheets or databases.
• Time-Series Data: Sequential data points indexed in time order, useful for forecasting and temporal pattern recognition.
• Graph Data: Data structured as nodes and edges, suitable for network analysis and relational tasks.

⚡ Quick Start

Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency.

You can try our demo by running the following command:

- 🐍 Create a Conda Environment
  
  - Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI):

    .. code-block:: sh
    
        conda create -n rdagent python=3.10

  - Activate the environment:

    .. code-block:: sh

        conda activate rdagent

- 📦 Install the RDAgent
    
  - You can install the RDAgent package from PyPI:

    .. code-block:: sh

        pip install rdagent


- 🚀 Run the Application
    
  - Prepare relevant files (in pdf format) by uploading papers to the directory below and copy the path as report_file_path.
      
    .. code-block:: sh

        rdagent/scenarios/general_model
    
  - Run the following command in your terminal within the same virtual environment:
  
    .. code-block:: sh

        rdagent general_model --report-file-path=<path_to_pdf_file>

---

### Scens Quant Agent Fin (`docs/scens/quant_agent_fin.rst`)

.. _quant_agent_fin:

=====================
Finance Quant Agent
=====================


**🥇The First Data-Centric Quant Multi-Agent Framework RD-Agent(Q)**
---------------------------------------------------------------------

R&D-Agent for Quantitative Finance, in short **RD-Agent(Q)**, is the first data-centric, multi-agent framework designed to automate the full-stack research and development of quantitative strategies via coordinated factor-model co-optimization.

You can learn more details about **RD-Agent(Q)** through the `paper <https://arxiv.org/abs/2505.15155>`_.

⚡ Quick Start

Before you start, please make sure you have installed RD-Agent and configured the environment for RD-Agent correctly. If you want to know how to install and configure the RD-Agent, please refer to the documentation <../installation_and_configuration.html>_.

Then, you can run the framework by running the following command:

• 🐍 Create a Conda Environment

  • Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI):

    .. code-block:: sh

    text Copy

    conda create -n rdagent python=3.10

  • Activate the environment:

    .. code-block:: sh

    text Copy

    conda activate rdagent

• 📦 Install the RDAgent

  • You can install the RDAgent package from PyPI:

    .. code-block:: sh

    text Copy

    pip install rdagent

• 🚀 Run the Application

  • You can directly run the application by using the following command:

    .. code-block:: sh

    text Copy

    rdagent fin_quant

🛠️ Usage of modules

.. _Env Config: 

- **Env Config**

The following environment variables can be set in the `.env` file to customize the application's behavior:

.. autopydantic_settings:: rdagent.app.qlib_rd_loop.conf.QuantBasePropSetting
    :settings-show-field-summary: False
    :exclude-members: Config

.. autopydantic_settings:: rdagent.components.coder.factor_coder.config.FactorCoSTEERSettings
    :settings-show-field-summary: False
    :members: coder_use_cache, data_folder, data_folder_debug, file_based_execution_timeout, select_method, max_loop, knowledge_base_path, new_knowledge_base_path
    :exclude-members: Config, fail_task_trial_limit, v1_query_former_trace_limit, v1_query_similar_success_limit, v2_query_component_limit, v2_query_error_limit, v2_query_former_trace_limit, v2_error_summary, v2_knowledge_sampler
    :no-index:

- **Qlib Configuration**
    - The `.yaml` files in both the `model_template` and `factor_template` directories contain some configurations for running the corresponding models or factors within the Qlib framework. Below is an overview of their contents and roles:
        - **General Settings**:
            - **provider_uri**: Specifies the local Qlib data path, set to `~/.qlib/qlib_data/cn_data`.
            - **market**: Configured to `csi300`, representing the CSI 300 index constituents.
            - **benchmark**: Set to `SH000300`, used for backtesting evaluation.
        
        - **Data Handling**:
            - **start_time** and **end_time**: Define the full data range, from `2008-01-01` to `2022-08-01`.
            - **fit_start_time**: The start date for fitting the model, set to `2008-01-01`.
            - **fit_end_time**: The end date for fitting the model, set to `2014-12-31`.
            - **features and labels**: Generated via a nested data loader combining `Alpha158DL` (for engineered features such as `RESI5`, `WVMA5`, `RSQR5`, `KLEN`, etc.) and a `StaticDataLoader` that loads precomputed factor files (`combined_factors_df.parquet`).
            -  **normalization**: The pipeline includes `RobustZScoreNorm` (with clipping) and `Fillna` for inference, and `DropnaLabel` with `CSZScoreNorm` for training.
        
        - **Training Configuration**:
            - **Model**: Uses `GeneralPTNN`, a PyTorch-based neural network model.
            - **Dataset Splits**:
                - **train**: `2008-01-01` to `2014-12-31`
                - **valid**: `2015-01-01` to `2016-12-31`
                - **test**: `2017-01-01` to `2020-08-01`

        - **Default Hyperparameters** (can be overridden by command-line arguments):
            - **n_epochs**: `100`
            - **lr**: `2e-4`
            - **early_stop**: `10`
            - **batch_size**: `256`
            - **weight_decay**: `0.0`
            - **metric**: `loss`
            - **loss**: `mse`
            - **n_jobs**: `20`
            - **GPU**: `0` (uses GPU 0 if available)
            
        - **Backtesting and Evaluation**:
            - **strategy**: `TopkDropoutStrategy`, which selects the top 50 stocks and randomly drops 5 to introduce exploration.
            - **backtest period**: `2017-01-01` to `2020-08-01`
            - **initial capital**: `100,000,000`
            - **cost configuration**: Includes open/close costs, minimum transaction costs, and slippage control.
            
        - **Recording and Analysis**:
            - **SignalRecord**: Logs predicted signals.
            - **SigAnaRecord**: Performs signal analysis without long-short separation.
            - **PortAnaRecord**: Conducts portfolio analysis using the configured strategy and backtest settings.

---

--- METRICS ---
- Files Extracted: 48
- Estimated Token Budget: ~66779 tokens
- Canonical Reference: https://codewiki.google/github.com/microsoft/RD-Agent