## 1. Project Overview & Quickstart (inclusionAI/AReaL) ## File: README.md

*AReaL*: A Large-Scale Asynchronous Reinforcement Learning System

**AReaL** is a reinforcement learning (RL) infrastructure designed to bridge foundation model training with modern agent-based applications. It was originally developed by researchers and engineers from Tsinghua IIIS and the AReaL Team at Ant Group. Built on a **fully asynchronous RL training paradigm**, AReaL is optimized for efficiency and scalability, making it particularly well-suited for training large-scale **reasoning and agentic models**. AReaL’s mission is to make building AI agents **accessible, efficient, and cost-effective** for a broad community of developers and researchers. > Like milk tea - customizable, scalable, and enjoyable - we hope AReaL brings both > flexibility and delight to your AI development experience. Cheers! **AReaL Highlights** - ⚑ **Flexibility**: Seamless customization for [agentic RL](https://areal-project.github.io/AReaL/en/tutorial/agentic_rl.html) and [online RL training](https://areal-ai.io/en/tutorial/online_proxy.html) for **black-box agent applications** by simply replacing the `base_url`. - πŸ“ˆ **Scalability**: **Stable** fully asynchronous RL training with **industry-leading speed**. - ✨ **Cutting-Edge Performance**: State-of-the-art [math](/blog/AReaL_v0_2.md), [coding](/blog/AReaL_v0_3.md), [search](https://github.com/inclusionAI/ASearcher), and [customer service](https://arxiv.org/abs/2601.22607) agents. ## πŸ“° News **\[2026/07/01\]** πŸŽ‰ We release **AReaL 2.0**, a major architectural milestone. This release refactors AReaL into a **microservice architecture** with independent [training](./areal/v2/training_service/), [inference](./areal/v2/inference_service/), [agent](./areal/v2/agent_service/), and [weight-update](./areal/v2/weight_update/) services, and ships the [Hermes online RL loop](./examples/hermes) and [end-to-end SWE RL training examples](./examples/swe). Check out our [technical report](https://arxiv.org/abs/2607.01120). **\[2026/06/17\]** πŸ”¬ Introducing **KPop** β€” bidirectional binary KL divergence token masking. Configured via `rejection_sampling.metric=binary_kl`. Also adding an [IcePop config](examples/math/gsm8k_icepop.yaml) (importance-ratio-based token masking). Check out [gsm8k_kpop.yaml](examples/math/gsm8k_kpop.yaml) and [gsm8k_icepop.yaml](examples/math/gsm8k_icepop.yaml) to get started! **πŸ“‹ Previous Releases** **\[2026/04/23\]** πŸš€ We’re excited to release our integration with [Scaffoldings](https://github.com/NVIDIA/TensorRT-LLM/tree/main/tensorrt_llm/scaffolding) for agentic RL training - now live in our [examples](https://github.com/areal-project/AReaL/tree/main/examples/scaffolding)! Huge shoutout to @narutolhy and @WeiHaocheng for making this happen πŸ™Œ. The modular design of the Scaffoldings enables it to achieve a thorough decoupling of agent execution, reward calculation, and trajectory acquisition. This enables developers to reuse existing modules when implementing an agentic RL method, allowing them to focus on their own innovative modules. **\[2026/04/18\]** We are thrilled to announce that **AReaL's first Community Biweekly Meeting** was successfully held! Thank you to everyone who joined us. Meeting materials are now available [here](https://github.com/areal-project/AReaL/tree/main/assets/community). Our next meeting is scheduled for **2026/05/01** and will also be conducted in Chinese; English-language meetings will be scheduled in the future. We warmly welcome everyone to participate! See [Community](./assets/community/README.md) for more details. **\[2026/03/02\]** We provide [a complete example](./examples/openclaw/) to train your own 🦞 OpenClaw agent by simply replacing the `base_url` and `api_key` with AReaL's RL service - no complicated dependencies, no code changes, works with any agentic runtime! **\[2026/02/06\]** We are delighted to introduce **AReaL-SEA**, a self-evolving data synthesis engine. Combined with RL training on AReaL, the 235B MoE model surpasses GPT 5 and achieves comparable performance with Gemini 3.0 Pro on $\\tau^2$-bench! Check out the [paper](https://arxiv.org/pdf/2601.22607), [model](https://huggingface.co/inclusionAI/AReaL-SEA-235B-A22B), [data](https://huggingface.co/datasets/inclusionAI/AReaL-tau2-data), and [code](https://github.com/areal-project/AReaL/tree/main/examples/tau2). **\[2026/01/15\]** Congrats to our friends at [CAMEL-AI](https://www.camel-ai.org/) for open-sourcing [SETA](https://github.com/camel-ai/seta), their terminal agent RL project trained with AReaL! Check out [their training workflow](https://github.com/camel-ai/seta/tree/main/training/tbench_areal_workflow) and the [announcement on X](https://x.com/guohao_li/status/2009678513574408636). **\[2026/01/01\]** Happy New Year! Thanks to the outstanding contribution from @HwVanICI, we are excited to officially announce stable support for AReaL training on **Ascend NPU devices**! The code is actively maintained and continuously updated in the [`ascend` branch](https://github.com/areal-project/AReaL/tree/ascend). Check out [our documentation](https://areal-project.github.io/AReaL/en/tutorial/installation_npu.html) to get started, and feel free to report any issues! **\[2025/08/30\]** Introducing ASearcher, a state-of-the-art search agent built with AReaL's end-to-end asynchronous RL training. Check out the [paper](assets/paper.pdf) and the [open-source repository](https://github.com/inclusionAI/ASearcher)! **\[2025/07/31\] (AReaL-lite)** We introduce AReaL-lite, a **lightweight** version of AReaL designed specifically for AI researchers and rapid prototyping. AReaL-lite features an **algorithm-first** API design that prioritizes ease of use and algorithm development, while natively supporting **fully asynchronous agentic RL**. With 80% fewer lines of code, AReaL-lite maintains 90% of AReaL's performance and core functionality. Check out [our documentation](https://areal-project.github.io/AReaL/en/intro.html) and [the quickstart guide](https://areal-project.github.io/AReaL/en/tutorial/quickstart.html) to begin your journey with **AReaL-lite**! **\[2025/06/03\] (v0.3, bobaΒ²)** We release **bobaΒ²** (double-boba) for fully asynchronous RL training, which achieves **2.77Γ— speedup while delivering comparable or superior training performance** compared to synchronous systems. Furthermore, asynchronous RL significantly simplifies multi-turn agentic RL training setup! Check out [our v0.3 overview blog](/blog/AReaL_v0_3.md) and the [research paper](assets/paper.pdf). **\[2025/03/31\] (v0.2, boba)** Introducing our milestone releaseβ€”boba! Please call it A-ReaL-boba! This release features significantly faster training with SGLang support and state-of-the-art 7B and 32B models for mathematical reasoning. Check out our [v0.2 technical blog](/blog/AReaL_v0_2.md). **\[2025/02/24\] (v0.1)** Our initial release includes reproducible results for 1.5B and 7B Large Reasoning Models (LRMs). Check out our [v0.1 technical blog](/blog/AReaL_v0_1.md). ## πŸš€ Getting Started First, install the package: ```bash git clone https://github.com/areal-project/AReaL cd AReaL pip install uv # Install flash-attn pre-built wheel first to avoid compiling from source # (pick the wheel matching your Python version; see https://github.com/mjun0812/flash-attention-prebuild-wheels/releases) uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl" uv sync --extra cuda # installs training packages + SGLang (default inference backend) # For vLLM instead: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda ``` Our training scripts automatically download the required dataset (openai/gsm8k) and model (Qwen/Qwen2-1.5B-Instruct). To run on a single node: ```bash python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local ``` If you prefer to run experiments on a Ray cluster, update paths in the YAML file to point to your shared storage, and run: ```bash python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml \ cluster.n_nodes=2 cluster.n_gpus_per_node=8 \ cluster.fileroot=/path/to/nfs \ scheduler.type=ray ``` For comprehensive setup instructions, see [our quickstart guide](https://areal-project.github.io/AReaL/en/tutorial/quickstart.html). For Megatron MoE router replay experiments, see the [R3 Router Replay guide](r3/README.md). ## πŸ“š Examples ### Math & Reasoning | Task | Description | Performance | | --------------------------------------------------- | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | **[Math](examples/math/)** | GSM8K math reasoning with GRPO, PPO, DAPO, REINFORCE, RLOO, LitePPO, DR-GRPO, GSPO, and more | - | | **[Multi-Turn Math](examples/multi_turn_math/)** | Multi-turn math agent with reward discounting across turns | [Training Curve](examples/multi_turn_math/reward_curve.png) | | **[LoRA Math](examples/math/gsm8k_grpo_lora.yaml)** | Parameter-efficient math training with LoRA (SGLang/vLLM backends) | - | | **[Countdown](examples/countdown/)** | Countdown numbers game with custom rewards | [Training Curve](examples/countdown/countdown_training_curve.png) | ### Agentic RL | Task | Description | Performance | | -------------------------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | **[General Agent](examples/agent_workflow/)** | General agentic training with any agentic frameworks | [Guide](docs/en/tutorial/agentic_rl.md) | | **[Hermes Online RL](examples/hermes/)** | End-to-end Online RL loop with Hermes agent under AReaL 2.0 | [Guide](examples/hermes/README.md) | | **[Coding Agent RL](examples/swe/)** | End-to-end coding-agent RL training with AReaL-SWEAgent/Claude Code Agent | [Guide](examples/swe/README.md) | | **[Tau2 Customer Service](examples/tau2/)** | Customer service agent on Tau2-Bench (retail, airline, telecom) | [Paper](https://arxiv.org/abs/2601.22607) | | **[Search Agent](examples/search_agent/)** | End-to-end search agent with Tongyi-DeepResearch workflow | [Training Curve](examples/search_agent/tongyi_deepresearch/reward_curve.png) | | **[Tool-Integrated Reasoning](examples/tir/)** | Multi-turn tool calling during reasoning (Python executor, calculator) | [Training Curve](examples/tir/figures/task_reward.png) | | **[OpenAI Agents Integration](examples/openai_agents/)** | Integration with OpenAI Agents SDK for agentic workflows | - | | **[CAMEL-AI Integration](examples/camel/)** | Integration with CAMEL-AI framework for agentic RL | - | ### Vision-Language Models | Task | Description | Performance | | ----------------------------------- | --------------------------------------------------------- | ----------------------------------------------- | | **[VLM](examples/vlm/)** | Geometry3K and CLEVR Count 70K visual reasoning with GRPO | - | | **[VLM on NPU](examples/vlm_npu/)** | VLM training on Huawei NPU hardware | [Benchmark Results](examples/vlm_npu/README.md) | ### Alignment & Infrastructure | Task | Description | Performance | | ----------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------- | | **[RLHF Reward Modeling](examples/alignment/)** | Bradley-Terry reward modeling on Anthropic HH-RLHF | [Training Curve](examples/alignment/rw_curve.png) | | **[SkyPilot Deployment](examples/skypilot/)** | Cloud deployment with SkyPilot (GCP, AWS, Kubernetes) | [Screenshots](examples/skypilot/README.md) | ## πŸ”§ Support Matrix ### 🧠 Algorithms All RL algorithms support both asynchronous and synchronous versions by setting `max_head_offpolicyness=0`. See [Asynchronous RL Guide](docs/en/algorithms/async.md). | Algorithm | Documentation | Paper | Configuration | | ------------------------ | --------------------------------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------- | | **GRPO** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Paper](https://arxiv.org/pdf/2402.03300) | [πŸ”— GSM8K Example](examples/math/gsm8k_grpo.yaml) | | **GSPO** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Paper](https://arxiv.org/abs/2507.18071) | [πŸ”— GSM8K Example](examples/math/gsm8k_gspo.yaml) | | **PPO** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Paper](https://arxiv.org/pdf/2203.02155) | [πŸ”— GSM8K Example](examples/math/gsm8k_ppo.yaml) | | **DAPO** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Paper](https://arxiv.org/abs/2503.14476) | [πŸ”— GSM8K Example](examples/math/gsm8k_dapo_dynamic_bs.yaml) | | **LitePPO** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Paper](https://arxiv.org/abs/2508.08221) | [πŸ”— GSM8K Example](examples/math/gsm8k_liteppo.yaml) | | **Dr.GRPO** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Paper](https://arxiv.org/abs/2503.20783) | [πŸ”— GSM8K Example](examples/math/gsm8k_drgrpo.yaml) | | **REINFORCE++** | - | [πŸ“„ Paper](https://arxiv.org/pdf/2501.03262) | [πŸ”— GSM8K Example](examples/math/gsm8k_reinforce.yaml) | | **RLOO** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Paper](https://arxiv.org/pdf/2402.14740v1) | [πŸ”— GSM8K Example](examples/math/gsm8k_rloo.yaml) | | **SAPO** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Paper](https://arxiv.org/abs/2511.20347) | [πŸ”— GSM8K Example](examples/math/gsm8k_sapo.yaml) | | **IcePop** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Blog](https://ringtech.notion.site/icepop) | [πŸ”— GSM8K Example](examples/math/gsm8k_icepop.yaml) | | **KPop** | [πŸ“– Docs](docs/en/algorithms/grpo_series.md) | [πŸ“„ Blog](https://ringtech.notion.site/kpop) | [πŸ”— GSM8K Example](examples/math/gsm8k_kpop.yaml) | | **M2PO** | [πŸ“– Docs](docs/en/algorithms/m2po.md) | [πŸ“„ Paper](https://arxiv.org/abs/2510.01161) | [πŸ”— GSM8K Example](examples/math/gsm8k_m2po.yaml) | | **DPO** | [πŸ“– Docs](docs/en/algorithms/dpo.md) | [πŸ“„ Paper](https://arxiv.org/abs/2305.18290) | [πŸ”— HH-RLHF Example](examples/alignment/hhrlhf_dpo.yaml) | | **RLHF Reward Modeling** | - | - | [πŸ”— RLHF Example](examples/alignment/hhrlhf_rw.yaml) | | **SFT** | - | - | [πŸ”— GSM8K Example](examples/math/gsm8k_sft.py) | | **Distillation** | [πŸ“– Docs](docs/en/algorithms/distillation.md) | [πŸ“„ Paper](https://arxiv.org/pdf/2506.02208) | [πŸ”— GSM8K Example](examples/distillation/gsm8k_grpo_distill_mode_trainEngine.yaml) | ### Models | Model Family | Megatron | PyTorch FSDP | PyTorch Archon | Notes | | -------------------------- | -------- | ------------ | -------------- | -------------------------------------------------------- | | **Qwen2/3** | βœ… | βœ… | βœ… | - | | **Qwen3-MoE** | βœ… | βœ… | βœ… | - | | **Qwen3.5** | βœ… | βœ… | ❌ | Vision-language model | | **Qwen3.5-MoE** | βœ… | βœ… | ❌ | Vision-language model | | **Bailing V3** | βœ… | ❌ | ❌ | - | | **Qwen2.5-VL** | βœ… | βœ… | ❌ | Vision-language model | | **Qwen3-VL** | βœ… | βœ… | ❌ | Vision-language model | | **Gemma 3** | ❌ | βœ… | ❌ | Vision-language model | | **Other Hugging Face LLM** | ❌ | βœ… | ❌ | Compatibility depending on the version of `transformers` | Qwen2 and Qwen2.5 **text models** support RL weight updates with `actor.weight_update_mode=disk` in both v1 and v2. XCCL weight updates require `actor._version=v1`, `rollout._version=v1`, and `actor.weight_update_mode=xccl`. AWEX weight updates are not supported for these models. This restriction does not apply to Qwen2.5-VL. Check the [AI Coding Assistant Guide](docs/en/reference/ai_assisted_dev.md) and [Archon Reference](docs/en/tutorial/archon.md) for how to integrate new models into AReaL. ### Training Backends | Backend | DP | Tensor Parallel | Sequence Parallel within TP | Context Parallel | Pipeline Parallel | Expert Parallel | 1D Sequence Packing | LoRA | | ------------------ | ----------- | --------------- | --------------------------- | ---------------- | ----------------- | --------------- | ------------------- | -------------------------------- | | **Megatron** | βœ… (ZeRO-1) | βœ… | βœ… | βœ… | βœ… | βœ… | βœ… | βœ… (with vLLM inference backend) | | **PyTorch FSDP** | βœ… (FSDP2) | βœ… | βœ… | βœ… | ❌ | ❌ | βœ… | βœ… | | **PyTorch Archon** | βœ… (FSDP2) | βœ… | βœ… | βœ… | βœ… | βœ… | βœ… | ❌ | ### Inference Backends | Backend | Tensor Parallel | Context Parallel | Pipeline Parallel | Data Parallel Attention | Expert Parallel | | ---------- | --------------- | ---------------- | ----------------- | ----------------------- | --------------- | | **vLLM** | βœ… | ❓ | βœ… | ❓ | ❓ | | **SGLang** | βœ… | ❌ | ❌ | βœ… | βœ… | ## πŸ“– Resources ### Tutorial - [Installation](docs/en/tutorial/installation.md) - [Quickstart](docs/en/tutorial/quickstart.md) - [AReaL CLI Guide](docs/en/best_practices/cli_guide.md) - [Agentic RL](docs/en/tutorial/agentic_rl.md) - [Evaluation](docs/en/tutorial/eval.md) - [Large MoE with Megatron](docs/en/tutorial/megatron.md) - [Large MoE with PyTorch Archon](docs/en/tutorial/archon.md) ### Code Walkthrough - [Running GRPO on GSM8K dataset](docs/en/tutorial/gsm8k_grpo.md) ### Best Practices - [Improving Algorithm Performance](docs/en/best_practices/algo_perf.md) - [Agent Workflow Best Practices](docs/en/best_practices/workflow.md) - [Debugging](docs/en/best_practices/debugging.md) - [Handling OOM Issues](docs/en/best_practices/handling_oom.md) - [Performance Profiling](docs/en/best_practices/perf_profiling.md) ### Customization - [Customize Dataset](docs/en/customization/dataset.md) - [Customize Agentic/RVLR Rollout Workflows](docs/en/customization/agent.md) ### Algorithms - [Asynchronous RL Explained](docs/en/algorithms/async.md) - [PPO, GRPO, and Related Algorithms](docs/en/algorithms/grpo_series.md) - [M2PO](docs/en/algorithms/m2po.md) ### Reference - [CLI Configurations](docs/en/cli_reference.md) - [LoRA RL](docs/en/reference/lora.md) - [Checkpointing](docs/en/reference/checkpointing.md) - [Metrics Tracking](docs/en/reference/metrics_tracking.md) - [Allocation Mode](docs/en/reference/alloc_mode.md) - [Rollout Workflow](docs/en/reference/rollout_workflow.md) - [Agent Workflow](docs/en/reference/agent_workflow.md) - [AI-Assisted Development](docs/en/reference/ai_assisted_dev.md) ## 🀝 Contributing We warmly welcome contributions from the community! Whether you're fixing bugs, adding features, improving documentation, or helping others, your contribution is valued. Please check our **[Contributing Guide](https://github.com/areal-project/community/blob/main/CONTRIBUTING.md)** for detailed information. ```bash # Fork and clone the repository git clone https://github.com/YOUR-USERNAME/AReaL cd AReaL # Install uv and sync dependencies pip install uv # Install flash-attn pre-built wheel to avoid compiling from source uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl" # Use `--extra cuda` on Linux with CUDA (installs training packages + SGLang) uv sync --extra cuda --group dev # For vLLM instead: # cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda --group dev # Or without CUDA support # uv sync --group dev # Set up pre-commit hooks (formatting, linting, commit message checks) pre-commit install --install-hooks # Make changes git checkout -b feat/gpt-o5 git add . # `git commit` will automatically check your files and commit messages git commit -m "feat: implement gpt-o5 training loop" git push ``` ## πŸ—ΊοΈ Future Roadmap - **[Full Roadmap](ROADMAP.md)** - **[2026 H2 Roadmap](https://github.com/areal-project/AReaL/issues/1381)** - **[2026 Q2 Roadmap](https://github.com/areal-project/AReaL/issues/1302)** AReaL is under active development with planned minor releases weekly and major releases monthly. We warmly welcome community engagement and contributions. We are also **actively hiring interns and full-time employees** with open positions in both the US and China. ## πŸ“’ Community We warmly welcome everyone to join our community! We have a _**WeChat group**_ for real-time discussions and a [GitHub discussion board](https://github.com/areal-project/AReaL/discussions) for asynchronous conversations. Check out our [Community Repository](https://github.com/areal-project/community) for more details and meeting materials. ## πŸ™ Acknowledgments We gratefully acknowledge that major contributors are from the AReaL Team at the Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University and Ant Group. We have also received invaluable assistance from the following groups (listed alphabetically): - The Data Intelligence Lab at Ant Research for their data support - @HwVanICI for support on vLLM, LoRA, NPU integration, and more - The [Relaxed System Lab](https://github.com/Relaxed-System-Lab) at HKUST for seamless collaboration on numerous system-related aspects - The [SGLang team](https://github.com/sgl-project/sglang) for supporting custom weight update features and their contributions during AReaL-lite development - The Super Computing Technology (SCT) team at Ant Group for their expertise in large-scale cluster operations and maintenance - Special thanks to @Lyken17 for providing valuable suggestions throughout the API design process We also deeply appreciate all pioneering work from the community, particularly the [ReaLHF](https://github.com/openpsi-project/ReaLHF) project from OpenPsi Inc. and other outstanding projects, including but not limited to [DeepScaleR](https://github.com/agentica-project/deepscaler), [Open-Reasoner-Zero](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/tree/main), [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF), [VeRL](https://github.com/volcengine/verl), [SGLang](https://github.com/sgl-project/sglang), [QwQ](https://github.com/QwenLM/QwQ), [Light-R1](https://github.com/Qihoo360/Light-R1), and [DAPO](https://github.com/BytedTsinghua-SIA/DAPO). ## πŸ“œ License This project is licensed under the [Apache License 2.0](LICENSE). ## πŸ“„ Citation ```bibtex @inproceedings{mei2025real, author = {Mei, Zhiyu and Fu, Wei and Li, Kaiwei and Wang, Guangju and Zhang, Huanchen and Wu, Yi}, title = {ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation}, booktitle = {Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025}, publisher = {mlsys.org}, year = {2025}, } ``` ```bibtex @misc{fu2025areal, title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning}, author={Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu}, year={2025}, eprint={2505.24298}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2505.24298}, } ``` --- ## File: AGENTS.md # AGENTS.md -- AReaL Agent Operations Guide ## Quick reference **Tech stack**: Python 3.12+ | PyTorch | FSDP2 / Megatron / Archon | SGLang / vLLM ```bash # Environment uv sync --extra cuda # CUDA + SGLang inference (default); for vLLM: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda uv sync --extra sandbox # Daytona cloud sandbox backend (optional) source .venv/bin/activate # activate venv BEFORE pre-commit or git commit if venv exists pre-commit install --install-hooks # hooks: Ruff, clang-format, mdformat, nbstripout, conventional-commits pre-commit run --all-files # lint + format everything # Tests uv run pytest tests/test_.py # CLI docs uv run python docs/generate_cli_docs.py # Docs build (canonical, release-aligned) ./docs/build_all.sh # Do NOT use `jupyter-book build docs/en|docs/zh` directly for final preview/release, # because it skips AReaL-specific static setup and output packaging. ``` **Hard rules** -- never violate: - No wildcard imports (`from x import *`). - No hardcoded secrets, paths, or endpoints. - No skipping pre-commit hooks. - No guessing cluster configs or rebuilding CUDA/driver stacks. - Integration tests require multi-node hardware -- explain skips explicitly. **Always do**: - Read relevant files before modifying code. - Run `pre-commit run --all-files` before committing. - Follow existing code patterns in the same module. - Add tests for new functionality. - Ask for decisions and clarifications with short, structured options instead of broad open-ended questions. Use the platform's native question/clarification tool if available. **Ask first** before: - Modifying config structures in `areal/api/cli_args.py`. - Adding new dependencies. - Changing launcher or scheduler logic. - Deleting or renaming public APIs. When unsure, leave a `TODO(agent)` comment and note the constraint in your response. ______________________________________________________________________ ## Repository map ``` areal/ Core Python package |-- api/ Config dataclasses, contracts, IO structs |-- dataset/ Stateful dataset loaders (GSM8K, Geometry3K, CLEVR, ...) |-- engine/ Training backends (FSDP2, Megatron) + inference adapters |-- experimental/ Prototype engines/workflows (Archon MoE engine) |-- infra/ Launchers (Local/Ray/Slurm), schedulers, utilities |-- models/ Model adapters (Megatron-Core, Transformers, custom heads) |-- reward/ Built-in reward functions + math parsers |-- tests/ Unit/integration test suites |-- trainer/ High-level orchestrators (PPOTrainer, SFTTrainer) |-- utils/ Cross-cutting helpers (logging, data, checkpoints, RL ops) +-- workflow/ RolloutWorkflow implementations (RLVR, multi-turn, vision) docs/ Jupyter Book docs (https://areal-project.github.io/AReaL/) examples/ Training scripts and launcher recipes ``` ______________________________________________________________________ ## Code style & patterns - **Composition over inheritance** -- keep hierarchies \<= 2 levels; prefer delegation. | Type | Pattern | Example | | ---------------- | --------------- | ----------------------------------------- | | Config dataclass | `XxxConfig` | `GRPOConfig`, `FSDPEngineConfig` | | Engine class | `XxxEngine` | `FSDPEngine`, `ArchonEngine` | | Workflow class | `XxxWorkflow` | `RLVRWorkflow`, `MultiTurnWorkflow` | | Reward function | `xxx_reward_fn` | `gsm8k_reward_fn`, `geometry3k_reward_fn` | **Logging**: `areal.utils.logging.getLogger(name)` with **PascalCase** names -- never `print` or `logging.__name__`. Per-rank format: `[{Component} Rank {N}]`. Register new loggers with color in `areal/utils/logging.py`. **Performance**: - No GPU-CPU sync in hot paths (`.item()`, `.tolist()`, `print(tensor)`). - Batch ops over Python loops on tensor elements. - Explicit `dtype`/`device`; `torch.Size` assertions for shape validation. **Typing & imports**: explicit type hints; reuse `areal/api/cli_args.py` dataclasses; no wildcard imports; heavy optional deps inside functions. **Async**: rollout workflows must stay non-blocking (`await` + `aiofiles`); no sync I/O in `arun_episode`. ______________________________________________________________________ ## Domain experts & skills Fire the appropriate **expert subagent** or **load a skill** based on what you're working on. Experts are read-only consultants with deep domain knowledge; skills are step-by-step implementation guides. | Working on... | Fire subagent | Load skill | | ---------------------------- | ------------------ | ------------------- | | FSDP engine code | `fsdp-expert` | -- | | Archon engine / new model | `archon-expert` | `add-archon-model` | | Megatron engine code | `megatron-expert` | -- | | RL algorithms / PPO / GRPO | `algorithm-expert` | -- | | Launcher / scheduler / infra | `launcher-expert` | `debug-distributed` | | New reward function | -- | `add-reward` | | New dataset loader | -- | `add-dataset` | | New rollout workflow | -- | `add-workflow` | | Unit tests | -- | `add-unit-tests` | | Distributed debugging | -- | `debug-distributed` | **How to invoke experts and skills** (platform-specific): | Platform | Fire expert subagent | Load skill | | -------- | ---------------------------------------------------------------------------------- | -------------------------------------------------- | | OpenCode | `task(subagent_type="", load_skills=[], run_in_background=true, prompt="…")` | `skill(name="")` or `load_skills=[""]` | | Codex | Invoke registered subagent by canonical name (see `.codex/config.toml`) | Reference `.agents/skills//SKILL.md` | **Harness layout**: | Component | OpenCode | Codex | | ----------------- | --------------------------------------- | ------------------------------------------------------ | | Root instructions | `AGENTS.md` | `AGENTS.md` | | Agent configs | `.opencode/agents/*.md` (frontmatter) | `.codex/config.toml` + `.codex/agents/*.toml` + `*.md` | | Skills | `.opencode/skills/` + `.agents/skills/` | `.agents/skills//SKILL.md` | Directly executable workflows (both platforms): `add-workflow`, `review-pr`, `create-pr`, `translate-doc-zh`. ______________________________________________________________________ ## Core concepts **Trainer** orchestrator (`areal/trainer/`, `PPOTrainer`, `SFTTrainer`): manages the training loop, dataset loading, and workflow execution. Entry point: `examples/math/gsm8k_rl.py`. **Rollout workflows** (`areal/workflow/`, `RolloutWorkflow.arun_episode`): define how episodes are generated. Use `add-workflow` skill for step-by-step guide. **Engines**: *Inference engines* handle async generation via `engine.agenerate()` and manage weight updates. *Training engines* consume rollout tensors, compute PPO/GRPO updates, and broadcast weight versions (FSDP2, Megatron, or Archon). **Weight versioning**: async workflows require version alignment via `WeightUpdateMeta` (`areal/api/engine_api.py`). Critical for correctness across distributed training. **Observability**: emit metrics via `stats_tracker.get()`, persist artifacts under `dump_dir`, checkpoint via `areal/utils/saver.py` / `recover.py`. **Launcher / scheduler**: training requires cluster setup (local / Ray / Slurm) via configs in `areal/infra/launcher/`. See `launcher-expert` for deployment guidance. ______________________________________________________________________ ## API & config rules *Applies to: `areal/api/**`* - **Field ordering**: required -> common optional -> rare optional -> internal (`_` prefix). - **Validation**: `__post_init__` with `ValueError` and clear message. - **Backward compat**: add fields with defaults; deprecate before removing; avoid type changes. - **CLI**: use `Literal` for enum choices; all public configs need docstrings with constraints. ______________________________________________________________________ ## Distributed code rules *Applies to: `areal/engine/**`, `areal/experimental/**`* - Never create global process groups at module level; always pass `process_group` explicitly. - `dist.get_rank(group)` not `dist.get_rank()` when group matters. - DeviceMesh dimensions must match `ArchonParallelDims`: `dp_shard`, `tp`, `cp`, `ep`, `etp`. - All-reduce: all ranks must call. Broadcast: explicit `src`. Barrier: debugging only. | Issue | Cause | Fix | | ------------- | -------------------------------- | ------------------------- | | Hang | Mismatched collective calls | All ranks call same op | | Wrong results | Incorrect `ReduceOp` | Check SUM vs MEAN | | OOM | Unsharded tensor on wrong device | Verify DTensor placements | Debug env vars: `TORCH_DISTRIBUTED_DEBUG=DETAIL`, `NCCL_DEBUG=INFO`, `CUDA_LAUNCH_BLOCKING=1`. See the `debug-distributed` skill for the full workflow. ______________________________________________________________________ ## Testing rules *Applies to: `**/tests/**`, `test_*.py`* | Marker | When | | --------------------------------------- | -------------------------------- | | `@pytest.mark.slow` | > 10s (excluded from default CI) | | `@pytest.mark.slow` + `@pytest.mark.ci` | Slow but must run in CI | | `@pytest.mark.asyncio` | Async tests | - Naming: `test___()` with Arrange/Act/Assert. - GPU: skip gracefully (`@pytest.mark.skipif(not CUDA_AVAILABLE, reason="...")`). - Distributed mocking: `torch.distributed.fake_pg`; don't mock FSDP/DTensor internals. - Assertions: `torch.testing.assert_close()` with explicit `rtol`/`atol`; prefer `tmp_path`, `monkeypatch`. | Suite | Command | GPU | | ----------- | ----------------------------- | --------- | | Unit | `pytest tests/test_*.py` | No | | GRPO | `pytest tests/grpo/` | Yes | | FSDP | `pytest tests/test_fsdp_*.py` | Yes | | Distributed | `pytest tests/torchrun/` | Multi-GPU | ______________________________________________________________________ ## Collaboration & review - **Branches**: kebab-case (`feature/multi-turn-metrics`, `bugfix/fsdp-weight-sync`). - **Commits**: Conventional Commits (e.g., `feat:`, `fix:`, `docs:`, `gov:`), ~72 char subject, imperative voice. Squash WIP before PR. - **Pre-merge**: full pre-commit stack; doc-only edits need at least `mdformat --check`. - **PRs**: tie to issue, highlight risk areas, list test commands executed, note skipped suites with reasons. | Skill | Purpose | | -------------------- | ------------------------------------------------------ | | `create-pr` | Rebase, squash, and create or update a PR | | `commit-conventions` | Commit message conventions to load before `git commit` | | `review-pr` | Dynamic PR review with targeted expert consultation | | `translate-doc-zh` | Translate English docs to Chinese | ______________________________________________________________________ ## Reference material - **Docs portal**: - **Quickstart**: `docs/tutorial/quickstart.md` - **Architecture**: `docs/tutorial/gsm8k_grpo.md` - **Customization**: `docs/customization/*.md` - **Algorithms**: `docs/algorithms/*.md` - **Best practices**: `docs/best_practices/*.md` - **CLI reference**: `docs/cli_reference.md` - **Agent workflow**: `docs/customization/agent.md` --- ## File: CLAUDE.md # CLAUDE.md - AReaL ## WHAT: Project Overview AReaL is a distributed RL training framework for LLM alignment via reinforcement learning. **Tech Stack**: Python 3.12+ | PyTorch | FSDP2/Megatron | SGLang/vLLM **Core Directories**: - `areal/` - Core package - `api/` - Config dataclasses, workflow/engine contracts - `engine/` - FSDP2, Megatron, SGLang/vLLM adapters - `fsdp_utils/` - FSDP2-specific utilities (checkpoint, grad, optimizer, parallel) - `megatron_utils/` - Megatron/FP8 utilities (checkpoint, pipeline, quantization) - `core/` - Engine-shared utilities (distributed, lock, model, offload) - `infra/` - Infrastructure (launcher, scheduler, RPC) - `utils/` - Infrastructure utilities (launcher, proc, http, concurrent, slurm, ray) - `workflow/` - RolloutWorkflow implementations - `reward/` - Reward functions - `dataset/` - Dataset loaders - `utils/` - Cross-cutting utilities (logging, data, checkpoints, network, RL functional) - `examples/` - Training scripts and configs - `docs/` - Jupyter Book source ## WHY: Purpose - Enable efficient RL training for LLM alignment at scale - Async rollout + distributed training for high throughput - Modular design: workflows, engines, rewards, and datasets are independently extensible ## HOW: Core Commands ```bash # Check environment python --version # Requires 3.12+ uv --version # Install: https://docs.astral.sh/uv/ # Sync dependencies uv sync --extra cuda # CUDA + SGLang inference (default) # For vLLM: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda uv sync --group dev # Include dev/test packages uv run python3 areal/tools/validate_installation.py # Validate installation # Pre-commit hooks pre-commit install --install-hooks # Set up hooks (run once) pre-commit run --all-files # Format and lint # Run tests # First check GPU availability (many tests require GPU) python -c "import torch; print('GPU available:', torch.cuda.is_available())" uv run pytest tests/test_.py # Generate CLI docs uv run python docs/generate_cli_docs.py # Build docs (canonical, release-aligned) ./docs/build_all.sh # Do NOT use `jupyter-book build docs/en|docs/zh` directly for final preview/release, # because it skips AReaL-specific static setup and output packaging. ``` ## Boundaries ### Constraints - Designed for distributed GPU clusters; assume containerized execution - Integration tests require multi-node hardware; explain skips when unavailable - Secrets and endpoints are managed outside the repo ### Always Do - Read relevant files before modifying code - Run `pre-commit run --all-files` before committing - Follow existing code patterns in the same module - Add tests for new functionality ### Ask First - Modifying config structures in `areal/api/cli_args.py` - Adding new dependencies - Changing launcher or scheduler logic - Deleting or renaming public APIs - Running GPU/distributed tests (check GPU first: `python -c "import torch; print('GPU available:', torch.cuda.is_available())"`) ### Never Do - Hardcode secrets, paths, or endpoints - Skip pre-commit hooks - Guess cluster configs or rebuild CUDA/driver stacks - Use wildcard imports (`from x import *`) ## Progressive Disclosure: Detailed Guides | Task | Reference | | ---------------------- | ------------------------------------------------------------- | | Add Workflow | `docs/customization/agent.md`, `areal/workflow/multi_turn.py` | | Add Dataset | `docs/customization/`, `areal/dataset/gsm8k.py` | | Add Reward | `areal/api/reward_api.py`, `areal/reward/geometry3k.py` | | Add Archon Model | `areal/experimental/models/archon/qwen2/`, `qwen3/` | | Algorithm Details | `docs/algorithms/*.md` | | R3 Router Replay | `r3/README.md` | | Quickstart | `docs/tutorial/quickstart.md` | | Architecture Deep Dive | `docs/tutorial/gsm8k_grpo.md` | | CLI Reference | `docs/cli_reference.md` | ## Git Workflow - **Commits**: Conventional Commits (e.g., `feat:`, `fix:`, `docs:`, `gov:`), ~72 chars subject, imperative voice, reasoning in body - **Squash**: Squash WIP commits before opening PR - **PR requirements**: Run pre-commit, document test coverage, note hardware limitations ## Extended Configuration See `.claude/agents/`, `.claude/skills/`, `.claude/commands/`, and `.claude/rules/` for specialized instructions. ### Agents | Agent | Purpose | Activation Trigger | | --------------------------- | ----------------------------------------- | ------------------------------------------------------------------- | | `planner` | Implementation planning | Before multi-file changes, new features, or architectural decisions | | `simple-code-reviewer` | Quick code quality checks | After code changes, before committing | | `code-verifier` | Formatting/linting/tests | After code changes, before committing | | `fsdp-engine-expert` | FSDPEngine implementation | FSDPEngine code changes or questions | | `archon-engine-expert` | ArchonEngine implementation | ArchonEngine code changes or questions | | `megatron-engine-expert` | MegatronEngine implementation | MegatronEngine code changes or questions | | `algorithm-expert` | RL algorithms | GRPO/PPO/DAPO questions | | `launcher-scheduler-expert` | Cluster launching and resource scheduling | Launcher/scheduler code changes or configuration questions | **Stage-by-Stage Agent Guidance**: 1. **Planning Stage** (Before coding): Use `planner` for architecture design and implementation planning 1. **Code Formatting & Linting** (After coding): Use `code-verifier` to automatically run formatting, linting, and tests, catching syntax errors and style issues quickly 1. **Code Quality Check** (After formatting): Use `simple-code-reviewer` for quick code quality checks, focusing on logic issues and code smells ### Skills (Guided Development Workflows) Skills provide step-by-step guides for common development tasks: - `/add-dataset` - Dataset loader creation guide - `/add-workflow` - Workflow implementation guide - `/add-reward` - Reward function guide - `/add-archon-model` - Archon engine model architecture guide - `/debug-distributed` - Distributed debugging guide - `/add-unit-tests` - Test development guide (NEW) ### Commands (User-invoked Actions) Commands perform specific actions when invoked: - `/create-pr` - Rebase, squash commits, and create/update PR with intelligent messages - `/gen-commit-msg` - Generate commit messages from staged changes - `/review-pr` - Intelligent PR code review with dynamic agent allocation - `/translate-doc-zh` - Translate English documentation to Chinese ### Rules (Code Quality Standards) Project-wide standards enforced across all code changes: - `api-config.md` - Configuration dataclass design patterns - `code-style.md` - Coding conventions beyond pre-commit hooks - `distributed.md` - Distributed training patterns and constraints - `testing.md` - Testing strategy and coverage requirements --- ## File: CODE_OF_CONDUCT.md The latest version of this code of conduct can be found at https://github.com/areal-project/community/blob/main/CODE_OF_CONDUCT.md --- ## File: CONTRIBUTING.md The latest version of this file can be found at https://github.com/areal-project/community/blob/main/CONTRIBUTING.md --- ## File: GOVERNANCE.md The latest version of this file can be found at https://github.com/areal-project/community/blob/main/GOVERNANCE.md --- ## File: LEGAL.md The latest version of this file can be found at https://github.com/areal-project/community/blob/main/LEGAL.md --- ## File: ROADMAP.md # AReaL Roadmap This roadmap outlines the planned features and improvements for AReaL in the next quarter. We welcome community feedback and contributions to help shape the future direction of the project. **Latest Release:** Check [releases](https://github.com/areal-project/AReaL/releases) for the most recent version. ## 2026 H2 Roadmap (due December 31, 2026) [GitHub Issue #1381](https://github.com/areal-project/AReaL/issues/1381). This roadmap tracks major planned enhancements for the second half of 2026. Items are organized into two categories: - **On-going:** Features currently under active development by the core AReaL team - **Planned but not in progress:** Features that are good to have but currently lacking bandwidth We use `[CC]` to mark items suitable for community contributions. If you're interested in contributing, please reach out to discuss implementation details. ### Backends **On-going** - [ ] Support training the latest large MoE models with the Megatron backend (including Kimi 2.5 / GLM 5 / DeepSeek V3 / Qwen 3.6, etc.) ([#1372](https://github.com/areal-project/AReaL/pull/1372), [#1373](https://github.com/areal-project/AReaL/pull/1373)) - [ ] More distribution strategies for colocated deployment **Planned but not in progress** - [ ] Omini model RL support with FSDP backend - [ ] Proxy server support via RayScheduler (HTTP launcher actor) `[CC]` - [ ] Weight delta update with `awex` - [ ] Memory service for self-evolving agent ### Usability **On-going** - [ ] Effective AReaL CLI support ([#1374](https://github.com/areal-project/AReaL/issues/1374)) - [ ] Profiling toolkit (distributed trace) - [ ] Online RL training example based on AReaL 2.0 architecture **Planned but not in progress** - [ ] More RL post-training paradigms: IcePop Plus algorithm (verified in Ling MoE training) and stepwise reward rubrics - [ ] GUI Agent VLM training example - [ ] OS-bench VLM training example `[CC]` - [ ] Diffusion image/video generation model RL post-training `[CC]` - [ ] Comprehensive LoRA support over the existing policy model training - [ ] AReaL autopilot (automatic performance-optimized RL deployment suggestions and systematic bottleneck analysis) ### Documentation **Planned but not in progress** - [ ] Explain benchmarking results from nightly CI and how to extract issues ## 2026 Q2 Roadmap (due July 31, 2026) [GitHub Issue #1302](https://github.com/areal-project/AReaL/issues/1302). This roadmap tracks major planned enhancements through July 31, 2026. Items are organized into two categories: - **On-going:** Features currently under active development by the core AReaL team - **Planned but not in progress:** Features that are good to have but currently lacking bandwidth We use `[CC]` to mark items suitable for community contributions. If you're interested in contributing, please reach out to discuss implementation details. ### Backends **On-going** - [ ] Full training example with AReaL 2.0 architecture - [ ] Deprecate support for SPMD mode (launcher, sglang/vllm server, etc.) - [ ] Initial support of colocation weight transfer with `awex` **Planned but not in progress** - [ ] Migrate primary Megatron integration lib from `mbridge` to `megatron-bridge` ([#1260](https://github.com/areal-project/AReaL/issues/1260)) `[CC]` - [ ] Full support for colocation/separation weight transfer with `awex` as the backend (`areal/v2/weight_update/`) `[CC]` - [ ] Migrate legacy NCCL broadcast weight transfer approach from `areal/engine` into `areal/v2/weight_update/` `[CC]` - [ ] Omini model RL support with FSDP backend ([#879](https://github.com/areal-project/AReaL/issues/879)) `[CC]` - [ ] Support training the latest large MoE models with the Megatron backend, including dpsk-v3/v4, Kimi-2.5, GLM-4/5 `[CC]` - [ ] Native Kubernetes (K8S) scheduler `[CC]` ### Usability **On-going** - [ ] Nightly CI workflow for performance benchmarking ([#1284](https://github.com/areal-project/AReaL/issues/1284)) - [ ] Refactor unit tests for faster execution **Planned but not in progress** - [ ] OS-bench VLM training example `[CC]` - [ ] Multi-agent training example (single LLM, different prompts, e.g., planner agent with sub-agents) `[CC]` - [ ] Migrate legacy multi-turn agent examples to new API (`agenerate` β†’ `ArealOpenAI` or URL-based `AgentWorkflow`) `[CC]` - [ ] Publish PyPI packages and CLI for running experiments - [ ] Support distributed training and debugging in Jupyter notebooks - [ ] Implement controller construction with model-centric API similar to `transformers` ### Documentation **Planned but not in progress** - [ ] Explain benchmarking results from nightly CI and how to extract issues - [ ] Document AReaL 2.0 architecture ## 2026 Q1 Roadmap (due April 30, 2026) [GitHub Issue #907](https://github.com/areal-project/AReaL/issues/907). This roadmap tracks major planned enhancements through April 30, 2026. Items are organized into two categories: - **On-going:** Features currently under active development by the core AReaL team - **Planned but not in progress:** Features with concrete implementation plans where we welcome community contributions ### Backends **On-going** - [ ] ZBPP & ZBPP-V support for the Archon backend - [ ] FP8 training for Archon **Planned but not in progress** - [ ] Support for agentic training with large VLM MoE models (Archon backend) - [ ] Omini model RL support with FSDP/Archon backend - [ ] Decoupling agent service from the inference service - [ ] Online RL training with the proxy server - [ ] LoRA support for the Archon backend - [ ] Colocation mode with `awex` as the weight sync engine - [ ] Multi-LLM training (different agents with different parameters) - [ ] Auto-scaling inference engines in single-controller mode - [ ] Elastic weight update setup and acceleration - [ ] RL training with cross-node vLLM pipeline/context parallelism ### Usability **On-going** - [ ] Flatten the import structure of areal modules **Planned but not in progress** - [ ] Publishing PyPI packages - [ ] Support distributed training and debugging in Jupyter notebooks - [ ] Example of using a generative or critic-like reward model - [ ] Support directly constructing inference/training engines without config objects - [ ] Add router in rollout controller for simpler proxy server usage - [ ] Integrate `aenvironment` for environment handling ### Documentation **Planned but not in progress** - [ ] Use case guides: multi-agent training - [ ] Guide for online proxy mode training ## Historical Roadmaps ### 2025 Q4 [GitHub Issue #542](https://github.com/areal-project/AReaL/issues/542). **Backends** Completed: - Single-controller mode - Detailed profiling for optimal performance across different scales - Low-precision RL training (Megatron FP8) - Data transfer optimization in single-controller mode - New PyTorch-native backend: Archon Carried over to Q1 2026: - Multi-LLM training (different agents with different parameters) - Auto-scaling inference engines in single-controller mode - Elastic weight update setup and acceleration - RL training with cross-node vLLM pipeline/context parallelism **Usability** Completed: - Add CI pipeline to build Docker images upon release - Wrap training scripts into trainers - Refactor FSDP/Megatron engine/controller APIs to finer granularity - Fully respect allocation mode in trainers/training scripts Carried over to Q1 2026: - Flatten the import structure of areal modules - Support distributed training and debugging in Jupyter notebooks - Example of using a generative or critic-like reward model Canceled: - Rename `RemoteSGLang/vLLMEngine` as `SGLang/vLLMEngine` **Documentation** Completed: - Tutorial on how to write efficient async rollout workflows - Benchmarking and profiling guide - Use case guides: offline inference, offline evaluation - AReaL performance tuning guide - Device allocation strategies for training and inference - Parallelism strategy configuration for training and inference Carried over to Q1 2026: - Use case guides: multi-agent training ### 2025 Q3 [GitHub Issue #257](https://github.com/areal-project/AReaL/issues/257). **Backends** Completed: - Megatron training backend support - SGLang large expert parallelism (EP) inference support - Remote vLLM inference engine - Ulysses context parallelism & tensor parallelism for FSDP backend - End-to-end MoE RL training with large EP inference and Megatron expert parallelism - Distributed weight resharder for Megatron training backend Canceled: - Local SGLang inference engine with inference/training colocation (hybrid engine) - RL training with SGLang pipeline parallelism **Usability** Completed: - OpenAI-compatible client support - Support RLOO - Provide benchmarking configuration examples: - DAPO - Bradley-Terry reward modeling - PPO with critic models - REINFORCE++ **Documentation** Completed: - OpenAI-compatible client documentation - Out-of-memory (OOM) troubleshooting guide - AReaL debugging best practices: - LLM server-only debugging - How to launch LLM servers independently and debug agent workflows - Mock data and torchrun debugging - Creating synthetic data and using `torchrun` for algorithm debugging - Training-free evaluation experiments - Running evaluations without training or additional GPUs ## How to Influence the Roadmap We value community input! Here's how you can help shape AReaL's future: ### πŸ’‘ Propose New Features 1. **Check Existing Issues:** Search [issues](https://github.com/areal-project/AReaL/issues) and [discussions](https://github.com/areal-project/AReaL/discussions) to see if your idea already exists 1. **Create a Feature Request:** Use our [feature request template](https://github.com/areal-project/AReaL/issues/new?template=feature.md) 1. **Discuss in GitHub Discussions:** Post in [Ideas category](https://github.com/areal-project/AReaL/discussions/categories/ideas) for early feedback 1. **Vote on Features:** Use πŸ‘ reactions on issues to show support ### πŸ› οΈ Contribute Implementation Check our [contribution guide](CONTRIBUTING.md). ## Release Cycle **Minor Releases:** Bi-weekly - Bug fixes, small improvements, and new features **Major Releases:** Quarterly - Important milestones and significant changes ## Historical Milestones Check [our historical milestone summaries since open-source](docs/en/version_history.md). ## Long-Term Vision Our vision for AReaL is to become the **go-to framework for training reasoning and agentic AI systems** that is: 1. **Accessible:** Easy to get started, whether you're a researcher or practitioner 1. **Scalable:** Scales from laptop to 1000+ GPU clusters seamlessly 1. **Flexible:** Supports diverse algorithms, models, and use cases 1. **Performant:** Industry-leading training speed and efficiency 1. **Open:** Fully open-source with transparent development ______________________________________________________________________ **Last Updated:** 2026-06-05 **Questions about the roadmap?** Open a discussion in [GitHub Discussions](https://github.com/areal-project/AReaL/discussions) or ask in our [WeChat group](./assets/figures/wechat_qrcode.png). --- ## File: .pre-commit-config.yaml default_install_hook_types: - pre-commit - commit-msg repos: - repo: https://github.com/pre-commit/mirrors-clang-format rev: v19.1.7 hooks: - id: clang-format files: \.(c|cc|cxx|cpp|h|hpp|hxx|cu|cuh)$ args: - --style=file - --fallback-style=Google - repo: https://github.com/pre-commit/pre-commit-hooks rev: v6.0.0 hooks: - id: check-yaml files: \.(yaml|yml)$ exclude: \.github/workflows/ - id: end-of-file-fixer files: \.(py|md|yaml|yml|c|cc|cxx|cpp|h|hpp|hxx|cu|cuh|ipynb)$ - id: trailing-whitespace files: \.(py|md|yaml|yml|c|cc|cxx|cpp|h|hpp|hxx|cu|cuh|ipynb)$ - id: check-added-large-files name: Check for large files args: ['--maxkb=1000'] exclude: uv.lock - id: check-json name: Check JSON files - id: detect-private-key name: Detect private keys - repo: https://github.com/executablebooks/mdformat rev: 0.7.17 hooks: - id: mdformat args: - --wrap=88 additional_dependencies: - mdformat-gfm - mdformat-tables - mdformat-frontmatter exclude: ^(docs/(en|zh)/algorithms/.*\.md|docs/(en|zh)/best_practices/algo_perf\.md)$ - repo: https://github.com/astral-sh/ruff-pre-commit # Ruff version. rev: v0.14.9 hooks: - id: ruff # Run the linter. name: Run Linter Check (Ruff) types_or: [ python, pyi, jupyter ] args: [ --fix ] - id: ruff-format # Run the formatter. name: Run Formatter (Ruff) types_or: [ python, pyi, jupyter ] # Clean notebook outputs and metadata - repo: https://github.com/kynan/nbstripout rev: 0.7.1 hooks: - id: nbstripout name: nbstripout - Strip notebook output # Check consistency between pyproject.toml variants (sglang vs vllm) - repo: local hooks: - id: check-pyproject-consistency name: Check pyproject.toml consistency entry: python3 areal/tools/check_pyproject_consistency.py language: system files: ^pyproject(\.vllm)?\.toml$ pass_filenames: false always_run: false require_serial: true # Regenerate uv lockfiles when pyproject files change - repo: local hooks: - id: uv-lock name: Regenerate uv lockfiles entry: bash scripts/uv_lock.sh language: system files: ^pyproject(\.vllm)?\.toml$ pass_filenames: false always_run: false require_serial: true # Generate CLI documentation - repo: local hooks: - id: generate-cli-docs name: Generate CLI documentation entry: python docs/generate_cli_docs.py language: system files: ^(areal/api/cli_args\.py|docs/generate_cli_docs\.py)$ pass_filenames: false always_run: false require_serial: true # Ensure SPDX license header in Python source files - repo: local hooks: - id: check-license-header name: Check SPDX license header entry: python3 areal/tools/check_license_header.py language: system files: ^areal/.*\.py$ types: [python] # Format and lint .github/CODEOWNERS - repo: local hooks: - id: format-codeowners name: Format CODEOWNERS entry: python3 areal/tools/format_codeowners.py language: system files: ^\.github/CODEOWNERS$ pass_filenames: false always_run: false require_serial: true # Conventional Commits message check - repo: https://github.com/compilerla/conventional-pre-commit rev: v4.4.0 hooks: - id: conventional-pre-commit stages: [commit-msg] args: - feat - fix - docs - gov - style - refactor - perf - test - build - ci - chore - revert --- ## File: areal/v2/agent_service/README.md [Binary file] --- ## File: assets/community/README.md This file has been moved to the [community repository](https://github.com/areal-project/community), and is no longer maintained here. Please refer to [the new location](https://github.com/areal-project/community/tree/main/meeting) for the latest information and updates regarding the community. --- ## File: benchmark/README.md # Benchmark This directory contains guides for reproducing AReaL's throughput and algorithmic performance benchmarks, as reported in the paper. As of January 2026, AReaL has been fully refactored from the original `realhf` directory to the new `areal` directory. The legacy configurations and instructions are no longer compatible with the current codebase. For reference, you can still access the legacy configs and run experiments within `realhf` by reverting to commit `820ca49` or earlier. Updated benchmark results for the new codebase will be available soon. --- ## File: blog/AReaL_v0_1.md

*AReaL* v0.1: Reliable RL Training of 1.5B and 7B LLMs on Math Reasoning Tasks

Our initial release AReaL v0.1 includes reproducible experiments based on our AReaL system for 1.5B and 7B LRMs, validated across diverse computational budgets. We demonstrate that with AReaL, users can: 1. **Reliably train a 1.5B distilled model** with RL to **surpass o1-Preview on math reasoning within 40 hours** πŸš€ 1. **Reliably perform an R1-Zero-style experiment with a 7B model** by running RL training on Qwen2.5-7B, and **observe emergent thinking tokens and continuous model improvement on math reasoning** ______________________________________________________________________ ## Training a 1.5B LRM from the Distilled Model Starting with the [`DeepSeek-R1-Distill-Qwen-1.5B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) model, we follow DeepScaleR to iteratively increase the output context length from 8K to 16K and eventually 24K. The training reward continuously increases during RL training. We observe that the response length first **shrinks in the 8K training stage**, then **increases in the 16K and 24K training stages**. Our experiments are conducted on 16 nodes, each equipped with 8 H800 GPUs. The results, along with the associated training curves, are presented below. *Figure 1. Training rewards and response lengths during RL training. The base model is DeepSeek-R1-Distill-Qwen-1.5B. Curves are averaged with a window size of 25.* These three stages progressively enhanced the performance of the base model, as demonstrated below: | Model | MATH500 | AIME 2024 | AMC 2023 | Download | | ----------------------------- | ------- | --------- | -------- | ------------------------------------------------------------------------------- | | o1-Preview | 81.4 | 40.0 | - | - | | DeepSeek-R1-Distill-Qwen-1.5B | 82.8 | 28.8 | 62.9 | - | | DeepScaleR (Official) | 87.8 | 43.1 | 73.6 | - | | AReaL Stage 1: 8K (Ours) | 85.7 | 33.2 | 74.7 | [πŸ€— HuggingFace](https://huggingface.co/inclusionAI/AReaL-1.5B-Preview-Stage-1) | | AReaL Stage 2: 16K (Ours) | 87.4 | 34.2 | 79.6 | [πŸ€— HuggingFace](https://huggingface.co/inclusionAI/AReaL-1.5B-Preview-Stage-2) | | AReaL Stage 3: 24K (Ours) | 88.0 | 40.2 | 81.2 | [πŸ€— HuggingFace](https://huggingface.co/inclusionAI/AReaL-1.5B-Preview-Stage-3) | *Table 1. Evaluation on competition-level mathematics benchmarks including AIME 2024, AMC 2023, and MATH-500. Results report Pass@1 accuracy averaged over 32 samples per problem at temperature 0.6.* ### RL Training Details Our RL training process incorporates insights from [DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1), [DeepScaleR](https://github.com/agentica-project/deepscaler), and [ReaLHF](https://github.com/openpsi-project/ReaLHF). Key components include: - **Base Model**: [`DeepSeek-R1-Distill-Qwen-1.5B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) - **Dataset**: 40k high-quality mathematical reasoning tasks from [DeepScaleR](https://huggingface.co/datasets/agentica-org/DeepScaleR-Preview-Dataset) - **Reward Design**: +5 for correct responses, -5 for incorrect responses (evaluated using Sympy following [Qwen's protocol](https://github.com/QwenLM/Qwen2.5-Math/tree/main/evaluation)) - **RL Algorithm**: Simplified PPO variant without the critic network - **Key Hyperparameters**: - Discount factor (Ξ³) and GAE parameter (Ξ») both set to 1 (following [Open-Reasoner-Zero](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero)) - Small KL reward coefficient - **Iterative Context Lengthening**: Three-stage training (8Kβ†’16Kβ†’24K) following [DeepScaleR](https://github.com/agentica-project/deepscaler) - **Batch Size**: 1024 questions per step with 8 rollouts per question, which is **larger** than prior works ### AReaL with Various Computational Resources AReaL adapts efficiently to different computational budgets. Increased resources accelerate RL training, significantly boosting research progress. We provide detailed hardware requirements and environment setup guides for different configurations in [our tutorials](/examples/README.md). *Figure 2. Total RL training time for 10 epochs across different resource configurations.* ## Qwen2.5-7B-Zero RL Training We conduct RL training starting from Qwen2.5-7B using DeepSeek-R1-Zero style training. Initial results using data from [Open-Reasoner-Zero](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero) show both rewards and response lengths gradually increasing during training. This **simultaneous growth of response lengths and rewards** suggests **emergent deep thinking capabilities** in solving complex reasoning problems. *Figure 3. Qwen2.5-7B-Zero RL training curve* Evaluation of intermediate checkpoints on MATH500 and AIME24 datasets shows continuous improvement in both accuracy and response length: *Figure 4. Test accuracy and response length on MATH500 and AIME24 datasets* Additional experiments on the [DeepScaleR](https://github.com/agentica-project/deepscaler) dataset show similar trends: | Model | MATH-500 | AIME 2024 | AMC 2023 | | -------------------------------------- | -------- | --------- | -------- | | Qwen2.5-7B | 34.0 | 2.0 | 17.8 | | Open-Reasoner-Zero-7B (Step 150) | ~77 | ~15 | - | | Qwen2.5-7B-Zero (Step 150, DeepScaleR) | 75.9 | 13.9 | 56.1 | | Qwen2.5-7B-Zero (Step 150, ORZ) | 77.3 | 13.9 | 57.1 | | Qwen2.5-7B-Zero (Step 200, ORZ) | 78.0 | 14.5 | 58.7 | *Table 2. Evaluation on AIME 2024, AMC 2023, and MATH-500 (Pass@1 accuracy averaged over 32 samples at temperature 1.0).* ## Open Research Questions on Algorithm and Model Based on AReaL, our team is also actively exploring the research frontier of LRMs. We highlight some of the major research topics we are working on. - **Effective Training Datasets for LRMs** How do we design training tasks that enable LRMs to progressively evolve sophisticated reasoning skills via RL? - For more advanced distilled models, such as DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Qwen-32B, what kind of task distribution could drive further enhancements? - How to properly curate the distillation dataset for cold-start to achieve the best RL results? - **Boosting Reasoning with Auxiliary Training Signals.** Can auxiliary signals - such as Process Reward Models (PRMs) and expert-curated labels - be adopted in RL training to refine the reasoning processes of LRMs? - **Adaptive Response Lengths: Quality Over Quantity.** How can RL training schemes teach LRMs to dynamically adjust response complexity based on task difficulty? The goal is to eliminate redundant "over-reasoning" for simpler inputs while preserving rigor for challenging problems. - **Sample Efficient RL for LRMs** In addition to system-level acceleration, can we further speed up the LRM training process with a more sample-efficient advanced RL algorithm? Can we get the critic back to reduce gradient variance without compromising the computation efficiency? How to better encourage exploration? --- ## File: blog/AReaL_v0_2.md

*AReaL* v0.2: Training a SOTA 7B LRM with 1.5x Throughput Improvement

## Introduction We are excited to release AReaL v0.2 (boba), featuring three major milestones: - **SGLang Support**: With the addition of SGlang support and a series of engineering optimizations, AReaL v0.2 achieves a speed improvement of 1.5x over AReaL v0.1 on 7B models. - **SOTA 7B Model**: AReaL's RL training becomes more stable and sample-efficient. We obtain a SOTA 7B model in mathematical reasoning, which achieves pass@1 score of 61.9 on aime24 and 48.3 on aime25 respectively. - **Competitive 32B Model**: We train a highly competitive 32B model at an extremely low cost, achieving results comparable to QwQ-32B using only 200 data samples. | **Model (7B)** | **AIME 2024** | **AIME 2025** | **GPQA-Diamond** | | :----------------------------------------------------------------------------: | :-----------: | :-----------: | :--------------: | | R1-Distill-Qwen-7B | 55.0 | 39.7 | 47.1 | | Light-R1-7B-DS | 56.7 | 44.9 | 40.9 | | [AReaL-boba-RL-7B πŸ€—](https://huggingface.co/inclusionAI/AReaL-boba-RL-7B) | **61.9** | **48.3** | **47.6** | | **Model (32B)** | **AIME 2024** | **AIME 2025** | **GPQA-Diamond** | | R1-Distill-Qwen-32B | 72.6 | 54.9 | 63.2 | | QwQ-32B | 78.9 | 70.2 | 64.6 | | Light-R1-32B-DS | 76.2 | 67.8 | 63.5 | | [AReaL-boba-SFT-32B πŸ€—](https://huggingface.co/inclusionAI/AReaL-boba-SFT-32B) | 78.8 | 62.1 | 60.1 | *Table 1: The performance of AReaL-boba-RL-7B and AReaL-boba-SFT-32B. We obtain SOTA 7B model using RL on math reasoning. Although our dataset primarily consists of math and logic problems, we observed that RL training led to measurable improvements on the challenging STEM benchmark GPQA. Additionally, We train a highly competitive 32B model using only 200 data samples, replicating **QwQ-32B's** inference performance on AIME 2024.* We declare that the reported numbers are based on our re-testing, with each number representing the average results of 32 sampling responses. The evaluation code was available in the `evaluation/` folder. Please roll back to [commit f55fe68](https://github.com/areal-project/AReaL/commit/f55fe685c661b0117d888fcae47d6f46e9a85087) (Feb 4, 2026) or earlier. For the baselines and the SFT model AReaL-boba-SFT-32B, we follow the recommended configuration (temperature = 0.6, top_p = 0.95, suggested by DeepSeek) and use the default R1-Distill-Qwen template for prompts. For the RL-trained models, we retaine the same temperature as during RL rollout (1.0). Notably, during GPQA testing, we found that all answers are "A." To address this bias, we randomized the answer options. ### Training Speed Comparison AReaL v0.2.0 features the following system optimizations: - **Upgraded Generation Backend: vLLM 0.6.3 β†’ SGLang v0.4.0** The generation backend has been upgraded from vLLM 0.6.3 to SGLang v0.4.0, leveraging its radix attention mechanism to significantly improve throughput in scenarios where multiple responses are sampled from the same prompt. Additionally, SGLang automatically flushes radix caches upon weight updates, ensuring correctness in on-policy reinforcement learning (RL). We will continue tracking advancements in the community to integrate further optimizations. - **Optimized Training for Variable-Length Sequences & Large Batches** To handle variable sequence lengths efficiently, we eliminate padding and instead pack sequences into 1D tensors. A dynamic allocation algorithm (approximately) optimally distributes sequences under a maximum token budget, balancing micro-batch sizes while minimizing the number of micro-batches. This approach maximizes GPU memory utilization, enabling efficient computation across a large batch of variable-length inputs. - **High-Performance Data Transfer for 1K-GPU Scaling** AReaL employs NCCL with GPU-Direct RDMA (GDRDMA) over InfiniBand/RoCE, enabling direct GPU-to-GPU communication that bypasses costly CPU-mediated transfers and PCIe bottlenecks. Compared to traditional Ethernet-based approaches, this reduces latency and increases throughput, keeping generation-to-training data transfer overhead below 3 seconds even in a 1,000-GPU cluster. ## Training Recipe ### SOTA 7B model using RL on math reasoning #### Base Model We use [R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) as our foundation model. #### Dataset Curation Our training dataset ([AReaL-boba-106k](https://huggingface.co/datasets/inclusionAI/AReaL-RL-Data/blob/main/data/boba_106k_0319.jsonl)) combines resources from multiple open-source projects: - [DeepScaleR](https://github.com/agentica-project/deepscaler) - [Open-Reasoner-Zero](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/) - [Light-R1](https://github.com/Qihoo360/Light-R1) - [DAPO](https://github.com/BytedTsinghua-SIA/DAPO) We enhanced this with challenging problems from: - [NuminaMath](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT) (AoPS/Olympiad subsets) - [ZebraLogic](https://hf-mirror.com/datasets/WildEval/ZebraLogic) To maintain an appropriate difficulty level, we filtered out overly simple questions. Specifically, We generate 8 solutions per question using DeepSeek-R1-Distill-Qwen-7B and filter out questions where all solutions were correct. #### Reward Function We adopt a sparse sequence-level reward mechanism. The model is instructed to enclose the final answer within \\boxed{}, and the boxed answer is then verified. Correct responses receive a reward of +5, while incorrect ones are penalized with -5. Notably, we observe that the KL reward can impair performance, particularly in long chain-of-thought training, so we set it to zero. #### RL Algorithm We employ Proximal Policy Optimization (PPO) as our training algorithm. We remove the critic model to save compute. We set both the discount factor Ξ³ and the GAE parameter Ξ» to 1. Such practices are also adopted by the [Open-Reasoner-Zero](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero) project. #### Token-Level Loss Normalization Averaging the loss in the sequence level can underweight the overall contribution of longer texts. To address this, we normalize the loss at the token level, this practice is also highlighted in [DAPO](https://github.com/BytedTsinghua-SIA/DAPO). #### Rollout Strategy During the rollout phase, we sample 512 questions per batch, and the LLM generates 16 responses per questionβ€”resulting in a total batch size of 8,192. To minimize output truncation, we set the maximum generation length to 27K tokens. In our experiment, the truncation rate remained below 5%. #### Advantage Normalization During the training phase, we compute advantages using GAE and normalize them across all generated tokens. #### Key Hyperparameters | **Parameter** | **Value** | | :-------------: | :-------: | | PPO Minibatches | 4 | | Learning Rate | 2e-5 | | Adam Ξ΅ | 1e-5 | This configuration balances convergence speed with training stability, avoiding collapse risks from higher learning rates or smaller Ξ΅ values. ### Approaching QwQ-32B's performance using only 200 data samples At the 32B model size, we further refine the training data and released [AReaL-boba-SFT-200](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data/blob/main/AReaL-boba-SFT-200.jsonl), a high-quality dataset with only **200 data points**. Accompanied by relevant training scripts, we replicated **QwQ-32B's** inference performance on **AIME 2024** via Supervised Fine-Tuning (SFT). ### Evaluation Best Practices During evaluation, we use vLLM v0.6.3 as the generation framework. We identified several settings that influence evaluation performance, particularly for long-context generation. We recommend manually configuring the following options: ```bash enforce_eager=True no_enable_chunked_prefill=True disable_custom_all_reduce=True disable_sliding_window=True ``` Following the practice of DeepSeek models, we incorporate a directive in the prompt: "Please reason step by step, and enclose your final answer in \\boxed{}." To encourage long context reasoning, we also enforce that the model begins each response with "\\n" at the start of its output. To ensure reliable pass@1 estimation, we: - Sample 32 answers per problem - Use temperature=0.6 and top_p=0.95 for SFT models - Maintain training temperature (1.0) for RL models ## Conclusion & Future Work Our results demonstrate that **high-quality data is equally critical as algorithmic innovations**. When conducting RL training on a powerful base model, we require more challenging problems to facilitate learning. Therefore, we integrate resources from multiple recent open-source projects and filter problems by difficulty. A straightforward strategy for data filtering involves removing problems that the base model consistently solves correctly across multiple sampling attempts, as they no longer contribute to improving the model's performance. AReaL delivers stable and fast training with cutting-edge model performances. Since initial release, we've continuously improved system efficiency, training stability, and accessibility. All aforementioned techniques have been implemented in AReaL, with [**reproducible configurations**](/examples/configs/) for various model sizes and different hardware setups. Looking ahead, the AReaL team will: - Further optimize the RL training throughput - Introduce new algorithmic features - Continue open-sourcing training data - Expand to broader reasoning tasks We believe these contributions lower the barrier for high-quality RL training while pushing the boundaries of reasoning capabilities. The project will continue evolving - we welcome community feedback and collaboration to drive further progress in this exciting field. --- ## File: blog/AReaL_v0_3.md

*AReaL* v0.3: SOTA Coding Models with 2.77x Faster Asynchronous RL Training

## Introduction We now release AReaL v0.3, featuring three major milestones: - **A fully asynchronous RL training pipeline with system and RL algorithm co-design**, achieving over 2.77x speedup without any performance drop - **SOTA coding models**, i.e., a 14B model with a **69.1 score on LCB-v5**. Reproducible results with fully open-sourced datasets are also provided - Experimental support for **multi-turn agentic RL training** ### Performance Results | **Model (8B)** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | | :---------------------------------------------------------------------------------: | :-----------------------------------: | :------------: | :--------------: | | Qwen3-8B | 58.8 | 1879/96.7% | 31.4 | | DeepSeek-R1-0528-Qwen3-8B | 58.4 | 1945/97.3% | 31.0 | | [πŸ€— AReaL-bobaΒ²-8B-Open](https://huggingface.co/inclusionAI/AReaL-boba-2-8B-subset) | 62.0 | 1933/97.2% | **41.4** | | [πŸ€— AReaL-bobaΒ²-8B](https://huggingface.co/inclusionAI/AReaL-boba-2-8B) | **63.0** | **1962/97.5%** | 40.8 | | **Model (14B)** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | | :-----------------------------------------------------------------------------------: | :-----------------------------------: | :------------: | :--------------: | | Qwen3-14B | 65.4 | 1978/97.7% | 38.3 | | DeepCoder-14B-Preview | 60.6 | 1936/95.3% | 40.1 | | [πŸ€— AReaL-bobaΒ²-14B-Open](https://huggingface.co/inclusionAI/AReaL-boba-2-14B-subset) | 67.3 | 1990/97.8% | **46.2** | | [πŸ€— AReal-bobaΒ²-14B](https://huggingface.co/inclusionAI/AReaL-boba-2-14B) | **69.1** | **2044/98.2%** | 46.1 | | **Larger Models** | **LiveCodeBench v5** (2024.10-2025.2) | **Codeforce** | **CodeContests** | | :---------------------: | :-----------------------------------: | :-----------: | :--------------: | | Qwen3-235B | 70.7 | 2056 | - | | DeepSeek-R1 | 64.3 | 2029 | - | | OpenAI-o3-mini (Medium) | 66.3 | 2036 | - | *Table 1: Coding Task Performance Comparison. AReaL-bobaΒ²-8B/14B-Open denotes training results on open-sourced data. AReaL-bobaΒ²-8B/14B models are trained with an additional small amount of internal data and achieve SOTA performance on LiveCodeBench, Codeforce & CodeContests.* To access our latest models and training data, please visit this [Huggingface Link](https://huggingface.co/collections/inclusionAI/areal-boba-2-683f0e819ccb7bb2e1b2f2d5). ## Motivation for Asynchronous RL System ### Inference devices are underutilized During the synchronous RL training process, the generation step must wait until the finish of the longest output within a batch. Due to the varying output lengths for LRM, a synchronous RL system suffers from severe training inefficiency. Some works ([DeepCoder](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51), [Intellect](https://www.primeintellect.ai/blog/intellect-2)) use outputs generated from a previous model version to update the current model. For the best performances, the model version used for rollout generation is limited to only one or two steps older. However, all these systems still follow a batched generation setting, the issue of system inefficiency during the generation phase still remains unaddressed. *Fig.1. Left: Execution timeline of a synchronous RL training. Right: Execution timeline of one-step overlap RL system.* ### Scalability is poor in synchronous RL systems Synchronous systems distribute generation across all devices, reducing the per-GPU decoding batch size. This pushes the decoding process into a memory-IO-bound regime where additional devices fail to improve throughput. *Fig2. Left: Strong scaling of batched generation throughput for a 1.5B LRM. Right: Generation becomes memory-IO bound as GPU count increases.* ## Asynchronous AReaL Overview We design a system that fully decouples generation and training across separate GPU clusters. This system should be hardware-efficient, scalable, and equipped with the flexibility for a customized RL workflow. We implement these principles in AReaL. Fig.3 presents the architecture and data flow of AREAL. The system comprises 4 core components: *Fig.3 The architecture featuring asynchronous generation and training components.* ### Core Components - **Interruptible Rollout Worker** handles two types of requests: (1) The *generate* request generates responses given prompts. (2) The *update_weights* request interrupts all ongoing generations and loads parameters of new versions. Upon the interruption, the rollout workers discard KV caches computed by old weights, and re-compute them using the new weights. Afterwards, the rollout workers continue to decode the unfinished sequences until the next interruption or termination. We emphasize that such interruptions and in-flight weight updates would result in trajectories composed of segments produced by different model versions. This introduces a novel algorithmic challenge, which will be addressed in the next section. - **Reward Service** evaluates the accuracy of the responses generated by the model. For example, in the coding task, this service extracts the code and executes unit tests to verify its accuracy. - **Trainer Workers** continuously sample from the replay buffer, accumulating data until reaching the configured training batch size. They then perform PPO updates and store the resulting parameters in distributed storage. To ensure data freshness, data from the replay buffer is used only once. - **Rollout Controller** serves as a critical bridge between the rollout workers, reward service, and the model workers. During the training process, it reads data from the dataset and invokes the rollout worker's *generate* request. The received response is then sent to the reward service to obtain the reward. The trajectory, along with the reward, is stored in the replay buffer, waiting to be trained by the model worker. After the model worker updates the parameters, the controller calls the rollout worker's *update_weight*. We illustrate the generation and training management in Fig.4. This asynchronous pipeline ensures continuous full utilization of both generation and training resources. *Fig 4. Execution timeline of our fully asynchronous RL system.* ## Algorithmic Challenges and Solutions for Asynchronous RL ### Challenges While the asynchronous system design offers significant acceleration through improved device utilization, it introduces several technical challenges that require algorithmic considerations. - **Challenge 1: Data Staleness** - Due to the asynchronous nature of AREAL, each training batch contains data from multiple prior policy versions. Data staleness would lead to a distribution gap between the training data and the latest model. In asynchronous RL training for LRMs, this issue could be even more severe for long trajectories due to extended decoding time. - **Challenge 2: Inconsistent Policy Versions** - As shown in Fig.4, a single generated trajectory may involve segments produced by different policy versions. This inconsistency fundamentally violates the formulation of standard PPO that assumes all actions being generated by a single policy. ### Solutions To overcome these two challenges, we propose two solutions: - **Solution 1: Staleness-Aware Training** - To avoid negative impact of the data staleness issue, we constrain the version discrepancy between the policy version of generated trajectories and the training policy. We introduce a hyperparameter Ξ· representing the maximum permitted staleness. When Ξ· = 0, the system degenerates to the synchronous RL setting where generation exactly matches training batches. When Ξ· = 1, the system recovers to the previous one-step overlap methods. - **Solution 2: Decoupled PPO Objective** - We apply a decoupled PPO objective that disentangles the behavior policy and the proximal policy. The behavior policy represents the policy used for sampling trajectories and the proxy policy is a proximal policy serving as a recent target to regularize the update of online policy. ## Validating Asynchronous AReaL We first evaluate Asynchronous AReaL on a math task to validate our system and algorithmic choices. Building on our previous release, AReaL-boba, we adopt R1-Distill-Qwen as the base model and [AReaL-boba-106k](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data/blob/main/AReaL-boba-106k.jsonl) as the training dataset. ### End-to-End Performance Comparison We compared synchronous and asynchronous training on 1.5B and 7B models. Under equal resource constraints and training steps, the asynchronous system was over twice as fast as the synchronous one. Evaluations on AIME24 confirmed that this acceleration does not compromise performance. | **Model (1.5B)** | **AIME24** | **# Nodes** | **PPO Steps** | **Training Hours** | | :----------------: | :--------: | :---------: | :-----------: | :----------------: | | basemodel | 29.3 | - | - | - | | Synchronous AReaL | 42.0 | 16 | 1000 | 41.0 | | Asynchronous AReaL | 42.2 | 16 | 1000 | **14.8** | | **Model (7B)** | **AIME24** | **# Nodes** | **PPO Steps** | **Training Hours** | | :----------------: | :--------: | :---------: | :-----------: | :----------------: | | basemodel | 54.3 | - | - | - | | Synchronous AReaL | 63.0 | 24 | 1000 | 57.7 | | Asynchronous AReaL | 63.1 | 24 | 1000 | **25.4** | *Table 2: End-to-end comparison between asynchronous AReaL and synchronous AReaL. Following last release AReaL-boba, we adopt R1-Distill-Qwen as the basemodel and [AReaL-boba-106k](https://huggingface.co/datasets/inclusionAI/AReaL-boba-Data/blob/main/AReaL-boba-106k.jsonl) as the training data. Each performance number represents the average results of 32 sampling responses.* ### Ablation Study #### System Ablations We ablate interruptible generation and present the resulting generation throughput in Fig.5. Without interruptible generation, the controller must wait for the longest response. In particular, interruptible generation leads to a 12% and 17% throughput increase for 1.5B and 7B models respectively on 4 nodes. *Fig.5 Ablation study of interruptible generation.* #### Algorithm Ablations We vary the maximum allowed staleness Ξ· and compare configurations with and without the decoupled PPO objective. Fig.6 shows the learning curves after 1000 PPO updates, demonstrating that naive PPO fails to match the performance of the synchronous RL oracle (i.e., the performance when Ξ· = 0). Even slight staleness can significantly degrade final performance due to the improper clipping center and policy changes during interruptible generation. Furthermore, increasing data staleness consistently degrades learning performance. *Fig.6 Ablation Study on Decoupled PPO Objective with DeepSeek-R1-Distill-Qwen-1.5B. Left: Learning curves with naive PPO. Right: Learning curves with decoupled PPO objective.* | Ξ· | **AIME 24** | | **AIME 25** | | | :-: | :---------: | :-------------------: | :---------: | :-------------------: | | | naive PPO | + Decoupled Objective | naive PPO | + Decoupled Objective | | 0 | 42.0 | | 32.9 | | | 1 | 41.8 | 42.1 | 30.7 | 31.9 | | 2 | 40.0 | 41.8 | 32.1 | 32.5 | | 4 | 23.3 | 42.2 | 23.1 | 32.0 | | 8 | 35.7 | 41.0 | 27.8 | 31.1 | | 16 | 35.8 | 38.7 | 26.2 | 32.5 | | ∞ | 34.0 | 36.9 | 26.9 | 29.9 | *Table 3: Evaluation scores when varying data staleness, comparing performance with and without the decoupled objective.* ```{note} In practice, the effect of staleness could depend on various factors, such as learning rate, batch size, the number of mini batches, generation length, and also the RL algorithm. Here we aim to illustrate the impact of staleness on the training performance of a synchronous RL system, and show that stalenss control and decoupled PPO could well mitigate the negative impact of staleness. Here we follow a large batch size setting with `batch_size=512`, `n_rollouts=16`, `n_mini_batch=4`, `max_new_tokens=8192`, and `lr=2e-5`. The staleness is defined as the gap of model versions, where the model version increases by 1 when a batch of `batch_size` X `n_rollouts` samples are trained on. ``` However, the decoupled PPO objective substantially improves training stability when handling stale data. Notably, even with the decoupled objective, unbounded staleness (maximum staleness β†’ ∞) still results in inferior performance compared to the zero-staleness oracle. When properly constrained, moderate staleness (e.g., Ξ· ≀ 4) has minimal impact on final performance while significantly accelerating training through the asynchronous pipeline, as demonstrated in Tab.3 and Fig.7. *Fig.7 The relationship between Ξ· and training throughput. Larger Ξ· leads to higher throughput.* ## Training Recipe with Asynchronous AReaL ### Dataset Curation We train the 7B and 14B model using open-source data from [DeepCoder](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51). After deduplication and quality filtering, approximately 7,600 coding problems remained. Additionally, we have around 10,000 internal data entries that can further improve the model's performance. We are actively working on open-sourcing this portion of the data. ### Rollout Strategy During the rollout phase, the LLM generates 16 responses per question. To minimize output truncation, we set the maximum generation length to 27K tokens. ### Filtering Strategy In the design of our asynchronous system, we can naturally incorporate strategies to filter some rollouts. Specifically, we exclude the questions where all 16 answers are either entirely correct or entirely incorrect. ### Key Hyperparameters | **Parameter** | **Value** | | :----------------------: | :-------: | | PPO Minibatches | 1 | | Learning Rate | 2e-5 | | Adam Ξ΅ | 1e-5 | | Batch Size | 2,048 | | max_head_offpolicyness Ξ· | 16 | | success_rate_ub | 0.95 | | success_rate_lb | 0.05 | ## Resources - [bobaΒ² Code dataset](https://huggingface.co/datasets/inclusionAI/AReaL-boba-2-RL-Code) - **Reproduce bobaΒ² Code Models** - πŸ€— **Model weights**: [8B-code](https://huggingface.co/inclusionAI/AReaL-boba-2-8B), [14B-code](https://huggingface.co/inclusionAI/AReaL-boba-2-14B), [8B-code-open](https://huggingface.co/inclusionAI/AReaL-boba-2-8B-subset), [14B-code-open](https://huggingface.co/inclusionAI/AReaL-boba-2-14B-subset) - [Evaluation Guide](https://areal-project.github.io/AReaL/en/tutorial/eval.html) - [Training configs](https://github.com/areal-project/AReaL/tree/main/examples/configs/v0.3-qwen3-code) and instructions (`reference/reproduce.html`, revert to a version before v0.5.1 to see the reproduction guide) --- ## File: docs/adapt-bailing-moe-v2.5.md # BailingMoeV2.5 Adaptation Guide for AReaL ## 1. Architecture Overview BailingMoeV2.5 is a **heterogeneous** Transformer with 4 types of layer combinations: ``` BailingMoeV2_5ForCausalLM Embedding β†’ N TransformerLayers (heterogeneous) β†’ Final LayerNorm β†’ lm_head ``` Each layer = **Attention** (Lightning or MLA) + **MLP** (Dense or MoE). ### Layer Assignment Rules - **Attention**: Every `layer_group_size`-th layer uses MLA; others use Lightning Attention. Formula: `is_lightning = (layer_number + 1) % layer_group_size != 0` - **MLP**: First `first_k_dense_replace` layers use Dense; rest use MoE (256 experts, sigmoid routing, grouped TopK with `n_group=8`, `topk_group=4`, shared experts, expert bias). ### Attention Mechanisms **Lightning Attention** (linear, O(n)): - Fused QKV β†’ Q/K LayerNorm β†’ Partial RoPE (first half of head_dim only) β†’ `chunk_simple_gla` kernel (fla library) β†’ Gate (sigmoid gating with `g_proj` + `g_norm` on attn output) β†’ Output proj - Per-head decay via ALiBi slopes, scaled by layer depth: `g_gamma = -alibi_slopes * (1 - (layer_idx - 1) / (N - 1) + 1e-5)` **MLA (Multi-Latent Attention)** (standard softmax): - Low-rank KV compression via `q_a_proj β†’ q_a_layernorm β†’ q_b_proj` and `kv_a_proj β†’ kv_a_layernorm β†’ kv_b_proj` - Reuses megatron-core `MLASelfAttention` directly ### Model Variants | Model | model_type | Layers | Hidden | q_lora_rank | rope_theta | layer_group_size | | ----- | ------------------ | ------ | ------ | ----------- | ---------- | ---------------- | | Mini | bailing_moe_linear | 20 | 2048 | null | 10000 | 5 | | Flash | bailing_hybrid | 32 | 4096 | 1536 | 6000000 | 8 | ______________________________________________________________________ ## 2. Implementation: Key Files & Design ### File Map | File | Purpose | | ------------------------------------------- | --------------------------------------------------------------------- | | `areal/models/mcore/lightning_attention.py` | Custom Lightning Attention module for megatron-core | | `areal/models/mcore/bailing_moe.py` | HF config β†’ `MLATransformerConfig` + heterogeneous layer spec builder | | `areal/models/mcore/bailing_moe_bridge.py` | mbridge weight mapping (HF ↔ mcore) | | `areal/models/mcore/registry.py` | Architecture dispatch (`BailingMoeV2_5ForCausalLM` branch) | | `areal/engine/core/model.py` | `VALID_MOE_MODELS` list | | `areal/engine/megatron_utils/megatron.py` | `convert_bailingmoe_to_hf()` for NCCL weight push | | `areal/models/mcore/hf_load.py` | `_load_fused_qkv_weight()` for HFβ†’megatron QKV format | ### Heterogeneous Layer Spec Construction Cannot use single `get_gpt_decoder_block_spec()` (assumes uniform layers). Build 4 spec variants: 1. Generate base specs with `get_gpt_layer_with_transformer_engine_spec()` for MLA+Dense, MLA+MoE, Standard+Dense, Standard+MoE 1. Deep-copy Standard specs and replace `self_attention` with custom `LightningSelfAttention` 1. Per-layer: select spec based on `is_lightning_layer()` and `is_moe` ### QKV Format Conversion HF stores `[Q_all, K_all, V_all]` concatenated `[3, H, D, hidden]`. Megatron expects per-head interleaved `[H, 3, D, hidden]`. Conversion: `permute(1, 0, 2, 3)`. Inverse for NCCL push is the same permute in reverse direction. Both load and save paths verified as precise inverses. ### Key Design Decisions - `rotary_percent=1.0` in AReaL (not 0.5 as in HybridEngine) because megatron-core MLA passes `qk_pos_emb_head_dim=64`, so `64 * 1.0 = 64` gives correct RoPE dim - Lightning layers create their own `RotaryEmbedding` (different RoPE dim from MLA) - Uses `chunk_simple_gla` (not `chunk_lightning_attn`) to pass custom ALiBi g_gamma ______________________________________________________________________ ## 3. Critical Bug Fixes ### Bug A: `layer_idx` Offset Error (root cause of loss=7.8) **Symptom**: Initial loss 7.8 instead of expected \<=2. **Root cause**: HF reference uses `(self.layer_idx - 1)` in slope formula (0-indexed minus 1), which gives layer 0 a scale of `1.0526`. AReaL intentionally uses `self.layer_idx` directly (layer 0 scale = `1.0`), matching SGLang and HybridEngine behavior. The original bug was that the slope formula was missing entirely. **Fix**: `layer_scale = 1.0 - self.layer_idx / max(self.num_layers - 1, 1) + 1e-5` ### Bug B: `g` vs `g_gamma` Parameter `g` and `g_gamma` are mathematically equivalent for constant decay. But the `g` path computes backward gradients on the constant buffer via `chunk_local_cumsum(dg, reverse=True)`, introducing gradient noise that raised loss from 7.8 to 14.42. `g_gamma` is the correct choice. ### Bug C: `gate_norm` Applied Twice Gate norm was applied to both gate and attn_output using the same weight. HF reference norms only `attn_output`, then multiplies by `sigmoid(raw_gate)`. **Fix**: Remove `gate = self.gate_norm(gate)`, keep only `attn_output = self.gate_norm(attn_output)`. ### Bug D: Missing QKV Format Conversion TP>1 weight loading silently produced garbage without format conversion. **Fix**: Added `_load_fused_qkv_weight()` implementing `[3,H,D,hidden] β†’ [H,3,D,hidden]` permutation + per-head TP slicing. ### Bug E: `q_lora_rank` Default Inconsistency `bailing_moe.py` defaulted to 512, bridge defaulted to None. Unified to None. ______________________________________________________________________ ## 4. Context Parallelism (CP) Support ### Strategy: All-to-All Head Parallelism For Lightning Attention layers (linear attention is head-independent): ``` Input: [S/CP, B, H_local, D] β†’ all_to_all CPβ†’HP β†’ [S, B, H_local/CP, D] (full seq, fewer heads) β†’ zigzag undo (restore sequential order) β†’ RoPE (BSHD path, continuous positions) β†’ chunk_simple_gla (g_gamma CP-sliced) β†’ zigzag redo β†’ all_to_all HPβ†’CP β†’ [S/CP, B, H_local, D] ``` For MLA layers: megatron-core/TE ring attention (built-in CP). ### MLA RoPE Monkey-Patch **Problem**: megatron-core 0.13.1 truncates `rotary_pos_emb` to `[0:total_tokens/CP]`, but `_get_thd_freqs_on_this_cp_rank` needs the full-length freq table for zigzag position indexing. **Solution**: Detect truncated freqs, rebuild to full length using `freqs[p] = p * freqs[1]` (valid for standard RoPE where `outer(positions, inv_freq)` is linear in position). ### Sequence Alignment Constraint All CP paths require: `each sub-sequence length ≑ 0 (mod tp_size Γ— cp_size Γ— 2)`. ### Gradient Propagation `postprocess_packed_seqs_context_parallel` uses `output.detach()` for all-gather, then replaces the local rank's slice with the original (gradient-carrying) tensor. This is a correct gradient-partitioning strategy: each CP rank backpropagates only through its own tokens. ______________________________________________________________________ ## 5. CP=8 NCCL SIGSEGV Crash **Symptom**: CP=8 with `megatron:(attn:d2p2t2c8|ffn:d2p2e16)` on 64 GPUs crashes at step 110-112 with SIGSEGV. **Root cause**: Ray scheduler does not set `NCCL_ASYNC_ERROR_HANDLING=1` (unlike torch elastic agent). Without this, NCCL communicators run in **blocking mode**. With 5+ communicators (TP, PP, CP, EP, DP) competing on the same GPU, blocking `ncclGroupEnd()` calls create resource contention that gradually corrupts NCCL internal state. **Fix**: Add `NCCL_ASYNC_ERROR_HANDLING: "1"` to worker env_vars. Recommended as global default. **Why CP=4 didn't crash**: Fewer communicators (3-4), insufficient contention. **Why `NCCL_DEBUG=INFO` also prevented crash**: Extra logging I/O shifted timing (Heisenbug). ______________________________________________________________________ ## 6. Configuration Examples ### SFT (no CP) ```yaml allocation_mode: megatron:d8p4t2e8 # 64 GPUs actor: dtype: bfloat16 gradient_checkpointing: true ``` ### CP=8 ```yaml allocation_mode: megatron:(attn:d2p2t2c8|ffn:d2p2e16) env_vars: NCCL_ASYNC_ERROR_HANDLING: "1" # Required for CP>=8 with Ray ``` ### Performance Results | Metric | CP=4 | CP=8 | | ------------------- | -------------- | ------------------------- | | Step time | 106.66s | **47.28s** (2.25x faster) | | GPU mem (allocated) | 57.88 GB (73%) | **41.00 GB (52%)** | | Loss convergence | Normal | Normal (0.696 β†’ 0.380) | ### Environment Dependencies | Component | Version | | ---------------------------- | ----------- | | flash-linear-attention (fla) | >= 0.3.0 | | SGLang | >= 0.5.9 | | TransformerEngine | 2.11.0 | | torch | 2.6.0+cu126 | ______________________________________________________________________ ## 7. Validation: AReaL vs HybridEngine Mini model comparison (real text, seq_len=512, EP=8): | Metric | AReaL | HybridEngine | Diff | | ------ | ----- | ------------ | ----- | | Loss | 1.512 | 1.491 | 1.33% | | PPL | 4.54 | 4.44 | 2.2% | Per-layer: Lightning layers cosine 0.999+, MLA layers 0.92-0.95 (from TE/flash_attn version differences, not a bug). ______________________________________________________________________ ## 8. Known Open Issues 1. ~~**expert_bias HF name**~~: Fixed β€” NCCL path now uses `expert_bias` (matching HF checkpoint). 1. **kv_channels may be None**: `hf_to_mcore_base_args()` reads `head_dim` which BailingMoe doesn't have. Should explicitly set `kv_channels`. 1. **BailingHybridForCausalLM not in registry.py**: Bridge registers `bailing_hybrid` but `registry.py` only checks `BailingMoeV2_5ForCausalLM` and `BailingMoeLinearForCausalLM`. --- ## File: docs/requirements.txt jupyter-book matplotlib numpy --- ## File: docs/en/_config.yml # Book settings title: AReaL Documentation author: AReaL Team copyright: "2026" logo: ../figures/logo.png execute: execute_notebooks: force latex: latex_documents: targetname: book.tex url: https://github.com/areal-project/AReaL path_to_book: docs branch: main html: use_issues_button: true use_repository_button: true extra_footerjs: - _static/js/lang-toggle.js extra_css: - _static/css/lang-toggle.css build: dir: ../_build/en figureurl: /figures sphinx: config: html_extra_path: - _static - ../figures --- ## File: docs/en/_toc.yml # English Table of Contents format: jb-book root: intro parts: - caption: Version History chapters: - file: version_history - caption: Tutorial chapters: - file: tutorial/installation - file: tutorial/installation_npu - file: tutorial/quickstart - file: tutorial/agentic_rl - file: tutorial/online_proxy - file: tutorial/eval - file: tutorial/megatron - file: tutorial/archon - file: cli_reference - caption: Code Walkthrough chapters: - file: tutorial/gsm8k_grpo - caption: Best Practices chapters: - file: best_practices/algo_perf - file: best_practices/workflow - file: best_practices/debugging - file: best_practices/handling_oom - file: best_practices/perf_profiling - caption: Customization chapters: - file: customization/dataset - file: customization/data_engineering - file: customization/agent - file: customization/advantage - caption: Algorithms chapters: - file: algorithms/async - file: algorithms/distillation - file: algorithms/dpo - file: algorithms/grpo_series - file: algorithms/m2po - file: algorithms/prox_approx - file: algorithms/prm - caption: Reference chapters: - file: reference/checkpointing - file: reference/metrics_tracking - file: reference/moe_visualization - file: reference/alloc_mode - file: reference/lora - file: reference/bridge_backend - file: reference/tree_training - file: reference/rollout_workflow - file: reference/agent_workflow - file: reference/ai_assisted_dev ## 2. Official Technical Reference & Guides (inclusionAI/inclusionAI.github.io) # Landscape project logos Drop project logo images in this folder to have them appear on the [`/landscape`](../../../src/pages/landscape.tsx) page. ## How to add a logo 1. Add the image file here, e.g. `static/img/landscape/aworld.png` (PNG/SVG with a transparent background works best; square aspect ratio, ideally β‰₯ 104Γ—104 px so it stays crisp inside the 52Γ—52 slot). 2. Open `src/data/landscape.ts` and set the matching project's `logo` field to the site-root path, e.g.: ```ts { name: "AWorld", description: "Search, understand, reproduce, and improve an idea with ease", logo: "/img/landscape/aworld.png", } ``` Until a `logo` is set, the page renders a dashed "Logo" placeholder box so the slot is visually reserved. ## Suggested file names | Layer | Project | Suggested file | | --------------------------- | ------------- | ------------------- | | AI Service | Ling Guang | `ling-guang.png` | | AI Service | MA XIAO CAI | `ma-xiao-cai.png` | | AI Service | AQ | `aq.png` | | AI Service | Life Services | `life-services.png` | | Agent Infra | AWorld | `aworld.png` | | Agent Infra | AEnvironment | `aenvironment.png` | | Agent Infra | ASearcher | `asearcher.png` | | Model Infra | cuLA | `cula.png` | | Model Infra | Humming | `humming.png` | | Model Infra | DLRover | `dlrover.png` | | Model Infra | ATorch | `atorch.png` | | Model Infra | AReaL | `areal.png` | | Model Infra | Asystem | `asystem.png` | | Model Infra | dInfer | `dinfer.png` | | Model Infra | dFactory | `dfactory.png` | | Model | Ling | `ling.png` | | Model | Ming | `ming.png` | | Model | Ring | `ring.png` | | Model | LLaDA | `llada.png` | | Embodied Intelligence Model | LingBot-Depth | `lingbot-depth.png` | | Embodied Intelligence Model | LingBot-Map | `lingbot-map.png` | | Embodied Intelligence Model | LingBot-VLA | `lingbot-vla.png` | | Embodied Intelligence Model | LingBot-VA | `lingbot-va.png` | | Embodied Intelligence Model | LingBot-World | `lingbot-world.png` |