File: README.md Copy Chapter

*AReaL*: A Large-Scale Asynchronous Reinforcement Learning System

AReaL is a reinforcement learning (RL) infrastructure designed to bridge foundation
model training with modern agent-based applications. It was originally developed by
researchers and engineers from Tsinghua IIIS and the AReaL Team at Ant Group.

Built on a fully asynchronous RL training paradigm, AReaL is optimized for
efficiency and scalability, making it particularly well-suited for training large-scale
reasoning and agentic models.

AReaL’s mission is to make building AI agents accessible, efficient, and
cost-effective for a broad community of developers and researchers.

Like milk tea - customizable, scalable, and enjoyable - we hope AReaL brings both
flexibility and delight to your AI development experience. Cheers!

AReaL Highlights

• ⚡ Flexibility: Seamless customization for
  agentic RL and
  online RL training for
  black-box agent applications by simply replacing the base_url.
• 📈 Scalability: Stable fully asynchronous RL training with industry-leading
  speed.
• ✨ Cutting-Edge Performance: State-of-the-art math,
  coding, search, and
  customer service agents.

📰 News Copy Chapter

[2026/07/01] 🎉 We release AReaL 2.0, a major architectural milestone. This
release refactors AReaL into a microservice architecture with independent
training, inference,
agent, and weight-update
services, and ships the Hermes online RL loop and
end-to-end SWE RL training examples. Check out our
technical report.

[2026/06/17] 🔬 Introducing KPop — bidirectional binary KL divergence token
masking. Configured via rejection_sampling.metric=binary_kl. Also adding an
IcePop config (importance-ratio-based token masking).
Check out gsm8k_kpop.yaml and
gsm8k_icepop.yaml to get started!

📋 Previous Releases

[2026/04/23] 🚀 We’re excited to release our integration with
Scaffoldings
for agentic RL training - now live in our
examples! Huge
shoutout to @narutolhy and @WeiHaocheng for making this happen 🙌. The modular design of
the Scaffoldings enables it to achieve a thorough decoupling of agent execution, reward
calculation, and trajectory acquisition. This enables developers to reuse existing
modules when implementing an agentic RL method, allowing them to focus on their own
innovative modules.

[2026/04/18] We are thrilled to announce that AReaL's first Community Biweekly
Meeting was successfully held! Thank you to everyone who joined us. Meeting materials
are now available
here. Our next
meeting is scheduled for 2026/05/01 and will also be conducted in Chinese;
English-language meetings will be scheduled in the future. We warmly welcome everyone to
participate! See Community for more details.

[2026/03/02] We provide a complete example to train your
own 🦞 OpenClaw agent by simply replacing the base_url and api_key with AReaL's RL
service - no complicated dependencies, no code changes, works with any agentic runtime!

[2026/02/06] We are delighted to introduce AReaL-SEA, a self-evolving data
synthesis engine. Combined with RL training on AReaL, the 235B MoE model surpasses GPT 5
and achieves comparable performance with Gemini 3.0 Pro on $\tau^2$-bench! Check out
the paper,
model,
data, and
code.

[2026/01/15] Congrats to our friends at CAMEL-AI for
open-sourcing SETA, their terminal agent RL project
trained with AReaL! Check out
their training workflow
and the announcement on X.

[2026/01/01] Happy New Year! Thanks to the outstanding contribution from
@HwVanICI, we are excited to officially announce stable support for AReaL training on
Ascend NPU devices! The code is actively maintained and continuously updated in the
ascend branch. Check out
our documentation
to get started, and feel free to report any issues!

[2025/08/30] Introducing ASearcher, a state-of-the-art search agent built with
AReaL's end-to-end asynchronous RL training. Check out the paper and
the open-source repository!

[2025/07/31] (AReaL-lite) We introduce AReaL-lite, a lightweight version of
AReaL designed specifically for AI researchers and rapid prototyping. AReaL-lite
features an algorithm-first API design that prioritizes ease of use and algorithm
development, while natively supporting fully asynchronous agentic RL. With 80% fewer
lines of code, AReaL-lite maintains 90% of AReaL's performance and core functionality.
Check out our documentation and
the quickstart guide
to begin your journey with AReaL-lite!

[2025/06/03] (v0.3, boba²) We release boba² (double-boba) for fully
asynchronous RL training, which achieves 2.77× speedup while delivering comparable or
superior training performance compared to synchronous systems. Furthermore,
asynchronous RL significantly simplifies multi-turn agentic RL training setup! Check out
our v0.3 overview blog and the
research paper.

[2025/03/31] (v0.2, boba) Introducing our milestone release—boba! Please call it
A-ReaL-boba! This release features significantly faster training with SGLang support and
state-of-the-art 7B and 32B models for mathematical reasoning. Check out our
v0.2 technical blog.

[2025/02/24] (v0.1) Our initial release includes reproducible results for 1.5B and
7B Large Reasoning Models (LRMs). Check out our
v0.1 technical blog.

🚀 Getting Started Copy Chapter

First, install the package:

git clone https://github.com/areal-project/AReaL
cd AReaL
pip install uv
# Install flash-attn pre-built wheel first to avoid compiling from source
# (pick the wheel matching your Python version; see https://github.com/mjun0812/flash-attention-prebuild-wheels/releases)
uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl"
uv sync --extra cuda  # installs training packages + SGLang (default inference backend)
# For vLLM instead: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda

Our training scripts automatically download the required dataset (openai/gsm8k) and
model (Qwen/Qwen2-1.5B-Instruct). To run on a single node:

python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local

If you prefer to run experiments on a Ray cluster, update paths in the YAML file to
point to your shared storage, and run:

python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml \
  cluster.n_nodes=2 cluster.n_gpus_per_node=8 \
  cluster.fileroot=/path/to/nfs \
  scheduler.type=ray

For comprehensive setup instructions, see
our quickstart guide.

For Megatron MoE router replay experiments, see the
R3 Router Replay guide.

📚 Examples Copy Chapter

Math & Reasoning Copy Section

Agentic RL Copy Section

Vision-Language Models Copy Section

Alignment & Infrastructure Copy Section

🔧 Support Matrix Copy Chapter

🧠 Algorithms Copy Section

All RL algorithms support both asynchronous and synchronous versions by setting
max_head_offpolicyness=0. See Asynchronous RL Guide.

Models Copy Section

Qwen2 and Qwen2.5 text models support RL weight updates with
actor.weight_update_mode=disk in both v1 and v2. XCCL weight updates require
actor._version=v1, rollout._version=v1, and actor.weight_update_mode=xccl. AWEX
weight updates are not supported for these models. This restriction does not apply to
Qwen2.5-VL.

Check the AI Coding Assistant Guide and
Archon Reference for how to integrate new models into
AReaL.

Training Backends Copy Section

Inference Backends Copy Section

📖 Resources Copy Chapter

Tutorial Copy Section

• Installation
• Quickstart
• AReaL CLI Guide
• Agentic RL
• Evaluation
• Large MoE with Megatron
• Large MoE with PyTorch Archon

Code Walkthrough Copy Section

• Running GRPO on GSM8K dataset

Best Practices Copy Section

• Improving Algorithm Performance
• Agent Workflow Best Practices
• Debugging
• Handling OOM Issues
• Performance Profiling

Customization Copy Section

• Customize Dataset
• Customize Agentic/RVLR Rollout Workflows

Algorithms Copy Section

• Asynchronous RL Explained
• PPO, GRPO, and Related Algorithms
• M2PO

Reference Copy Section

• CLI Configurations
• LoRA RL
• Checkpointing
• Metrics Tracking
• Allocation Mode
• Rollout Workflow
• Agent Workflow
• AI-Assisted Development

🤝 Contributing Copy Chapter

We warmly welcome contributions from the community! Whether you're fixing bugs, adding
features, improving documentation, or helping others, your contribution is valued.
Please check our
Contributing Guide
for detailed information.

# Fork and clone the repository
git clone https://github.com/YOUR-USERNAME/AReaL
cd AReaL

# Install uv and sync dependencies
pip install uv
# Install flash-attn pre-built wheel to avoid compiling from source
uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl"
# Use `--extra cuda` on Linux with CUDA (installs training packages + SGLang)
uv sync --extra cuda --group dev
# For vLLM instead:
# cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda --group dev
# Or without CUDA support
# uv sync --group dev

# Set up pre-commit hooks (formatting, linting, commit message checks)
pre-commit install --install-hooks

# Make changes
git checkout -b feat/gpt-o5
git add .
# `git commit` will automatically check your files and commit messages
git commit -m "feat: implement gpt-o5 training loop"
git push

🗺️ Future Roadmap Copy Chapter

• Full Roadmap
• 2026 H2 Roadmap
• 2026 Q2 Roadmap

AReaL is under active development with planned minor releases weekly and major releases
monthly. We warmly welcome community engagement and contributions. We are also
actively hiring interns and full-time employees with open positions in both the US
and China.

📢 Community Copy Chapter

We warmly welcome everyone to join our community! We have a WeChat group for
real-time discussions and a
GitHub discussion board for
asynchronous conversations.

Check out our Community Repository for
more details and meeting materials.

🙏 Acknowledgments Copy Chapter

We gratefully acknowledge that major contributors are from the AReaL Team at the
Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University and Ant
Group.

We have also received invaluable assistance from the following groups (listed
alphabetically):

• The Data Intelligence Lab at Ant Research for their data support

• @HwVanICI for support on vLLM, LoRA, NPU integration, and more

• The Relaxed System Lab at HKUST for seamless
  collaboration on numerous system-related aspects

• The SGLang team for supporting custom weight
  update features and their contributions during AReaL-lite development

• The Super Computing Technology (SCT) team at Ant Group for their expertise in
  large-scale cluster operations and maintenance

• Special thanks to @Lyken17 for providing valuable suggestions throughout the API
  design process

We also deeply appreciate all pioneering work from the community, particularly the
ReaLHF project from OpenPsi Inc. and other
outstanding projects, including but not limited to
DeepScaleR,
Open-Reasoner-Zero,
OpenRLHF,
VeRL,
SGLang, QwQ,
Light-R1, and
DAPO.

📜 License Copy Chapter

This project is licensed under the Apache License 2.0.

📄 Citation Copy Chapter

@inproceedings{mei2025real,
  author       = {Mei, Zhiyu and Fu, Wei and Li, Kaiwei and Wang, Guangju and Zhang, Huanchen and Wu, Yi},
  title        = {ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation},
  booktitle    = {Proceedings of the Eighth Conference on Machine Learning and Systems,
                  MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025},
  publisher    = {mlsys.org},
  year         = {2025},
}

@misc{fu2025areal,
      title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning},
      author={Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu},
      year={2025},
      eprint={2505.24298},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.24298},
}

File: AGENTS.md Copy Chapter

AGENTS.md -- AReaL Agent Operations Guide

Quick reference Copy Chapter

Tech stack: Python 3.12+ | PyTorch | FSDP2 / Megatron / Archon | SGLang / vLLM

# Environment
uv sync --extra cuda            # CUDA + SGLang inference (default); for vLLM: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda
uv sync --extra sandbox         # Daytona cloud sandbox backend (optional)
source .venv/bin/activate        # activate venv BEFORE pre-commit or git commit if venv exists
pre-commit install --install-hooks  # hooks: Ruff, clang-format, mdformat, nbstripout, conventional-commits
pre-commit run --all-files       # lint + format everything

# Tests
uv run pytest tests/test_<topic>.py

# CLI docs
uv run python docs/generate_cli_docs.py

# Docs build (canonical, release-aligned)
./docs/build_all.sh
# Do NOT use `jupyter-book build docs/en|docs/zh` directly for final preview/release,
# because it skips AReaL-specific static setup and output packaging.

Hard rules -- never violate:

• No wildcard imports (from x import *).
• No hardcoded secrets, paths, or endpoints.
• No skipping pre-commit hooks.
• No guessing cluster configs or rebuilding CUDA/driver stacks.
• Integration tests require multi-node hardware -- explain skips explicitly.

Always do:

• Read relevant files before modifying code.
• Run pre-commit run --all-files before committing.
• Follow existing code patterns in the same module.
• Add tests for new functionality.
• Ask for decisions and clarifications with short, structured options instead of broad
  open-ended questions. Use the platform's native question/clarification tool if
  available.

Ask first before:

• Modifying config structures in areal/api/cli_args.py.
• Adding new dependencies.
• Changing launcher or scheduler logic.
• Deleting or renaming public APIs.

When unsure, leave a TODO(agent) comment and note the constraint in your response.

Repository map Copy Chapter

areal/                     Core Python package
|-- api/                   Config dataclasses, contracts, IO structs
|-- dataset/               Stateful dataset loaders (GSM8K, Geometry3K, CLEVR, ...)
|-- engine/                Training backends (FSDP2, Megatron) + inference adapters
|-- experimental/          Prototype engines/workflows (Archon MoE engine)
|-- infra/                 Launchers (Local/Ray/Slurm), schedulers, utilities
|-- models/                Model adapters (Megatron-Core, Transformers, custom heads)
|-- reward/                Built-in reward functions + math parsers
|-- tests/                 Unit/integration test suites
|-- trainer/               High-level orchestrators (PPOTrainer, SFTTrainer)
|-- utils/                 Cross-cutting helpers (logging, data, checkpoints, RL ops)
+-- workflow/              RolloutWorkflow implementations (RLVR, multi-turn, vision)

docs/                      Jupyter Book docs (https://areal-project.github.io/AReaL/)
examples/                  Training scripts and launcher recipes

Code style & patterns Copy Chapter

• Composition over inheritance -- keep hierarchies <= 2 levels; prefer delegation.

Logging: areal.utils.logging.getLogger(name) with PascalCase names -- never
print or logging.__name__. Per-rank format: [{Component} Rank {N}]. Register new
loggers with color in areal/utils/logging.py.

Performance:

• No GPU-CPU sync in hot paths (.item(), .tolist(), print(tensor)).
• Batch ops over Python loops on tensor elements.
• Explicit dtype/device; torch.Size assertions for shape validation.

Typing & imports: explicit type hints; reuse areal/api/cli_args.py dataclasses; no
wildcard imports; heavy optional deps inside functions.

Async: rollout workflows must stay non-blocking (await + aiofiles); no sync I/O
in arun_episode.

Domain experts & skills Copy Chapter

Fire the appropriate expert subagent or load a skill based on what you're
working on. Experts are read-only consultants with deep domain knowledge; skills are
step-by-step implementation guides.

How to invoke experts and skills (platform-specific):

Harness layout:

Directly executable workflows (both platforms): add-workflow, review-pr,
create-pr, translate-doc-zh.

Core concepts Copy Chapter

Trainer orchestrator (areal/trainer/, PPOTrainer, SFTTrainer): manages the
training loop, dataset loading, and workflow execution. Entry point:
examples/math/gsm8k_rl.py.

Rollout workflows (areal/workflow/, RolloutWorkflow.arun_episode): define how
episodes are generated. Use add-workflow skill for step-by-step guide.

Engines: Inference engines handle async generation via engine.agenerate() and
manage weight updates. Training engines consume rollout tensors, compute PPO/GRPO
updates, and broadcast weight versions (FSDP2, Megatron, or Archon).

Weight versioning: async workflows require version alignment via WeightUpdateMeta
(areal/api/engine_api.py). Critical for correctness across distributed training.

Observability: emit metrics via stats_tracker.get(), persist artifacts under
dump_dir, checkpoint via areal/utils/saver.py / recover.py.

Launcher / scheduler: training requires cluster setup (local / Ray / Slurm) via
configs in areal/infra/launcher/. See launcher-expert for deployment guidance.

API & config rules Copy Chapter

Applies to: areal/api/**

• Field ordering: required -> common optional -> rare optional -> internal (_
  prefix).
• Validation: __post_init__ with ValueError and clear message.
• Backward compat: add fields with defaults; deprecate before removing; avoid type
  changes.
• CLI: use Literal for enum choices; all public configs need docstrings with
  constraints.

Distributed code rules Copy Chapter

Applies to: areal/engine/**, areal/experimental/**

• Never create global process groups at module level; always pass process_group
  explicitly.
• dist.get_rank(group) not dist.get_rank() when group matters.
• DeviceMesh dimensions must match ArchonParallelDims: dp_shard, tp, cp, ep,
  etp.
• All-reduce: all ranks must call. Broadcast: explicit src. Barrier: debugging only.

Debug env vars: TORCH_DISTRIBUTED_DEBUG=DETAIL, NCCL_DEBUG=INFO,
CUDA_LAUNCH_BLOCKING=1. See the debug-distributed skill for the full workflow.

Testing rules Copy Chapter

Applies to: **/tests/**, test_*.py

• Naming: test_<what>_<condition>_<expected>() with Arrange/Act/Assert.
• GPU: skip gracefully (@pytest.mark.skipif(not CUDA_AVAILABLE, reason="...")).
• Distributed mocking: torch.distributed.fake_pg; don't mock FSDP/DTensor internals.
• Assertions: torch.testing.assert_close() with explicit rtol/atol; prefer
  tmp_path, monkeypatch.

Collaboration & review Copy Chapter

• Branches: kebab-case (feature/multi-turn-metrics, bugfix/fsdp-weight-sync).
• Commits: Conventional Commits (e.g., feat:, fix:, docs:, gov:), ~72 char
  subject, imperative voice. Squash WIP before PR.
• Pre-merge: full pre-commit stack; doc-only edits need at least mdformat --check.
• PRs: tie to issue, highlight risk areas, list test commands executed, note skipped
  suites with reasons.

Reference material Copy Chapter

• Docs portal: https://areal-project.github.io/AReaL/
• Quickstart: docs/tutorial/quickstart.md
• Architecture: docs/tutorial/gsm8k_grpo.md
• Customization: docs/customization/*.md
• Algorithms: docs/algorithms/*.md
• Best practices: docs/best_practices/*.md
• CLI reference: docs/cli_reference.md
• Agent workflow: docs/customization/agent.md

File: CLAUDE.md Copy Chapter

CLAUDE.md - AReaL

WHAT: Project Overview Copy Chapter

AReaL is a distributed RL training framework for LLM alignment via reinforcement
learning.

Tech Stack: Python 3.12+ | PyTorch | FSDP2/Megatron | SGLang/vLLM

Core Directories:

• areal/ - Core package
  • api/ - Config dataclasses, workflow/engine contracts
  • engine/ - FSDP2, Megatron, SGLang/vLLM adapters
    • fsdp_utils/ - FSDP2-specific utilities (checkpoint, grad, optimizer, parallel)
    • megatron_utils/ - Megatron/FP8 utilities (checkpoint, pipeline, quantization)
    • core/ - Engine-shared utilities (distributed, lock, model, offload)
  • infra/ - Infrastructure (launcher, scheduler, RPC)
    • utils/ - Infrastructure utilities (launcher, proc, http, concurrent, slurm, ray)
  • workflow/ - RolloutWorkflow implementations
  • reward/ - Reward functions
  • dataset/ - Dataset loaders
  • utils/ - Cross-cutting utilities (logging, data, checkpoints, network, RL
    functional)
• examples/ - Training scripts and configs
• docs/ - Jupyter Book source

WHY: Purpose Copy Chapter

• Enable efficient RL training for LLM alignment at scale
• Async rollout + distributed training for high throughput
• Modular design: workflows, engines, rewards, and datasets are independently extensible

HOW: Core Commands Copy Chapter

# Check environment
python --version              # Requires 3.12+
uv --version                  # Install: https://docs.astral.sh/uv/

# Sync dependencies
uv sync --extra cuda          # CUDA + SGLang inference (default)
# For vLLM: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda
uv sync --group dev           # Include dev/test packages
uv run python3 areal/tools/validate_installation.py  # Validate installation

# Pre-commit hooks
pre-commit install --install-hooks  # Set up hooks (run once)
pre-commit run --all-files    # Format and lint

# Run tests
# First check GPU availability (many tests require GPU)
python -c "import torch; print('GPU available:', torch.cuda.is_available())"
uv run pytest tests/test_<topic>.py

# Generate CLI docs
uv run python docs/generate_cli_docs.py

# Build docs (canonical, release-aligned)
./docs/build_all.sh
# Do NOT use `jupyter-book build docs/en|docs/zh` directly for final preview/release,
# because it skips AReaL-specific static setup and output packaging.

Boundaries Copy Chapter

Constraints Copy Section

• Designed for distributed GPU clusters; assume containerized execution
• Integration tests require multi-node hardware; explain skips when unavailable
• Secrets and endpoints are managed outside the repo

Always Do Copy Section

• Read relevant files before modifying code
• Run pre-commit run --all-files before committing
• Follow existing code patterns in the same module
• Add tests for new functionality

Ask First Copy Section

• Modifying config structures in areal/api/cli_args.py
• Adding new dependencies
• Changing launcher or scheduler logic
• Deleting or renaming public APIs
• Running GPU/distributed tests (check GPU first:
  python -c "import torch; print('GPU available:', torch.cuda.is_available())")

Never Do Copy Section

• Hardcode secrets, paths, or endpoints
• Skip pre-commit hooks
• Guess cluster configs or rebuild CUDA/driver stacks
• Use wildcard imports (from x import *)

Progressive Disclosure: Detailed Guides Copy Chapter

Git Workflow Copy Chapter

• Commits: Conventional Commits (e.g., feat:, fix:, docs:, gov:), ~72 chars
  subject, imperative voice, reasoning in body
• Squash: Squash WIP commits before opening PR
• PR requirements: Run pre-commit, document test coverage, note hardware limitations

Extended Configuration Copy Chapter

See .claude/agents/, .claude/skills/, .claude/commands/, and .claude/rules/ for
specialized instructions.

Agents Copy Section

Stage-by-Stage Agent Guidance:

1. Planning Stage (Before coding): Use planner for architecture design and
   implementation planning
2. Code Formatting & Linting (After coding): Use code-verifier to automatically
   run formatting, linting, and tests, catching syntax errors and style issues quickly
3. Code Quality Check (After formatting): Use simple-code-reviewer for quick code
   quality checks, focusing on logic issues and code smells

Skills (Guided Development Workflows) Copy Section

Skills provide step-by-step guides for common development tasks:

• /add-dataset - Dataset loader creation guide
• /add-workflow - Workflow implementation guide
• /add-reward - Reward function guide
• /add-archon-model - Archon engine model architecture guide
• /debug-distributed - Distributed debugging guide
• /add-unit-tests - Test development guide (NEW)

Commands (User-invoked Actions) Copy Section

Commands perform specific actions when invoked:

• /create-pr - Rebase, squash commits, and create/update PR with intelligent messages
• /gen-commit-msg - Generate commit messages from staged changes
• /review-pr - Intelligent PR code review with dynamic agent allocation
• /translate-doc-zh - Translate English documentation to Chinese

Rules (Code Quality Standards) Copy Section

Project-wide standards enforced across all code changes:

• api-config.md - Configuration dataclass design patterns
• code-style.md - Coding conventions beyond pre-commit hooks
• distributed.md - Distributed training patterns and constraints
• testing.md - Testing strategy and coverage requirements

File: CODE_OF_CONDUCT.md Copy Chapter

The latest version of this code of conduct can be found at
https://github.com/areal-project/community/blob/main/CODE_OF_CONDUCT.md

File: CONTRIBUTING.md Copy Chapter

The latest version of this file can be found at
https://github.com/areal-project/community/blob/main/CONTRIBUTING.md

File: GOVERNANCE.md Copy Chapter

The latest version of this file can be found at
https://github.com/areal-project/community/blob/main/GOVERNANCE.md

File: LEGAL.md Copy Chapter

The latest version of this file can be found at
https://github.com/areal-project/community/blob/main/LEGAL.md

File: ROADMAP.md Copy Chapter

AReaL Roadmap

This roadmap outlines the planned features and improvements for AReaL in the next
quarter. We welcome community feedback and contributions to help shape the future
direction of the project.

Latest Release: Check releases
for the most recent version.

2026 H2 Roadmap (due December 31, 2026) Copy Chapter

GitHub Issue #1381.

This roadmap tracks major planned enhancements for the second half of 2026. Items are
organized into two categories:

• On-going: Features currently under active development by the core AReaL team
• Planned but not in progress: Features that are good to have but currently lacking
  bandwidth

We use [CC] to mark items suitable for community contributions. If you're interested
in contributing, please reach out to discuss implementation details.

Backends Copy Section

On-going

• Support training the latest large MoE models with the Megatron backend (including
  Kimi 2.5 / GLM 5 / DeepSeek V3 / Qwen 3.6, etc.)
  (#1372,
  #1373)
• More distribution strategies for colocated deployment

Planned but not in progress

• Omini model RL support with FSDP backend
• Proxy server support via RayScheduler (HTTP launcher actor) [CC]
• Weight delta update with awex
• Memory service for self-evolving agent

Usability Copy Section

On-going

• Effective AReaL CLI support
  (#1374)
• Profiling toolkit (distributed trace)
• Online RL training example based on AReaL 2.0 architecture

Planned but not in progress

• More RL post-training paradigms: IcePop Plus algorithm (verified in Ling MoE
  training) and stepwise reward rubrics
• GUI Agent VLM training example
• OS-bench VLM training example [CC]
• Diffusion image/video generation model RL post-training [CC]
• Comprehensive LoRA support over the existing policy model training
• AReaL autopilot (automatic performance-optimized RL deployment suggestions and
  systematic bottleneck analysis)

Documentation Copy Section

Planned but not in progress

• Explain benchmarking results from nightly CI and how to extract issues

2026 Q2 Roadmap (due July 31, 2026) Copy Chapter

GitHub Issue #1302.

This roadmap tracks major planned enhancements through July 31, 2026. Items are
organized into two categories:

• On-going: Features currently under active development by the core AReaL team
• Planned but not in progress: Features that are good to have but currently lacking
  bandwidth

We use [CC] to mark items suitable for community contributions. If you're interested
in contributing, please reach out to discuss implementation details.

Backends Copy Section

On-going

• Full training example with AReaL 2.0 architecture
• Deprecate support for SPMD mode (launcher, sglang/vllm server, etc.)
• Initial support of colocation weight transfer with awex

Planned but not in progress

• Migrate primary Megatron integration lib from mbridge to megatron-bridge
  (#1260) [CC]
• Full support for colocation/separation weight transfer with awex as the backend
  (areal/v2/weight_update/) [CC]
• Migrate legacy NCCL broadcast weight transfer approach from areal/engine into
  areal/v2/weight_update/ [CC]
• Omini model RL support with FSDP backend
  (#879) [CC]
• Support training the latest large MoE models with the Megatron backend, including
  dpsk-v3/v4, Kimi-2.5, GLM-4/5 [CC]
• Native Kubernetes (K8S) scheduler [CC]

Usability Copy Section

On-going

• Nightly CI workflow for performance benchmarking
  (#1284)
• Refactor unit tests for faster execution

Planned but not in progress

• OS-bench VLM training example [CC]
• Multi-agent training example (single LLM, different prompts, e.g., planner agent
  with sub-agents) [CC]
• Migrate legacy multi-turn agent examples to new API (agenerate → ArealOpenAI
  or URL-based AgentWorkflow) [CC]
• Publish PyPI packages and CLI for running experiments
• Support distributed training and debugging in Jupyter notebooks
• Implement controller construction with model-centric API similar to transformers

Documentation Copy Section

Planned but not in progress

• Explain benchmarking results from nightly CI and how to extract issues
• Document AReaL 2.0 architecture

2026 Q1 Roadmap (due April 30, 2026) Copy Chapter

GitHub Issue #907.

This roadmap tracks major planned enhancements through April 30, 2026. Items are
organized into two categories:

• On-going: Features currently under active development by the core AReaL team
• Planned but not in progress: Features with concrete implementation plans where we
  welcome community contributions

Backends Copy Section

On-going

• ZBPP & ZBPP-V support for the Archon backend
• FP8 training for Archon

Planned but not in progress

• Support for agentic training with large VLM MoE models (Archon backend)
• Omini model RL support with FSDP/Archon backend
• Decoupling agent service from the inference service
• Online RL training with the proxy server
• LoRA support for the Archon backend
• Colocation mode with awex as the weight sync engine
• Multi-LLM training (different agents with different parameters)
• Auto-scaling inference engines in single-controller mode
• Elastic weight update setup and acceleration
• RL training with cross-node vLLM pipeline/context parallelism

Usability Copy Section

On-going

• Flatten the import structure of areal modules

Planned but not in progress

• Publishing PyPI packages
• Support distributed training and debugging in Jupyter notebooks
• Example of using a generative or critic-like reward model
• Support directly constructing inference/training engines without config objects
• Add router in rollout controller for simpler proxy server usage
• Integrate aenvironment for environment handling

Documentation Copy Section

Planned but not in progress

• Use case guides: multi-agent training
• Guide for online proxy mode training

Historical Roadmaps Copy Chapter

2025 Q4 Copy Section

GitHub Issue #542.

Backends

Completed:

• Single-controller mode
• Detailed profiling for optimal performance across different scales
• Low-precision RL training (Megatron FP8)
• Data transfer optimization in single-controller mode
• New PyTorch-native backend: Archon

Carried over to Q1 2026:

• Multi-LLM training (different agents with different parameters)
• Auto-scaling inference engines in single-controller mode
• Elastic weight update setup and acceleration
• RL training with cross-node vLLM pipeline/context parallelism

Usability

Completed:

• Add CI pipeline to build Docker images upon release
• Wrap training scripts into trainers
• Refactor FSDP/Megatron engine/controller APIs to finer granularity
• Fully respect allocation mode in trainers/training scripts

Carried over to Q1 2026:

• Flatten the import structure of areal modules
• Support distributed training and debugging in Jupyter notebooks
• Example of using a generative or critic-like reward model

Canceled:

• Rename RemoteSGLang/vLLMEngine as SGLang/vLLMEngine

Documentation

Completed:

• Tutorial on how to write efficient async rollout workflows
• Benchmarking and profiling guide
• Use case guides: offline inference, offline evaluation
• AReaL performance tuning guide
  • Device allocation strategies for training and inference
  • Parallelism strategy configuration for training and inference

Carried over to Q1 2026:

• Use case guides: multi-agent training

2025 Q3 Copy Section

GitHub Issue #257.

Backends

Completed:

• Megatron training backend support
• SGLang large expert parallelism (EP) inference support
• Remote vLLM inference engine
• Ulysses context parallelism & tensor parallelism for FSDP backend
• End-to-end MoE RL training with large EP inference and Megatron expert parallelism
• Distributed weight resharder for Megatron training backend

Canceled:

• Local SGLang inference engine with inference/training colocation (hybrid engine)
• RL training with SGLang pipeline parallelism

Usability

Completed:

• OpenAI-compatible client support
• Support RLOO
• Provide benchmarking configuration examples:
  • DAPO
  • Bradley-Terry reward modeling
  • PPO with critic models
  • REINFORCE++

Documentation

Completed:

• OpenAI-compatible client documentation
• Out-of-memory (OOM) troubleshooting guide
• AReaL debugging best practices:
  • LLM server-only debugging - How to launch LLM servers independently and debug agent
    workflows
  • Mock data and torchrun debugging - Creating synthetic data and using torchrun for
    algorithm debugging
  • Training-free evaluation experiments - Running evaluations without training or
    additional GPUs

How to Influence the Roadmap Copy Chapter

We value community input! Here's how you can help shape AReaL's future:

💡 Propose New Features Copy Section

1. Check Existing Issues: Search
   issues and
   discussions to see if your idea
   already exists
2. Create a Feature Request: Use our
   feature request template
3. Discuss in GitHub Discussions: Post in
   Ideas category
   for early feedback
4. Vote on Features: Use 👍 reactions on issues to show support

🛠️ Contribute Implementation Copy Section

Check our contribution guide.

Release Cycle Copy Chapter

Minor Releases: Bi-weekly - Bug fixes, small improvements, and new features

Major Releases: Quarterly - Important milestones and significant changes

Historical Milestones Copy Chapter

Check
our historical milestone summaries since open-source.

Long-Term Vision Copy Chapter

Our vision for AReaL is to become the go-to framework for training reasoning and
agentic AI systems that is:

1. Accessible: Easy to get started, whether you're a researcher or practitioner
2. Scalable: Scales from laptop to 1000+ GPU clusters seamlessly
3. Flexible: Supports diverse algorithms, models, and use cases
4. Performant: Industry-leading training speed and efficiency
5. Open: Fully open-source with transparent development

Last Updated: 2026-06-05

Questions about the roadmap? Open a discussion in
GitHub Discussions or ask in our
WeChat group.

File: .pre-commit-config.yaml Copy Chapter

default_install_hook_types:

• pre-commit
• commit-msg

repos:

• repo: https://github.com/pre-commit/mirrors-clang-format
  rev: v19.1.7
  hooks:

  • id: clang-format
    files: .(c|cc|cxx|cpp|h|hpp|hxx|cu|cuh)$
    args:
    • --style=file
    • --fallback-style=Google

• repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v6.0.0
  hooks:

  • id: check-yaml
    files: .(yaml|yml)$
    exclude: .github/workflows/

  • id: end-of-file-fixer
    files: .(py|md|yaml|yml|c|cc|cxx|cpp|h|hpp|hxx|cu|cuh|ipynb)$

  • id: trailing-whitespace
    files: .(py|md|yaml|yml|c|cc|cxx|cpp|h|hpp|hxx|cu|cuh|ipynb)$

  • id: check-added-large-files
    name: Check for large files
    args: ['--maxkb=1000']
    exclude: uv.lock

  • id: check-json
    name: Check JSON files

  • id: detect-private-key
    name: Detect private keys

• repo: https://github.com/executablebooks/mdformat
  rev: 0.7.17
  hooks:

  • id: mdformat
    args:
    • --wrap=88
      additional_dependencies:
    • mdformat-gfm
    • mdformat-tables
    • mdformat-frontmatter
      exclude: ^(docs/(en|zh)/algorithms/.*.md|docs/(en|zh)/best_practices/algo_perf.md)$

• repo: https://github.com/astral-sh/ruff-pre-commit

  Ruff version.

  rev: v0.14.9
  hooks:

  • id: ruff # Run the linter.
    name: Run Linter Check (Ruff)
    types_or: [ python, pyi, jupyter ]
    args: [ --fix ]
  • id: ruff-format # Run the formatter.
    name: Run Formatter (Ruff)
    types_or: [ python, pyi, jupyter ]

Clean notebook outputs and metadata

• repo: https://github.com/kynan/nbstripout
  rev: 0.7.1
  hooks:
  • id: nbstripout
    name: nbstripout - Strip notebook output

Check consistency between pyproject.toml variants (sglang vs vllm)

• repo: local
  hooks:
  • id: check-pyproject-consistency
    name: Check pyproject.toml consistency
    entry: python3 areal/tools/check_pyproject_consistency.py
    language: system
    files: ^pyproject(.vllm)?.toml$
    pass_filenames: false
    always_run: false
    require_serial: true

Regenerate uv lockfiles when pyproject files change

• repo: local
  hooks:
  • id: uv-lock
    name: Regenerate uv lockfiles
    entry: bash scripts/uv_lock.sh
    language: system
    files: ^pyproject(.vllm)?.toml$
    pass_filenames: false
    always_run: false
    require_serial: true

Generate CLI documentation

• repo: local
  hooks:
  • id: generate-cli-docs
    name: Generate CLI documentation
    entry: python docs/generate_cli_docs.py
    language: system
    files: ^(areal/api/cli_args.py|docs/generate_cli_docs.py)$
    pass_filenames: false
    always_run: false
    require_serial: true

Ensure SPDX license header in Python source files

• repo: local
  hooks:
  • id: check-license-header
    name: Check SPDX license header
    entry: python3 areal/tools/check_license_header.py
    language: system
    files: ^areal/.*.py$
    types: [python]

Format and lint .github/CODEOWNERS

• repo: local
  hooks:
  • id: format-codeowners
    name: Format CODEOWNERS
    entry: python3 areal/tools/format_codeowners.py
    language: system
    files: ^.github/CODEOWNERS$
    pass_filenames: false
    always_run: false
    require_serial: true

Conventional Commits message check

• repo: https://github.com/compilerla/conventional-pre-commit
  rev: v4.4.0
  hooks:
  • id: conventional-pre-commit
    stages: [commit-msg]
    args:
    • feat
    • fix
    • docs
    • gov
    • style
    • refactor
    • perf
    • test
    • build
    • ci
    • chore
    • revert

File: areal/v2/agent_service/README.md Copy Chapter

[Binary file]

File: assets/community/README.md Copy Chapter

This file has been moved to the
community repository, and is no longer
maintained here. Please refer to
the new location for the
latest information and updates regarding the community.

File: benchmark/README.md Copy Chapter

Benchmark

This directory contains guides for reproducing AReaL's throughput and algorithmic
performance benchmarks, as reported in the paper.

As of January 2026, AReaL has been fully refactored from the original realhf directory
to the new areal directory. The legacy configurations and instructions are no longer
compatible with the current codebase. For reference, you can still access the legacy
configs and run experiments within realhf by reverting to commit 820ca49 or earlier.

Updated benchmark results for the new codebase will be available soon.

File: blog/AReaL_v0_1.md Copy Chapter

*AReaL* v0.1: Reliable RL Training of 1.5B and 7B LLMs on Math Reasoning Tasks

Our initial release AReaL v0.1 includes reproducible experiments based on our AReaL
system for 1.5B and 7B LRMs, validated across diverse computational budgets. We
demonstrate that with AReaL, users can:

1. Reliably train a 1.5B distilled model with RL to surpass o1-Preview on math
   reasoning within 40 hours 🚀
2. Reliably perform an R1-Zero-style experiment with a 7B model by running RL
   training on Qwen2.5-7B, and observe emergent thinking tokens and continuous model
   improvement on math reasoning

Training a 1.5B LRM from the Distilled Model Copy Chapter

Starting with the
DeepSeek-R1-Distill-Qwen-1.5B
model, we follow DeepScaleR to iteratively increase the output context length from 8K to
16K and eventually 24K. The training reward continuously increases during RL training.
We observe that the response length first shrinks in the 8K training stage, then
increases in the 16K and 24K training stages.

Our experiments are conducted on 16 nodes, each equipped with 8 H800 GPUs. The results,
along with the associated training curves, are presented below.

Figure 1. Training rewards and response lengths during RL training. The base model is
DeepSeek-R1-Distill-Qwen-1.5B. Curves are averaged with a window size of 25.

These three stages progressively enhanced the performance of the base model, as
demonstrated below:

Table 1. Evaluation on competition-level mathematics benchmarks including AIME 2024,
AMC 2023, and MATH-500. Results report Pass@1 accuracy averaged over 32 samples per
problem at temperature 0.6.

RL Training Details Copy Section

Our RL training process incorporates insights from
DeepSeek-R1,
DeepScaleR, and
ReaLHF. Key components include:

• Base Model:
  DeepSeek-R1-Distill-Qwen-1.5B
• Dataset: 40k high-quality mathematical reasoning tasks from
  DeepScaleR
• Reward Design: +5 for correct responses, -5 for incorrect responses (evaluated
  using Sympy following
  Qwen's protocol)
• RL Algorithm: Simplified PPO variant without the critic network
• Key Hyperparameters:
  • Discount factor (γ) and GAE parameter (λ) both set to 1 (following
    Open-Reasoner-Zero)
  • Small KL reward coefficient
• Iterative Context Lengthening: Three-stage training (8K→16K→24K) following
  DeepScaleR
• Batch Size: 1024 questions per step with 8 rollouts per question, which is
  larger than prior works

AReaL with Various Computational Resources Copy Section

AReaL adapts efficiently to different computational budgets. Increased resources
accelerate RL training, significantly boosting research progress. We provide detailed
hardware requirements and environment setup guides for different configurations in
our tutorials.

Figure 2. Total RL training time for 10 epochs across different resource
configurations.

Qwen2.5-7B-Zero RL Training Copy Chapter

We conduct RL training starting from Qwen2.5-7B using DeepSeek-R1-Zero style training.
Initial results using data from
Open-Reasoner-Zero show both
rewards and response lengths gradually increasing during training. This simultaneous
growth of response lengths and rewards suggests emergent deep thinking
capabilities in solving complex reasoning problems.

Figure 3.
Qwen2.5-7B-Zero RL training curve

Evaluation of intermediate checkpoints on MATH500 and AIME24 datasets shows continuous
improvement in both accuracy and response length:

Figure 4. Test accuracy
and response length on MATH500 and AIME24 datasets

Additional experiments on the
DeepScaleR dataset show similar
trends:

Table 2. Evaluation on AIME 2024, AMC 2023, and MATH-500 (Pass@1 accuracy averaged over
32 samples at temperature 1.0).

Open Research Questions on Algorithm and Model Copy Chapter

Based on AReaL, our team is also actively exploring the research frontier of LRMs. We
highlight some of the major research topics we are working on.

• Effective Training Datasets for LRMs How do we design training tasks that enable
  LRMs to progressively evolve sophisticated reasoning skills via RL?
  • For more advanced distilled models, such as DeepSeek-R1-Distill-Qwen-7B and
    DeepSeek-R1-Distill-Qwen-32B, what kind of task distribution could drive further
    enhancements?
  • How to properly curate the distillation dataset for cold-start to achieve the best
    RL results?
• Boosting Reasoning with Auxiliary Training Signals. Can auxiliary signals - such
  as Process Reward Models (PRMs) and expert-curated labels - be adopted in RL training
  to refine the reasoning processes of LRMs?
• Adaptive Response Lengths: Quality Over Quantity. How can RL training schemes
  teach LRMs to dynamically adjust response complexity based on task difficulty? The
  goal is to eliminate redundant "over-reasoning" for simpler inputs while preserving
  rigor for challenging problems.
• Sample Efficient RL for LRMs In addition to system-level acceleration, can we
  further speed up the LRM training process with a more sample-efficient advanced RL
  algorithm? Can we get the critic back to reduce gradient variance without compromising
  the computation efficiency? How to better encourage exploration?

File: blog/AReaL_v0_2.md Copy Chapter

*AReaL* v0.2: Training a SOTA 7B LRM with 1.5x Throughput Improvement

Introduction Copy Chapter

We are excited to release AReaL v0.2 (boba), featuring three major milestones:

• SGLang Support: With the addition of SGlang support and a series of engineering
  optimizations, AReaL v0.2 achieves a speed improvement of 1.5x over AReaL v0.1 on 7B
  models.
• SOTA 7B Model: AReaL's RL training becomes more stable and sample-efficient. We
  obtain a SOTA 7B model in mathematical reasoning, which achieves pass@1 score of 61.9
  on aime24 and 48.3 on aime25 respectively.
• Competitive 32B Model: We train a highly competitive 32B model at an extremely low
  cost, achieving results comparable to QwQ-32B using only 200 data samples.

Table 1: The performance of AReaL-boba-RL-7B and AReaL-boba-SFT-32B. We obtain SOTA 7B
model using RL on math reasoning. Although our dataset primarily consists of math and
logic problems, we observed that RL training led to measurable improvements on the
challenging STEM benchmark GPQA. Additionally, We train a highly competitive 32B model
using only 200 data samples, replicating QwQ-32B's inference performance on AIME
2024.

We declare that the reported numbers are based on our
re-testing, with each number representing the average results of 32 sampling responses.
The evaluation code was available in the evaluation/ folder. Please roll back to
commit f55fe68
(Feb 4, 2026) or earlier.

For the baselines and the SFT model AReaL-boba-SFT-32B, we follow the recommended
configuration (temperature = 0.6, top_p = 0.95, suggested by DeepSeek) and use the
default R1-Distill-Qwen template for prompts. For the RL-trained models, we retaine the
same temperature as during RL rollout (1.0).

Notably, during GPQA testing, we found that all answers are "A." To address this bias,
we randomized the answer options.

Training Speed Comparison Copy Section

AReaL v0.2.0 features the following system optimizations:

• Upgraded Generation Backend: vLLM 0.6.3 → SGLang v0.4.0

The generation backend has been upgraded from vLLM 0.6.3 to SGLang v0.4.0, leveraging
its radix attention mechanism to significantly improve throughput in scenarios where
multiple responses are sampled from the same prompt. Additionally, SGLang automatically
flushes radix caches upon weight updates, ensuring correctness in on-policy
reinforcement learning (RL). We will continue tracking advancements in the community to
integrate further optimizations.

• Optimized Training for Variable-Length Sequences & Large Batches

To handle variable sequence lengths efficiently, we eliminate padding and instead pack
sequences into 1D tensors. A dynamic allocation algorithm (approximately) optimally
distributes sequences under a maximum token budget, balancing micro-batch sizes while
minimizing the number of micro-batches. This approach maximizes GPU memory utilization,
enabling efficient computation across a large batch of variable-length inputs.

• High-Performance Data Transfer for 1K-GPU Scaling

AReaL employs NCCL with GPU-Direct RDMA (GDRDMA) over InfiniBand/RoCE, enabling direct
GPU-to-GPU communication that bypasses costly CPU-mediated transfers and PCIe
bottlenecks. Compared to traditional Ethernet-based approaches, this reduces latency and
increases throughput, keeping generation-to-training data transfer overhead below 3
seconds even in a 1,000-GPU cluster.

Training Recipe Copy Chapter

SOTA 7B model using RL on math reasoning Copy Section

Base Model

We use
R1-Distill-Qwen-7B as
our foundation model.

Dataset Curation

Our training dataset
(AReaL-boba-106k)
combines resources from multiple open-source projects:

• DeepScaleR
• Open-Reasoner-Zero
• Light-R1
• DAPO

We enhanced this with challenging problems from:

• NuminaMath (AoPS/Olympiad
  subsets)
• ZebraLogic

To maintain an appropriate difficulty level, we filtered out overly simple questions.
Specifically, We generate 8 solutions per question using DeepSeek-R1-Distill-Qwen-7B and
filter out questions where all solutions were correct.

Reward Function

We adopt a sparse sequence-level reward mechanism. The model is instructed to enclose
the final answer within \boxed{}, and the boxed answer is then verified. Correct
responses receive a reward of +5, while incorrect ones are penalized with -5.

Notably, we observe that the KL reward can impair performance, particularly in long
chain-of-thought training, so we set it to zero.

RL Algorithm

We employ Proximal Policy Optimization (PPO) as our training algorithm. We remove the
critic model to save compute. We set both the discount factor γ and the GAE parameter λ
to 1. Such practices are also adopted by the
Open-Reasoner-Zero project.

Token-Level Loss Normalization

Averaging the loss in the sequence level can underweight the overall contribution of
longer texts. To address this, we normalize the loss at the token level, this practice
is also highlighted in DAPO.

Rollout Strategy

During the rollout phase, we sample 512 questions per batch, and the LLM generates 16
responses per question—resulting in a total batch size of 8,192. To minimize output
truncation, we set the maximum generation length to 27K tokens. In our experiment, the
truncation rate remained below 5%.

Advantage Normalization

During the training phase, we compute advantages using GAE and normalize them across all
generated tokens.

Key Hyperparameters

This configuration balances convergence speed with training stability, avoiding collapse
risks from higher learning rates or smaller ε values.

Approaching QwQ-32B's performance using only 200 data samples Copy Section

At the 32B model size, we further refine the training data and released
AReaL-boba-SFT-200,
a high-quality dataset with only 200 data points. Accompanied by relevant training
scripts, we replicated QwQ-32B's inference performance on AIME 2024 via
Supervised Fine-Tuning (SFT).

Evaluation Best Practices Copy Section

During evaluation, we use vLLM v0.6.3 as the generation framework. We identified several
settings that influence evaluation performance, particularly for long-context
generation. We recommend manually configuring the following options:

enforce_eager=True
no_enable_chunked_prefill=True
disable_custom_all_reduce=True
disable_sliding_window=True

Following the practice of DeepSeek models, we incorporate a directive in the prompt:
"Please reason step by step, and enclose your final answer in \boxed{}." To encourage
long context reasoning, we also enforce that the model begins each response with "\n"
at the start of its output.

To ensure reliable pass@1 estimation, we:

• Sample 32 answers per problem
• Use temperature=0.6 and top_p=0.95 for SFT models
• Maintain training temperature (1.0) for RL models

Conclusion & Future Work Copy Chapter

Our results demonstrate that high-quality data is equally critical as algorithmic
innovations. When conducting RL training on a powerful base model, we require more
challenging problems to facilitate learning. Therefore, we integrate resources from
multiple recent open-source projects and filter problems by difficulty. A
straightforward strategy for data filtering involves removing problems that the base
model consistently solves correctly across multiple sampling attempts, as they no longer
contribute to improving the model's performance.

AReaL delivers stable and fast training with cutting-edge model performances. Since
initial release, we've continuously improved system efficiency, training stability, and
accessibility. All aforementioned techniques have been implemented in AReaL, with
reproducible configurations for various model sizes and
different hardware setups.

Looking ahead, the AReaL team will:

• Further optimize the RL training throughput
• Introduce new algorithmic features
• Continue open-sourcing training data
• Expand to broader reasoning tasks

We believe these contributions lower the barrier for high-quality RL training while
pushing the boundaries of reasoning capabilities. The project will continue evolving -
we welcome community feedback and collaboration to drive further progress in this
exciting field.

File: blog/AReaL_v0_3.md Copy Chapter

*AReaL* v0.3: SOTA Coding Models with 2.77x Faster Asynchronous RL Training

Introduction Copy Chapter

We now release AReaL v0.3, featuring three major milestones:

• A fully asynchronous RL training pipeline with system and RL algorithm co-design,
  achieving over 2.77x speedup without any performance drop
• SOTA coding models, i.e., a 14B model with a 69.1 score on LCB-v5.
  Reproducible results with fully open-sourced datasets are also provided
• Experimental support for multi-turn agentic RL training

Performance Results Copy Section

Table 1: Coding Task Performance Comparison. AReaL-boba²-8B/14B-Open denotes training
results on open-sourced data. AReaL-boba²-8B/14B models are trained with an additional
small amount of internal data and achieve SOTA performance on LiveCodeBench, Codeforce &
CodeContests.

To access our latest models and training data, please visit this
Huggingface Link.

Motivation for Asynchronous RL System Copy Chapter

Inference devices are underutilized Copy Section

During the synchronous RL training process, the generation step must wait until the
finish of the longest output within a batch. Due to the varying output lengths for LRM,
a synchronous RL system suffers from severe training inefficiency.

Some works
(DeepCoder,
Intellect) use outputs generated from
a previous model version to update the current model. For the best performances, the
model version used for rollout generation is limited to only one or two steps older.
However, all these systems still follow a batched generation setting, the issue of
system inefficiency during the generation phase still remains unaddressed.

Fig.1. Left: Execution timeline of a synchronous RL training. Right: Execution timeline
of one-step overlap RL system.

Scalability is poor in synchronous RL systems Copy Section

Synchronous systems distribute generation across all devices, reducing the per-GPU
decoding batch size. This pushes the decoding process into a memory-IO-bound regime
where additional devices fail to improve throughput.

Fig2. Left: Strong scaling of batched generation throughput for a 1.5B LRM. Right:
Generation becomes memory-IO bound as GPU count increases.

Asynchronous AReaL Overview Copy Chapter

We design a system that fully decouples generation and training across separate GPU
clusters. This system should be hardware-efficient, scalable, and equipped with the
flexibility for a customized RL workflow. We implement these principles in AReaL. Fig.3
presents the architecture and data flow of AREAL. The system comprises 4 core
components:

Fig.3 The architecture featuring asynchronous generation and training components.

Core Components Copy Section

• Interruptible Rollout Worker handles two types of requests: (1) The generate
  request generates responses given prompts. (2) The update_weights request interrupts
  all ongoing generations and loads parameters of new versions. Upon the interruption,
  the rollout workers discard KV caches computed by old weights, and re-compute them
  using the new weights. Afterwards, the rollout workers continue to decode the
  unfinished sequences until the next interruption or termination. We emphasize that
  such interruptions and in-flight weight updates would result in trajectories composed
  of segments produced by different model versions. This introduces a novel algorithmic
  challenge, which will be addressed in the next section.

• Reward Service evaluates the accuracy of the responses generated by the model. For
  example, in the coding task, this service extracts the code and executes unit tests to
  verify its accuracy.

• Trainer Workers continuously sample from the replay buffer, accumulating data
  until reaching the configured training batch size. They then perform PPO updates and
  store the resulting parameters in distributed storage. To ensure data freshness, data
  from the replay buffer is used only once.

• Rollout Controller serves as a critical bridge between the rollout workers, reward
  service, and the model workers. During the training process, it reads data from the
  dataset and invokes the rollout worker's generate request. The received response is
  then sent to the reward service to obtain the reward. The trajectory, along with the
  reward, is stored in the replay buffer, waiting to be trained by the model worker.
  After the model worker updates the parameters, the controller calls the rollout
  worker's update_weight. We illustrate the generation and training management in
  Fig.4. This asynchronous pipeline ensures continuous full utilization of both
  generation and training resources.

Fig 4. Execution timeline of our fully asynchronous RL system.

Algorithmic Challenges and Solutions for Asynchronous RL Copy Chapter

Challenges Copy Section

While the asynchronous system design offers significant acceleration through improved
device utilization, it introduces several technical challenges that require algorithmic
considerations.

• Challenge 1: Data Staleness - Due to the asynchronous nature of AREAL, each
  training batch contains data from multiple prior policy versions. Data staleness would
  lead to a distribution gap between the training data and the latest model. In
  asynchronous RL training for LRMs, this issue could be even more severe for long
  trajectories due to extended decoding time.

• Challenge 2: Inconsistent Policy Versions - As shown in Fig.4, a single generated
  trajectory may involve segments produced by different policy versions. This
  inconsistency fundamentally violates the formulation of standard PPO that assumes all
  actions being generated by a single policy.

Solutions Copy Section

To overcome these two challenges, we propose two solutions:

• Solution 1: Staleness-Aware Training - To avoid negative impact of the data
  staleness issue, we constrain the version discrepancy between the policy version of
  generated trajectories and the training policy. We introduce a hyperparameter η
  representing the maximum permitted staleness. When η = 0, the system degenerates to
  the synchronous RL setting where generation exactly matches training batches. When η =
  1, the system recovers to the previous one-step overlap methods.

• Solution 2: Decoupled PPO Objective - We apply a decoupled PPO objective that
  disentangles the behavior policy and the proximal policy. The behavior policy
  represents the policy used for sampling trajectories and the proxy policy is a
  proximal policy serving as a recent target to regularize the update of online policy.

Validating Asynchronous AReaL Copy Chapter

We first evaluate Asynchronous AReaL on a math task to validate our system and
algorithmic choices. Building on our previous release, AReaL-boba, we adopt
R1-Distill-Qwen as the base model and
AReaL-boba-106k
as the training dataset.

End-to-End Performance Comparison Copy Section

We compared synchronous and asynchronous training on 1.5B and 7B models. Under equal
resource constraints and training steps, the asynchronous system was over twice as fast
as the synchronous one. Evaluations on AIME24 confirmed that this acceleration does not
compromise performance.

Table 2: End-to-end comparison between asynchronous AReaL and synchronous AReaL.
Following last release AReaL-boba, we adopt R1-Distill-Qwen as the basemodel and
AReaL-boba-106k
as the training data. Each performance number represents the average results of 32
sampling responses.

Ablation Study Copy Section

System Ablations

We ablate interruptible generation and present the resulting generation throughput in
Fig.5. Without interruptible generation, the controller must wait for the longest
response. In particular, interruptible generation leads to a 12% and 17% throughput
increase for 1.5B and 7B models respectively on 4 nodes.

Fig.5 Ablation study of interruptible generation.

Algorithm Ablations

We vary the maximum allowed staleness η and compare configurations with and without the
decoupled PPO objective. Fig.6 shows the learning curves after 1000 PPO updates,
demonstrating that naive PPO fails to match the performance of the synchronous RL oracle
(i.e., the performance when η = 0). Even slight staleness can significantly degrade
final performance due to the improper clipping center and policy changes during
interruptible generation. Furthermore, increasing data staleness consistently degrades
learning performance.

Fig.6 Ablation Study on Decoupled PPO Objective with DeepSeek-R1-Distill-Qwen-1.5B.
Left: Learning curves with naive PPO. Right: Learning curves with decoupled PPO
objective.

Table 3: Evaluation scores when varying data staleness, comparing performance with and
without the decoupled objective.

In practice, the effect of staleness could depend on various factors, such as learning rate, batch size, the number of mini batches, generation length, and also the RL algorithm. Here we aim to illustrate the impact of staleness on the training performance of a synchronous RL system, and show that stalenss control and decoupled PPO could well mitigate the negative impact of staleness. Here we follow a large batch size setting with `batch_size=512`, `n_rollouts=16`, `n_mini_batch=4`, `max_new_tokens=8192`, and `lr=2e-5`. The staleness is defined as the gap of model versions, where the model version increases by 1 when a batch of `batch_size` X `n_rollouts` samples are trained on.

However, the decoupled PPO objective substantially improves training stability when
handling stale data. Notably, even with the decoupled objective, unbounded staleness
(maximum staleness → ∞) still results in inferior performance compared to the
zero-staleness oracle. When properly constrained, moderate staleness (e.g., η ≤ 4) has
minimal impact on final performance while significantly accelerating training through
the asynchronous pipeline, as demonstrated in Tab.3 and Fig.7.

Fig.7 The relationship between η and training throughput. Larger η leads to higher
throughput.

Training Recipe with Asynchronous AReaL Copy Chapter

Dataset Curation Copy Section

We train the 7B and 14B model using open-source data from
DeepCoder.
After deduplication and quality filtering, approximately 7,600 coding problems remained.
Additionally, we have around 10,000 internal data entries that can further improve the
model's performance. We are actively working on open-sourcing this portion of the data.

Rollout Strategy Copy Section

During the rollout phase, the LLM generates 16 responses per question. To minimize
output truncation, we set the maximum generation length to 27K tokens.

Filtering Strategy Copy Section

In the design of our asynchronous system, we can naturally incorporate strategies to
filter some rollouts. Specifically, we exclude the questions where all 16 answers are
either entirely correct or entirely incorrect.

Key Hyperparameters Copy Section

Resources Copy Chapter

• boba² Code dataset
• Reproduce boba² Code Models
  • 🤗 Model weights: 8B-code,
    14B-code,
    8B-code-open,
    14B-code-open
  • Evaluation Guide
  • Training configs
    and instructions (reference/reproduce.html, revert to a version before v0.5.1 to
    see the reproduction guide)

File: docs/adapt-bailing-moe-v2.5.md Copy Chapter

BailingMoeV2.5 Adaptation Guide for AReaL

1. Architecture Overview Copy Chapter

BailingMoeV2.5 is a heterogeneous Transformer with 4 types of layer combinations:

BailingMoeV2_5ForCausalLM
  Embedding → N TransformerLayers (heterogeneous) → Final LayerNorm → lm_head

Each layer = Attention (Lightning or MLA) + MLP (Dense or MoE).

Layer Assignment Rules Copy Section

• Attention: Every layer_group_size-th layer uses MLA; others use Lightning
  Attention. Formula: is_lightning = (layer_number + 1) % layer_group_size != 0
• MLP: First first_k_dense_replace layers use Dense; rest use MoE (256 experts,
  sigmoid routing, grouped TopK with n_group=8, topk_group=4, shared experts, expert
  bias).

Attention Mechanisms Copy Section

Lightning Attention (linear, O(n)):

• Fused QKV → Q/K LayerNorm → Partial RoPE (first half of head_dim only) →
  chunk_simple_gla kernel (fla library) → Gate (sigmoid gating with g_proj +
  g_norm on attn output) → Output proj
• Per-head decay via ALiBi slopes, scaled by layer depth:
  g_gamma = -alibi_slopes * (1 - (layer_idx - 1) / (N - 1) + 1e-5)

MLA (Multi-Latent Attention) (standard softmax):

• Low-rank KV compression via q_a_proj → q_a_layernorm → q_b_proj and
  kv_a_proj → kv_a_layernorm → kv_b_proj
• Reuses megatron-core MLASelfAttention directly

Model Variants Copy Section

2. Implementation: Key Files & Design Copy Chapter

File Map Copy Section

Heterogeneous Layer Spec Construction Copy Section

Cannot use single get_gpt_decoder_block_spec() (assumes uniform layers). Build 4 spec
variants:

1. Generate base specs with get_gpt_layer_with_transformer_engine_spec() for
   MLA+Dense, MLA+MoE, Standard+Dense, Standard+MoE
2. Deep-copy Standard specs and replace self_attention with custom
   LightningSelfAttention
3. Per-layer: select spec based on is_lightning_layer() and is_moe

QKV Format Conversion Copy Section

HF stores [Q_all, K_all, V_all] concatenated [3, H, D, hidden]. Megatron expects
per-head interleaved [H, 3, D, hidden]. Conversion: permute(1, 0, 2, 3). Inverse for
NCCL push is the same permute in reverse direction. Both load and save paths verified as
precise inverses.

Key Design Decisions Copy Section

• rotary_percent=1.0 in AReaL (not 0.5 as in HybridEngine) because megatron-core MLA
  passes qk_pos_emb_head_dim=64, so 64 * 1.0 = 64 gives correct RoPE dim
• Lightning layers create their own RotaryEmbedding (different RoPE dim from MLA)
• Uses chunk_simple_gla (not chunk_lightning_attn) to pass custom ALiBi g_gamma

3. Critical Bug Fixes Copy Chapter

Bug A: `layer_idx` Offset Error (root cause of loss=7.8) Copy Section

Symptom: Initial loss 7.8 instead of expected <=2.

Root cause: HF reference uses (self.layer_idx - 1) in slope formula (0-indexed
minus 1), which gives layer 0 a scale of 1.0526. AReaL intentionally uses
self.layer_idx directly (layer 0 scale = 1.0), matching SGLang and HybridEngine
behavior. The original bug was that the slope formula was missing entirely.

Fix: layer_scale = 1.0 - self.layer_idx / max(self.num_layers - 1, 1) + 1e-5

Bug B: `g` vs `g_gamma` Parameter Copy Section

g and g_gamma are mathematically equivalent for constant decay. But the g path
computes backward gradients on the constant buffer via
chunk_local_cumsum(dg, reverse=True), introducing gradient noise that raised loss from
7.8 to 14.42. g_gamma is the correct choice.

Bug C: `gate_norm` Applied Twice Copy Section

Gate norm was applied to both gate and attn_output using the same weight. HF reference
norms only attn_output, then multiplies by sigmoid(raw_gate).

Fix: Remove gate = self.gate_norm(gate), keep only
attn_output = self.gate_norm(attn_output).

Bug D: Missing QKV Format Conversion Copy Section

TP>1 weight loading silently produced garbage without format conversion.

Fix: Added _load_fused_qkv_weight() implementing [3,H,D,hidden] → [H,3,D,hidden]
permutation + per-head TP slicing.

Bug E: `q_lora_rank` Default Inconsistency Copy Section

bailing_moe.py defaulted to 512, bridge defaulted to None. Unified to None.

4. Context Parallelism (CP) Support Copy Chapter

Strategy: All-to-All Head Parallelism Copy Section

For Lightning Attention layers (linear attention is head-independent):

Input: [S/CP, B, H_local, D]
  → all_to_all CP→HP → [S, B, H_local/CP, D]  (full seq, fewer heads)
  → zigzag undo (restore sequential order)
  → RoPE (BSHD path, continuous positions)
  → chunk_simple_gla (g_gamma CP-sliced)
  → zigzag redo
  → all_to_all HP→CP → [S/CP, B, H_local, D]

For MLA layers: megatron-core/TE ring attention (built-in CP).

MLA RoPE Monkey-Patch Copy Section

Problem: megatron-core 0.13.1 truncates rotary_pos_emb to [0:total_tokens/CP],
but _get_thd_freqs_on_this_cp_rank needs the full-length freq table for zigzag
position indexing.

Solution: Detect truncated freqs, rebuild to full length using
freqs[p] = p * freqs[1] (valid for standard RoPE where outer(positions, inv_freq) is
linear in position).

Sequence Alignment Constraint Copy Section

All CP paths require: each sub-sequence length ≡ 0 (mod tp_size × cp_size × 2).

Gradient Propagation Copy Section

postprocess_packed_seqs_context_parallel uses output.detach() for all-gather, then
replaces the local rank's slice with the original (gradient-carrying) tensor. This is a
correct gradient-partitioning strategy: each CP rank backpropagates only through its own
tokens.

5. CP=8 NCCL SIGSEGV Crash Copy Chapter

Symptom: CP=8 with megatron:(attn:d2p2t2c8|ffn:d2p2e16) on 64 GPUs crashes at step
110-112 with SIGSEGV.

Root cause: Ray scheduler does not set NCCL_ASYNC_ERROR_HANDLING=1 (unlike torch
elastic agent). Without this, NCCL communicators run in blocking mode. With 5+
communicators (TP, PP, CP, EP, DP) competing on the same GPU, blocking ncclGroupEnd()
calls create resource contention that gradually corrupts NCCL internal state.

Fix: Add NCCL_ASYNC_ERROR_HANDLING: "1" to worker env_vars. Recommended as global
default.

Why CP=4 didn't crash: Fewer communicators (3-4), insufficient contention.

Why NCCL_DEBUG=INFO also prevented crash: Extra logging I/O shifted timing
(Heisenbug).

6. Configuration Examples Copy Chapter

SFT (no CP) Copy Section

allocation_mode: megatron:d8p4t2e8   # 64 GPUs
actor:
  dtype: bfloat16
  gradient_checkpointing: true

CP=8 Copy Section

allocation_mode: megatron:(attn:d2p2t2c8|ffn:d2p2e16)
env_vars:
  NCCL_ASYNC_ERROR_HANDLING: "1"   # Required for CP>=8 with Ray

Performance Results Copy Section

Environment Dependencies Copy Section

7. Validation: AReaL vs HybridEngine Copy Chapter

Mini model comparison (real text, seq_len=512, EP=8):

Per-layer: Lightning layers cosine 0.999+, MLA layers 0.92-0.95 (from TE/flash_attn
version differences, not a bug).

8. Known Open Issues Copy Chapter

1. expert_bias HF name: Fixed — NCCL path now uses expert_bias (matching HF
   checkpoint).
2. kv_channels may be None: hf_to_mcore_base_args() reads head_dim which
   BailingMoe doesn't have. Should explicitly set kv_channels.
3. BailingHybridForCausalLM not in registry.py: Bridge registers bailing_hybrid
   but registry.py only checks BailingMoeV2_5ForCausalLM and
   BailingMoeLinearForCausalLM.

File: docs/requirements.txt Copy Chapter

jupyter-book
matplotlib
numpy

File: docs/en/_config.yml Copy Chapter

Book settings

title: AReaL Documentation
author: AReaL Team
copyright: "2026"
logo: ../figures/logo.png

execute:
execute_notebooks: force

latex:
latex_documents:
targetname: book.tex
url: https://github.com/areal-project/AReaL
path_to_book: docs
branch: main

html:
use_issues_button: true
use_repository_button: true
extra_footerjs:
- _static/js/lang-toggle.js
extra_css:
- _static/css/lang-toggle.css

build:
dir: ../_build/en
figureurl: /figures

sphinx:
config:
html_extra_path:
- _static
- ../figures

File: docs/en/_toc.yml Copy Chapter

English Table of Contents

format: jb-book
root: intro
parts:

• caption: Version History
  chapters:
  • file: version_history
• caption: Tutorial
  chapters:
  • file: tutorial/installation
  • file: tutorial/installation_npu
  • file: tutorial/quickstart
  • file: tutorial/agentic_rl
  • file: tutorial/online_proxy
  • file: tutorial/eval
  • file: tutorial/megatron
  • file: tutorial/archon
  • file: cli_reference
• caption: Code Walkthrough
  chapters:
  • file: tutorial/gsm8k_grpo
• caption: Best Practices
  chapters:
  • file: best_practices/algo_perf
  • file: best_practices/workflow
  • file: best_practices/debugging
  • file: best_practices/handling_oom
  • file: best_practices/perf_profiling
• caption: Customization
  chapters:
  • file: customization/dataset
  • file: customization/data_engineering
  • file: customization/agent
  • file: customization/advantage
• caption: Algorithms
  chapters:
  • file: algorithms/async
  • file: algorithms/distillation
  • file: algorithms/dpo
  • file: algorithms/grpo_series
  • file: algorithms/m2po
  • file: algorithms/prox_approx
  • file: algorithms/prm
• caption: Reference
  chapters:
  • file: reference/checkpointing
  • file: reference/metrics_tracking
  • file: reference/moe_visualization
  • file: reference/alloc_mode
  • file: reference/lora
  • file: reference/bridge_backend
  • file: reference/tree_training
  • file: reference/rollout_workflow
  • file: reference/agent_workflow
  • file: reference/ai_assisted_dev