## 1. Project Overview & Quickstart (rllm-org/rllm)

## File: README.md

rLLM is an open-source framework for training language agents with reinforcement learning. Bring any harness, run it in any sandbox, and switch training backends with one flag — the same agent code drives both eval and training.

## Core features

- **Any harness.** 10+ CLI harnesses (Claude Code, Codex, Terminus-2, mini-swe-agent, opencode, ...) plus Harbor-compatible task dirs. Or wrap your own agent — LangGraph, OpenAI Agents SDK, `openai.OpenAI` — with `@rllm.rollout`.
- **Any sandbox.** Docker, Daytona, Modal, or local — with snapshot + warm-pool acceleration to keep rollouts cheap at training-scale.
- **Multiple training backends, one API.** `verl` (distributed multi-GPU), `tinker` (single-machine), `fireworks` (Fireworks platform). Switch with one flag.
- **60+ integrated benchmarks.** Math, code, MCQ, QA, search, VLM, translation, agentic — Terminal-Bench 2.0, SWE-bench, SkillsBench, AIME, MATH-500, GPQA, and more. `rllm eval <name>` auto-pulls and runs.
- **Multiple training methods.** GRPO, REINFORCE, RLOO, SFT, on-policy distillation, and more.
- **Battle-tested.** State-of-the-art open-source results (DeepScaleR-1.5B, DeepCoder-14B, DeepSWE-32B, FinQA-4B). Adopted by academic labs and industry research teams (see [Community Projects](#community-projects) below).

Read more on our [documentation site](https://docs.rllm-project.com/).

## Installation

rLLM requires `Python >= 3.11`. You can install it either directly via pip or build from source.

```bash
uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git"
```

This installs dependencies for running `rllm` CLI with the `tinker` backend (single-machine, Tinker API). For other backends:

```bash
# Distributed multi-GPU training (verl + vLLM/SGLang)
uv pip install "rllm[verl] @ git+https://github.com/rllm-org/rllm.git"

# Fireworks training platform
uv pip install "rllm[fireworks] @ git+https://github.com/rllm-org/rllm.git"
```

For building from source or Docker, see the [installation guide](https://docs.rllm-project.com/installation).

## Quickstart

### Option A: CLI (no code needed)

```bash
# 1. Configure your model provider
rllm model setup

# 2. Evaluate on a benchmark
rllm eval gsm8k

# 3. Train with RL
rllm train gsm8k
```

### Option B: Python API

Define a rollout (your agent) and an evaluator (your reward function), then hand them to the trainer:

```python
# my_flow.py
from openai import OpenAI
import rllm
from rllm.types import AgentConfig, Episode, Task, Trajectory

@rllm.rollout
def solve(task: Task, config: AgentConfig) -> Episode:
    client = OpenAI(base_url=config.base_url, api_key="EMPTY")
    response = client.chat.completions.create(
        model=config.model,
        messages=[{"role": "user", "content": task.instruction}],
    )
    answer = response.choices[0].message.content or ""
    return Episode(
        trajectories=[Trajectory(name="solver", steps=[])],
        artifacts={"answer": answer},
    )
```

```python
# my_evaluator.py
import rllm
from rllm.eval.types import EvalOutput, Signal
from rllm.types import Episode

@rllm.evaluator
def score(task: dict, episode: Episode) -> EvalOutput:
    answer = str(episode.artifacts.get("answer", ""))
    is_correct = answer.strip() == task["ground_truth"].strip()
    reward = 1.0 if is_correct else 0.0
    return EvalOutput(reward=reward, is_correct=is_correct,
                      signals=[Signal(name="accuracy", value=reward)])
```

```python
# train.py
from rllm.trainer import AgentTrainer
trainer = AgentTrainer(
    backend="tinker",
    agent_flow=solve,
    evaluator=score,
    config=config,
    train_dataset=dataset,
)
trainer.train()
```

During training, `config.base_url` points to a gateway that transparently captures token IDs and logprobs — your agent code stays the same for eval and training.

See the [cookbooks](./cookbooks) for complete working examples (single-turn VLM solver, multi-agent solver-judge, and more).

## Architecture

rLLM follows a simple pipeline: **run your agent → collect traces → compute rewards → update the model**.

```
┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  Your Agent  │───▶│    Traces     │───▶│   Rewards    │───▶│  RL Update   │
│  (any code)  │    │  (auto-logged)│    │ (your logic) │    │  (GRPO etc.) │
└──────────────┘    └──────────────┘    └──────────────┘    └──────────────┘
```

Your agent runs as-is — rLLM's model gateway captures LLM calls (token IDs + logprobs) by URL-routed sessions and structures them into **Episodes** (one task) containing **Trajectories** (one agent run) made of **Steps** (one LLM call). A reward function scores the result, and the RL algorithm updates the model weights. The same agent code works for both eval and training.

Under the hood:
- **Workflow Engine** runs N parallel agent instances to collect rollouts
- **Model Gateway** routes requests and captures token IDs + logprobs
- **Transform Pipeline** groups trajectories for advantage computation
- **Training Backend** (verl, tinker, or fireworks) handles the policy update

## Community Projects

- [Tongyi DeepResearch](https://github.com/Alibaba-NLP/DeepResearch) — Open-source AI researchers by Alibaba NLP [](https://github.com/Alibaba-NLP/DeepResearch)
- [Terminal-Bench-RL](https://github.com/Danau5tin/terminal-bench-rl) — Training long-horizon terminal agents with RL [](https://github.com/Danau5tin/terminal-bench-rl)
- [PettingLLMs](https://github.com/pettingllms-ai/PettingLLMs) — Multi-agent RL with on-policy training [](https://github.com/pettingllms-ai/PettingLLMs)
- [SETA](https://github.com/camel-ai/seta) — Scaling environments for terminal agents [](https://github.com/camel-ai/seta)
- [LLM-in-Sandbox](https://github.com/llm-in-sandbox/llm-in-sandbox) — Building general agents by running LLMs in a sandbox [](https://github.com/llm-in-sandbox/llm-in-sandbox)
- [Vision-DeepResearch](https://github.com/Osilly/Vision-DeepResearch) — The first long-horizon multimodal deep-research MLLM [](https://github.com/Osilly/Vision-DeepResearch)
- [OpenSearch-VL](https://github.com/shawn0728/OpenSearch-VL) - An Open Recipe for Frontier Multimodal Search Agents [](https://github.com/shawn0728/OpenSearch-VL)
- [Cogito, Ergo Ludo](https://www.arxiv.org/abs/2509.25052) — An agent that learns to play by reasoning and planning
- [Cut the Bill, Keep the Turns](https://agate-slipper-ef0.notion.site/Cut-the-Bill-Keep-the-Turns-Affordable-Multi-Turn-Search-RL-003f78214a4d451fb06f453d084e666c) — Affordable multi-turn search RL
- [Experiential Reinforcement Learning](https://arxiv.org/abs/2602.13949) — Experience-reflection-consolidation loop for RL with sparse rewards
- [V1: Unifying Generation and Self-Verification](https://arxiv.org/abs/2603.04304) — Pairwise self-verification for parallel test-time scaling
- [TherapyGym](https://therapygym.stanford.edu/) - Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots
- [SandMLE](https://arxiv.org/pdf/2604.04872) - Synthetic Sandbox for Training MLE Agents
- [AxPO](https://byungkwanlee.github.io/AXPO-page/) - Agent Explorative Policy Optimization for Multimodal Agentic Reasoning
- [Remember When It Matters](https://arxiv.org/abs/2607.08716) — Proactive Memory Agent for Long-Horizon Agents
  
## Articles & Blog Posts

- [rLLM UI: Real-Time Observability Tool for Agent Training & Evaluation](https://rllm-project.com/post.html?post=rllm_ui.md) — Mar 2026
- [rLLM On-Policy Distillation: Training Smaller Students from Stronger Teachers](https://rllm-project.com/post.html?post=opd.md) — Mar 2026
- [Faster and Better: Open-Source Recipe for Deep Research Agents with Fully Async Training](https://rllm-project.com/post.html?post=async_rl.md) — Feb 2026
- [rLLM-FinQA: How a 4B Model Outperforms 235B and Rivals Gemini 2.5 Pro on Financial Analysis](https://rllm-project.com/post.html?post=finqa.md) — Feb 2026
- [rLLM SDK: Training Any Agentic Program without Code Changes](https://rllm-project.com/post.html?post=sdk.md) — Dec 2025
- [rLLM v0.2: RL Training for General Agentic Programs](https://rllm-project.com/post.html?post=rllm_v0.2.md) — Oct 2025
- [DeepSWE: Open-source SWE Agent via RL](https://pretty-radio-b75.notion.site/DeepSWE-Training-a-Fully-Open-sourced-State-of-the-Art-Coding-Agent-by-Scaling-RL-22281902c1468193aabbe9a8c59bbe33) — Jul 2025
- [DeepCoder: 14B Coder at O3-mini Level](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51) — Apr 2025
- [DeepScaleR: 1.5B Surpasses O1-Preview](https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2) — Feb 2025

## Acknowledgements

Our work is done as part of [Berkeley Sky Computing Lab](https://sky.cs.berkeley.edu/). The rLLM team is generously supported by grants from [Laude Institute](https://www.laude.org/), [AWS](https://aws.amazon.com/), [Hyperbolic](https://www.hyperbolic.ai/), [Fireworks AI](https://fireworks.ai/), and [Modal](https://modal.com/). We pay special thanks to [Together AI](https://www.together.ai/) for the research partnership and compute support.

## Citation

```bibtex
@misc{rllm2025,
  title={rLLM: A Framework for Post-Training Language Agents},
  author={Sijun Tan and Michael Luo and Colin Cai and Tarun Venkat and Kyle Montgomery and Aaron Hao and Tianhao Wu and Arnav Balyan and Manan Roongta and Chenguang Wang and Li Erran Li and Raluca Ada Popa and Ion Stoica},
  year={2025},
  howpublished={\url{https://pretty-radio-b75.notion.site/rLLM-A-Framework-for-Post-Training-Language-Agents-21b81902c146819db63cd98a54ba5f31}},
  note={Notion Blog},
}
```

You may also cite our prior work [DeepScaleR](https://scholar.googleusercontent.com/scholar.bib?q=info:PrmBADk39GwJ:scholar.google.com/&output=citation&scisdr=CgIJFx-xEMCQ6zOgcuI:AAZF9b8AAAAAaPCmauIfzg8Rm9ImNYDad0uPUK8&scisig=AAZF9b8AAAAAaPCmahXsNqb1jTQBw2iPfw2vm9g&scisf=4&ct=citation&cd=-1&hl=en&scfhb=1), [DeepCoder](https://scholar.googleusercontent.com/scholar.bib?q=info:xpZNEPI6opAJ:scholar.google.com/&output=citation&scisdr=CgIJFx-xEMCQ6zOgjM8:AAZF9b8AAAAAaPCmlM_hb3S0tzBSVrRYBZYDLWg&scisig=AAZF9b8AAAAAaPCmlG109SG8d8230AiDP4jMxlw&scisf=4&ct=citation&cd=-1&hl=en&scfhb=1), and [DeepSWE](https://scholar.googleusercontent.com/scholar.bib?q=info:J9rT3SnY_aMJ:scholar.google.com/&output=citation&scisdr=CgIJFx-xEMCQ6zOg3D4:AAZF9b8AAAAAaPCmxD7Nl0xA_AcAeydpcE1BXCo&scisig=AAZF9b8AAAAAaPCmxE2Spzf5lf-2Toys5xEpnuA&scisf=4&ct=citation&cd=-1&hl=en&scfhb=1).

---

## File: cookbooks/workflow/solver_judge/README.md

# Solver-Judge (legacy Workflow API)

Solver-judge multi-agent flow for the countdown task, built on the **legacy `Workflow` abstraction** (`rllm.workflows.workflow.Workflow` + `RolloutEngine` + `AgentWorkflowEngine`). The training side runs through the **unified trainer** (`rllm.trainer.unified_trainer.AgentTrainer`), which keeps the `workflow_class=...` interface while routing to the maintained backend launchers. The old `rllm.trainer.AgentTrainer(backend="tinker")` path has been removed.

For the equivalent example written against the newer `@rllm.rollout` AgentFlow protocol, see [`cookbooks/solver_judge_flow/`](../../solver_judge_flow/).

## Pattern

- **Solver** generates `n_solutions` candidate answers in parallel.
- **Judge** picks the index of the best candidate; its answer is the workflow's final answer.
- The workflow emits one episode with `n_solutions + 1` trajectories (named `solver` / `judge`), each carrying its own reward so GRPO can group advantages per role.

## Layout

| File | Description |
|------|-------------|
| `solver_judge_flow.py` | `SolverJudgeWorkflow` — subclass of `rllm.workflows.workflow.Workflow` |
| `run.py` | Inference / eval driver via `AgentWorkflowEngine` + `OpenAIEngine` |
| `train.py` | Hydra entrypoint — `unified_trainer.AgentTrainer(workflow_class=...)` |
| `train_tinker.sh` | Tinker-backend launch (`rllm/backend=tinker`) |
| `train_verl.sh` | Verl-backend launch (`rllm/backend=verl`) |

## Data

```bash
rllm dataset pull countdown
```

This registers `countdown` train/test splits in the `DatasetRegistry`. The workflow expects each task to carry `question`, `target`, `nums`, and `ground_truth = {"target", "numbers"}` (see `process_countdown_fn` in `run.py`).

## Eval / inference

Start an OpenAI-compatible server (sglang, vllm, ...) on `http://localhost:30000/v1`, then run from this cookbook directory:

```bash
cd cookbooks/workflow/solver_judge
python run.py
```

Reports pass@1 / pass@k over the countdown test split and dumps the full episode log to `logs/solver_judge_countdown.json`.

## Train

### Tinker (single-machine)

```bash
cd cookbooks/workflow/solver_judge
bash train_tinker.sh
```

### Verl (distributed GPU)

```bash
cd cookbooks/workflow/solver_judge
bash train_verl.sh
```

Both scripts call the same `train.py` with `rllm/backend=tinker|verl` and any additional Hydra overrides. Append more overrides on the command line — they will be forwarded.

---

## File: cookbooks/workflow/geo3k/README.md

# Geo3K (legacy Workflow API)

Single-turn VLM geometry solver for the [Geometry3K](https://huggingface.co/datasets/hiyouga/geometry3k) dataset, built on the **legacy `Workflow` abstraction** (`rllm.workflows.workflow.Workflow` + `RolloutEngine` + `AgentWorkflowEngine`). The training side runs through the **unified trainer** (`rllm.trainer.unified_trainer.AgentTrainer`), which keeps the `workflow_class=...` interface while routing to the maintained backend launchers. The old `rllm.trainer.AgentTrainer(backend="tinker")` path has been removed.

For the equivalent example written against the newer `@rllm.rollout` AgentFlow protocol, see [`cookbooks/geo3k/`](../../geo3k/).

## Pattern

- **Solver** receives a geometry problem with a diagram image. The workflow assembles a `{"role": "user", "content": <text>, "images": [PIL.Image]}` message and calls `rollout_engine.get_model_response` directly. Backend-specific image rendering (OpenAI multimodal blocks for verl/vLLM, the tinker-renderer-aware path for tinker) happens inside the `RolloutEngine`.
- One trajectory per episode, named `solver`, scored by `math_reward_fn` (boxed-answer extraction + symbolic math grading).

## Layout

| File | Description |
|------|-------------|
| `geo3k_flow.py` | `Geo3KWorkflow` — subclass of `rllm.workflows.workflow.Workflow` |
| `run.py` | Inference / eval driver via `AgentWorkflowEngine` + `OpenAIEngine` |
| `train.py` | Hydra entrypoint — `unified_trainer.AgentTrainer(workflow_class=...)` |
| `train_tinker.sh` | Tinker-backend launch (`rllm/backend=tinker`) |
| `train_verl.sh` | Verl-backend launch (`rllm/backend=verl`) |

## VLM dependency note

`Qwen/Qwen3-VL-30B-A3B-Instruct` (and the wider Qwen3-VL family) ships a `Qwen3VLVideoProcessor` that imports torchvision at load time. Even though the tinker rollout only uses the image side, `AutoProcessor.from_pretrained` will fail without it:

```bash
uv pip install torchvision
```

The tinker backend tries to load the processor lazily and now fails fast with an actionable message if torchvision is missing — install it once before kicking off training.

## Data

```bash
rllm dataset pull geo3k
```

## Eval / inference

Start an OpenAI-compatible VLM server (sglang, vllm, ...) on `http://localhost:30000/v1`, then run from this cookbook directory:

```bash
cd cookbooks/workflow/geo3k
python run.py
```

Reports pass@1 / pass@k over the geo3k test split and dumps the full episode log to `logs/geo3k.json`.

## Train

### Tinker (single-machine)

```bash
cd cookbooks/workflow/geo3k
bash train_tinker.sh
```

### Verl (distributed GPU)

```bash
cd cookbooks/workflow/geo3k
bash train_verl.sh
```

Both scripts call the same `train.py` with `rllm/backend=tinker|verl` and any additional Hydra overrides. Append more overrides on the command line — they will be forwarded.

## Config parity with `cookbooks/geo3k/`

This cookbook intentionally matches the AgentFlow geo3k cookbook's hyperparameters so the two are apples-to-apples comparable:

- Model: `Qwen/Qwen3-VL-30B-A3B-Instruct`
- LoRA: rank 32 (alpha 32, merge true) on verl
- `training.group_size`: 8
- Tinker: `total_epochs=3`, `test_freq=10`, train/val temperature 0.6
- Verl: `actor_rollout_ref.rollout.n=4`, `data.train_batch_size=32`, `lr=2e-5`, `loss_agg_mode=token-mean`, vLLM async rollout, `tensor_model_parallel_size=2`

The only intentional differences are `project_name` (`geo3k_workflow` vs `geo3k`) and the underlying agent shape (legacy `Workflow` vs `@rllm.rollout` AgentFlow).

---

## File: cookbooks/solver_judge_flow/README.md

# Solver-Judge Flow

A multi-agent flow for rLLM that trains a solver-judge system on the countdown task using the **AgentFlow protocol**.

## Overview

The solver-judge pattern uses two agents cooperatively:

1. **Solver** — generates N candidate solutions to a countdown problem in parallel
2. **Judge** — evaluates the candidates and selects the best one

Both agents use a plain `OpenAI` client pointed at `config.base_url`. During training, this URL points to the model gateway which transparently captures token IDs and logprobs for RL optimization. During eval, it points directly to the model provider. The agent code is identical in both cases.

## Architecture

```
AgentFlow.run(task, config)
  │
  ├── Solver (N parallel threads)
  │     └── OpenAI(base_url=config.base_url).chat.completions.create(...)
  │         → Trajectory(name="solver", steps=[Step(action=parsed_answer)])
  │
  └── Judge
        └── OpenAI(base_url=config.base_url).chat.completions.create(...)
            → Trajectory(name="judge", steps=[Step(action=selected_answer)])
  │
  └── Episode(trajectories=[solver_0, solver_1, ..., judge])
```

The evaluator scores each trajectory independently:
- Solver trajectories are scored by whether their answer is correct
- Judge trajectory is scored by whether the selected answer is correct
- GRPO computes advantages separately for the `solver` and `judge` trajectory groups

## Installation

```bash
# From the rllm repo root
uv pip install -e ".[tinker]"                    # rllm + tinker backend
uv pip install -e cookbooks/solver_judge_flow    # this cookbook
```

After installation, the agent and evaluator are discoverable by the CLI:

```bash
rllm agent list    # should show "solver_judge" as a plugin
```

## Dataset

Pull the countdown dataset (one-time):

```bash
rllm dataset pull countdown
```

## Training

### Tinker (single-machine)

```bash
bash cookbooks/solver_judge_flow/train_tinker.sh
```

Or directly via the Python API:

```bash
python cookbooks/solver_judge_flow/train.py \
    rllm/backend=tinker \
    model.name=Qwen/Qwen3-4B-Instruct-2507 \
    model.lora_rank=32 \
    training.group_size=4
```

### Verl (distributed GPU)

Requires verl extras and megatron:

```bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
```

Then:

```bash
bash cookbooks/solver_judge_flow/train_verl.sh
```

## Eval

```bash
rllm eval countdown \
    --agent solver_judge \
    --evaluator solver_judge_countdown \
    --model Qwen/Qwen3-8B
```

## Files

| File | Description |
|------|-------------|
| `solver_judge_flow.py` | `SolverJudgeFlow` — AgentFlow implementation |
| `evaluator.py` | `SolverJudgeCountdownEvaluator` — per-trajectory reward scoring |
| `train.py` | Python API training script (Hydra config) |
| `train_tinker.sh` | Tinker backend — single-machine training |
| `train_verl.sh` | Verl backend — distributed multi-GPU training |
| `pyproject.toml` | Plugin metadata and entry points |
| `test.py` | Unit tests for parsing and evaluation |

---

## File: cookbooks/migrationbench/README.md

# MigrationBench Agent (AWS AgentCore Runtime)

Train a Java 8→17 migration agent on [MigrationBench](https://github.com/amazon-science/MigrationBench) with rLLM. The agent runs **inside an AWS Bedrock AgentCore Runtime** (a serverless remote runtime), not in-process — rLLM drives rollouts over the network and trains the policy locally with the **verl** backend.

## Overview

The agent code lives in the [agentcore-rl-toolkit](https://github.com/awslabs/agentcore-rl-toolkit) repo, not here. You build and deploy it from there ([`examples/strands_migration_agent`](https://github.com/awslabs/agentcore-rl-toolkit/tree/main/examples/strands_migration_agent)), which produces the inputs this cookbook needs:

- **An AgentCore agent runtime ARN** — the deployed container where the agent loop to perform migrations is hosted.
- **A data S3 bucket** — holds the prepared MigrationBench repo tarballs and their `metadata.json`, written by the toolkit's `preprocess.py`. The container downloads repos from here at runtime, and `prepare_migrationbench_data.py` reads its metadata to register the dataset.
- **An output S3 bucket** — where the agent writes rollout results (rewards, etc.); rLLM polls it for each rollout. This can be a different bucket from the data bucket.

This cookbook only handles the rLLM side: registering the dataset from the data bucket's metadata and launching verl training against the remote runtime. The repo tarballs stay in S3 and are pulled at runtime by the container.

```
agentcore-rl-toolkit (separate repo)         rLLM (this cookbook)
──────────────────────────────────          ─────────────────────────
build + deploy agent  ──► agent ARN ─────►   train_…_verl.sh  (drive rollouts + train)
preprocess.py         ──► data bucket ───►   prepare_migrationbench_data.py  (register dataset)
                          output bucket ◄──  agent writes rewards; rLLM polls
```

> **Note:** this example supports **verl only** (distributed multi-GPU). It is tested on a single 8×B200 node with verl 0.8.0 / Qwen3-Coder-30B-A3B-Instruct (LoRA).

## Installation

```bash
# rLLM + verl backend (vLLM) + AgentCore runtime client
uv pip install -e ".[verl,agentcore]"

# Megatron deps for the verl training backend
bash scripts/install_megatron.sh <cu129|cu130|...>
```

## Prerequisites (agentcore-rl-toolkit)

Clone the toolkit, then preprocess the dataset and build/deploy the agent. Follow the
[strands_migration_agent README](https://github.com/awslabs/agentcore-rl-toolkit/tree/main/examples/strands_migration_agent):

```bash
git clone https://github.com/awslabs/agentcore-rl-toolkit
cd agentcore-rl-toolkit/examples/strands_migration_agent

# Upload prepared MigrationBench repos + metadata to your data bucket
python preprocess.py --s3-bucket-name <data-bucket>

# Build + deploy the agent container (see the example README) → records the agent runtime ARN
```

When this finishes you have the **agent runtime ARN**, the **data bucket**, and an **output bucket** — the only inputs the rest of this cookbook needs. Run the remaining steps from this cookbook folder (`cd cookbooks/migrationbench`); the train script sources `.env` from the current directory. Copy the example and fill it in:

```bash
cp .env.example .env
# edit .env:
#   AGENTCORE_AGENT_ARN=arn:aws:bedrock-agentcore:<region>:<account>:runtime/<name>
#   AGENTCORE_S3_BUCKET=<output-bucket>
```

## Dataset

Run from this cookbook folder (`cd cookbooks/migrationbench`):

```bash
python prepare_migrationbench_data.py --s3-bucket-name <data-bucket>
```

This downloads only the small `metadata.json` files from `s3://<data-bucket>/tars/{train,test}/`, then registers `migration_bench/{train,test}` with the rLLM `DatasetRegistry`:

- **Train** — repos under `tars/train/` with `num_test_cases > 0`.
- **Test** — all repos under `tars/test/`.

## Training

From this cookbook folder (`cd cookbooks/migrationbench`):

```bash
bash train_agentcore_migrationbench_verl.sh
```

The script sources `.env` for the agent ARN and output bucket, then runs verl with `rllm.remote_runtime.backend=agentcore` so rollouts execute in the deployed container. Tune `MODEL_PATH`, parallelism (`TP`/`EP`/`CP`), batch sizes, and `trainer.n_gpus_per_node`/`nnodes` in the script to match your hardware.

## Files

| File | Description |
|------|-------------|
| `prepare_migrationbench_data.py` | Register `migration_bench/{train,test}` from the data bucket's metadata |
| `train_agentcore_migrationbench_verl.py` | Python API training entrypoint (Hydra config, verl backend) |
| `train_agentcore_migrationbench_verl.sh` | Launch verl training against the AgentCore Runtime |

---

## File: cookbooks/math_tool_agent/README.md

# Math Tool Agent

A multi-turn agent flow for rLLM that trains a math agent to solve arithmetic problems using a **calculator tool**, demonstrating the **AgentFlow protocol** with tool use.

## Overview

The agent solves competition math problems from [DeepScaleR-Preview](https://huggingface.co/datasets/agentica-org/DeepScaleR-Preview-Dataset) (train, ~40K problems blending AIME/AMC/Omni-MATH) and [MATH-500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500) (test) by calling a calculator tool for each arithmetic step. These problems are harder than GSM8K and require multi-turn reasoning, making them a better test for multi-turn RL training. This cookbook serves two purposes:

1. **End-to-end system test** — a multi-turn tool-use example that exercises the full training loop
2. **Onboarding** — shows new users how to build a tool-calling agent with rLLM

## Architecture

```
AgentFlow.run(task, config)
  │
  └── Multi-turn loop (up to 5 turns)
        │
        ├── LLM call via OpenAI(base_url=config.base_url)
        │     Model outputs reasoning + <tool_call>...</tool_call>
        │
        ├── Parse tool call → execute calculator → inject result
        │
        └── Repeat until model outputs <answer>NUMBER</answer>
```

The evaluator checks the final `<answer>` against the ground truth via numeric comparison.

## Installation

```bash
# From the rllm repo root
uv pip install -e ".[tinker]"                              # rllm + tinker backend
uv pip install --no-deps -e cookbooks/math_tool_agent      # this cookbook
```

After installation, the agent and evaluator are discoverable by the CLI:

```bash
rllm agent list    # should show "math_tool_agent" as a plugin
```

## Dataset

Pull the datasets (one-time):

```bash
rllm dataset pull deepscaler_math   # ~40K competition math problems (AIME/AMC/Omni-MATH/STILL)
rllm dataset pull math500           # 500-problem test benchmark
```

## Training

### Tinker (single-machine)

```bash
bash cookbooks/math_tool_agent/train_tinker.sh
```

Or directly via the Python API:

```bash
python cookbooks/math_tool_agent/train.py \
    rllm/backend=tinker \
    model.name=Qwen/Qwen3-4B-Instruct-2507 \
    model.lora_rank=32 \
    training.group_size=8
```

### Verl (distributed GPU)

Requires verl extras and megatron:

```bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
```

Then:

```bash
bash cookbooks/math_tool_agent/train_verl.sh
```

## Tests

```bash
pytest cookbooks/math_tool_agent/test.py -v
```

## Files

| File | Description |
|------|-------------|
| `math_tool_agent.py` | `math_tool_agent` — multi-turn AgentFlow with calculator tool |
| `math_tool_eval.py` | `math_tool_evaluator` — numeric answer comparison |
| `train.py` | Python API training script (Hydra config) |
| `train_tinker.sh` | Tinker backend — single-machine training |
| `train_verl.sh` | Verl backend — distributed multi-GPU training |
| `pyproject.toml` | Plugin metadata and entry points |
| `test.py` | Unit tests for calculator, parsing, and evaluation |

---

## File: cookbooks/math/README.md

# Math Agent

A single-turn math agent for rLLM that solves competition / textbook math problems via the **AgentFlow protocol**, replacing the four legacy `examples/{deepscaler, simple_math, gsm8k_lora, math_tinker}/` examples (all of which used `MathAgent` + `SingleTurnEnvironment` + `AgentExecutionEngine`).

This is the no-tool counterpart to [`cookbooks/math_tool_agent/`](../math_tool_agent/), which uses a calculator tool and multi-turn reasoning. Use `math` for chain-of-thought-only training; use `math_tool_agent` if you want the model to learn explicit tool use.

## Overview

Each task is a math problem. The agent emits one assistant message containing reasoning followed by a final answer in `\boxed{...}` notation. The evaluator extracts the boxed answer and grades it against ground truth via [`rllm.eval.reward_fns.math`](../../rllm/eval/reward_fns/math.py) (which uses symbolic + numeric equivalence — `0.5` matches `\frac{1}{2}`, etc.).

## Architecture

```
AgentFlow.run(task, config)
  │
  ├── one LLM call via OpenAI(base_url=config.base_url)
  │     model outputs reasoning + \boxed{ANSWER}
  │
  └── store full response in episode.artifacts["answer"]

Evaluator.evaluate(task, episode)
  │
  └── extract last \boxed{...}, grade against task.metadata["ground_truth"]
      via mathd + sympy.
```

## Installation

```bash
uv pip install -e ".[tinker]"                  # rllm + tinker backend
uv pip install --no-deps -e cookbooks/math     # this cookbook
```

After installation:

```bash
rllm agent list      # should show "math"
```

## Datasets

```bash
rllm dataset pull hendrycks_math    # train (Hendrycks MATH)
rllm dataset pull math500           # 500-problem test
rllm dataset pull gsm8k             # alternative train
rllm dataset pull deepscaler_math   # ~40K AIME/AMC/Omni-MATH/STILL
rllm dataset pull aime2024          # AIME 2024 (eval)
```

## Eval (rllm CLI)

```bash
rllm eval math500 \
    --agent math \
    --evaluator math \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --base-url http://localhost:8000/v1 \
    --max-examples 20
```

For `aime2024` pass@k-style eval (the original `examples/deepscaler/` use case), use `--max-examples` and run multiple times, or use the harness's repeat flag if exposed.

Episode JSONs land under `~/.rllm/eval_results/` for inspection (`rllm view`).

## Training (rllm CLI)

```bash
rllm train hendrycks_math \
    --agent math \
    --evaluator math \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --group-size 8 \
    --batch-size 32 \
    --lora-rank 32 \
    --epochs 1 \
    --val-freq 10
```

For LoRA-only training (the legacy `examples/gsm8k_lora/` use case), set `--lora-rank` higher and pass `--train-dataset gsm8k`. For deeper customization (verl backend, full PPO knobs), use the shell scripts.

## Training (shell scripts)

### Tinker (single-machine)

```bash
bash cookbooks/math/train_tinker.sh
```

### Verl (distributed GPU)

```bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
bash cookbooks/math/train_verl.sh
```

## Tests

```bash
pytest cookbooks/math/test.py -v
```

## Files

| File | Description |
|------|-------------|
| `math_flow.py` | `math_flow` — single-turn AgentFlow |
| `evaluator.py` | `math_evaluator` — wraps `rllm.eval.reward_fns.math` |
| `train.py` | Python API training script (Hydra config) |
| `train_tinker.sh` | Tinker backend — single-machine training |
| `train_verl.sh` | Verl backend — distributed multi-GPU training |
| `pyproject.toml` | Plugin metadata and entry points |
| `test.py` | Unit tests for evaluator scoring |

## Migration notes

This cookbook replaces:

- **`examples/deepscaler/`** — `MathAgent` eval on `aime2024` → use `rllm eval aime2024 --agent math --evaluator math`
- **`examples/simple_math/`** — `MathAgent` train on `hendrycks_math` → see `train_tinker.sh` / `train_verl.sh`
- **`examples/gsm8k_lora/`** — `MathAgent` LoRA train on `gsm8k` → `rllm train gsm8k --agent math --evaluator math --lora-rank 32`
- **`examples/math_tinker/`** — `MathAgent` + few-shot prompts → fork `math_flow.py` and customize the `SYSTEM_PROMPT`

All four shared the same `MathAgent` + `SingleTurnEnvironment` + `AgentExecutionEngine` plumbing. The cookbook collapses them into one self-contained AgentFlow + entry-point with no dependency on `rllm.agents` or `rllm.environments`.

---

## File: cookbooks/geo3k/README.md

# Geo3K Flow

A VLM geometry problem solver for rLLM that trains on the [Geometry3K](https://huggingface.co/datasets/hiyouga/geometry3k) dataset using the **AgentFlow protocol**.

## Overview

A single-turn VLM agent that receives a geometry problem with a diagram image and produces a step-by-step solution with a boxed final answer. Uses a plain `OpenAI` client with multimodal content blocks (base64-encoded images).

During training, `config.base_url` points to the model gateway which transparently captures token IDs and logprobs. During eval, it points directly to the model provider. The agent code is identical in both cases.

## Architecture

```
AgentFlow.run(task, config)
  │
  └── Solver
        └── OpenAI(base_url=config.base_url).chat.completions.create(
                messages=[system_prompt, {images + question}]
            )
            → Trajectory(name="solver", steps=[Step(action=response)])
  │
  └── Episode(trajectories=[solver], artifacts={"answer": response})
```

The evaluator extracts `\boxed{}` from the response and grades it against the ground truth using symbolic math grading.

## Installation

```bash
# From the rllm repo root
uv pip install -e ".[tinker]"          # rllm + tinker backend
uv pip install -e cookbooks/geo3k      # this cookbook
```

After installation, the agent and evaluator are discoverable by the CLI:

```bash
rllm agent list    # should show "geo3k" as a plugin
```

## VLM dependency note

`Qwen/Qwen3-VL-30B-A3B-Instruct` (and the wider Qwen3-VL family) ships a `Qwen3VLVideoProcessor` that imports torchvision at load time. Even though the tinker rollout only uses the image side, `AutoProcessor.from_pretrained` will fail without it:

```bash
uv pip install torchvision
```

The tinker backend tries to load the processor lazily and fails fast with an actionable message if torchvision is missing — install it once before kicking off training.

## Dataset

Pull the Geometry3K dataset (one-time):

```bash
rllm dataset pull geo3k
```

## Training

### Tinker (single-machine)

```bash
bash cookbooks/geo3k/train_tinker.sh
```

Or directly via the Python API:

```bash
python cookbooks/geo3k/train.py \
    rllm/backend=tinker \
    model.name=Qwen/Qwen3-VL-30B-A3B-Instruct \
    model.lora_rank=32 \
    training.group_size=8
```

### Verl (distributed GPU)

Requires verl extras and megatron:

```bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
```

Then:

```bash
bash cookbooks/geo3k/train_verl.sh
```

## Eval

```bash
rllm eval geo3k \
    --agent geo3k \
    --evaluator geo3k_math \
    --model Qwen/Qwen3-VL-30B-A3B-Instruct
```

## Files

| File | Description |
|------|-------------|
| `geo3k_flow.py` | `Geo3KFlow` — AgentFlow implementation (VLM single-turn solver) |
| `geo3k_eval.py` | `geo3k_evaluator` — math answer grading with `\boxed{}` extraction |
| `train.py` | Python API training script (Hydra config) |
| `train_tinker.sh` | Tinker backend — single-machine training |
| `train_verl.sh` | Verl backend — distributed multi-GPU training |
| `pyproject.toml` | Plugin metadata and entry points |
| `test.py` | Unit tests for image handling and evaluation |

---

## File: cookbooks/frozenlake/README.md

# FrozenLake Agent

A multi-turn agent flow for rLLM that trains a model to navigate procedurally-generated FrozenLake puzzles end-to-end via the **AgentFlow protocol**, demonstrating gym-environment integration without the legacy `Agent + Environment + AgentExecutionEngine` stack.

## Overview

Each task is a randomly-generated `size x size` FrozenLake grid (with `S`tart, `G`oal, frozen tiles `_`, and `H`oles `O`). The agent reads the rendered grid each turn and outputs `Up`/`Down`/`Left`/`Right`. Reward is `1.0` if the player reaches the goal, `0.0` otherwise.

The map is regenerated deterministically from `(seed, size, p)` inside the flow, so the dataset stores only those parameters — no map serialization.

## Architecture

```
AgentFlow.run(task, config)
  │
  ├── generate_random_map(seed, size, p)   # deterministic, in-process
  ├── env = gymnasium.make("FrozenLake-v1", desc=…, is_slippery=…)
  └── Multi-turn loop (up to max_steps turns)
        │
        ├── LLM call via OpenAI(base_url=config.base_url)
        │     Model outputs reasoning + ```Up|Down|Left|Right```
        │
        ├── parse_action() → env.step(action)
        │
        └── Repeat until terminated (G or H), truncated, or max_steps
```

The evaluator just reads `episode.artifacts["won"]` set by the flow.

## Installation

```bash
uv pip install -e ".[tinker]"                          # rllm + tinker backend
uv pip install --no-deps -e cookbooks/frozenlake       # this cookbook (gymnasium pulled in transitively)
```

After installation:

```bash
rllm agent list      # should show "frozenlake"
```

## Dataset

The dataset is procedurally generated — there's nothing to download. Run once:

```bash
python cookbooks/frozenlake/prepare_frozenlake_data.py
# or, with custom sizes:
python cookbooks/frozenlake/prepare_frozenlake_data.py --train-size 5000 --test-size 200 --slippery
```

This registers `frozenlake/{train,test}` with `DatasetRegistry`.

## Eval (rllm CLI)

Once the cookbook is installed and the dataset is generated, evaluate a model on the test split with one command:

```bash
rllm eval frozenlake \
    --agent frozenlake \
    --evaluator frozenlake \
    --split test \
    --max-examples 20            # smoke-test on 20 puzzles before running the full split
```

`--agent` and `--evaluator` resolve via the entry points declared in `pyproject.toml`. Drop `--base-url` if you want the CLI to auto-start a proxy from your `rllm setup` config. Episode JSONs are written under `~/.rllm/eval_results/` for inspection (`rllm view`).

## Training (rllm CLI)

The simplest training entrypoint is the CLI, which mirrors the eval flags:

```bash
rllm train frozenlake \
    --agent frozenlake \
    --evaluator frozenlake \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --group-size 8 \
    --batch-size 32 \
    --lora-rank 32 \
    --epochs 1 \
    --val-freq 10
```

This dispatches through `AgentTrainer` with the tinker backend by default. For deeper customization (verl backend, full PPO knobs, megatron, …) use the shell scripts below.

## Training (shell scripts)

### Tinker (single-machine)

```bash
bash cookbooks/frozenlake/train_tinker.sh
```

Or directly via the Python API with Hydra overrides:

```bash
python cookbooks/frozenlake/train.py \
    rllm/backend=tinker \
    model.name=Qwen/Qwen3-4B-Instruct-2507 \
    training.group_size=8
```

### Verl (distributed GPU)

```bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
bash cookbooks/frozenlake/train_verl.sh
```

## Tests

```bash
pytest cookbooks/frozenlake/test.py -v
```

## Files

| File | Description |
|------|-------------|
| `frozenlake_flow.py` | `frozenlake_flow` — multi-turn AgentFlow over a gym env |
| `frozenlake_eval.py` | `frozenlake_evaluator` — reward = won? 1 : 0 |
| `prepare_frozenlake_data.py` | Generate + register the train/test splits |
| `train.py` | Python API training script (Hydra config) |
| `train_tinker.sh` | Tinker backend — single-machine training |
| `train_verl.sh` | Verl backend — distributed multi-GPU training |
| `pyproject.toml` | Plugin metadata and entry points |
| `test.py` | Unit tests for map gen, action parsing, rendering, evaluator |

## Migration notes

This cookbook is the AgentFlow port of `examples/frozenlake/`, which uses the legacy `BaseAgent` + `Environment` + `AgentExecutionEngine` stack. The two are functionally equivalent on FrozenLake; the cookbook is fully self-contained — no imports from `rllm.agents`, `rllm.environments`, or `rllm.engine.agent_execution_engine` — so the legacy stack can be deleted without touching this directory.

---

## File: cookbooks/finqa/README.md

# FinQA Agent

A multi-turn ReAct-style financial-QA agent for rLLM, ported to the **AgentFlow protocol**. Replaces `projects/finqa/` (which depended on `BaseAgent` + `ToolAgent` + `ToolEnvironment` + `AgentExecutionEngine`).

## Overview

The agent answers questions about SEC 10-K financial statements by querying structured tables extracted from filings. Four tools are exposed via native OpenAI function calling:

| Tool | Description |
|------|-------------|
| `get_table_names` | List queryable tables for a given company |
| `get_table_info` | Column metadata, dtypes, sample values for a table |
| `sql_query` | Run a filtered SQLite query against the in-memory store |
| `calculator` | Safely evaluate a numeric expression (asteval) |

All ~6,900 tables across 207 companies are pre-loaded into a process-wide `:memory:` SQLite at module import; the tools just look up cached metadata or run a read-only query under a thread lock.

The agent's final reply ends with a `FINAL ANSWER: …` block. The evaluator extracts it and grades against ground truth via a judge LLM (gpt-5-nano for single-table, gpt-5-mini with a structured rubric for multi-table), called directly through the OpenAI API. A small bonus is awarded when the agent inspects the *expected* tables via `get_table_info` (the `right_table_access_reward`).

[Model Weights](https://huggingface.co/rLLM/rLLM-FinQA-4B) | [Dataset](https://huggingface.co/datasets/rLLM/finqa)

## Architecture

```
AgentFlow.run(task, config)
  │
  └── Multi-turn loop (up to 20 turns, native OpenAI tool calls)
        │
        ├── client.chat.completions.create(messages, tools=TOOL_SPECS)
        │
        ├── If msg.tool_calls is empty → that's the final answer, break.
        │
        └── Else: dispatch each tool call → append a `tool` message → repeat.
              (track table_name in `accessed_tables` whenever
               `get_table_info` is invoked, for the table-access bonus)
  │
  └── episode.artifacts = {"answer": full_response, "accessed_tables": [...], "turns": N}

Evaluator.evaluate(task, episode)
  │
  └── extract FINAL ANSWER from artifacts["answer"], grade via judge LLM,
      add table-access bonus, return EvalOutput.
```

## Installation

```bash
uv pip install -e ".[tinker]"                      # rllm + tinker backend
uv pip install --no-deps -e cookbooks/finqa        # this cookbook
```

After installation:

```bash
rllm agent list      # should show "finqa"
```

## Dataset

```bash
python cookbooks/finqa/prepare_finqa_data.py
```

This will:
- Download the dataset tarball from [rLLM/finqa](https://huggingface.co/datasets/rLLM/finqa)
- Extract company tables to `cookbooks/finqa/data/company_tables/` (207 companies, 6,923 tables)
- Register `finqa/{train, val, test}` with `DatasetRegistry` (4,030 / 522 / 558 examples)

## Eval (rllm CLI)

Set the judge key (used by the evaluator), then run:

```bash
export OPENAI_API_KEY=sk-…

rllm eval finqa \
    --agent finqa \
    --evaluator finqa \
    --model rLLM/rLLM-FinQA-4B \
    --base-url http://localhost:30000/v1 \
    --split test \
    --max-examples 20
```

Episode JSONs land under `~/.rllm/eval_results/`. If `OPENAI_API_KEY` is missing, the evaluator silently returns `reward=0` rather than crashing.

## Training (rllm CLI)

```bash
export OPENAI_API_KEY=sk-…

rllm train finqa \
    --agent finqa \
    --evaluator finqa \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --group-size 8 \
    --batch-size 16 \
    --lora-rank 32 \
    --epochs 1 \
    --val-freq 20
```

## Training (shell scripts)

### Tinker (single-machine, LoRA on 30B)

```bash
bash cookbooks/finqa/train_tinker.sh
```

### Verl (distributed GPU, 4B base)

```bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
bash cookbooks/finqa/train_verl.sh
```

## Tests

```bash
pytest cookbooks/finqa/test.py -v
```

## Files

| File | Description |
|------|-------------|
| `finqa_flow.py` | `finqa_flow` — multi-turn AgentFlow with native tool calling |
| `finqa_tools.py` | The four tools as plain functions + OpenAI tool specs |
| `finqa_eval.py` | `finqa_evaluator` — judge-LLM correctness + table-access bonus |
| `finqa_constants.py` | Path constants (data, prompts) |
| `prepare_finqa_data.py` | HF download + register train / val / test splits |
| `train.py` | Python API training script (Hydra config) |
| `train_tinker.sh` | Tinker backend — single-machine LoRA training |
| `train_verl.sh` | Verl backend — distributed multi-GPU training |
| `pyproject.toml` | Plugin metadata and entry points |
| `test.py` | Unit tests (calculator, FINAL ANSWER parsing, table-access scoring) |
| `prompts/` | System prompt + correctness rubric prompts |

## Migration notes

This cookbook replaces `projects/finqa/`, which used the legacy stack:

| Legacy | Replacement |
|---|---|
| `FinQAAgent(ToolAgent)` (qwen tool-call parsing) | `finqa_flow` AgentFlow + native OpenAI function calling |
| `FinQAEnvironment(ToolEnvironment)` (in-loop reward) | Evaluator outside the loop, called once on the final episode |
| `MultiTurnWorkflow` + `AgentExecutionEngine` | Plain async function — `client.chat.completions.create(tools=…)` in a `for` loop |
| `Tool` base class wrappers | Plain Python callables (`get_table_names`, `get_table_info`, `sql_query`, `calculator`) |

The cookbook has zero imports from `rllm.agents`, `rllm.environments`, `rllm.engine.agent_execution_engine`, or `rllm.workflows`. After this migration, `ToolAgent` / `ToolEnvironment` / `MCPToolAgent` are no longer reachable from any active code path and can be removed.

## 2. Official Technical Reference & Guides (rllm-org/rllm-docs)

# Mintlify Starter Kit

Use the starter kit to get your docs deployed and ready to customize.

Click the green **Use this template** button at the top of this repo to copy the Mintlify starter kit. The starter kit contains examples with

- Guide pages
- Navigation
- Customizations
- API reference pages
- Use of popular components

**[Follow the full quickstart guide](https://starter.mintlify.com/quickstart)**

## AI-assisted writing

Set up your AI coding tool to work with Mintlify:

```bash
npx skills add https://mintlify.com/docs
```

This command installs Mintlify's documentation skill for your configured AI tools like Claude Code, Cursor, Windsurf, and others. The skill includes component reference, writing standards, and workflow guidance.

See the [AI tools guides](/ai-tools) for tool-specific setup.

## Development

Install the [Mintlify CLI](https://www.npmjs.com/package/mint) to preview your documentation changes locally. To install, use the following command:

```
npm i mint
```

Run the following command at the root of your documentation, where your `docs.json` is located:

```
mint dev
```

View your local preview at `http://localhost:3000`.

## Publishing changes

Install our GitHub app from your [dashboard](https://dashboard.mintlify.com/settings/organization/github-app) to propagate changes from your repo to your deployment. Changes are deployed to production automatically after pushing to the default branch.

## Need help?

### Troubleshooting

- If your dev environment isn't running: Run `mint update` to ensure you have the most recent version of the CLI.
- If a page loads as a 404: Make sure you are running in a folder with a valid `docs.json`.

### Resources
- [Mintlify documentation](https://mintlify.com/docs)