rllm Documentation

Democratizing Reinforcement Learning for LLMs

5,770 stars Python Python 1 file · ~11,840 tokens #agent-framework#agentic-workflow#coding-agent#distributed-training#llm-reasoning#llm-training#machine-learning#ml-infrastructure
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy rllm on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name rllm --restart always rllm-org/rllm:latest

1. Project Overview & Quickstart (rllm-org/rllm)

File: README.md

rLLM is an open-source framework for training language agents with reinforcement learning. Bring any harness, run it in any sandbox, and switch training backends with one flag — the same agent code drives both eval and training.

Core features

  • Any harness. 10+ CLI harnesses (Claude Code, Codex, Terminus-2, mini-swe-agent, opencode, ...) plus Harbor-compatible task dirs. Or wrap your own agent — LangGraph, OpenAI Agents SDK, openai.OpenAI — with @rllm.rollout.
  • Any sandbox. Docker, Daytona, Modal, or local — with snapshot + warm-pool acceleration to keep rollouts cheap at training-scale.
  • Multiple training backends, one API. verl (distributed multi-GPU), tinker (single-machine), fireworks (Fireworks platform). Switch with one flag.
  • 60+ integrated benchmarks. Math, code, MCQ, QA, search, VLM, translation, agentic — Terminal-Bench 2.0, SWE-bench, SkillsBench, AIME, MATH-500, GPQA, and more. rllm eval <name> auto-pulls and runs.
  • Multiple training methods. GRPO, REINFORCE, RLOO, SFT, on-policy distillation, and more.
  • Battle-tested. State-of-the-art open-source results (DeepScaleR-1.5B, DeepCoder-14B, DeepSWE-32B, FinQA-4B). Adopted by academic labs and industry research teams (see Community Projects below).

Read more on our documentation site.

Installation

rLLM requires Python >= 3.11. You can install it either directly via pip or build from source.

bash
uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git"

This installs dependencies for running rllm CLI with the tinker backend (single-machine, Tinker API). For other backends:

bash
# Distributed multi-GPU training (verl + vLLM/SGLang)
uv pip install "rllm[verl] @ git+https://github.com/rllm-org/rllm.git"

# Fireworks training platform
uv pip install "rllm[fireworks] @ git+https://github.com/rllm-org/rllm.git"

For building from source or Docker, see the installation guide.

Quickstart

Option A: CLI (no code needed)

bash
# 1. Configure your model provider
rllm model setup

# 2. Evaluate on a benchmark
rllm eval gsm8k

# 3. Train with RL
rllm train gsm8k

Option B: Python API

Define a rollout (your agent) and an evaluator (your reward function), then hand them to the trainer:

python
# my_flow.py
from openai import OpenAI
import rllm
from rllm.types import AgentConfig, Episode, Task, Trajectory

@rllm.rollout
def solve(task: Task, config: AgentConfig) -> Episode:
    client = OpenAI(base_url=config.base_url, api_key="EMPTY")
    response = client.chat.completions.create(
        model=config.model,
        messages=[{"role": "user", "content": task.instruction}],
    )
    answer = response.choices[0].message.content or ""
    return Episode(
        trajectories=[Trajectory(name="solver", steps=[])],
        artifacts={"answer": answer},
    )
python
# my_evaluator.py
import rllm
from rllm.eval.types import EvalOutput, Signal
from rllm.types import Episode

@rllm.evaluator
def score(task: dict, episode: Episode) -> EvalOutput:
    answer = str(episode.artifacts.get("answer", ""))
    is_correct = answer.strip() == task["ground_truth"].strip()
    reward = 1.0 if is_correct else 0.0
    return EvalOutput(reward=reward, is_correct=is_correct,
                      signals=[Signal(name="accuracy", value=reward)])
python
# train.py
from rllm.trainer import AgentTrainer
trainer = AgentTrainer(
    backend="tinker",
    agent_flow=solve,
    evaluator=score,
    config=config,
    train_dataset=dataset,
)
trainer.train()

During training, config.base_url points to a gateway that transparently captures token IDs and logprobs — your agent code stays the same for eval and training.

See the cookbooks for complete working examples (single-turn VLM solver, multi-agent solver-judge, and more).

Architecture

rLLM follows a simple pipeline: run your agent → collect traces → compute rewards → update the model.

text
┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  Your Agent  │───▶│    Traces     │───▶│   Rewards    │───▶│  RL Update   │
│  (any code)  │    │  (auto-logged)│    │ (your logic) │    │  (GRPO etc.) │
└──────────────┘    └──────────────┘    └──────────────┘    └──────────────┘

Your agent runs as-is — rLLM's model gateway captures LLM calls (token IDs + logprobs) by URL-routed sessions and structures them into Episodes (one task) containing Trajectories (one agent run) made of Steps (one LLM call). A reward function scores the result, and the RL algorithm updates the model weights. The same agent code works for both eval and training.

Under the hood:

  • Workflow Engine runs N parallel agent instances to collect rollouts
  • Model Gateway routes requests and captures token IDs + logprobs
  • Transform Pipeline groups trajectories for advantage computation
  • Training Backend (verl, tinker, or fireworks) handles the policy update

Community Projects

Articles & Blog Posts

Acknowledgements

Our work is done as part of Berkeley Sky Computing Lab. The rLLM team is generously supported by grants from Laude Institute, AWS, Hyperbolic, Fireworks AI, and Modal. We pay special thanks to Together AI for the research partnership and compute support.

Citation

bibtex
@misc{rllm2025,
  title={rLLM: A Framework for Post-Training Language Agents},
  author={Sijun Tan and Michael Luo and Colin Cai and Tarun Venkat and Kyle Montgomery and Aaron Hao and Tianhao Wu and Arnav Balyan and Manan Roongta and Chenguang Wang and Li Erran Li and Raluca Ada Popa and Ion Stoica},
  year={2025},
  howpublished={\url{https://pretty-radio-b75.notion.site/rLLM-A-Framework-for-Post-Training-Language-Agents-21b81902c146819db63cd98a54ba5f31}},
  note={Notion Blog},
}

You may also cite our prior work DeepScaleR, DeepCoder, and DeepSWE.

File: cookbooks/workflow/solver_judge/README.md

Solver-Judge (legacy Workflow API)

Solver-judge multi-agent flow for the countdown task, built on the legacy Workflow abstraction (rllm.workflows.workflow.Workflow + RolloutEngine + AgentWorkflowEngine). The training side runs through the unified trainer (rllm.trainer.unified_trainer.AgentTrainer), which keeps the workflow_class=... interface while routing to the maintained backend launchers. The old rllm.trainer.AgentTrainer(backend="tinker") path has been removed.

For the equivalent example written against the newer @rllm.rollout AgentFlow protocol, see cookbooks/solver_judge_flow/.

Pattern

  • Solver generates n_solutions candidate answers in parallel.
  • Judge picks the index of the best candidate; its answer is the workflow's final answer.
  • The workflow emits one episode with n_solutions + 1 trajectories (named solver / judge), each carrying its own reward so GRPO can group advantages per role.

Layout

File Description
solver_judge_flow.py SolverJudgeWorkflow — subclass of rllm.workflows.workflow.Workflow
run.py Inference / eval driver via AgentWorkflowEngine + OpenAIEngine
train.py Hydra entrypoint — unified_trainer.AgentTrainer(workflow_class=...)
train_tinker.sh Tinker-backend launch (rllm/backend=tinker)
train_verl.sh Verl-backend launch (rllm/backend=verl)

Data

bash
rllm dataset pull countdown

This registers countdown train/test splits in the DatasetRegistry. The workflow expects each task to carry question, target, nums, and ground_truth = {"target", "numbers"} (see process_countdown_fn in run.py).

Eval / inference

Start an OpenAI-compatible server (sglang, vllm, ...) on http://localhost:30000/v1, then run from this cookbook directory:

bash
cd cookbooks/workflow/solver_judge
python run.py

Reports pass@1 / pass@k over the countdown test split and dumps the full episode log to logs/solver_judge_countdown.json.

Train

Tinker (single-machine)

bash
cd cookbooks/workflow/solver_judge
bash train_tinker.sh

Verl (distributed GPU)

bash
cd cookbooks/workflow/solver_judge
bash train_verl.sh

Both scripts call the same train.py with rllm/backend=tinker|verl and any additional Hydra overrides. Append more overrides on the command line — they will be forwarded.

File: cookbooks/workflow/geo3k/README.md

Geo3K (legacy Workflow API)

Single-turn VLM geometry solver for the Geometry3K dataset, built on the legacy Workflow abstraction (rllm.workflows.workflow.Workflow + RolloutEngine + AgentWorkflowEngine). The training side runs through the unified trainer (rllm.trainer.unified_trainer.AgentTrainer), which keeps the workflow_class=... interface while routing to the maintained backend launchers. The old rllm.trainer.AgentTrainer(backend="tinker") path has been removed.

For the equivalent example written against the newer @rllm.rollout AgentFlow protocol, see cookbooks/geo3k/.

Pattern

  • Solver receives a geometry problem with a diagram image. The workflow assembles a {"role": "user", "content": <text>, "images": [PIL.Image]} message and calls rollout_engine.get_model_response directly. Backend-specific image rendering (OpenAI multimodal blocks for verl/vLLM, the tinker-renderer-aware path for tinker) happens inside the RolloutEngine.
  • One trajectory per episode, named solver, scored by math_reward_fn (boxed-answer extraction + symbolic math grading).

Layout

File Description
geo3k_flow.py Geo3KWorkflow — subclass of rllm.workflows.workflow.Workflow
run.py Inference / eval driver via AgentWorkflowEngine + OpenAIEngine
train.py Hydra entrypoint — unified_trainer.AgentTrainer(workflow_class=...)
train_tinker.sh Tinker-backend launch (rllm/backend=tinker)
train_verl.sh Verl-backend launch (rllm/backend=verl)

VLM dependency note

Qwen/Qwen3-VL-30B-A3B-Instruct (and the wider Qwen3-VL family) ships a Qwen3VLVideoProcessor that imports torchvision at load time. Even though the tinker rollout only uses the image side, AutoProcessor.from_pretrained will fail without it:

bash
uv pip install torchvision

The tinker backend tries to load the processor lazily and now fails fast with an actionable message if torchvision is missing — install it once before kicking off training.

Data

bash
rllm dataset pull geo3k

Eval / inference

Start an OpenAI-compatible VLM server (sglang, vllm, ...) on http://localhost:30000/v1, then run from this cookbook directory:

bash
cd cookbooks/workflow/geo3k
python run.py

Reports pass@1 / pass@k over the geo3k test split and dumps the full episode log to logs/geo3k.json.

Train

Tinker (single-machine)

bash
cd cookbooks/workflow/geo3k
bash train_tinker.sh

Verl (distributed GPU)

bash
cd cookbooks/workflow/geo3k
bash train_verl.sh

Both scripts call the same train.py with rllm/backend=tinker|verl and any additional Hydra overrides. Append more overrides on the command line — they will be forwarded.

Config parity with `cookbooks/geo3k/`

This cookbook intentionally matches the AgentFlow geo3k cookbook's hyperparameters so the two are apples-to-apples comparable:

  • Model: Qwen/Qwen3-VL-30B-A3B-Instruct
  • LoRA: rank 32 (alpha 32, merge true) on verl
  • training.group_size: 8
  • Tinker: total_epochs=3, test_freq=10, train/val temperature 0.6
  • Verl: actor_rollout_ref.rollout.n=4, data.train_batch_size=32, lr=2e-5, loss_agg_mode=token-mean, vLLM async rollout, tensor_model_parallel_size=2

The only intentional differences are project_name (geo3k_workflow vs geo3k) and the underlying agent shape (legacy Workflow vs @rllm.rollout AgentFlow).

File: cookbooks/solver_judge_flow/README.md

Solver-Judge Flow

A multi-agent flow for rLLM that trains a solver-judge system on the countdown task using the AgentFlow protocol.

Overview

The solver-judge pattern uses two agents cooperatively:

  1. Solver — generates N candidate solutions to a countdown problem in parallel
  2. Judge — evaluates the candidates and selects the best one

Both agents use a plain OpenAI client pointed at config.base_url. During training, this URL points to the model gateway which transparently captures token IDs and logprobs for RL optimization. During eval, it points directly to the model provider. The agent code is identical in both cases.

Architecture

text
AgentFlow.run(task, config)
  │
  ├── Solver (N parallel threads)
  │     └── OpenAI(base_url=config.base_url).chat.completions.create(...)
  │         → Trajectory(name="solver", steps=[Step(action=parsed_answer)])
  │
  └── Judge
        └── OpenAI(base_url=config.base_url).chat.completions.create(...)
            → Trajectory(name="judge", steps=[Step(action=selected_answer)])
  │
  └── Episode(trajectories=[solver_0, solver_1, ..., judge])

The evaluator scores each trajectory independently:

  • Solver trajectories are scored by whether their answer is correct
  • Judge trajectory is scored by whether the selected answer is correct
  • GRPO computes advantages separately for the solver and judge trajectory groups

Installation

bash
# From the rllm repo root
uv pip install -e ".[tinker]"                    # rllm + tinker backend
uv pip install -e cookbooks/solver_judge_flow    # this cookbook

After installation, the agent and evaluator are discoverable by the CLI:

bash
rllm agent list    # should show "solver_judge" as a plugin

Dataset

Pull the countdown dataset (one-time):

bash
rllm dataset pull countdown

Training

Tinker (single-machine)

bash
bash cookbooks/solver_judge_flow/train_tinker.sh

Or directly via the Python API:

bash
python cookbooks/solver_judge_flow/train.py \
    rllm/backend=tinker \
    model.name=Qwen/Qwen3-4B-Instruct-2507 \
    model.lora_rank=32 \
    training.group_size=4

Verl (distributed GPU)

Requires verl extras and megatron:

bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>

Then:

bash
bash cookbooks/solver_judge_flow/train_verl.sh

Eval

bash
rllm eval countdown \
    --agent solver_judge \
    --evaluator solver_judge_countdown \
    --model Qwen/Qwen3-8B

Files

File Description
solver_judge_flow.py SolverJudgeFlow — AgentFlow implementation
evaluator.py SolverJudgeCountdownEvaluator — per-trajectory reward scoring
train.py Python API training script (Hydra config)
train_tinker.sh Tinker backend — single-machine training
train_verl.sh Verl backend — distributed multi-GPU training
pyproject.toml Plugin metadata and entry points
test.py Unit tests for parsing and evaluation

File: cookbooks/migrationbench/README.md

MigrationBench Agent (AWS AgentCore Runtime)

Train a Java 8→17 migration agent on MigrationBench with rLLM. The agent runs inside an AWS Bedrock AgentCore Runtime (a serverless remote runtime), not in-process — rLLM drives rollouts over the network and trains the policy locally with the verl backend.

Overview

The agent code lives in the agentcore-rl-toolkit repo, not here. You build and deploy it from there (examples/strands_migration_agent), which produces the inputs this cookbook needs:

  • An AgentCore agent runtime ARN — the deployed container where the agent loop to perform migrations is hosted.
  • A data S3 bucket — holds the prepared MigrationBench repo tarballs and their metadata.json, written by the toolkit's preprocess.py. The container downloads repos from here at runtime, and prepare_migrationbench_data.py reads its metadata to register the dataset.
  • An output S3 bucket — where the agent writes rollout results (rewards, etc.); rLLM polls it for each rollout. This can be a different bucket from the data bucket.

This cookbook only handles the rLLM side: registering the dataset from the data bucket's metadata and launching verl training against the remote runtime. The repo tarballs stay in S3 and are pulled at runtime by the container.

text
agentcore-rl-toolkit (separate repo)         rLLM (this cookbook)
──────────────────────────────────          ─────────────────────────
build + deploy agent  ──► agent ARN ─────►   train_…_verl.sh  (drive rollouts + train)
preprocess.py         ──► data bucket ───►   prepare_migrationbench_data.py  (register dataset)
                          output bucket ◄──  agent writes rewards; rLLM polls

Note: this example supports verl only (distributed multi-GPU). It is tested on a single 8×B200 node with verl 0.8.0 / Qwen3-Coder-30B-A3B-Instruct (LoRA).

Installation

bash
# rLLM + verl backend (vLLM) + AgentCore runtime client
uv pip install -e ".[verl,agentcore]"

# Megatron deps for the verl training backend
bash scripts/install_megatron.sh <cu129|cu130|...>

Prerequisites (agentcore-rl-toolkit)

Clone the toolkit, then preprocess the dataset and build/deploy the agent. Follow the
strands_migration_agent README:

bash
git clone https://github.com/awslabs/agentcore-rl-toolkit
cd agentcore-rl-toolkit/examples/strands_migration_agent

# Upload prepared MigrationBench repos + metadata to your data bucket
python preprocess.py --s3-bucket-name <data-bucket>

# Build + deploy the agent container (see the example README) → records the agent runtime ARN

When this finishes you have the agent runtime ARN, the data bucket, and an output bucket — the only inputs the rest of this cookbook needs. Run the remaining steps from this cookbook folder (cd cookbooks/migrationbench); the train script sources .env from the current directory. Copy the example and fill it in:

bash
cp .env.example .env
# edit .env:
#   AGENTCORE_AGENT_ARN=arn:aws:bedrock-agentcore:<region>:<account>:runtime/<name>
#   AGENTCORE_S3_BUCKET=<output-bucket>

Dataset

Run from this cookbook folder (cd cookbooks/migrationbench):

bash
python prepare_migrationbench_data.py --s3-bucket-name <data-bucket>

This downloads only the small metadata.json files from s3://<data-bucket>/tars/{train,test}/, then registers migration_bench/{train,test} with the rLLM DatasetRegistry:

  • Train — repos under tars/train/ with num_test_cases > 0.
  • Test — all repos under tars/test/.

Training

From this cookbook folder (cd cookbooks/migrationbench):

bash
bash train_agentcore_migrationbench_verl.sh

The script sources .env for the agent ARN and output bucket, then runs verl with rllm.remote_runtime.backend=agentcore so rollouts execute in the deployed container. Tune MODEL_PATH, parallelism (TP/EP/CP), batch sizes, and trainer.n_gpus_per_node/nnodes in the script to match your hardware.

Files

File Description
prepare_migrationbench_data.py Register migration_bench/{train,test} from the data bucket's metadata
train_agentcore_migrationbench_verl.py Python API training entrypoint (Hydra config, verl backend)
train_agentcore_migrationbench_verl.sh Launch verl training against the AgentCore Runtime

File: cookbooks/math_tool_agent/README.md

Math Tool Agent

A multi-turn agent flow for rLLM that trains a math agent to solve arithmetic problems using a calculator tool, demonstrating the AgentFlow protocol with tool use.

Overview

The agent solves competition math problems from DeepScaleR-Preview (train, ~40K problems blending AIME/AMC/Omni-MATH) and MATH-500 (test) by calling a calculator tool for each arithmetic step. These problems are harder than GSM8K and require multi-turn reasoning, making them a better test for multi-turn RL training. This cookbook serves two purposes:

  1. End-to-end system test — a multi-turn tool-use example that exercises the full training loop
  2. Onboarding — shows new users how to build a tool-calling agent with rLLM

Architecture

text
AgentFlow.run(task, config)
  │
  └── Multi-turn loop (up to 5 turns)
        │
        ├── LLM call via OpenAI(base_url=config.base_url)
        │     Model outputs reasoning + <tool_call>...</tool_call>
        │
        ├── Parse tool call → execute calculator → inject result
        │
        └── Repeat until model outputs <answer>NUMBER</answer>

The evaluator checks the final <answer> against the ground truth via numeric comparison.

Installation

bash
# From the rllm repo root
uv pip install -e ".[tinker]"                              # rllm + tinker backend
uv pip install --no-deps -e cookbooks/math_tool_agent      # this cookbook

After installation, the agent and evaluator are discoverable by the CLI:

bash
rllm agent list    # should show "math_tool_agent" as a plugin

Dataset

Pull the datasets (one-time):

bash
rllm dataset pull deepscaler_math   # ~40K competition math problems (AIME/AMC/Omni-MATH/STILL)
rllm dataset pull math500           # 500-problem test benchmark

Training

Tinker (single-machine)

bash
bash cookbooks/math_tool_agent/train_tinker.sh

Or directly via the Python API:

bash
python cookbooks/math_tool_agent/train.py \
    rllm/backend=tinker \
    model.name=Qwen/Qwen3-4B-Instruct-2507 \
    model.lora_rank=32 \
    training.group_size=8

Verl (distributed GPU)

Requires verl extras and megatron:

bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>

Then:

bash
bash cookbooks/math_tool_agent/train_verl.sh

Tests

bash
pytest cookbooks/math_tool_agent/test.py -v

Files

File Description
math_tool_agent.py math_tool_agent — multi-turn AgentFlow with calculator tool
math_tool_eval.py math_tool_evaluator — numeric answer comparison
train.py Python API training script (Hydra config)
train_tinker.sh Tinker backend — single-machine training
train_verl.sh Verl backend — distributed multi-GPU training
pyproject.toml Plugin metadata and entry points
test.py Unit tests for calculator, parsing, and evaluation

File: cookbooks/math/README.md

Math Agent

A single-turn math agent for rLLM that solves competition / textbook math problems via the AgentFlow protocol, replacing the four legacy examples/{deepscaler, simple_math, gsm8k_lora, math_tinker}/ examples (all of which used MathAgent + SingleTurnEnvironment + AgentExecutionEngine).

This is the no-tool counterpart to cookbooks/math_tool_agent/, which uses a calculator tool and multi-turn reasoning. Use math for chain-of-thought-only training; use math_tool_agent if you want the model to learn explicit tool use.

Overview

Each task is a math problem. The agent emits one assistant message containing reasoning followed by a final answer in \boxed{...} notation. The evaluator extracts the boxed answer and grades it against ground truth via rllm.eval.reward_fns.math (which uses symbolic + numeric equivalence — 0.5 matches \frac{1}{2}, etc.).

Architecture

text
AgentFlow.run(task, config)
  │
  ├── one LLM call via OpenAI(base_url=config.base_url)
  │     model outputs reasoning + \boxed{ANSWER}
  │
  └── store full response in episode.artifacts["answer"]

Evaluator.evaluate(task, episode)
  │
  └── extract last \boxed{...}, grade against task.metadata["ground_truth"]
      via mathd + sympy.

Installation

bash
uv pip install -e ".[tinker]"                  # rllm + tinker backend
uv pip install --no-deps -e cookbooks/math     # this cookbook

After installation:

bash
rllm agent list      # should show "math"

Datasets

bash
rllm dataset pull hendrycks_math    # train (Hendrycks MATH)
rllm dataset pull math500           # 500-problem test
rllm dataset pull gsm8k             # alternative train
rllm dataset pull deepscaler_math   # ~40K AIME/AMC/Omni-MATH/STILL
rllm dataset pull aime2024          # AIME 2024 (eval)

Eval (rllm CLI)

bash
rllm eval math500 \
    --agent math \
    --evaluator math \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --base-url http://localhost:8000/v1 \
    --max-examples 20

For aime2024 pass@k-style eval (the original examples/deepscaler/ use case), use --max-examples and run multiple times, or use the harness's repeat flag if exposed.

Episode JSONs land under ~/.rllm/eval_results/ for inspection (rllm view).

Training (rllm CLI)

bash
rllm train hendrycks_math \
    --agent math \
    --evaluator math \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --group-size 8 \
    --batch-size 32 \
    --lora-rank 32 \
    --epochs 1 \
    --val-freq 10

For LoRA-only training (the legacy examples/gsm8k_lora/ use case), set --lora-rank higher and pass --train-dataset gsm8k. For deeper customization (verl backend, full PPO knobs), use the shell scripts.

Training (shell scripts)

Tinker (single-machine)

bash
bash cookbooks/math/train_tinker.sh

Verl (distributed GPU)

bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
bash cookbooks/math/train_verl.sh

Tests

bash
pytest cookbooks/math/test.py -v

Files

File Description
math_flow.py math_flow — single-turn AgentFlow
evaluator.py math_evaluator — wraps rllm.eval.reward_fns.math
train.py Python API training script (Hydra config)
train_tinker.sh Tinker backend — single-machine training
train_verl.sh Verl backend — distributed multi-GPU training
pyproject.toml Plugin metadata and entry points
test.py Unit tests for evaluator scoring

Migration notes

This cookbook replaces:

  • examples/deepscaler/ — MathAgent eval on aime2024 → use rllm eval aime2024 --agent math --evaluator math
  • examples/simple_math/ — MathAgent train on hendrycks_math → see train_tinker.sh / train_verl.sh
  • examples/gsm8k_lora/ — MathAgent LoRA train on gsm8k → rllm train gsm8k --agent math --evaluator math --lora-rank 32
  • examples/math_tinker/ — MathAgent + few-shot prompts → fork math_flow.py and customize the SYSTEM_PROMPT

All four shared the same MathAgent + SingleTurnEnvironment + AgentExecutionEngine plumbing. The cookbook collapses them into one self-contained AgentFlow + entry-point with no dependency on rllm.agents or rllm.environments.

File: cookbooks/geo3k/README.md

Geo3K Flow

A VLM geometry problem solver for rLLM that trains on the Geometry3K dataset using the AgentFlow protocol.

Overview

A single-turn VLM agent that receives a geometry problem with a diagram image and produces a step-by-step solution with a boxed final answer. Uses a plain OpenAI client with multimodal content blocks (base64-encoded images).

During training, config.base_url points to the model gateway which transparently captures token IDs and logprobs. During eval, it points directly to the model provider. The agent code is identical in both cases.

Architecture

text
AgentFlow.run(task, config)
  │
  └── Solver
        └── OpenAI(base_url=config.base_url).chat.completions.create(
                messages=[system_prompt, {images + question}]
            )
            → Trajectory(name="solver", steps=[Step(action=response)])
  │
  └── Episode(trajectories=[solver], artifacts={"answer": response})

The evaluator extracts \boxed{} from the response and grades it against the ground truth using symbolic math grading.

Installation

bash
# From the rllm repo root
uv pip install -e ".[tinker]"          # rllm + tinker backend
uv pip install -e cookbooks/geo3k      # this cookbook

After installation, the agent and evaluator are discoverable by the CLI:

bash
rllm agent list    # should show "geo3k" as a plugin

VLM dependency note

Qwen/Qwen3-VL-30B-A3B-Instruct (and the wider Qwen3-VL family) ships a Qwen3VLVideoProcessor that imports torchvision at load time. Even though the tinker rollout only uses the image side, AutoProcessor.from_pretrained will fail without it:

bash
uv pip install torchvision

The tinker backend tries to load the processor lazily and fails fast with an actionable message if torchvision is missing — install it once before kicking off training.

Dataset

Pull the Geometry3K dataset (one-time):

bash
rllm dataset pull geo3k

Training

Tinker (single-machine)

bash
bash cookbooks/geo3k/train_tinker.sh

Or directly via the Python API:

bash
python cookbooks/geo3k/train.py \
    rllm/backend=tinker \
    model.name=Qwen/Qwen3-VL-30B-A3B-Instruct \
    model.lora_rank=32 \
    training.group_size=8

Verl (distributed GPU)

Requires verl extras and megatron:

bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>

Then:

bash
bash cookbooks/geo3k/train_verl.sh

Eval

bash
rllm eval geo3k \
    --agent geo3k \
    --evaluator geo3k_math \
    --model Qwen/Qwen3-VL-30B-A3B-Instruct

Files

File Description
geo3k_flow.py Geo3KFlow — AgentFlow implementation (VLM single-turn solver)
geo3k_eval.py geo3k_evaluator — math answer grading with \boxed{} extraction
train.py Python API training script (Hydra config)
train_tinker.sh Tinker backend — single-machine training
train_verl.sh Verl backend — distributed multi-GPU training
pyproject.toml Plugin metadata and entry points
test.py Unit tests for image handling and evaluation

File: cookbooks/frozenlake/README.md

FrozenLake Agent

A multi-turn agent flow for rLLM that trains a model to navigate procedurally-generated FrozenLake puzzles end-to-end via the AgentFlow protocol, demonstrating gym-environment integration without the legacy Agent + Environment + AgentExecutionEngine stack.

Overview

Each task is a randomly-generated size x size FrozenLake grid (with Start, Goal, frozen tiles _, and Holes O). The agent reads the rendered grid each turn and outputs Up/Down/Left/Right. Reward is 1.0 if the player reaches the goal, 0.0 otherwise.

The map is regenerated deterministically from (seed, size, p) inside the flow, so the dataset stores only those parameters — no map serialization.

Architecture

text
AgentFlow.run(task, config)
  │
  ├── generate_random_map(seed, size, p)   # deterministic, in-process
  ├── env = gymnasium.make("FrozenLake-v1", desc=…, is_slippery=…)
  └── Multi-turn loop (up to max_steps turns)
        │
        ├── LLM call via OpenAI(base_url=config.base_url)
        │     Model outputs reasoning + ```Up|Down|Left|Right```
        │
        ├── parse_action() → env.step(action)
        │
        └── Repeat until terminated (G or H), truncated, or max_steps

The evaluator just reads episode.artifacts["won"] set by the flow.

Installation

bash
uv pip install -e ".[tinker]"                          # rllm + tinker backend
uv pip install --no-deps -e cookbooks/frozenlake       # this cookbook (gymnasium pulled in transitively)

After installation:

bash
rllm agent list      # should show "frozenlake"

Dataset

The dataset is procedurally generated — there's nothing to download. Run once:

bash
python cookbooks/frozenlake/prepare_frozenlake_data.py
# or, with custom sizes:
python cookbooks/frozenlake/prepare_frozenlake_data.py --train-size 5000 --test-size 200 --slippery

This registers frozenlake/{train,test} with DatasetRegistry.

Eval (rllm CLI)

Once the cookbook is installed and the dataset is generated, evaluate a model on the test split with one command:

bash
rllm eval frozenlake \
    --agent frozenlake \
    --evaluator frozenlake \
    --split test \
    --max-examples 20            # smoke-test on 20 puzzles before running the full split

--agent and --evaluator resolve via the entry points declared in pyproject.toml. Drop --base-url if you want the CLI to auto-start a proxy from your rllm setup config. Episode JSONs are written under ~/.rllm/eval_results/ for inspection (rllm view).

Training (rllm CLI)

The simplest training entrypoint is the CLI, which mirrors the eval flags:

bash
rllm train frozenlake \
    --agent frozenlake \
    --evaluator frozenlake \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --group-size 8 \
    --batch-size 32 \
    --lora-rank 32 \
    --epochs 1 \
    --val-freq 10

This dispatches through AgentTrainer with the tinker backend by default. For deeper customization (verl backend, full PPO knobs, megatron, …) use the shell scripts below.

Training (shell scripts)

Tinker (single-machine)

bash
bash cookbooks/frozenlake/train_tinker.sh

Or directly via the Python API with Hydra overrides:

bash
python cookbooks/frozenlake/train.py \
    rllm/backend=tinker \
    model.name=Qwen/Qwen3-4B-Instruct-2507 \
    training.group_size=8

Verl (distributed GPU)

bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
bash cookbooks/frozenlake/train_verl.sh

Tests

bash
pytest cookbooks/frozenlake/test.py -v

Files

File Description
frozenlake_flow.py frozenlake_flow — multi-turn AgentFlow over a gym env
frozenlake_eval.py frozenlake_evaluator — reward = won? 1 : 0
prepare_frozenlake_data.py Generate + register the train/test splits
train.py Python API training script (Hydra config)
train_tinker.sh Tinker backend — single-machine training
train_verl.sh Verl backend — distributed multi-GPU training
pyproject.toml Plugin metadata and entry points
test.py Unit tests for map gen, action parsing, rendering, evaluator

Migration notes

This cookbook is the AgentFlow port of examples/frozenlake/, which uses the legacy BaseAgent + Environment + AgentExecutionEngine stack. The two are functionally equivalent on FrozenLake; the cookbook is fully self-contained — no imports from rllm.agents, rllm.environments, or rllm.engine.agent_execution_engine — so the legacy stack can be deleted without touching this directory.

File: cookbooks/finqa/README.md

FinQA Agent

A multi-turn ReAct-style financial-QA agent for rLLM, ported to the AgentFlow protocol. Replaces projects/finqa/ (which depended on BaseAgent + ToolAgent + ToolEnvironment + AgentExecutionEngine).

Overview

The agent answers questions about SEC 10-K financial statements by querying structured tables extracted from filings. Four tools are exposed via native OpenAI function calling:

Tool Description
get_table_names List queryable tables for a given company
get_table_info Column metadata, dtypes, sample values for a table
sql_query Run a filtered SQLite query against the in-memory store
calculator Safely evaluate a numeric expression (asteval)

All ~6,900 tables across 207 companies are pre-loaded into a process-wide :memory: SQLite at module import; the tools just look up cached metadata or run a read-only query under a thread lock.

The agent's final reply ends with a FINAL ANSWER: … block. The evaluator extracts it and grades against ground truth via a judge LLM (gpt-5-nano for single-table, gpt-5-mini with a structured rubric for multi-table), called directly through the OpenAI API. A small bonus is awarded when the agent inspects the expected tables via get_table_info (the right_table_access_reward).

Model Weights | Dataset

Architecture

text
AgentFlow.run(task, config)
  │
  └── Multi-turn loop (up to 20 turns, native OpenAI tool calls)
        │
        ├── client.chat.completions.create(messages, tools=TOOL_SPECS)
        │
        ├── If msg.tool_calls is empty → that's the final answer, break.
        │
        └── Else: dispatch each tool call → append a `tool` message → repeat.
              (track table_name in `accessed_tables` whenever
               `get_table_info` is invoked, for the table-access bonus)
  │
  └── episode.artifacts = {"answer": full_response, "accessed_tables": [...], "turns": N}

Evaluator.evaluate(task, episode)
  │
  └── extract FINAL ANSWER from artifacts["answer"], grade via judge LLM,
      add table-access bonus, return EvalOutput.

Installation

bash
uv pip install -e ".[tinker]"                      # rllm + tinker backend
uv pip install --no-deps -e cookbooks/finqa        # this cookbook

After installation:

bash
rllm agent list      # should show "finqa"

Dataset

bash
python cookbooks/finqa/prepare_finqa_data.py

This will:

  • Download the dataset tarball from rLLM/finqa
  • Extract company tables to cookbooks/finqa/data/company_tables/ (207 companies, 6,923 tables)
  • Register finqa/{train, val, test} with DatasetRegistry (4,030 / 522 / 558 examples)

Eval (rllm CLI)

Set the judge key (used by the evaluator), then run:

bash
export OPENAI_API_KEY=sk-…

rllm eval finqa \
    --agent finqa \
    --evaluator finqa \
    --model rLLM/rLLM-FinQA-4B \
    --base-url http://localhost:30000/v1 \
    --split test \
    --max-examples 20

Episode JSONs land under ~/.rllm/eval_results/. If OPENAI_API_KEY is missing, the evaluator silently returns reward=0 rather than crashing.

Training (rllm CLI)

bash
export OPENAI_API_KEY=sk-…

rllm train finqa \
    --agent finqa \
    --evaluator finqa \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --group-size 8 \
    --batch-size 16 \
    --lora-rank 32 \
    --epochs 1 \
    --val-freq 20

Training (shell scripts)

Tinker (single-machine, LoRA on 30B)

bash
bash cookbooks/finqa/train_tinker.sh

Verl (distributed GPU, 4B base)

bash
uv pip install -e ".[verl]"
bash scripts/install_megatron.sh <cu128|cu129|...>
bash cookbooks/finqa/train_verl.sh

Tests

bash
pytest cookbooks/finqa/test.py -v

Files

File Description
finqa_flow.py finqa_flow — multi-turn AgentFlow with native tool calling
finqa_tools.py The four tools as plain functions + OpenAI tool specs
finqa_eval.py finqa_evaluator — judge-LLM correctness + table-access bonus
finqa_constants.py Path constants (data, prompts)
prepare_finqa_data.py HF download + register train / val / test splits
train.py Python API training script (Hydra config)
train_tinker.sh Tinker backend — single-machine LoRA training
train_verl.sh Verl backend — distributed multi-GPU training
pyproject.toml Plugin metadata and entry points
test.py Unit tests (calculator, FINAL ANSWER parsing, table-access scoring)
prompts/ System prompt + correctness rubric prompts

Migration notes

This cookbook replaces projects/finqa/, which used the legacy stack:

Legacy Replacement
FinQAAgent(ToolAgent) (qwen tool-call parsing) finqa_flow AgentFlow + native OpenAI function calling
FinQAEnvironment(ToolEnvironment) (in-loop reward) Evaluator outside the loop, called once on the final episode
MultiTurnWorkflow + AgentExecutionEngine Plain async function — client.chat.completions.create(tools=…) in a for loop
Tool base class wrappers Plain Python callables (get_table_names, get_table_info, sql_query, calculator)

The cookbook has zero imports from rllm.agents, rllm.environments, rllm.engine.agent_execution_engine, or rllm.workflows. After this migration, ToolAgent / ToolEnvironment / MCPToolAgent are no longer reachable from any active code path and can be removed.

2. Official Technical Reference & Guides (rllm-org/rllm-docs)

Mintlify Starter Kit

Use the starter kit to get your docs deployed and ready to customize.

Click the green Use this template button at the top of this repo to copy the Mintlify starter kit. The starter kit contains examples with

  • Guide pages
  • Navigation
  • Customizations
  • API reference pages
  • Use of popular components

Follow the full quickstart guide

AI-assisted writing

Set up your AI coding tool to work with Mintlify:

bash
npx skills add https://mintlify.com/docs

This command installs Mintlify's documentation skill for your configured AI tools like Claude Code, Cursor, Windsurf, and others. The skill includes component reference, writing standards, and workflow guidance.

See the AI tools guides for tool-specific setup.

Development

Install the Mintlify CLI to preview your documentation changes locally. To install, use the following command:

text
npm i mint

Run the following command at the root of your documentation, where your docs.json is located:

text
mint dev

View your local preview at http://localhost:3000.

Publishing changes

Install our GitHub app from your dashboard to propagate changes from your repo to your deployment. Changes are deployed to production automatically after pushing to the default branch.

Need help?

Troubleshooting

  • If your dev environment isn't running: Run mint update to ensure you have the most recent version of the CLI.
  • If a page loads as a 404: Make sure you are running in a folder with a valid docs.json.

Resources

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#agent-framework Browse agent-framework catalog → #agentic-workflow Browse agentic-workflow catalog → #coding-agent Browse coding-agent catalog → #distributed-training Browse distributed-training catalog → #llm-reasoning Browse llm-reasoning catalog → #llm-training Browse llm-training catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →