verl Documentation

1. Project Overview & Quickstart (verl-project/verl) Copy Chapter

File: docker/verl0.4-cu124-torch2.6-fa2.7.4/README.md Copy Chapter

verl image with verl v0.4.x

Important packages version Copy Chapter

cuda==12.4
cudnn==9.8.0
torch==2.6.0
flash_attn=2.7.4
sglang==0.4.6.post5
vllm==0.8.5.post1
nvidia-cudnn-cu12==9.8.0.87
transformer_engine==2.3
megatron.core==core_v0.12.2
# Preview
transformer_engine==2.5
megatron.core==core_r0.13.0

Target Copy Chapter

• Base image:
  • verlai/verl:base-verl0.4-cu124-cudnn9.8-torch2.6-fa2.7.4
• App image:
  • verlai/verl:app-verl0.4-sglang0.4.6.post5-vllm0.8.5-mcore0.12.2-te2.2: SGLang requires vLLM in 0.4.6.post5 version, vLLM can have some package conflicts with SGLang
  • verlai/verl:app-verl0.4-sglang0.4.6.post5-vllm0.8.5-mcore0.12.2-te2.2-deepep: Built with deepep
  • verlai/verl:app-verl0.4-vllm0.8.5-mcore0.12.2-te2.2
  • verlai/verl:app-verl0.4-vllm0.8.5-mcore0.12.2-te2.2-deepep: Built with deepep
• Preview image:
  • verlai/verl:app-verl0.4-sglang0.4.6.post5-vllm0.8.5-mcore0.13.0-te2.2-preview
  • verlai/verl:app-verl0.4-vllm0.8.5-mcore0.13.0-te2.2-preview

File: docker/verl0.5-cu126-torch2.7.1-fa2.8.0/README.md Copy Chapter

verl image with verl v0.5

Important packages version Copy Chapter

cuda==12.6
cudnn==9.8.0
torch==2.7.1
flash_attn=2.8.0    ##
sglang==0.4.8
vllm==0.8.5.post1
nvidia-cudnn-cu12==9.8.0.87
transformer_engine==2.3
megatron.core==core_v0.12.2
# Preview
transformer_engine==2.5
megatron.core==core_r0.13.0

Target Copy Chapter

• Base image:
  • verlai/verl:base-verl0.5-cu126-cudnn9.8-torch2.7.1-fa2.8.0: We offer a base image with deep ep built in
• App image:
  • verlai/verl:app-verl0.5-sglang0.4.9-mcore0.12.2
  • verlai/verl:app-verl0.5-sglang0.4.9-mcore0.13.0-preview
• vllm temporarily not support latest version

File: docker/verl0.5-cu126-torch2.7-fa2.7.4/README.md Copy Chapter

verl image with verl v0.5

Important packages version Copy Chapter

cuda==12.6
cudnn==9.8.0
torch==2.7.1
flash_attn=2.7.4.post1
sglang==0.4.9.post6
vllm==0.8.5.post1
nvidia-cudnn-cu12==9.8.0.87
transformer_engine==2.3
megatron.core==core_v0.12.2
# Preview
transformer_engine==2.5
megatron.core==core_r0.13.0

Target Copy Chapter

• Base image:
  • verlai/verl:base-verl0.5-cu126-cudnn9.8-torch2.7.1-fa2.7.4: We offer a base image with deep ep built in, for vllm/sglang
• App image:
  • verlai/verl:app-verl0.5-transformers4.55.4-vllm0.10.0-mcore0.13.0-te2.2
  • verlai/verl:app-verl0.5-transformers4.55.4-sglang0.4.10.post2-mcore0.13.0-te2.2
  • iseekyan/verl:app-verl0.5-transformers4.55.4-vllm0.10.0-mcore0.15.0-te2.7

File: docker/verl0.5-preview-cu128-torch2.7.1-fa2.8.0/README.md Copy Chapter

verl image with verl v0.5

Important packages version Copy Chapter

cuda==12.8
cudnn==9.8.0
torch==2.7.1
flash_attn=2.8.0    ##
sglang==0.4.8
transformer_engine==2.5
megatron.core==core_r0.13.0
nvidia-cudnn-cu12==9.8.0.87

Target Copy Chapter

• Base image:
  • verlai/verl:base-verl0.5-preview-cu128-cudnn9.8-torch2.7.1-fa2.8.0: We offer a base image with flash infer 0.2.6.post1 built in
• App image:
  • verlai/verl:app-verl0.5-preview-sglang0.4.8-mcore0.13.0-preview
• vllm temporarily not support latest version

!!!Notice!!! Copy Chapter

• pyext is lack of maintainace and cannot work with python 3.12, consider using replacement and deprecating this package.

File: docker/README.md Copy Chapter

Dockerfiles of verl

We provide pre-built Docker images for quick setup. And from this version, we utilize a new image release hierarchy for productivity and stability.

Start from v0.6.0, we use vllm and sglang release image as our base image.

Start from v0.7.0, since vllm/vllm-openai:v0.12.0 is a minimal image without some essential libraries, we use nvidia/cuda:12.9.1-devel-ubuntu22.04 as our base image for vllm.

Base Image Copy Chapter

• vLLM: https://hub.docker.com/r/nvidia/cuda
• SGLang: https://hub.docker.com/r/lmsysorg/sglang

Application Image Copy Chapter

Upon base image, the following packages are added:

• flash_attn
• Megatron-LM
• Apex
• TransformerEngine
• DeepEP

Latest docker file:

• Dockerfile.stable.vllm
• Dockerfile.stable.sglang

All pre-built images are available in dockerhub: https://hub.docker.com/r/verlai/verl. For example, verlai/verl:sgl0512.latest, verlai/verl:vllm024.latest.

You can find the latest images used for development and ci in our github workflows:

• .github/workflows/vllm.yml
• .github/workflows/sgl.yml

Building Locally Copy Chapter

To build an image from source:

docker build -f docker/Dockerfile.stable.vllm -t verl:vllm-local .

For users in China who need an apt mirror to speed up package downloads, pass APT_MIRROR:

docker build -f docker/Dockerfile.stable.vllm \
    --build-arg APT_MIRROR=https://mirrors.tuna.tsinghua.edu.cn \
    -t verl:vllm-local .

GB200 / aarch64 Copy Section

Pre-built images for GB200 (aarch64) are not yet published. Users should build locally on an aarch64 machine. Pre-built images will be added once available. This applies to the uv image below too — it supports aarch64 from the same Dockerfile and the same uv.lock.

docker build -f docker/Dockerfile.stable.vllm -t verl:vllm-arm64 .
docker build -f docker/Dockerfile.uv.cu130 -t verl:uv-cu130-arm64 .   # uv image

Check uname -m first: on an aarch64 host the plain docker build above is
all you need, and everything below is irrelevant.

From an x86_64 host you have two options.

Remote arm64 machine, built natively (preferred). Attach it as a buildx node
and drive the build from x86_64 — no emulation. Replace user@arm-host with a
real, resolvable arm64 host: this is a placeholder, and buildx reports a bare
Could not resolve hostname if it is pasted verbatim. The node needs Docker
18.09+, key-based SSH (buildx runs ssh ... docker system dial-stdio
non-interactively, so a password prompt fails), and an SSH user in the docker
group. Verify the connection before creating the builder:

ssh user@arm-host docker version          # must succeed without a password
docker buildx create --name verl-arm --driver docker-container \
    --platform linux/arm64 ssh://user@arm-host --use
docker buildx build --platform linux/arm64 -f docker/Dockerfile.uv.cu130 \
    -t verl:uv-cu130-arm64 --load .

Note that --platform on buildx create only declares what the node
advertises; it does not make an x86_64 node build arm64 natively.

No arm64 machine at all: QEMU emulation. Works, but slow — the apt install,
the GDRCopy build and the prefetch stage's megatron-core source build
all run emulated:

docker run --privileged --rm tonistiigi/binfmt --install arm64
docker buildx create --name verl-qemu --driver docker-container --use
docker buildx build --platform linux/arm64 -f docker/Dockerfile.uv.cu130 \
    -t verl:uv-cu130-arm64 --load .

Building behind an egress proxy Copy Section

--build-arg http_proxy=... reaches the RUN steps only. The base image in
FROM is resolved by the BuildKit daemon itself — in the
buildx_buildkit_* container, before any build arg applies — so a proxy passed
that way cannot fix failed to resolve source metadata for docker.io/nvidia/cuda:.... Put the proxy on the builder, and keep the build
args for the RUN steps:

docker buildx rm verl-qemu 2>/dev/null || true
docker buildx create --name verl-qemu --driver docker-container --use \
    --driver-opt env.http_proxy=http://proxy.example.com:8118 \
    --driver-opt env.https_proxy=http://proxy.example.com:8118 \
    --driver-opt env.no_proxy=localhost,127.0.0.1

docker buildx build --platform linux/arm64 -f docker/Dockerfile.uv.cu130 \
    --build-arg http_proxy=http://proxy.example.com:8118 \
    --build-arg https_proxy=http://proxy.example.com:8118 \
    -t verl:uv-cu130-arm64 --load .

If Docker Hub stays unreachable, skip it entirely by pointing the base at an
internal mirror (the mirror must carry the arm64 manifest for an arm64 build):

docker buildx build --platform linux/arm64 -f docker/Dockerfile.uv.cu130 \
    --build-arg CUDA_BASE_IMAGE=<mirror>/nvidia/cuda \
    -t verl:uv-cu130-arm64 --load .

One tag for both arches Copy Section

Both arches can and should share a single tag, published as a multi-arch
manifest list — docker pull verlai/verl:uv.cu130 then resolves to whichever
arch the puller is on, and FROM verlai/verl:uv.cu130 in a downstream image
resolves per build platform. This is how nvidia/cuda:13.0.2-devel-ubuntu24.04
itself works, and what lets one Dockerfile and one job spec cover both.

If a single builder can reach both platforms (e.g. a native x86_64 node plus a
native arm64 node in the same builder), one command does it:

docker buildx build --platform linux/amd64,linux/arm64 \
    -f docker/Dockerfile.uv.cu130 -t verlai/verl:uv.cu130 --push .

More often the two builds happen on different machines at different times. Then
push arch-suffixed tags and join them afterwards — the join is metadata only, so
it is instant and re-runnable:

# on each host, natively:
docker buildx build -f docker/Dockerfile.uv.cu130 -t verlai/verl:uv.cu130-amd64 --push .
docker buildx build -f docker/Dockerfile.uv.cu130 -t verlai/verl:uv.cu130-arm64 --push .

# then, from anywhere:
docker buildx imagetools create -t verlai/verl:uv.cu130 \
    verlai/verl:uv.cu130-amd64 verlai/verl:uv.cu130-arm64
docker buildx imagetools inspect verlai/verl:uv.cu130   # verify both platforms

Three things to know:

• A multi-platform build must --push; --load cannot put an index into
  the local docker image store (load one arch at a time instead).
• Add --provenance=false if your registry or runtime trips over the extra
  unknown/unknown attestation entries buildx puts in the index by default.
• Pin by tag, not digest, to keep the multi-arch behaviour — a digest names
  one specific manifest, so FROM ...@sha256:... pins you to a single arch.

uv image (`Dockerfile.uv.cu130`) Copy Chapter

docker/Dockerfile.uv.cu130 builds one image around verl's universal
uv.lock (GPU: CUDA 13.0 / torch 2.11 — vllm, sglang, fsdp, megatron — plus
the GPU-free cpu slice), on x86_64 and aarch64 alike. The build bakes the
full uv package cache for every backend into the image (the prefetch
stage); it does not bake a fixed .venv. Build with BuildKit:

DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.uv.cu130 -t verl:uv-cu130 .

You pick the backend combination at run time (not build time), and it must be
conflict-free (see [tool.uv].conflicts). There is no install step: the
container starts in a shell, and the first uv run --extra ... installs the
requested extras into /workspace/verl/.venv from the baked cache (fast /
offline) and runs from it — that venv is already on PATH:

docker run --rm -it --gpus all verl:uv-cu130 bash
# then, inside the container:
bash examples/grpo_trainer/run_qwen3_8b_fsdp.sh
# ... or spell the command out:
uv run --frozen --all-packages --extra sglang --extra megatron \
    python3 -m verl.trainer.main_ppo ...

Optional named stages: --target=prefetch builds just the baked cache (every
backend, no source), and --target=lock regenerates uv.lock. A companion
docker/Dockerfile.uv.cu129 builds the cu12.9 / torch-2.9.1 backends (veomni,
nemoautomodel) the same way; trtllm stays deferred. For the full story — the
launch flow, the baked-cache mechanics, and re-locking — see
"Install with uv" in
docs/start/install.rst.

Installation from Docker Copy Chapter

After pulling the desired Docker image and installing desired inference and training frameworks, you can run it with the following steps:

1. Launch the desired Docker image and attach into it:

docker create --runtime=nvidia --gpus all --net=host --shm-size="10g" --cap-add=SYS_ADMIN -v .:/workspace/verl --name verl <image:tag> sleep infinity
docker start verl
docker exec -it verl bash

2. If you use the images provided, you only need to install verl itself without dependencies:

# install the nightly version (recommended)
git clone https://github.com/verl-project/verl && cd verl
pip3 install --no-deps -e .

[Optional] If you hope to switch between different frameworks, you can install verl with the following command:

# install the nightly version (recommended)
git clone https://github.com/verl-project/verl && cd verl
pip3 install -e .[vllm]
pip3 install -e .[sglang]

Release History Copy Chapter

• 2026/07/21: update vllm stable image to vllm==0.24.0 (torch==2.11.0, CUDA 13.0.2, Ubuntu 24.04); update sglang stable image to sglang==0.5.12
• 2026/03/10: update vllm stable image to vllm==0.17.0; update sglang stable image to sglang==0.5.9
• 2026/01/17: update vllm stable image to torch==2.9.1, cudnn==9.16, deepep==1.2.1
• 2025/12/23: update vllm stable image to vllm==0.12.0; update sglang stable image to sglang==0.5.6
• 2025/11/18: update vllm stable image to vllm==0.11.1; update sglang stable image to sglang==0.5.5

File: docs/blog/v0.7.md Copy Chapter

verl 0.7 release blog

Author: verl team

Last updated: 01/03/2026.

Overview Copy Chapter

verl adopts a Hybrid-Controller architecture (also known as HybridFlow). Sharing design principles with asynchronous sharded dataflow systems like Google Pathways, verl models Reinforcement Learning (RL) algorithms, such as PPO, GRPO, DAPO, and others, as a multi-stage, multi-model and parallelizable dataflow graph.

To balance flexibility with performance, verl unifies two distinct programming models:

High-Level Single-Controller (MPMD): At the orchestration level, a single process RLTrainer manages the global computation graph. It handles macro-tasks such as scheduling rollout generation, triggering reward scoring, and dispatching distributed training jobs.

Internal Multi-Controller (SPMD): Internally, the Model Engine operates in standard distributed training mode. Workers execute identical programs, via trainer backends like FSDP, Megatron, or VeOmni, or rollout executors (not rollout server) like vLLM/SGLang/TensorRT-LLM, to perform heavy distributed computation, synchronizing via collective communication.

This hybrid approach offers significant advantages:

Flexible Orchestration: The single-controller design allows verl to dynamically manage complex constraints within the computation graph, including flexible data dependencies, diverse resource allocation and model placement, and fine-grained asynchronous staleness control.

Abstraction of Complexity: We encapsulate complex parallel strategies—such as 5D parallelism (DP, TP, CP, PP, and EP)—strictly within the Model Engine. This allows users to focus entirely on RL algorithm implementation without getting bogged down by the details of distributed training.

Furthermore, leveraging Ray placement groups, verl provides ResourcePool and WorkerGroup abstractions. These enable flexible GPU sharing among the various roles in the RL process—such as actor, critic, reward, and rollout—allowing components to share resources efficiently while remaining isolated.

As illustrated in the diagram below, the overall architecture of verl is divided into two layers:

• verl-core: provides four components required for the RL pipeline: model engine, rollout engine, checkpoint engine, and transfer queue. Each component exposes abstract interfaces, making them both extensible and pluggable.
• verl-trainer: builds upon these components, construct various RL pipelines—such as on-policy, one-step-off-policy, and fully asynchronous—tailored to meet the demands of diverse scenarios.

verl-core Copy Chapter

Model Engine Copy Section

The Model Engine serves as verl's core training engine, defining a set of abstract interfaces that support pluggable backends. It operates in SPMD mode:

• SFT: Workers are launched via torchrun.
• RL: Workers are executed via the WorkerGroup API, invoked by the single-controller.

The abstract interfaces include methods like initialize, forward, optimizer_step, and load/offload. Integrating a new training engine simply requires inheriting and implementing these interfaces. Crucially, because all backends adhere to this unified abstraction, adding a new Model Engine requires absolutely no code modification on the caller side. The RLTrainer remains completely agnostic to the backend's specific parallel strategy when calling these interfaces, while the WorkerGroup automatically handles data dispatch and collection based on the underlying parallelism.

Currently, the Model Engine supports the following backends (more backend maybe supported in future, e.g torchtitan):

class BaseEngine:
    def initialize(self):
        """Instantiate or load the model, optimizer, and learning rate scheduler."""
        raise NotImplementedError

    def optimizer_zero_grad(self):
        """Zero the gradients of the optimizer."""
        raise NotImplementedError

    def optimizer_step(self):
        """Perform an optimization step using the optimizer."""
        raise NotImplementedError

    def lr_scheduler_step(self):
        """Advance the learning rate scheduler by one step."""
        raise NotImplementedError

    def forward_backward_batch(self, data: TensorDict, loss_function: Callable, forward_only=False) -> Any:
        """Perform a forward pass and optionally a backward pass on a batch of data."""
        raise NotImplementedError

    def get_per_tensor_param(self) -> tuple[Generator[tuple[str, torch.Tensor], None, None], Optional[dict]]:
        """Get a generator that yields per-tensor parameters and optional peft config."""
        raise NotImplementedError

    def to(self, device: str, model: bool = True, optimizer: bool = True, grad: bool = True):
        """Move model parameters, optimizer states, or both to the specified device."""
        raise NotImplementedError

Rollout Engine Copy Section

As LLM reinforcement learning evolves from single-turn, static tasks to multi-turn, dynamic, and interactive agentic tasks, the legacy SPMD rollout mode previously used by verl has become insufficient. Consequently, in verl v0.7, we have removed the SPMD rollout mode and switched to rollout server mode by default.

In the server mode, the LLM server operates as online serving rather than the traditional offline batch inference. Clients send per-sample requests to the server, enabling the engine to utilize dynamic batching. This significantly enhances throughput efficiency for multi-turn conversation. Furthermore, the server-based approach eliminates the need for intrusive modifications to the LLM inference engine, allowing for the seamless integration of modern inference backends such as vLLM, SGLang, and TensorRT-LLM.

On the client side, verl introduces an extensible AgentLoop abstraction designed to define custom agentic task loops. This abstraction manages the cycle of requesting responses from the LLM server and interacting with external environments to obtain feedback. We provide two default implementations:

• SingleTurnAgentLoop: Designed for standard single-turn tasks.
• ToolAgentLoop: Designed for classic ReAct architectures involving multi-turn tool invocation.

Users can implement custom AgentLoop logic tailored to their specific needs, such as SWEAgentLoop or GUIAgentLoop.

class AgentLoopBase(ABC):
    @abstractmethod
    async def run(self, sampling_params: dict[str, Any], **kwargs) -> AgentLoopOutput:
        """Run agent loop to interact with LLM server and environment.

        Args:
            sampling_params (Dict[str, Any]): LLM sampling params.
            **kwargs: dataset fields from `verl.utils.dataset.RLHFDataset`.

        Returns:
            AgentLoopOutput: Agent loop output.
        """
        raise NotImplementedError

TransferQueue Copy Section

As mentioned, verl uses a global single-controller RLTrainer to orchestrate the computation graph. A major limitation in the current implementation is that the RLTrainer handles both control and data flow, creating a bottleneck when dispatching data between components. This issue is amplified by the massive data volumes in multimodal training (images, video, audio) and complex algorithms like router replay, which requires transmitting large tensors per sample. Our earlier attempt to solve this using the Ray object store yielded poor performance due to the lack of tensor optimization and fine-grained column access.

In v0.7, we experimentally introduced TransferQueue to decouple control flow from data flow. The RLTrainer now only dispatch instructions and metadata, while TransferQueue handles data transmission via reference passing. TransferQueue is specifically optimized for PyTorch tensors (supporting zero-copy and RDMA) and allows for backend extensions like ZeroMQ, NIXL, and Ray RDT. We plan to make this the default transmission method in v0.8.

# In PPOTrainer
def fit(self):
    batch = next(dataloader)
    gen_batch: BatchMeta = self.rollout_manager.generate_sequences(batch)
    output: BatchMeta = self.actor_rollout_wg.compute_log_prob(gen_batch)
    gen_batch = gen_batch.union(output)
    output = self.actor_rollout_wg.update_actor(gen_batch)

# In Worker
def compute_log_prob(self, batch: BatchMeta) -> BatchMeta:
    data = tq.get(batch)
    output = self.actor.infer_batch(data=data)
    return tq.put(output)

Checkpoint Engine Copy Section

With the increase in LLM context lengths and the evolution of agentic tasks, the "long-tail" problem in rollout has become prominent, limiting the overall efficiency of RL training.

To mitigate this, a viable strategy is moving from on-policy synchronous training to off-policy asynchronous training, e.g Laminar, Areal, StreamRL, LlamaRL, PipelineRL. This involves separating the rollout and model engines onto different nodes (a disaggregated architecture, as opposed to colocated), with data transmitted via queues. This separation alleviates the rollout long-tail issue and enables rollout elastic scaling, fault tolerance, and heterogeneous hardware. However, it introduces a new challenge: efficient cross-node parameter synchronization.

To address this, we introduce the Checkpoint Engine: a unified abstraction layer designed to synchronize weights between various training and inference backends.

• It provides three unified APIs to implement the streaming transmission of parameters.
• Users can extend the Transport Layer implementation based on their specific infrastructure requirements (device, network, local cache, etc.).

Currently, we provide two transport backends: NCCL (for broadcast collective communication) and NIXL (for P2P point-to-point communication).

class CheckpointEngine(ABC):
    @abstractmethod
    async def send_weights(self, weights: Generator[tuple[str, torch.Tensor], None, None]):
        """Send the weights of the model.

        Args:
            weights: A generator that yields the name of the weight tensor and the tensor itself.
        """
        raise NotImplementedError

    @abstractmethod
    async def receive_weights(self) -> Generator[tuple[str, torch.Tensor], None, None]:
        """Receive the weights of the model.

        Yields:
            A tuple of the name of the weight tensor and the tensor itself.
        """
        raise NotImplementedError

verl-trainer Copy Chapter

Building upon the four core components provided by verl-core, verl-trainer constructs several RL training pipelines tailored to specific scenarios. These pipelines are designed to address training efficiency challenges across varying scales and requirements:

On-policy (Synchronous)

• Main Features: Executes rollout and training serially, typically sharing GPU resources (Colocate). It strictly adheres to standard on-policy algorithm definitions, where training must wait for all samples to be generated.
• Scenarios: Best for baseline implementations, scenarios where strict algorithmic correctness is prioritized over training throughput.

One-step-off-policy (Async)

• Main Features: Parallelizes generation and training by overlapping the current training step with the next batch's generation. It employs resource isolation and uses parameters from the previous step for rollout to minimize GPU idle time.
• Scenarios: Ideal for scenarios requiring moderate efficiency gains (20%–40%) while maintaining training stability very close to strict on-policy methods.

Fully async (Decoupled & Streaming)

• Main Features: Completely decouples the Trainer and Rollouter onto separate nodes. It utilizes streaming data transfer, staleness control, and partial rollout mechanisms to maximize throughput and mitigate long-tail generation latency.
• Scenarios: Essential for large-scale training (e.g., 128+ GPUs) or complex reasoning tasks (e.g., long chain-of-thought) where generation latency significantly bottlenecks performance.

roadmap Copy Chapter

v0.7 release Copy Section

Model Engine

• Integrate Megatron-Bridge and support LoRA/PEFT, see blog post: How We Build Trillion Parameter Reasoning RL with 10% GPUs
• Support experimental fp8 training for megatron backend
• Support new model for megatron backend: GPT-OSS, Qwen3-Next
• Comprehensive support for new mode engine, FSDP and Megatron engine are production ready.
  • Dispatch tensordict with nested tensor instead of padded DataProto
  • Add TrainingWorker that resembles Tinker-like API
  • Add VLM support for model engine, SFT and RL trainer
  • Add model engine based critic model
  • Implement ActorRolloutRefWorker by TrainingWorker, support different backend in one worker
• New VeOmni engine added, still in alpha status.

Rollout Engine

• Remove SPMD rollout mode
• Support blockwise fp8 rollout for vllm and sglang; support online quant for vllm with torchao
• Experimental router replay support for vllm
• Optimize multi-modal data fetch and preprocess, support video input
• Upgrade to vllm==0.12.0; sglang==0.5.6

Reward

• Support hybrid reward scenarios, including generative, discriminative, rule-based rewards, and their combinations.
• Refactor reward models into server mode, supporting both colocated and standalone deployments.
• Introduce new reward managers to handle more complex scenarios, limited mode for request rate control and remote mode for CPU-intensive tasks.

Algorithm

• Add CISPO: Clipped IS-weight Policy Optimization
• Add SAPO: Soft Adaptive Policy Optimization

Recipe

• [NEW] VLA: add experimental support for VLA model
• [NEW] rhymerl: History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL
• TransferQueue: support multiple data partition and optimize tensor zero-copy serialization
• One-step-off-policy/Fully async: optimize weight synchronization by checkpoint engine with bucket and pipeline support.

v0.8 Copy Section

Model Engine

• Deprecate DataProto by Tensordict for zero padding transmission
• Switch default to new model engine, mark legacy engine (fsdp_workers.py, megatron_workers.py) as deprecated
• Feature parity between new and legacy model engine: LoRA/PEFT, etc
• Polish VeOmni engine to production ready status
• Support MTP RL training
• Optimize GPU memory for long context: fine-grained activation recompuation/offload
• New model support: DeepSeek V3.2, etc

Rollout Engine

• New rollout engine TensorRT-LLM
• Separate vllm worker from trainer process, update weights by cuda ipc

TransferQueue

• Merge TransferQueue recipe into main
• Optimize e2e image/video vlm training pipeline by TransferQueue
• Optimize router replay transmission by TransferQueue

Checkpoint Engine

• Add checkpoint engine abstract interface
• Add NCCL and NIXL transport backend
• Add more transport backend

v0.9 Copy Section

Trainer

• Merge Full async into main: refactor with verl-core component

Model Engine

• Remove legacy model engine (fsdp_workers.py, megatron_workers.py)
• Support omni-model RL training: Qwen3-Omni, BAGEL, etc

Rollout Engine

• New rollout engine vllm-omni

More agentic training recipe

• SWEAgent
• GUIAgent

2. Official Technical Reference & Guides (verl-project/verl-project.github.io) Copy Chapter

File: content/posts/2026-08-04-introducing-the-verl-blog/index.md Copy Chapter

title: "Introducing the verl blog"
date: 2026-08-04
authors:

• "verl-project"
  summary: "verl now has its own blog. It exists so that the people who build features can explain them properly."
  image: "cover.png"
  tags:
• announcement
• community
  math: false
  toc: true

verl now has a blog, and it is open to contributors from day one. If you have
landed something in verl and want to explain how it works, you can publish it
here by opening a pull request.

What gets published here Copy Chapter

Writing that helps someone understand verl better. Some of the shapes that
takes:

• Feature deep dives — the design behind something that landed, including
  the approaches that were tried and rejected.
• Performance work — with hardware, model, and baseline stated, so the
  numbers mean something.
• Recipes and case studies — end-to-end accounts of training something real.
• Release deep dives — what changed and why it matters.
• … and anything else you think is worth explaining. The list above is a
  starting point, not a set of boxes to fit into.

None of this replaces the documentation.
The docs describe how the current version works and are kept up to date; a post
captures the reasoning behind a change at the time it was made. If what you are
writing is reference material, it belongs in the docs instead.

Publishing is a pull request Copy Chapter

The whole point of hosting this in the project's own repository is that the
contribution path is the one you already use. A post is a directory:

content/posts/2026-08-04-your-post-slug/
├── index.md
└── cover.png

Its front matter declares who wrote it and what it says:

---
title: "Async rollout in verl"
date: 2026-08-04
authors:
  - "Jane Doe"
summary: "Async rollout overlaps generation with training and cuts idle GPU time on long-horizon tasks."
image: "cover.png"
tags:
  - performance
---

Open a pull request and CI validates the front matter and builds the site. A
maintainer reviews it for accuracy. On merge, the site redeploys on its own.

One thing worth doing first: open an issue with the idea before you start
drafting. A maintainer can tell you early whether the topic fits and whether
somebody else is already writing it, which is a lot cheaper than finding out
once the draft is done.

Full details are on the Write for us page.

Attribution Copy Chapter

Posts stay attributed to the people who wrote them. Your name stays on the
byline rather than being replaced by a generic project account, and each author
name becomes an archive page collecting that person's posts.

You also keep the copyright in what you write. Publishing here does not sign
anything away, and nothing stops you from putting the same piece on your own
site as well.

What happens next Copy Chapter

That is up to the community. A blog only works if people write for it, and the
person best placed to explain a piece of verl is usually the one who built it.

So if you have landed something in verl and found yourself explaining it more
than twice — in a Slack thread, in a code review, in an issue comment — that is
a post waiting to be written. Writing it down once, properly, saves the next
person the same conversation, and it is one of the more lasting ways to
contribute to an open source project.

Have a look at Write for us and open a pull request. We are
happy to help with drafts, and just as happy to see a rough one.

File: content/posts/2026-08-11-verl-tinker/index.md Copy Chapter

title: "Introducing verl-tinker: Your Tinker loop, your GPUs"
date: 2026-08-11
authors:

• "Tianle Zhong"
• "Huaye Zeng"
• "Xibin Wu"
• "Siping Tao"
• "Peng Wu"
• "Yifan Pi"
• "Xiao Yu"
  summary: "Keep the Tinker Cookbook loop you know, and run SFT, RL, and distillation on verl-managed GPU workers you control."
  image: "verl_tinker_logo.png"
  tags:
• training
• rollout
• recipes
• distillation
  math: false
  toc: true

What if you could keep a Tinker Cookbook training loop exactly where it is,
but run the heavy lifting on your own GPUs?

That is what verl-tinker makes possible. The official Tinker SDK and nearly
the full range of Tinker Cookbook recipes work as they are; in most cases, the
only client-side change is overriding the base URL. Point the program at a
local HTTP endpoint and it can drive supervised fine tuning (SFT),
reinforcement learning (RL), online and offline distillation, checkpointing,
and rollout on verl-managed workers.

In other words: the experiment loop stays simple; the infrastructure becomes
yours.

The implementation is available now in the
verl_tinker recipe.
It is deliberately lightweight: a thin forwarding layer exposes a
Tinker-compatible API, while presets make it easy to launch the server in one
step. Underneath, workloads run on mature, production-tested verl, VeOmni, and
vLLM configurations.

Keep the loop. Bring the infrastructure. Copy Chapter

Tinker Cookbook recipes describe an algorithm from the researcher's point of
view. They build data, ask a model to sample or compute log probabilities,
submit a loss-bearing batch, and take an optimizer step. It is an appealingly
compact way to express an experiment.

verl tackles the other side of the problem: distributed actor, reference,
rollout, and teacher workers; model-state movement; parallelism; offload; and
checkpoint storage.

verl-tinker joins those two worlds at the API boundary. Researchers keep the
concise Cookbook loop and choose the cluster, model weights, inference engine,
parallelism strategy, and storage behind it. The client environment needs only
tinker and tinker-cookbook; the server owns the distributed runtime.

The bridge stays intentionally small. It forwards Tinker operations into the
backend and handles the compatibility details at the API boundary; it does not
reimplement the training or inference stack. The distributed execution path
comes from the same proven backend configurations used by verl, VeOmni, and
vLLM workloads.

Start a workload in minutes Copy Chapter

Install the server environment from the root of verl-recipe, then start an
actor-and-rollout configuration:

./install_verl.sh --recipe verl_tinker
cd verl_tinker
python -m verl_tinker.start \
  --config configs/quick_start/actor_rollout.yaml

Once GET /api/v1/healthz returns {"status":"ready"}, point a client at the
server:

export TINKER_BASE_URL=http://127.0.0.1:8000/
export TINKER_API_KEY=tml-verl-tinker-local

cd verl_tinker/client_examples
uv sync
uv run run_single_test.py \
  --test-name sft_tulu3

That is the whole handoff. For an existing SDK or Cookbook workload, setting
TINKER_BASE_URL is usually all that is required. The recipe issues familiar
Tinker calls, and the server forwards them to the distributed backend.

The bundled client examples
are intentionally lightweight wrappers around recipes from the
Tinker Cookbook.
Together they cover most official Cookbook recipe patterns, including SFT,
supervised distillation, GSM8K RL, SFT followed by RL, and single- or
multi-teacher on-policy distillation.

They are examples, not a required client layer. Because verl-tinker speaks
the Tinker API, you can point your own program built with the Tinker SDK at the
same endpoint and run your own data, training loop, and workflow.

Three quick-start configurations cover the common shapes:

• actor.yaml for SFT and optimizer-only workflows;
• actor_rollout.yaml when the client calls asample;
• actor_rollout_ref.yaml for RL that requests reference log probabilities for KL penalties.

What happens after the HTTP call? Copy Chapter

A request passes through three layers:

Tinker / Tinker Cookbook client
              |
              | Tinker-compatible HTTP
              v
Thin Tinker-compatible forwarding layer
              |
              | preset-backed server configuration
              v
verl / VeOmni training + vLLM rollout workers

Bring your teachers, too Copy Chapter

Distillation is where this bridge gets especially interesting. A Tinker
program can use one or more larger, frozen models as teachers, ask them to
generate or return top-K log probabilities, and use those signals to train a
smaller actor—all through the same familiar client API.

That means you can explore single-teacher or multi-teacher on-policy
distillation without building a separate serving stack for every model. Your
training loop decides how to use each teacher; verl-tinker runs the actor and
teachers on the GPU resources you assign to them.

The recipe includes two ready-to-adapt starting points:

• The single-teacher config
  pairs a Qwen3-1.7B actor with a Qwen3-30B-A3B teacher on eight GPUs.
• The multi-teacher config
  shows a Qwen3-8B actor learning from Qwen3-32B and Qwen3-235B-A22B teachers
  across two eight-GPU nodes.

Use these configs as templates: swap in your models, choose the parallelism
that fits your cluster, and let your Tinker code decide what each teacher
contributes to the learning signal.

Where the boundary is today Copy Chapter

For most workflows, verl-tinker lines up closely with Tinker. The main
difference is that a server currently manages one trainable model and one
rollout replica.

Because that rollout is shared, a sampling session created from the trainable
model is tied to a particular version of its weights. If the rollout is later
synchronized to newer weights—or otherwise diverges from the version that
session expects—the earlier sampling resource may no longer be available.
Teacher sampling and reference-log-probability requests are separate and are
not affected by this restriction.

You will not silently sample from the wrong weights. verl-tinker keeps track
of which model version each sampling resource belongs to and tells you when a
resource is no longer available.

Within that boundary, the division of labor stays simple: your Tinker code
owns the experiment; verl owns distributed execution. You keep the loop you
want, while choosing the GPUs, rollout engine, placement, offload, and
checkpoints underneath it.

If that sounds like the missing bridge in your workflow, start with the
verl_tinker recipe
and run one of the included client examples.

verl-tinker was developed by the ByteDance AML/Seed Team.

File: content/posts/2026-08-13-verl-speco-release/index.md Copy Chapter

title: "Faster Draft Model Training: verl-SpeCo 0.1.0 Adds Native DSpark Support and Unlocks Standalone Training"
date: 2026-08-13
authors:

• "PeiXiang Tan"
• "Shuang Li"
• "Shuai Yuan"
• "RunPing Chen"
  author_links:
  "PeiXiang Tan": "https://github.com/tpx818"
  "Shuang Li": "https://github.com/755651978"
  "Shuai Yuan": "https://github.com/crp0128"
  "RunPing Chen": "https://github.com/ys2025-AI"
  summary: "verl-SpeCo 0.1.0 adds native DSpark support and standalone draft model training, with reusable feature storage, seven algorithm backends, vLLM/SGLang integration, and GPU/Ascend NPU support."
  image: "verl-sepco.png"
  tags:
• DSpark
• Dflash
• Speculative Decoding
• Co-Train
  math: false
  toc: true

1. Overview Copy Chapter

verl-SpeCo 0.1.0 advances draft model training along three major fronts:

First, it integrates the complete DSpark training and speculative decoding pipeline, resolving the semantic mismatch between conventional token-wise training and block-wise prediction.

Second, it introduces a standalone draft model training workflow with three modes—online training, feature collection only, and offline training—so draft models can be trained independently of a full reinforcement learning job.

Third, it expands algorithm coverage to seven widely used draft model families: DSpark, EAGLE-1, EAGLE-2, EAGLE3, DFlash, Domino, and P-EAGLE. It also supports both vLLM and SGLang inference runtimes across GPU and Ascend NPU backends.

The goal of this release is to provide native training support for advanced draft algorithms such as DSpark, establish complete workflows for both online co-training and standalone training, and broaden support for speculative decoding algorithms. The sections below walk through the four key areas of the release.

2. DSpark: Native Training for Block-Wise Speculative Decoding Copy Chapter

2.1 Training Semantics Designed for DSpark's Block-Wise Predictions Copy Section

DSpark uses block-wise generation to overcome performance limitations in conventional draft models. verl-SpeCo 0.1.0 introduces a dedicated DSpark training backend. During training, it starts from an anchor token in the sequence, constructs a fixed-length draft block, and aligns the anchor, preceding tokens, target tokens, and valid positions with DSpark's semantics. This allows even the first position in each block to contribute to the training objective.

This purpose-built data layout eliminates the semantic mismatch between generic token-wise training and DSpark's block-wise generation. The anchor, preceding tokens within the block, and supervised positions seen during training now match those used at inference time, providing a sound foundation for improving block-wise draft model performance.

2.2 Markov Heads for Explicit Intra-Block Token Dependencies Copy Section

Unlike approaches that predict draft tokens solely from backbone hidden states, DSpark adds a Markov bias to the base logits, directly incorporating preceding tokens within the block into subsequent token predictions.

The verl-SpeCo training path supports two types of Markov Head: Vanilla and Gated. The Vanilla Head models dependencies between adjacent tokens through a low-rank token transition structure. The Gated Head additionally uses the hidden state at the current position to control the Markov signal dynamically. Markov bias works with full-vocabulary, restricted-CE, and sampled-CE computation paths. For restricted-CE and sampled-CE training, only the required output rows are selected while DSpark's transition modeling is preserved.

2.3 Joint Optimization with Cross-Entropy and Distribution Distillation Copy Section

DSpark training can combine target-token cross-entropy with distribution distillation. The latter reconstructs the full output distribution from the target model's final-layer hidden states and computes the L1 distance between the draft and target distributions. The two losses can be weighted independently and decayed exponentially by position within the block, placing greater emphasis on the early positions that matter most for longer acceptance sequences.

To support L1 distribution distillation, verl-SpeCo extends the rollout/actor feature collection pipeline. In addition to multi-layer context features, it captures the target model's final-layer hidden states. The training backend then aligns them precisely with the label position of each anchor. This enables DSpark to optimize both token prediction accuracy and distributional consistency. Compared with training on token labels alone, this objective is better aligned with the acceptance-length decisions made during speculative decoding and helps the draft output more closely match the target model.

For large vocabularies, restricted CE and sampled CE reduce LM Head computation. When L1 distillation is enabled, the training backend can still reconstruct the full probability distribution on demand. verl-SpeCo also provides fine-grained diagnostics for identifying data issues and convergence problems in block-wise draft model training. It can truncate invalid position sequences, filter non-finite losses such as NaN and Inf, report loss and accuracy by block position, and track Top-1/Top-5 accuracy, valid token counts, and the sampled vocabulary size used by CE.

Figure: DSpark training workflow

2.4 Dedicated GPU and NPU Optimizations Copy Section

DSpark builds on DFlash's block-wise context backbone. The training backend selects the attention implementation according to the device: CUDA uses a block mask with the Flex Attention path, while non-CUDA devices use an equivalent dense attention mask. This preserves sparse block computation on GPUs while providing an executable compatibility path for Ascend NPUs.

For inference, users set speculative_algorithm=DSPARK. On GPUs, this maps to vLLM's native DSpark method. On NPUs, integration follows the compatible vLLM and vLLM-Ascend versions specified in the project README.

Figure: DSpark experiment results on NPU

Figure: DSpark experiment results on GPU

3. Standalone Training: Decoupling Draft Model Training from the Main RL Job Copy Chapter

3.1 Three Training Modes Copy Section

In addition to online co-training, verl-SpeCo now supports standalone draft model training. Given a feature dataset produced by the target model, users can independently train any supported draft model, including EAGLE, DSpark, and DFlash. This makes it possible to build draft models for target models such as Qwen and Llama without running a complete RL job.

verl-SpeCo 0.1.0 divides draft model training into three modes: online, collect_only, and offline. The online mode preserves the existing co-training experience: it collects features in a PPO/Ray job, trains periodically, and can publish updated weights to the rollout engine. The collect_only mode writes collected features to the Feature Store without starting draft model training inside the RL job. The offline mode reads persisted features and runs draft model training as a separate job.

This separation decouples resource scheduling and draft model training from a single long-running pipeline. Feature generation and draft model experiments can run independently, and collected features can be reused for hyperparameter comparisons, architecture validation, and reproducible debugging. Dedicated training resources can also be provisioned outside the RL workflow.

3.2 Feature Store: Persistent Training Features Copy Section

verl-SpeCo provides a Feature Store based on sharded PyTorch files. It persists input IDs, loss masks, target hidden states, and auxiliary features in shards while maintaining a unified schema and shard manifest. The collection side supports step-based flushing and shard-capacity controls. The training side provides read-only loading, sample shuffling, cyclic iteration, and strict schema validation, helping prevent failures caused by missing fields, shape drift, or configuration mismatches.

The release also adds the verl-speco-inspect-features command, which samples and validates stored features before training begins. Its strict exit codes make it suitable for automated pipelines, allowing users to verify data quality before committing training resources.

3.3 Distributed Training Copy Section

verl-SpeCo adds the verl-speco-draft-train command and the draft_trainer.yaml top-level configuration. The launcher normalizes single-node multi-GPU and multi-node parameters, and automatically selects a single-node distributed torchrun configuration when appropriate. The offline training loop initializes the distributed environment, selects a Drafter Backend, reads from the Feature Store, performs gradient accumulation, saves checkpoints at configured intervals, and writes the final checkpoint.

Standalone training uses the same Drafter Backend interface as online co-training. The project currently includes training backends for EAGLE-1, EAGLE-2, EAGLE3, DFlash, DSpark, Domino, and P-EAGLE.

For observability, the training loop records loss, vloss, ploss, the current learning rate, and cumulative optimizer steps. Block-wise algorithms such as DFlash, DSpark, and Domino additionally expose CE/L1 loss, Top-1/Top-5 accuracy, valid token counts, sampled vocabulary size, loss and accuracy by block position, and timing for batch preparation, forward computation, loss reduction, backward propagation, and optimizer updates.

3.4 Getting Started Copy Section

verl-SpeCo includes the run_qwen3-8b_drafter_separate_training.sh example script to demonstrate the standalone workflow in two stages. In the first stage, rollout runs in collect_only mode and writes features to a specified directory. In the second stage, the standalone trainer runs in offline mode, reads from the same Feature Store, and trains the draft model. To get started, users only need to replace the target model, draft model, dataset, feature directory, and checkpoint directory.

Figure: Standalone training data loop

4. Broader Support Across Algorithms and Runtimes Copy Chapter

4.1 Seven Draft Model Training Backends Copy Section

In addition to DSpark, verl-SpeCo 0.1.0 supports EAGLE-1, EAGLE-2, EAGLE3, DFlash, Domino, and P-EAGLE. EAGLE-1 and EAGLE-2 reuse vLLM's native EAGLE draft method, with EAGLE-2 adding dynamic tree decoding. Domino is trained as a DFlash projector sub-mode and deployed through a DFlash runtime that supports the Domino projector.

4.2 vLLM, SGLang, GPU, and NPU Examples Copy Section

On the inference side, vLLM support covers EAGLE-1, EAGLE-2, EAGLE3, DFlash, and DSpark, while SGLang support covers EAGLE3 and DFlash. The project provides launch examples for both GPUs and NPUs. Graph modes, draft-model parameters, and version requirements remain explicit in the configuration, making each hardware setup easier to verify and reproduce.

Table: Training, standalone training, and rollout runtime support across seven algorithms

5. A Unified, Non-Intrusive Architecture Copy Chapter

Beyond the two headline capabilities, verl-SpeCo 0.1.0 establishes a foundation for continued expansion. It is compatible with verl release/v0.8.0 and ships as a separate package that provides layered Hydra configuration, training Workers, and rollout adapters without modifying the installed verl source code. The inference side supports vLLM and SGLang, with examples for GPUs and NPUs. The training side uses a unified FSDP backend and extends support for draft models through pluggable backends.

At runtime, the framework records drafter training time, weight publication time, and vLLM speculative decoding acceptance metrics, including mean_acceptance_length. This gives users visibility into both draft model learning and end-to-end rollout performance.

6. What's Next Copy Chapter

verl-SpeCo 0.1.0 addresses the challenge of integrating draft models reliably into reinforcement learning workflows. Native DSpark integration broadens support for block-wise speculative decoding, while standalone training provides more flexible ways to organize feature collection and model optimization resources.

Future work will focus on end-to-end training and runtime support for additional draft algorithms, multi-hardware validation for standalone training, performance optimization, stronger feature storage and data governance, and more systematic benchmarking and observability.

File: content/posts/2026-08-14-verl-on-amd-rocm/index.md Copy Chapter

title: "verl on AMD: production-ready RL post-training on ROCm"
date: 2026-08-14
authors:

• "Fuwei Yang"
• "Zhaodong Bing"
• "Mingjie Lu"
• "Xiaohong Kou"
• "Yuhan Yang"
• "Wei Cai"
• "Liz Li"
• "Yuankai Chen"
• "Yao Fu"
• "Dong Li"
• "Zhenyu Gu"
  summary: "RL post-training on AMD Instinct GPUs is here: a turnkey ROCm container, AITER-accelerated vLLM and SGLang rollout, and accuracy validated on both MI300 and MI350 series."
  tags:
• amd
• rocm
• performance
• vllm
• sglang
• megatron
• fsdp
  image: "cover.png"
  math: false
  toc: true

Reinforcement learning post-training on AMD Instinct GPUs is here — with a
turnkey container, AITER-accelerated vLLM and SGLang rollout, and accuracy
validated on both MI300 and MI350 series.

verl — the open-source implementation of
the RL controller — has quickly become one of the most widely adopted RL
post-training frameworks, powering PPO, GRPO and DAPO pipelines with a flexible
mix of rollout and training engines. Today, that same power runs on AMD Instinct
GPUs. Whether you're on MI300 or the latest MI350 series, you can stand up verl
on ROCm in minutes, scale it across your rollout and training engines, and trust
the numbers. Here's what makes AMD a first-class home for verl.

Zero-friction setup: one container, no dependency headaches Copy Chapter

Getting a modern RL stack running usually means fighting a maze of dependencies
— matching CUDA/ROCm versions, building attention kernels from source, and
praying the wheels line up. On AMD, that pain is gone.

AMD ships a turnkey ROCm container, verlai/verl:rocm7.14_torch2.12_release_0724,
built from docker/rocm/Dockerfile.rocm,
so you can go from zero to training without hand-assembling a single dependency.
The image bundles the entire runtime stack — ROCm, PyTorch, Triton, vLLM,
SGLang, AITER, TransformerEngine and Megatron-core — all pinned to a set of
versions verified to work together. Pull it, mount your data, and run: RL
post-training on AMD is a docker run away, so you can spend your time on
reward design and experiments rather than on environment setup.

• No dependency installation. Everything RL post-training needs is baked in
  — no extra source builds, no extra version detective work. There's an
  end-to-end AMD tutorial
  covering build, run, and example PPO/GRPO commands.
• Built for multiple architectures. The image targets gfx942 (MI300 series
  — MI300X / MI308X / MI325X) and gfx950 (MI350 series — MI350X /
  MI355X) out of the box.
• Customizable for your setup. Teams that want to tailor the image can do so
  — target a specific GPU architecture, pin component versions, and tune the
  build to their environment — all documented in the
  AMD support section of the verl README.

This is all backed by CI on real AMD hardware: end-to-end tests now run on a
community-hosted runner, so changes to the verl repository are validated on AMD
hardware before merge. Coverage is a work in progress and actively expanding
toward parity with the other platforms.

Two rollout backends — vLLM and SGLang, both AITER-accelerated Copy Chapter

Rollout is the key component of RL post-training, and on AMD you're not locked
into a single engine: verl supports both vLLM and SGLang on ROCm. Both are
accelerated by AITER, AMD's AI Tensor Engine
for ROCm kernel library — powering attention, RMSNorm / RoPE, mixture-of-experts
(MoE), and quantization — with everything fully overridable if you want to
experiment.

• vLLM — the default, battle-tested rollout engine; enable the AITER path
  with a few environment variables.
• SGLang — a first-class alternative that runs on ROCm with AITER kernels
  enabled by default.

Both engines work across both of verl's integration modes — colocated (rollout
and training share GPUs with fast weight transfer) and fully async (rollout and
training run concurrently for maximum GPU utilization, with live parameter sync
back to the rollout workers). Whichever engine and mode fit your workload,
AMD-accelerated rollout is ready to go.

Validated accuracy — MI300 and MI350 series, colocated and fully-async Copy Chapter

Enablement is only half the story. What ultimately matters is whether the model
you train on AMD is as good as the model you'd train anywhere else. It is — and
we've measured it.

Training accuracy has been validated on both AMD MI300 and MI350 series, across
both colocated and fully-async modes. RL post-training on Instinct GPUs converges
to the quality you expect, so you can move real workloads onto AMD with
confidence rather than crossing your fingers.

The MI350-series runs below were measured on MI350. MI350X and MI355X are the
same gfx950 silicon and differ in board power and cooling rather than in
numerics, so the accuracy results carry over between them unchanged — only
throughput tracks the power envelope.

Case study: Qwen3-8B GRPO with SGLang rollout on MI350 Copy Section

Start with the SGLang rollout backend — the first-class alternative to vLLM
introduced above — to show it delivers the same accuracy on AMD. Key run
parameters:

Note that "async" here refers to SGLang's async rollout server, which serves
generation requests concurrently within a rollout step. It is not verl's
fully-async execution mode: rollout and training still share the same 8 GPUs and
alternate, which is what "colocated" in the table means. The fully-async mode,
where rollout and training run on disjoint GPUs at the same time, is covered in
the next two case studies.

The training curves show textbook convergence:

{{< figure src="qwen3_8b.jpg" alt="Four panels over 200 training steps: GSM8K and MATH validation accuracy, mean training reward, rollout-to-actor KL, and wall-clock time per step" caption="Qwen3-8B GRPO on GSM8K and MATH. SGLang async rollout server with AITER, FSDP trainer, colocated on 8xMI350." >}}

1. Strong accuracy on both benchmarks. GSM8K validation accuracy climbs from
   0.82 to ~0.955 (peak 0.956) within the first ~30 steps and holds there, while
   the harder MATH benchmark rises steadily from 0.37 to ~0.835 (peak 0.839).
   Mean training reward tracks them in lockstep, from 0.62 to ~0.90 — no
   reward-hacking, no collapse. An 8B model reaching ~95% on GSM8K and ~84% on
   MATH is exactly the quality bar you'd expect from a well-behaved GRPO run.
2. SGLang and the FSDP trainer stay numerically locked. The rollout↔actor
   mismatch is negligible: actor/ppo_kl hovers around 1e-4 during the entire
   run. In other words, the tokens that the AITER-accelerated SGLang samples
   during rollout are assigned essentially identical probabilities when the FSDP
   actor recomputes them.

Case study: Qwen2.5-Math-7B DAPO, fully-async on MI300 Copy Section

We reproduced verl's fully-async experiments on AMD MI300: DAPO post-training of
Qwen2.5-Math-7B on the MATH task. Key parameters are defined as follows:

The training curves tell the story:

{{< figure src="qwen2.5_7b.jpg" alt="Four panels over 400 logged steps: MATH validation accuracy, mean training reward, wall-clock time per step, and rollout-to-actor KL" caption="Qwen2.5-Math-7B DAPO on MATH. vLLM with AITER, FSDP2 trainer, fully-async on 32 MI300 GPUs (16 rollout + 16 training)." >}}

1. Convergence is clean, and the final score matches the reference.
   Validation accuracy climbs from 0.13 to a peak of 0.34 and holds around 0.32,
   while mean training reward rises from −0.98 to +0.17 in lockstep — no
   reward-hacking, no collapse. Critically, these numbers land right on top of
   verl's published reference for the same configuration
   (staleness_threshold = 0.5, partial rollout): the upstream
   fully-async-policy results
   report max accuracy 0.3302 for the same 16+16 GPU allocation on H20, while on
   AMD we reproduce the result with the same training settings and measure 0.34.
2. The KL curve confirms stable off-policy training. Rollout↔actor KL
   (actor/ppo_kl) sits at ~0.0007 for the first ~100 logged steps, then rises
   and settles onto a controlled 0.15–0.25 plateau once staleness kicks in. The
   run is stable end to end, with no explosive divergence despite training on
   deliberately stale samples.
3. Fully-async is dramatically faster per training step. Against a colocated
   DAPO run of the same model on the same 32 GPUs, we compare wall-clock time
   per training step over the first 64 steps:

A training step of fully-async is ~1.8× faster than the colocated baseline. In
this experiment we verified on the AMD platform that fully-async mode achieves
the expected accuracy and performance speedup that were originally reported on
H20.

{{< note type="info" title="One honest caveat" >}}
In the step-time panel above, per-step time is flat at ~190–200 s for the first
~260 steps, then climbs into the 600–1,200 s range toward the end of training. This isn't a regression or
instability — it's the expected response-length growth of reasoning RL: as model
capacity saturates, the model only learns to increase the response length
without improving accuracy. We observe its average response roughly double
(from ~890 to ~1,850 tokens) during the late stage of the experiment. The same
pattern shows up in the reported H20 experiment data.
{{< /note >}}

Case study: Qwen3.5-35B-A3B MoE on MI350 Copy Section

Now a more complex case: GRPO post-training of Qwen3.5-35B-A3B on the
geometry3k reasoning task on MI350, with the Megatron backend, validated in both
colocated and fully-async modes.

Both modes deliver clean, stable convergence:

{{< figure src="qwen3.5_35b_a3b.jpg" alt="Four panels comparing colocated and fully-async over 150 training steps: geometry3k validation accuracy, mean training reward, rollout-to-actor KL, and per-GPU throughput" caption="Qwen3.5-35B-A3B GRPO on geometry3k. vLLM with AITER, Megatron backend, 8xMI350, colocated versus fully-async." >}}

• Validation accuracy climbs steadily. Both modes rise from a ~0.37–0.48
  start to a peak of ~0.77–0.78 within the first ~75 steps and hold there.
• Training reward rises in lockstep, from ~0.34 to ~0.82 in both modes,
  tracking the validation gains rather than reward-hacking.
• Rollout↔actor KL stays tiny — ≈0.001 for fully-async and ≈0 for colocated
  — meaning the AITER-accelerated vLLM rollout and the Megatron training policy
  stay numerically almost identical.
• Fully-async sustains ~310 tok/s/GPU versus colocated's ~216 — roughly 1.4×
  higher per-GPU throughput. That's exactly the payoff of decoupling.

So a 35B-parameter MoE model trained end-to-end with GRPO on AMD MI350 converges
smoothly to strong accuracy in both colocated and fully-async modes.

The bottom line Copy Chapter

RL post-training on AMD Instinct is production-ready:

1. Effortless setup — a turnkey ROCm container with the full stack baked in,
   so there are no dependencies to wrangle, and end-to-end CI on real MI300
   hardware keeping it solid (with coverage still expanding).
2. Two rollout backends — both vLLM and SGLang, AITER-accelerated out of the
   box, in both colocated and fully-async modes.
3. Validated accuracy — confirmed on MI300 and MI350 series, in both execution
   modes, with AITER-accelerated rollout delivering speed without sacrificing
   convergence.

Looking ahead, AMD will continue working closely with the verl community to
support emerging models and capabilities, including architectures such as
DeepSeek-V4, agentic RL workloads, and integration with platforms such as
VeOmni. We will also improve rollout efficiency through technologies such as
speculative decoding and FP4 inference.

Our goal is simple: make the latest verl models and features work out of the box
on AMD Instinct GPUs, with strong performance, validated accuracy, and a
reliable experience for customers.

File: content/posts/2026-08-17-verl-omni-v0-2-0/index.md Copy Chapter

title: "VeRL-Omni v0.2.0: Faster Diffusion RL and Stable Omni Training"
date: 2026-08-17
authors:

• "Yongxiang Huang"
• "Didan Deng"
• "Kawai Cheung"
• "Nguyen Long"
• "Jingan Zhou"
• "Yingshu Chen"
• "Ziyang Hong"
  summary: "A release focused on higher-throughput diffusion rollout, reusable omni adapters, and broader recipe coverage."
  image: "cover.png"
  tags:
• release
• performance
• recipes
• training
• rollout
  math: false
  toc: true

VeRL-Omni v0.2.0 establishes a stronger foundation for production-grade
omni-modal reinforcement learning. This release improves the training stack
across rollout performance, model integration, reward support, hardware
coverage, and documentation, with two changes carrying the most impact:

• Faster diffusion RL, centered on higher-throughput Qwen-Image FlowGRPO rollout
  and V1 trainer support.
• Stable omni training, built around the omni V1 trainer, reusable model
  adapters, FSDP2, and vLLM-Omni rollout.

{{< figure src="verl_omni_v0_2_0_blog_overview.png" alt="VeRL-Omni v0.2.0 release overview" caption="VeRL-Omni v0.2.0 release overview" width="100%" >}}

1. Faster Diffusion RL Copy Chapter

Diffusion RL is expensive, but not in the same way as autoregressive
language-model RL. A single rollout carries many denoising steps, large latent
tensors, prompt embeddings, optional classifier-free guidance, reward-model
scoring, old-log-prob recomputation, and policy-weight synchronization. For
Qwen-Image FlowGRPO, there is no single villain in the profile. Step time is
shaped by rollout generation, old-log-prob computation, reward scoring, actor
update, and LoRA weight sync together.

Key Features Copy Section

The faster diffusion RL work has two main features.

• Request-level batching leads the rollout side. For supported diffusion
  adapters, it becomes the default batching path. Instead of sending diffusion
  generations through a serial loop, the engine now acts more like a traffic
  controller, with explicit concurrency knobs for scheduling rollout work.

• The trainer path matters just as much. Diffusion now has a V1 trainer path,
  bringing diffusion RL closer to the modern trainer architecture used elsewhere
  in VeRL-Omni and laying the groundwork for decoupled rollout and training
  execution.

Faster rollout only matters if the generated trajectories and log-probs still
describe the same policy. This release fixes several correctness-sensitive
areas: request-batched diffusion log-probs, async rollout semantics, rank-local
LoRA weight-update routes, and the hooks used by optional rollout-correction
recipes.

Newly Support Copy Section

The rollout batching guide
explains both diffusion batching modes, how to enable them, and when to choose
each mode. Current faster diffusion RL support is organized around these
recipes:

A full diffusion post training support table in VeRL-Omni is available at
README.md.

Recipe and Benchmark Copy Section

The Qwen-Image LoRA OCR recipe is a good place to see the change. In the v0.1
line, rollout was the core bottleneck: each request effectively ran as serial
B≈1 DiT forwards, with 10 denoising steps and True-CFG doubling each step
into two forwards. GPU utilization hovered around 80%, not because the model
was small, but because the engine could not keep enough diffusion work packed
together.

In v0.2.0, request-level packing changes that shape. Multiple complete
requests are packed into one transformer forward, GPU utilization rises to about
100%, and isolated generation time drops from 226s to 108s, a 52%
reduction. The same story shows up in per-image generation latency, which falls
with the packed rollout path. Reference runs:
Qwen-Image OCR LoRA v0.1
and
Qwen-Image OCR LoRA v0.2.

In the charts below, the blue curve is v0.1 and the green curve is v0.2.

{{< figure src="qwen-image-gpu-utilization.svg" alt="Qwen-Image FlowGRPO GPU utilization" caption="GPU utilization rises after request-level packing. Blue: v0.1; green: v0.2." >}}

{{< figure src="qwen-image-timing-gen.svg" alt="Qwen-Image FlowGRPO generation time" caption="Generation time drops from the v0.1 path to the v0.2 path. Blue: v0.1; green: v0.2." >}}

{{< figure src="qwen-image-timing-step.svg" alt="Qwen-Image FlowGRPO step time" caption="Step time follows the same trend. Blue: v0.1; green: v0.2." >}}

The production-style Qwen-Image FlowGRPO recipes enable request-level batching by
default. The main entry points are run_qwen_image_ocr_lora.sh for the
baseline OCR reward setup. The request-level rollout batching is enabled by default:

actor_rollout_ref.rollout.step_execution=false
++actor_rollout_ref.rollout.engine_kwargs.vllm_omni.max_num_seqs=32

For Qwen-Image LoRA with True-CFG at 512 px, a practical tuning range is
max_num_seqs=8 to 32; larger values can run into HBM pressure. SD3.5 has a
lighter request-level memory shape and can use max_num_seqs=256.

The recipe-level step-time numbers line up with that story: the baseline
Qwen-Image FlowGRPO LoRA run is about 420s per step on 4 × H800, while the
async reward variant reaches about 360s per step on 5 GPUs.

2. Stable Omni Training Copy Chapter

The other half of the release is stable omni training. Omni models are not just
bigger language models; they are small systems with processors, modality-specific
towers, trainable stages, and rollout-time behavior that has to stay aligned
with the actor. v0.2.0 moves the project from model-specific integrations
toward a reusable omni training stack, so multimodal autoregressive training
fits more naturally into VeRL-Omni's trainer, adapter, rollout, and recipe
structure.

Key Features Copy Section

Here, the release pulls on two levers.

One lever is the verl V1 trainer architecture. Omni recipes get clearer
worker orchestration, standard configuration overrides, and better alignment
with vLLM-Omni rollout.

The other is the reusable omni model adapter layer. Instead of wiring each
architecture as a one-off path, the trainer can rely on a shared interface for
model setup, processor setup, trainable-stage selection, FSDP preparation, and
rollout alignment.

The repository-level call flow is roughly:

{{< figure src="omni-ppo-adapter-flow.svg" alt="Omni PPO trainer and OmniModelBase adapter call flow" caption="Omni PPO trainer and OmniModelBase adapter call flow." >}}

The module boundary is intentionally narrow. main_omni.py only decides that an
online omni job should enter the verl PPO V1 path. The PPO trainer then owns the
generic RL loop: rollout scheduling, advantage computation, and policy updates.
When the actor model is built, the FSDP omni engine loads the Hugging Face model
and asks OmniModelBase to resolve the adapter for the configured architecture
and stage. That adapter is where model-specific work lives. For Qwen3-Omni
thinker training, Qwen3OmniThinkerAdapter strips inactive modules, redirects
forward to the thinker component, and prepares the processor and rollout
alignment hooks before control returns to the PPO loop.

Newly Support Copy Section

The current Qwen3-Omni adapter supports thinker-only training by redirecting
training to the target component, stripping unused modules such as Talker and
codec-related components, and working with FSDP/FSDP2 wrapping. Current stable
omni training support is organized around these recipes:

A full omni post training support table in VeRL-Omni is available at
README.md.

Recipe and Benchmark Copy Section

The best single recipe to highlight is MMK12. It exercises the new stable
Qwen3-Omni path with real multimodal input: image plus text prompt, text answer,
GSPO optimization, FSDP actor training, and vLLM-Omni rollout.

MMK12 anchor recipe. run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh trains
Qwen3-Omni on K12 visual math reasoning (image -> text) with GSPO, LoRA rank
32, and colocated actor-rollout workers on 4 × H800 80GB. The rollout shape is
128 prompts × 16 responses, or 2048 samples per rollout. After training, the
run reaches 0.833 validation reward, 0.998 actor-rollout Pearson correlation,
and about 59 GB GPU memory usage. See some training results in the reference
run: MMK12 (wandb).

{{< figure src="mmk12_training_rewards.svg" alt="MMK12 training rewards mean scores" caption="MMK12 training rewards mean scores." >}}

{{< figure src="mmk12_val_rewards.svg" alt="MMK12 validation rewards mean scores" caption="MMK12 validation rewards mean scores." >}}

The MMK12 data pipeline converts raw MMK12 parquet shards into the verl RL
parquet layout. Each row carries the image bytes inline and uses a prompt format
that asks the model to produce a structured answer. The reward combines
math_verify accuracy with a progressive format reward on the
<answer>...\boxed{}...</answer> template.

To run the recipe:

python examples/gspo_trainer/data_process/mmk12.py \
    --local_dataset_path /path/to/mmk12/ \
    --local_save_dir ~/data/mmk12

TRAIN_FILE=$HOME/data/mmk12/train.parquet \
VAL_FILE=$HOME/data/mmk12/test.parquet \
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh

This anchors the v0.2.0 stability story: Qwen3-Omni training is no longer
just a model-specific launch path. It is a V1 trainer recipe with a reusable
omni adapter, multimodal data handling, actor-rollout consistency metrics, and
a documented image-to-text benchmark.

Model and Algorithm Extensions Copy Chapter

The release also expands the broader VeRL-Omni model and algorithm surface:

Outside the model-algorithm matrix, v0.2.0 also adds Ascend NPU Dockerfiles
and install guidance.

Future Plan Copy Chapter

• Optimize omni-modal models via fully async training.
• Extend new models and algorithms, such as MiniMax-H3, MiniCPM-o models, and
  OPD/M-OPD trainers.
• Make video diffusion model training more efficient via batching, TQ, and the
  V1 trainer.
• Harden diffusion and omni-modal rollout code for async training.
• Support agentic RL with multi-stage and multi-turn generation.

File: content/posts/_index.md Copy Chapter

title: "All posts" summary: "All posts." Copy Chapter

Everything published on this blog, newest first. Posts are written by verl
maintainers and contributors; if you have shipped something worth explaining,
here is how to write it up.

File: content/contribute.md Copy Chapter

title: "Write for the verl blog" summary: "How to propose, write and ship a post about work you have done in verl." Copy Chapter

If you have landed something in verl and want to explain how it works,
this blog is the place to write it up.

{{< note type="tip" title="Talk to us before you write" >}}
Open an issue with your idea before you start drafting.
A maintainer can tell you early whether the topic fits and whether
somebody else is already writing it, which is a lot cheaper than finding out
once the draft is done.
{{< /note >}}

What we publish Copy Chapter

Posts that help someone else understand verl better. In practice that tends to
be one of:

• Feature deep dives. You landed something non-trivial and want to explain
  the design, the trade-offs you rejected, and how to use it.
• Performance work. A change with numbers attached. Say what hardware, what
  model, what batch shape, and what you measured against.
• Recipes and case studies. An end-to-end account of training something
  real, including the parts that did not work at first.
• Release notes with substance. What actually changed and why it matters.
• … and anything else worth explaining. Treat the list as a starting point
  rather than a set of boxes to fit into. If you are unsure whether an idea
  fits, open an issue and ask.

One thing this blog is not is a substitute for the
documentation. Docs describe how the
current version behaves and get updated as it changes; a post captures the
reasoning behind a change at the time it was made. Reference material belongs in
the docs — and both are good places to contribute.

The short version Copy Chapter

1. Open an issue on the blog repository.
2. Fork the repository and create a directory under content/posts/, named
   YYYY-MM-DD-short-slug.
3. Write index.md inside it. Put images in the same directory.
4. Open a pull request. CI checks the front matter and builds the site.
5. A maintainer reviews for accuracy and clarity. Once it is merged, the site
   redeploys automatically and your post is live within a couple of minutes.

Writing the post Copy Chapter

Every post is a directory, not a loose file, so that its images travel with it:

content/posts/2026-08-04-your-post-slug/
├── index.md
├── cover.png
└── throughput-comparison.png

The front matter at the top of index.md looks like this:

---
title: "Async rollout in verl"
date: 2026-08-04
authors:
  - "Jane Doe"
summary: "Async rollout overlaps generation with training and cuts idle GPU time on long-horizon tasks."
image: "cover.png"
tags:
  - performance
math: false
toc: true
---

title, date, authors and summary are required; CI will reject the pull
request without them. A few notes on the ones that trip people up:

• summary must be 240 characters or fewer and cannot contain line breaks.
  It is the card blurb, the meta description and the social card text all at
  once. Write what the post concludes, not what it is "about".
• authors is a list. Use real names, GitHub handles, or a team name —
  whatever you want readers to see. Each value gets an archive page.
• image is a filename relative to the post directory. Hugo resizes it and
  converts it to WebP at build time, so commit the original rather than a
  hand-shrunk copy. Aim for something around 1200×630 for the cover.
• date in the future keeps the post unpublished until that date. This is
  the mechanism to use when a post has to land with a release.

If you would rather not write the front matter by hand, scaffold it:

hugo new content posts/2026-08-04-your-post-slug/index.md

Things the theme gives you Copy Section

Fenced code blocks are syntax highlighted. Line numbers are off by default,
since they get in the way of copying shell commands — add {linenos=true} after
the language when you want to refer to a specific line. Plain markdown images
are turned into captioned figures automatically, with the alt text becoming the
caption. For anything more specific there are two shortcodes:

{{</* figure src="throughput.png" alt="Throughput by batch size" caption="Decode throughput, 8×H100, Qwen3-8B." width="80%" */>}}

{{</* note type="warning" title="Version note" */>}}
This API changed in v0.8. On earlier versions, use `foo` instead.
{{</* /note */>}}

Set math: true in the front matter to load KaTeX, then write $$...$$ for
display equations and \(...\) inline.

House style Copy Section

We are not precious about this, but a few things make review faster:

• Lead with the outcome. State the result in the first paragraph. Save the
  background for later — readers who need it will keep reading.
• Numbers need context. Hardware, model, sequence length, batch size,
  baseline. A speedup with no baseline is not a result.
• Link to the code. Pull requests, issues and file paths let readers verify
  what you are describing and go deeper.
• Write for someone who knows RL but not your subsystem. Expand an acronym
  the first time you use it.
• English is the default.

Previewing your work Copy Chapter

You need Hugo v0.153.0 or newer. It is a
single binary with no runtime dependencies:

git clone https://github.com/verl-project/verl-project.github.io.git
cd verl-project.github.io
hugo server --buildDrafts

The site is then at http://localhost:1313 and reloads as you save. Drafts and
future-dated posts only appear with --buildDrafts and --buildFuture.

Copyright Copy Chapter

You keep the copyright in your post. Opening a pull request grants verl-project
the right to publish it here under your byline, and that is all it grants: you
stay free to put the same piece on your own site or anywhere else. The post
remains attributed to you indefinitely.