## 1. Project Overview & Quickstart (verl-project/verl) ## File: docker/verl0.4-cu124-torch2.6-fa2.7.4/README.md # verl image with verl v0.4.x ## Important packages version ```txt cuda==12.4 cudnn==9.8.0 torch==2.6.0 flash_attn=2.7.4 sglang==0.4.6.post5 vllm==0.8.5.post1 nvidia-cudnn-cu12==9.8.0.87 transformer_engine==2.3 megatron.core==core_v0.12.2 # Preview transformer_engine==2.5 megatron.core==core_r0.13.0 ``` ## Target - Base image: - `verlai/verl:base-verl0.4-cu124-cudnn9.8-torch2.6-fa2.7.4` - App image: - `verlai/verl:app-verl0.4-sglang0.4.6.post5-vllm0.8.5-mcore0.12.2-te2.2`: SGLang requires vLLM in 0.4.6.post5 version, vLLM can have some package conflicts with SGLang - `verlai/verl:app-verl0.4-sglang0.4.6.post5-vllm0.8.5-mcore0.12.2-te2.2-deepep`: Built with deepep - `verlai/verl:app-verl0.4-vllm0.8.5-mcore0.12.2-te2.2` - `verlai/verl:app-verl0.4-vllm0.8.5-mcore0.12.2-te2.2-deepep`: Built with deepep - Preview image: - `verlai/verl:app-verl0.4-sglang0.4.6.post5-vllm0.8.5-mcore0.13.0-te2.2-preview` - `verlai/verl:app-verl0.4-vllm0.8.5-mcore0.13.0-te2.2-preview` --- ## File: docker/verl0.5-cu126-torch2.7.1-fa2.8.0/README.md # verl image with verl v0.5 ## Important packages version ```txt cuda==12.6 cudnn==9.8.0 torch==2.7.1 flash_attn=2.8.0 ## sglang==0.4.8 vllm==0.8.5.post1 nvidia-cudnn-cu12==9.8.0.87 transformer_engine==2.3 megatron.core==core_v0.12.2 # Preview transformer_engine==2.5 megatron.core==core_r0.13.0 ``` ## Target - Base image: - `verlai/verl:base-verl0.5-cu126-cudnn9.8-torch2.7.1-fa2.8.0`: We offer a base image with deep ep built in - App image: - `verlai/verl:app-verl0.5-sglang0.4.9-mcore0.12.2` - `verlai/verl:app-verl0.5-sglang0.4.9-mcore0.13.0-preview` - vllm temporarily not support latest version --- ## File: docker/verl0.5-cu126-torch2.7-fa2.7.4/README.md # verl image with verl v0.5 ## Important packages version ```txt cuda==12.6 cudnn==9.8.0 torch==2.7.1 flash_attn=2.7.4.post1 sglang==0.4.9.post6 vllm==0.8.5.post1 nvidia-cudnn-cu12==9.8.0.87 transformer_engine==2.3 megatron.core==core_v0.12.2 # Preview transformer_engine==2.5 megatron.core==core_r0.13.0 ``` ## Target - Base image: - `verlai/verl:base-verl0.5-cu126-cudnn9.8-torch2.7.1-fa2.7.4`: We offer a base image with deep ep built in, for vllm/sglang - App image: - `verlai/verl:app-verl0.5-transformers4.55.4-vllm0.10.0-mcore0.13.0-te2.2` - `verlai/verl:app-verl0.5-transformers4.55.4-sglang0.4.10.post2-mcore0.13.0-te2.2` - `iseekyan/verl:app-verl0.5-transformers4.55.4-vllm0.10.0-mcore0.15.0-te2.7` --- ## File: docker/verl0.5-preview-cu128-torch2.7.1-fa2.8.0/README.md # verl image with verl v0.5 ## Important packages version ```txt cuda==12.8 cudnn==9.8.0 torch==2.7.1 flash_attn=2.8.0 ## sglang==0.4.8 transformer_engine==2.5 megatron.core==core_r0.13.0 nvidia-cudnn-cu12==9.8.0.87 ``` ## Target - Base image: - `verlai/verl:base-verl0.5-preview-cu128-cudnn9.8-torch2.7.1-fa2.8.0`: We offer a base image with flash infer 0.2.6.post1 built in - App image: - `verlai/verl:app-verl0.5-preview-sglang0.4.8-mcore0.13.0-preview` - vllm temporarily not support latest version ## !!!Notice!!! - pyext is lack of maintainace and cannot work with python 3.12, consider using replacement and deprecating this package. --- ## File: docker/README.md # Dockerfiles of verl We provide pre-built Docker images for quick setup. And from this version, we utilize a new image release hierarchy for productivity and stability. Start from v0.6.0, we use vllm and sglang release image as our base image. Start from v0.7.0, since vllm/vllm-openai:v0.12.0 is a minimal image without some essential libraries, we use nvidia/cuda:12.9.1-devel-ubuntu22.04 as our base image for vllm. ## Base Image - vLLM: https://hub.docker.com/r/nvidia/cuda - SGLang: https://hub.docker.com/r/lmsysorg/sglang ## Application Image Upon base image, the following packages are added: - flash_attn - Megatron-LM - Apex - TransformerEngine - DeepEP Latest docker file: - [Dockerfile.stable.vllm](https://github.com/verl-project/verl/blob/main/docker/Dockerfile.stable.vllm) - [Dockerfile.stable.sglang](https://github.com/verl-project/verl/blob/main/docker/Dockerfile.stable.sglang) All pre-built images are available in dockerhub: https://hub.docker.com/r/verlai/verl. For example, `verlai/verl:sgl0512.latest`, `verlai/verl:vllm024.latest`. You can find the latest images used for development and ci in our github workflows: - [.github/workflows/vllm.yml](https://github.com/verl-project/verl/blob/main/.github/workflows/vllm.yml) - [.github/workflows/sgl.yml](https://github.com/verl-project/verl/blob/main/.github/workflows/sgl.yml) ## Building Locally To build an image from source: ```sh docker build -f docker/Dockerfile.stable.vllm -t verl:vllm-local . ``` For users in China who need an apt mirror to speed up package downloads, pass `APT_MIRROR`: ```sh docker build -f docker/Dockerfile.stable.vllm \ --build-arg APT_MIRROR=https://mirrors.tuna.tsinghua.edu.cn \ -t verl:vllm-local . ``` ### GB200 / aarch64 Pre-built images for GB200 (aarch64) are not yet published. Users should build locally on an aarch64 machine. Pre-built images will be added once available. This applies to the uv image below too — it supports aarch64 from the same Dockerfile and the same `uv.lock`. ```sh docker build -f docker/Dockerfile.stable.vllm -t verl:vllm-arm64 . docker build -f docker/Dockerfile.uv.cu130 -t verl:uv-cu130-arm64 . # uv image ``` Check `uname -m` first: on an `aarch64` host the plain `docker build` above is all you need, and everything below is irrelevant. From an `x86_64` host you have two options. **Remote arm64 machine, built natively (preferred).** Attach it as a buildx node and drive the build from x86_64 — no emulation. Replace `user@arm-host` with a real, resolvable arm64 host: this is a placeholder, and buildx reports a bare `Could not resolve hostname` if it is pasted verbatim. The node needs Docker 18.09+, key-based SSH (buildx runs `ssh ... docker system dial-stdio` non-interactively, so a password prompt fails), and an SSH user in the `docker` group. Verify the connection before creating the builder: ```sh ssh user@arm-host docker version # must succeed without a password docker buildx create --name verl-arm --driver docker-container \ --platform linux/arm64 ssh://user@arm-host --use docker buildx build --platform linux/arm64 -f docker/Dockerfile.uv.cu130 \ -t verl:uv-cu130-arm64 --load . ``` Note that `--platform` on `buildx create` only declares what the node advertises; it does not make an x86_64 node build arm64 natively. **No arm64 machine at all: QEMU emulation.** Works, but slow — the apt install, the GDRCopy build and the `prefetch` stage's `megatron-core` source build all run emulated: ```sh docker run --privileged --rm tonistiigi/binfmt --install arm64 docker buildx create --name verl-qemu --driver docker-container --use docker buildx build --platform linux/arm64 -f docker/Dockerfile.uv.cu130 \ -t verl:uv-cu130-arm64 --load . ``` ### Building behind an egress proxy `--build-arg http_proxy=...` reaches the `RUN` steps only. The base image in `FROM` is resolved by the BuildKit daemon itself — in the `buildx_buildkit_*` container, before any build arg applies — so a proxy passed that way cannot fix `failed to resolve source metadata for docker.io/nvidia/cuda:...`. Put the proxy on the **builder**, and keep the build args for the `RUN` steps: ```sh docker buildx rm verl-qemu 2>/dev/null || true docker buildx create --name verl-qemu --driver docker-container --use \ --driver-opt env.http_proxy=http://proxy.example.com:8118 \ --driver-opt env.https_proxy=http://proxy.example.com:8118 \ --driver-opt env.no_proxy=localhost,127.0.0.1 docker buildx build --platform linux/arm64 -f docker/Dockerfile.uv.cu130 \ --build-arg http_proxy=http://proxy.example.com:8118 \ --build-arg https_proxy=http://proxy.example.com:8118 \ -t verl:uv-cu130-arm64 --load . ``` If Docker Hub stays unreachable, skip it entirely by pointing the base at an internal mirror (the mirror must carry the arm64 manifest for an arm64 build): ```sh docker buildx build --platform linux/arm64 -f docker/Dockerfile.uv.cu130 \ --build-arg CUDA_BASE_IMAGE=/nvidia/cuda \ -t verl:uv-cu130-arm64 --load . ``` ### One tag for both arches Both arches can and should share a single tag, published as a multi-arch manifest list — `docker pull verlai/verl:uv.cu130` then resolves to whichever arch the puller is on, and `FROM verlai/verl:uv.cu130` in a downstream image resolves per build platform. This is how `nvidia/cuda:13.0.2-devel-ubuntu24.04` itself works, and what lets one Dockerfile and one job spec cover both. If a single builder can reach both platforms (e.g. a native x86_64 node plus a native arm64 node in the same builder), one command does it: ```sh docker buildx build --platform linux/amd64,linux/arm64 \ -f docker/Dockerfile.uv.cu130 -t verlai/verl:uv.cu130 --push . ``` More often the two builds happen on different machines at different times. Then push arch-suffixed tags and join them afterwards — the join is metadata only, so it is instant and re-runnable: ```sh # on each host, natively: docker buildx build -f docker/Dockerfile.uv.cu130 -t verlai/verl:uv.cu130-amd64 --push . docker buildx build -f docker/Dockerfile.uv.cu130 -t verlai/verl:uv.cu130-arm64 --push . # then, from anywhere: docker buildx imagetools create -t verlai/verl:uv.cu130 \ verlai/verl:uv.cu130-amd64 verlai/verl:uv.cu130-arm64 docker buildx imagetools inspect verlai/verl:uv.cu130 # verify both platforms ``` Three things to know: - A multi-platform build **must** `--push`; `--load` cannot put an index into the local docker image store (load one arch at a time instead). - Add `--provenance=false` if your registry or runtime trips over the extra `unknown/unknown` attestation entries buildx puts in the index by default. - Pin by **tag**, not digest, to keep the multi-arch behaviour — a digest names one specific manifest, so `FROM ...@sha256:...` pins you to a single arch. ## uv image (`Dockerfile.uv.cu130`) `docker/Dockerfile.uv.cu130` builds one image around verl's universal `uv.lock` (GPU: CUDA 13.0 / torch 2.11 — vllm, sglang, fsdp, megatron — plus the GPU-free `cpu` slice), on x86_64 and aarch64 alike. The build **bakes the full uv package cache for every backend** into the image (the `prefetch` stage); it does *not* bake a fixed `.venv`. Build with BuildKit: ```sh DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.uv.cu130 -t verl:uv-cu130 . ``` You pick the backend combination at **run** time (not build time), and it must be conflict-free (see `[tool.uv].conflicts`). There is no install step: the container starts in a shell, and the first `uv run --extra ...` installs the requested extras into `/workspace/verl/.venv` from the baked cache (fast / offline) and runs from it — that venv is already on `PATH`: ```sh docker run --rm -it --gpus all verl:uv-cu130 bash # then, inside the container: bash examples/grpo_trainer/run_qwen3_8b_fsdp.sh # ... or spell the command out: uv run --frozen --all-packages --extra sglang --extra megatron \ python3 -m verl.trainer.main_ppo ... ``` Optional named stages: `--target=prefetch` builds just the baked cache (every backend, no source), and `--target=lock` regenerates `uv.lock`. A companion `docker/Dockerfile.uv.cu129` builds the cu12.9 / torch-2.9.1 backends (veomni, nemoautomodel) the same way; trtllm stays deferred. For the full story — the launch flow, the baked-cache mechanics, and re-locking — see **"Install with uv"** in [`docs/start/install.rst`](../docs/start/install.rst). ## Installation from Docker After pulling the desired Docker image and installing desired inference and training frameworks, you can run it with the following steps: 1. Launch the desired Docker image and attach into it: ```sh docker create --runtime=nvidia --gpus all --net=host --shm-size="10g" --cap-add=SYS_ADMIN -v .:/workspace/verl --name verl sleep infinity docker start verl docker exec -it verl bash ``` 2. If you use the images provided, you only need to install verl itself without dependencies: ```sh # install the nightly version (recommended) git clone https://github.com/verl-project/verl && cd verl pip3 install --no-deps -e . ``` [Optional] If you hope to switch between different frameworks, you can install verl with the following command: ```sh # install the nightly version (recommended) git clone https://github.com/verl-project/verl && cd verl pip3 install -e .[vllm] pip3 install -e .[sglang] ``` ## Release History - 2026/07/21: update vllm stable image to vllm==0.24.0 (torch==2.11.0, CUDA 13.0.2, Ubuntu 24.04); update sglang stable image to sglang==0.5.12 - 2026/03/10: update vllm stable image to vllm==0.17.0; update sglang stable image to sglang==0.5.9 - 2026/01/17: update vllm stable image to torch==2.9.1, cudnn==9.16, deepep==1.2.1 - 2025/12/23: update vllm stable image to vllm==0.12.0; update sglang stable image to sglang==0.5.6 - 2025/11/18: update vllm stable image to vllm==0.11.1; update sglang stable image to sglang==0.5.5 --- ## File: docs/blog/v0.7.md # verl 0.7 release blog **Author:** verl team Last updated: 01/03/2026. ## Overview verl adopts a Hybrid-Controller architecture (also known as HybridFlow). Sharing design principles with asynchronous sharded dataflow systems like Google Pathways, verl models Reinforcement Learning (RL) algorithms, such as PPO, GRPO, DAPO, and others, as a multi-stage, multi-model and parallelizable dataflow graph. To balance flexibility with performance, verl unifies two distinct programming models: **High-Level Single-Controller (MPMD)**: At the orchestration level, a single process `RLTrainer` manages the global computation graph. It handles macro-tasks such as scheduling rollout generation, triggering reward scoring, and dispatching distributed training jobs. **Internal Multi-Controller (SPMD)**: Internally, the Model Engine operates in standard distributed training mode. Workers execute identical programs, via trainer backends like FSDP, Megatron, or VeOmni, or rollout executors (not rollout server) like vLLM/SGLang/TensorRT-LLM, to perform heavy distributed computation, synchronizing via collective communication. This hybrid approach offers significant advantages: **Flexible Orchestration**: The single-controller design allows verl to dynamically manage complex constraints within the computation graph, including flexible data dependencies, diverse resource allocation and model placement, and fine-grained asynchronous staleness control. **Abstraction of Complexity**: We encapsulate complex parallel strategies—such as 5D parallelism (DP, TP, CP, PP, and EP)—strictly within the Model Engine. This allows users to focus entirely on RL algorithm implementation without getting bogged down by the details of distributed training. Furthermore, leveraging Ray placement groups, verl provides `ResourcePool` and `WorkerGroup` abstractions. These enable flexible GPU sharing among the various roles in the RL process—such as actor, critic, reward, and rollout—allowing components to share resources efficiently while remaining isolated. As illustrated in the diagram below, the overall architecture of verl is divided into two layers: - **verl-core**: provides four components required for the RL pipeline: model engine, rollout engine, checkpoint engine, and transfer queue. Each component exposes abstract interfaces, making them both extensible and pluggable. - **verl-trainer**: builds upon these components, construct various RL pipelines—such as on-policy, one-step-off-policy, and fully asynchronous—tailored to meet the demands of diverse scenarios. ## verl-core ### Model Engine The Model Engine serves as verl's core training engine, defining a set of abstract interfaces that support pluggable backends. It operates in SPMD mode: - SFT: Workers are launched via torchrun. - RL: Workers are executed via the WorkerGroup API, invoked by the single-controller. The abstract interfaces include methods like `initialize`, `forward`, `optimizer_step`, and `load`/`offload`. Integrating a new training engine simply requires inheriting and implementing these interfaces. Crucially, because all backends adhere to this unified abstraction, adding a new Model Engine requires absolutely no code modification on the caller side. The RLTrainer remains completely agnostic to the backend's specific parallel strategy when calling these interfaces, while the WorkerGroup automatically handles data dispatch and collection based on the underlying parallelism. Currently, the Model Engine supports the following backends (more backend maybe supported in future, e.g torchtitan): |Backend|Parallelism|Performance|Support Model|New Model Support Time |-----|-----|----|----|----| |FSDP| FSDP+SP|Dense medium/MoE low| all transformer models|Day 0 |MCore| DP+TP+PP+EP+CP|High| see [Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge) support model list|few weeks or month |VeOmni| FSDP+SP+EP|Medium| see [VeOmni](https://github.com/ByteDance-Seed/VeOmni) support model list|~1 week ```python class BaseEngine: def initialize(self): """Instantiate or load the model, optimizer, and learning rate scheduler.""" raise NotImplementedError def optimizer_zero_grad(self): """Zero the gradients of the optimizer.""" raise NotImplementedError def optimizer_step(self): """Perform an optimization step using the optimizer.""" raise NotImplementedError def lr_scheduler_step(self): """Advance the learning rate scheduler by one step.""" raise NotImplementedError def forward_backward_batch(self, data: TensorDict, loss_function: Callable, forward_only=False) -> Any: """Perform a forward pass and optionally a backward pass on a batch of data.""" raise NotImplementedError def get_per_tensor_param(self) -> tuple[Generator[tuple[str, torch.Tensor], None, None], Optional[dict]]: """Get a generator that yields per-tensor parameters and optional peft config.""" raise NotImplementedError def to(self, device: str, model: bool = True, optimizer: bool = True, grad: bool = True): """Move model parameters, optimizer states, or both to the specified device.""" raise NotImplementedError ``` ### Rollout Engine As LLM reinforcement learning evolves from single-turn, static tasks to multi-turn, dynamic, and interactive agentic tasks, the legacy SPMD rollout mode previously used by verl has become insufficient. Consequently, in verl v0.7, we have removed the SPMD rollout mode and switched to rollout server mode by default. In the server mode, the LLM server operates as online serving rather than the traditional offline batch inference. Clients send per-sample requests to the server, enabling the engine to utilize dynamic batching. This significantly enhances throughput efficiency for multi-turn conversation. Furthermore, the server-based approach eliminates the need for intrusive modifications to the LLM inference engine, allowing for the seamless integration of modern inference backends such as vLLM, SGLang, and TensorRT-LLM. On the client side, verl introduces an extensible **AgentLoop** abstraction designed to define custom agentic task loops. This abstraction manages the cycle of requesting responses from the LLM server and interacting with external environments to obtain feedback. We provide two default implementations: - **SingleTurnAgentLoop**: Designed for standard single-turn tasks. - **ToolAgentLoop**: Designed for classic ReAct architectures involving multi-turn tool invocation. Users can implement custom AgentLoop logic tailored to their specific needs, such as [SWEAgentLoop](https://github.com/verl-project/verl/pull/4080) or GUIAgentLoop. ```python class AgentLoopBase(ABC): @abstractmethod async def run(self, sampling_params: dict[str, Any], **kwargs) -> AgentLoopOutput: """Run agent loop to interact with LLM server and environment. Args: sampling_params (Dict[str, Any]): LLM sampling params. **kwargs: dataset fields from `verl.utils.dataset.RLHFDataset`. Returns: AgentLoopOutput: Agent loop output. """ raise NotImplementedError ``` ### TransferQueue As mentioned, verl uses a global single-controller RLTrainer to orchestrate the computation graph. A major limitation in the current implementation is that the RLTrainer handles both control and data flow, creating a bottleneck when dispatching data between components. This issue is amplified by the massive data volumes in multimodal training (images, video, audio) and complex algorithms like router replay, which requires transmitting large tensors per sample. Our earlier attempt to solve this using the Ray object store yielded poor performance due to the lack of tensor optimization and fine-grained column access. In v0.7, we experimentally introduced **TransferQueue** to decouple control flow from data flow. The RLTrainer now only dispatch instructions and metadata, while TransferQueue handles data transmission via reference passing. TransferQueue is specifically optimized for PyTorch tensors (supporting zero-copy and RDMA) and allows for backend extensions like ZeroMQ, NIXL, and Ray RDT. We plan to make this the default transmission method in v0.8. ```python # In PPOTrainer def fit(self): batch = next(dataloader) gen_batch: BatchMeta = self.rollout_manager.generate_sequences(batch) output: BatchMeta = self.actor_rollout_wg.compute_log_prob(gen_batch) gen_batch = gen_batch.union(output) output = self.actor_rollout_wg.update_actor(gen_batch) # In Worker def compute_log_prob(self, batch: BatchMeta) -> BatchMeta: data = tq.get(batch) output = self.actor.infer_batch(data=data) return tq.put(output) ``` ### Checkpoint Engine With the increase in LLM context lengths and the evolution of agentic tasks, the "long-tail" problem in rollout has become prominent, limiting the overall efficiency of RL training. To mitigate this, a viable strategy is moving from on-policy synchronous training to off-policy asynchronous training, e.g [Laminar](https://arxiv.org/abs/2510.12633), [Areal](https://arxiv.org/abs/2505.24298), [StreamRL](https://arxiv.org/abs/2504.15930), [LlamaRL](https://arxiv.org/pdf/2505.24034), [PipelineRL](https://arxiv.org/abs/2509.19128). This involves separating the rollout and model engines onto different nodes (a disaggregated architecture, as opposed to colocated), with data transmitted via queues. This separation alleviates the rollout long-tail issue and enables rollout elastic scaling, fault tolerance, and heterogeneous hardware. However, it introduces a new challenge: efficient cross-node parameter synchronization. To address this, we introduce the Checkpoint Engine: a unified abstraction layer designed to synchronize weights between various training and inference backends. - It provides three unified APIs to implement the streaming transmission of parameters. - Users can extend the Transport Layer implementation based on their specific infrastructure requirements (device, network, local cache, etc.). Currently, we provide two transport backends: NCCL (for broadcast collective communication) and NIXL (for P2P point-to-point communication). ```python class CheckpointEngine(ABC): @abstractmethod async def send_weights(self, weights: Generator[tuple[str, torch.Tensor], None, None]): """Send the weights of the model. Args: weights: A generator that yields the name of the weight tensor and the tensor itself. """ raise NotImplementedError @abstractmethod async def receive_weights(self) -> Generator[tuple[str, torch.Tensor], None, None]: """Receive the weights of the model. Yields: A tuple of the name of the weight tensor and the tensor itself. """ raise NotImplementedError ``` ## verl-trainer Building upon the four core components provided by verl-core, verl-trainer constructs several RL training pipelines tailored to specific scenarios. These pipelines are designed to address training efficiency challenges across varying scales and requirements: **On-policy (Synchronous)** - Main Features: Executes rollout and training serially, typically sharing GPU resources (Colocate). It strictly adheres to standard on-policy algorithm definitions, where training must wait for all samples to be generated. - Scenarios: Best for baseline implementations, scenarios where strict algorithmic correctness is prioritized over training throughput. **One-step-off-policy (Async)** - Main Features: Parallelizes generation and training by overlapping the current training step with the next batch's generation. It employs resource isolation and uses parameters from the previous step for rollout to minimize GPU idle time. - Scenarios: Ideal for scenarios requiring moderate efficiency gains (20%–40%) while maintaining training stability very close to strict on-policy methods. **Fully async (Decoupled & Streaming)** - Main Features: Completely decouples the Trainer and Rollouter onto separate nodes. It utilizes streaming data transfer, staleness control, and partial rollout mechanisms to maximize throughput and mitigate long-tail generation latency. - Scenarios: Essential for large-scale training (e.g., 128+ GPUs) or complex reasoning tasks (e.g., long chain-of-thought) where generation latency significantly bottlenecks performance. ## roadmap ### v0.7 release **Model Engine** - Integrate Megatron-Bridge and support LoRA/PEFT, see blog post: [How We Build Trillion Parameter Reasoning RL with 10% GPUs](https://macaron.im/mindlab/research/building-trillion-parameter-reasoning-rl-with-10-gpus) - Support experimental fp8 training for megatron backend - Support new model for megatron backend: GPT-OSS, Qwen3-Next - Comprehensive support for new mode engine, FSDP and Megatron engine are production ready. - Dispatch tensordict with nested tensor instead of padded DataProto - Add TrainingWorker that resembles Tinker-like API - Add VLM support for model engine, SFT and RL trainer - Add model engine based critic model - Implement ActorRolloutRefWorker by TrainingWorker, support different backend in one worker - New VeOmni engine added, still in alpha status. **Rollout Engine** - Remove SPMD rollout mode - Support blockwise fp8 rollout for vllm and sglang; support online quant for vllm with torchao - Experimental router replay support for vllm - Optimize multi-modal data fetch and preprocess, support video input - Upgrade to vllm==0.12.0; sglang==0.5.6 **Reward** - Support hybrid reward scenarios, including generative, discriminative, rule-based rewards, and their combinations. - Refactor reward models into server mode, supporting both colocated and standalone deployments. - Introduce new reward managers to handle more complex scenarios, limited mode for request rate control and remote mode for CPU-intensive tasks. **Algorithm** - Add [CISPO](https://arxiv.org/pdf/2506.13585): Clipped IS-weight Policy Optimization - Add [SAPO](https://arxiv.org/abs/2511.20347): Soft Adaptive Policy Optimization **Recipe** - [NEW] VLA: add experimental support for VLA model - [NEW] [rhymerl](https://arxiv.org/abs/2508.18588): History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL - TransferQueue: support multiple data partition and optimize tensor zero-copy serialization - One-step-off-policy/Fully async: optimize weight synchronization by checkpoint engine with bucket and pipeline support. ### v0.8 **Model Engine** - Deprecate DataProto by Tensordict for zero padding transmission - Switch default to new model engine, mark legacy engine (fsdp_workers.py, megatron_workers.py) as deprecated - Feature parity between new and legacy model engine: LoRA/PEFT, etc - Polish VeOmni engine to production ready status - Support MTP RL training - Optimize GPU memory for long context: fine-grained activation recompuation/offload - New model support: DeepSeek V3.2, etc **Rollout Engine** - New rollout engine TensorRT-LLM - Separate vllm worker from trainer process, update weights by cuda ipc **TransferQueue** - Merge TransferQueue recipe into main - Optimize e2e image/video vlm training pipeline by TransferQueue - Optimize router replay transmission by TransferQueue **Checkpoint Engine** - Add checkpoint engine abstract interface - Add NCCL and NIXL transport backend - Add more transport backend ### v0.9 **Trainer** - Merge Full async into main: refactor with verl-core component **Model Engine** - Remove legacy model engine (fsdp_workers.py, megatron_workers.py) - Support omni-model RL training: Qwen3-Omni, BAGEL, etc **Rollout Engine** - New rollout engine vllm-omni **More agentic training recipe** - SWEAgent - GUIAgent ## 2. Official Technical Reference & Guides (verl-project/verl-project.github.io) ## File: content/posts/2026-08-04-introducing-the-verl-blog/index.md --- title: "Introducing the verl blog" date: 2026-08-04 authors: - "verl-project" summary: "verl now has its own blog. It exists so that the people who build features can explain them properly." image: "cover.png" tags: - announcement - community math: false toc: true --- verl now has a blog, and it is open to contributors from day one. If you have landed something in verl and want to explain how it works, you can publish it here by opening a pull request. ## What gets published here Writing that helps someone understand verl better. Some of the shapes that takes: - **Feature deep dives** — the design behind something that landed, including the approaches that were tried and rejected. - **Performance work** — with hardware, model, and baseline stated, so the numbers mean something. - **Recipes and case studies** — end-to-end accounts of training something real. - **Release deep dives** — what changed and why it matters. - **… and anything else you think is worth explaining.** The list above is a starting point, not a set of boxes to fit into. None of this replaces the [documentation](https://verl.readthedocs.io/en/latest/). The docs describe how the current version works and are kept up to date; a post captures the reasoning behind a change at the time it was made. If what you are writing is reference material, it belongs in the docs instead. ## Publishing is a pull request The whole point of hosting this in the project's own repository is that the contribution path is the one you already use. A post is a directory: ```text content/posts/2026-08-04-your-post-slug/ ├── index.md └── cover.png ``` Its front matter declares who wrote it and what it says: ```yaml --- title: "Async rollout in verl" date: 2026-08-04 authors: - "Jane Doe" summary: "Async rollout overlaps generation with training and cuts idle GPU time on long-horizon tasks." image: "cover.png" tags: - performance --- ``` Open a pull request and CI validates the front matter and builds the site. A maintainer reviews it for accuracy. On merge, the site redeploys on its own. One thing worth doing first: open an issue with the idea before you start drafting. A maintainer can tell you early whether the topic fits and whether somebody else is already writing it, which is a lot cheaper than finding out once the draft is done. Full details are on the [Write for us](/contribute/) page. ## Attribution Posts stay attributed to the people who wrote them. Your name stays on the byline rather than being replaced by a generic project account, and each author name becomes an archive page collecting that person's posts. You also keep the copyright in what you write. Publishing here does not sign anything away, and nothing stops you from putting the same piece on your own site as well. ## What happens next That is up to the community. A blog only works if people write for it, and the person best placed to explain a piece of verl is usually the one who built it. So if you have landed something in verl and found yourself explaining it more than twice — in a Slack thread, in a code review, in an issue comment — that is a post waiting to be written. Writing it down once, properly, saves the next person the same conversation, and it is one of the more lasting ways to contribute to an open source project. Have a look at [Write for us](/contribute/) and open a pull request. We are happy to help with drafts, and just as happy to see a rough one. --- ## File: content/posts/2026-08-11-verl-tinker/index.md --- title: "Introducing verl-tinker: Your Tinker loop, your GPUs" date: 2026-08-11 authors: - "Tianle Zhong" - "Huaye Zeng" - "Xibin Wu" - "Siping Tao" - "Peng Wu" - "Yifan Pi" - "Xiao Yu" summary: "Keep the Tinker Cookbook loop you know, and run SFT, RL, and distillation on verl-managed GPU workers you control." image: "verl_tinker_logo.png" tags: - training - rollout - recipes - distillation math: false toc: true --- What if you could keep a Tinker Cookbook training loop exactly where it is, but run the heavy lifting on your own GPUs? That is what `verl-tinker` makes possible. The official Tinker SDK and nearly the full range of Tinker Cookbook recipes work as they are; in most cases, the only client-side change is overriding the base URL. Point the program at a local HTTP endpoint and it can drive supervised fine tuning (SFT), reinforcement learning (RL), online and offline distillation, checkpointing, and rollout on verl-managed workers. In other words: **the experiment loop stays simple; the infrastructure becomes yours.** The implementation is available now in the [`verl_tinker` recipe](https://github.com/verl-project/verl-recipe/tree/main/verl_tinker). It is deliberately lightweight: a thin forwarding layer exposes a Tinker-compatible API, while presets make it easy to launch the server in one step. Underneath, workloads run on mature, production-tested verl, VeOmni, and vLLM configurations. ## Keep the loop. Bring the infrastructure. Tinker Cookbook recipes describe an algorithm from the researcher's point of view. They build data, ask a model to sample or compute log probabilities, submit a loss-bearing batch, and take an optimizer step. It is an appealingly compact way to express an experiment. verl tackles the other side of the problem: distributed actor, reference, rollout, and teacher workers; model-state movement; parallelism; offload; and checkpoint storage. `verl-tinker` joins those two worlds at the API boundary. Researchers keep the concise Cookbook loop and choose the cluster, model weights, inference engine, parallelism strategy, and storage behind it. The client environment needs only `tinker` and `tinker-cookbook`; the server owns the distributed runtime. The bridge stays intentionally small. It forwards Tinker operations into the backend and handles the compatibility details at the API boundary; it does not reimplement the training or inference stack. The distributed execution path comes from the same proven backend configurations used by verl, VeOmni, and vLLM workloads. ## Start a workload in minutes Install the server environment from the root of `verl-recipe`, then start an actor-and-rollout configuration: ```bash ./install_verl.sh --recipe verl_tinker cd verl_tinker python -m verl_tinker.start \ --config configs/quick_start/actor_rollout.yaml ``` Once `GET /api/v1/healthz` returns `{"status":"ready"}`, point a client at the server: ```bash export TINKER_BASE_URL=http://127.0.0.1:8000/ export TINKER_API_KEY=tml-verl-tinker-local cd verl_tinker/client_examples uv sync uv run run_single_test.py \ --test-name sft_tulu3 ``` That is the whole handoff. For an existing SDK or Cookbook workload, setting `TINKER_BASE_URL` is usually all that is required. The recipe issues familiar Tinker calls, and the server forwards them to the distributed backend. The bundled [client examples](https://github.com/verl-project/verl-recipe/tree/main/verl_tinker/client_examples) are intentionally lightweight wrappers around recipes from the [Tinker Cookbook](https://github.com/thinking-machines-lab/tinker-cookbook). Together they cover most official Cookbook recipe patterns, including SFT, supervised distillation, GSM8K RL, SFT followed by RL, and single- or multi-teacher on-policy distillation. They are examples, not a required client layer. Because `verl-tinker` speaks the Tinker API, you can point your own program built with the Tinker SDK at the same endpoint and run your own data, training loop, and workflow. Three quick-start configurations cover the common shapes: - `actor.yaml` for SFT and optimizer-only workflows; - `actor_rollout.yaml` when the client calls `asample`; - `actor_rollout_ref.yaml` for RL that requests reference log probabilities for KL penalties. ## What happens after the HTTP call? A request passes through three layers: ```text Tinker / Tinker Cookbook client | | Tinker-compatible HTTP v Thin Tinker-compatible forwarding layer | | preset-backed server configuration v verl / VeOmni training + vLLM rollout workers ``` ## Bring your teachers, too Distillation is where this bridge gets especially interesting. A Tinker program can use one or more larger, frozen models as teachers, ask them to generate or return top-K log probabilities, and use those signals to train a smaller actor—all through the same familiar client API. That means you can explore single-teacher or multi-teacher on-policy distillation without building a separate serving stack for every model. Your training loop decides how to use each teacher; `verl-tinker` runs the actor and teachers on the GPU resources you assign to them. The recipe includes two ready-to-adapt starting points: - The [single-teacher config](https://github.com/verl-project/verl-recipe/blob/main/verl_tinker/configs/advance/qwen3_1b7_actor_qwen3_30b_a3b_teacher.yaml) pairs a Qwen3-1.7B actor with a Qwen3-30B-A3B teacher on eight GPUs. - The [multi-teacher config](https://github.com/verl-project/verl-recipe/blob/main/verl_tinker/configs/advance/qwen3_8b_actor_qwen3_32b_qwen3_235b_teachers.yaml) shows a Qwen3-8B actor learning from Qwen3-32B and Qwen3-235B-A22B teachers across two eight-GPU nodes. Use these configs as templates: swap in your models, choose the parallelism that fits your cluster, and let your Tinker code decide what each teacher contributes to the learning signal. ## Where the boundary is today For most workflows, `verl-tinker` lines up closely with Tinker. The main difference is that a server currently manages one trainable model and one rollout replica. Because that rollout is shared, a sampling session created from the trainable model is tied to a particular version of its weights. If the rollout is later synchronized to newer weights—or otherwise diverges from the version that session expects—the earlier sampling resource may no longer be available. Teacher sampling and reference-log-probability requests are separate and are not affected by this restriction. You will not silently sample from the wrong weights. `verl-tinker` keeps track of which model version each sampling resource belongs to and tells you when a resource is no longer available. Within that boundary, the division of labor stays simple: **your Tinker code owns the experiment; verl owns distributed execution.** You keep the loop you want, while choosing the GPUs, rollout engine, placement, offload, and checkpoints underneath it. If that sounds like the missing bridge in your workflow, start with the [`verl_tinker` recipe](https://github.com/verl-project/verl-recipe/tree/main/verl_tinker) and run one of the included client examples. `verl-tinker` was developed by the ByteDance AML/Seed Team. --- ## File: content/posts/2026-08-13-verl-speco-release/index.md --- title: "Faster Draft Model Training: verl-SpeCo 0.1.0 Adds Native DSpark Support and Unlocks Standalone Training" date: 2026-08-13 authors: - "PeiXiang Tan" - "Shuang Li" - "Shuai Yuan" - "RunPing Chen" author_links: "PeiXiang Tan": "https://github.com/tpx818" "Shuang Li": "https://github.com/755651978" "Shuai Yuan": "https://github.com/crp0128" "RunPing Chen": "https://github.com/ys2025-AI" summary: "verl-SpeCo 0.1.0 adds native DSpark support and standalone draft model training, with reusable feature storage, seven algorithm backends, vLLM/SGLang integration, and GPU/Ascend NPU support." image: "verl-sepco.png" tags: - DSpark - Dflash - Speculative Decoding - Co-Train math: false toc: true --- ## 1. Overview verl-SpeCo 0.1.0 advances draft model training along three major fronts: First, it integrates the complete DSpark training and speculative decoding pipeline, resolving the semantic mismatch between conventional token-wise training and block-wise prediction. Second, it introduces a standalone draft model training workflow with three modes—online training, feature collection only, and offline training—so draft models can be trained independently of a full reinforcement learning job. Third, it expands algorithm coverage to seven widely used draft model families: DSpark, EAGLE-1, EAGLE-2, EAGLE3, DFlash, Domino, and P-EAGLE. It also supports both vLLM and SGLang inference runtimes across GPU and Ascend NPU backends. The goal of this release is to provide native training support for advanced draft algorithms such as DSpark, establish complete workflows for both online co-training and standalone training, and broaden support for speculative decoding algorithms. The sections below walk through the four key areas of the release. ## 2. DSpark: Native Training for Block-Wise Speculative Decoding ### 2.1 Training Semantics Designed for DSpark's Block-Wise Predictions DSpark uses block-wise generation to overcome performance limitations in conventional draft models. verl-SpeCo 0.1.0 introduces a dedicated DSpark training backend. During training, it starts from an anchor token in the sequence, constructs a fixed-length draft block, and aligns the anchor, preceding tokens, target tokens, and valid positions with DSpark's semantics. This allows even the first position in each block to contribute to the training objective. This purpose-built data layout eliminates the semantic mismatch between generic token-wise training and DSpark's block-wise generation. The anchor, preceding tokens within the block, and supervised positions seen during training now match those used at inference time, providing a sound foundation for improving block-wise draft model performance. ### 2.2 Markov Heads for Explicit Intra-Block Token Dependencies Unlike approaches that predict draft tokens solely from backbone hidden states, DSpark adds a Markov bias to the base logits, directly incorporating preceding tokens within the block into subsequent token predictions. The verl-SpeCo training path supports two types of Markov Head: Vanilla and Gated. The Vanilla Head models dependencies between adjacent tokens through a low-rank token transition structure. The Gated Head additionally uses the hidden state at the current position to control the Markov signal dynamically. Markov bias works with full-vocabulary, restricted-CE, and sampled-CE computation paths. For restricted-CE and sampled-CE training, only the required output rows are selected while DSpark's transition modeling is preserved. ### 2.3 Joint Optimization with Cross-Entropy and Distribution Distillation DSpark training can combine target-token cross-entropy with distribution distillation. The latter reconstructs the full output distribution from the target model's final-layer hidden states and computes the L1 distance between the draft and target distributions. The two losses can be weighted independently and decayed exponentially by position within the block, placing greater emphasis on the early positions that matter most for longer acceptance sequences. To support L1 distribution distillation, verl-SpeCo extends the rollout/actor feature collection pipeline. In addition to multi-layer context features, it captures the target model's final-layer hidden states. The training backend then aligns them precisely with the label position of each anchor. This enables DSpark to optimize both token prediction accuracy and distributional consistency. Compared with training on token labels alone, this objective is better aligned with the acceptance-length decisions made during speculative decoding and helps the draft output more closely match the target model. For large vocabularies, restricted CE and sampled CE reduce LM Head computation. When L1 distillation is enabled, the training backend can still reconstruct the full probability distribution on demand. verl-SpeCo also provides fine-grained diagnostics for identifying data issues and convergence problems in block-wise draft model training. It can truncate invalid position sequences, filter non-finite losses such as NaN and Inf, report loss and accuracy by block position, and track Top-1/Top-5 accuracy, valid token counts, and the sampled vocabulary size used by CE. *Figure: DSpark training workflow* ### 2.4 Dedicated GPU and NPU Optimizations DSpark builds on DFlash's block-wise context backbone. The training backend selects the attention implementation according to the device: CUDA uses a block mask with the Flex Attention path, while non-CUDA devices use an equivalent dense attention mask. This preserves sparse block computation on GPUs while providing an executable compatibility path for Ascend NPUs. For inference, users set `speculative_algorithm=DSPARK`. On GPUs, this maps to vLLM's native DSpark method. On NPUs, integration follows the compatible vLLM and vLLM-Ascend versions specified in the project README. *Figure: DSpark experiment results on NPU* *Figure: DSpark experiment results on GPU* ## 3. Standalone Training: Decoupling Draft Model Training from the Main RL Job ### 3.1 Three Training Modes In addition to online co-training, verl-SpeCo now supports standalone draft model training. Given a feature dataset produced by the target model, users can independently train any supported draft model, including EAGLE, DSpark, and DFlash. This makes it possible to build draft models for target models such as Qwen and Llama without running a complete RL job. verl-SpeCo 0.1.0 divides draft model training into three modes: `online`, `collect_only`, and `offline`. The `online` mode preserves the existing co-training experience: it collects features in a PPO/Ray job, trains periodically, and can publish updated weights to the rollout engine. The `collect_only` mode writes collected features to the Feature Store without starting draft model training inside the RL job. The `offline` mode reads persisted features and runs draft model training as a separate job. This separation decouples resource scheduling and draft model training from a single long-running pipeline. Feature generation and draft model experiments can run independently, and collected features can be reused for hyperparameter comparisons, architecture validation, and reproducible debugging. Dedicated training resources can also be provisioned outside the RL workflow. ### 3.2 Feature Store: Persistent Training Features verl-SpeCo provides a Feature Store based on sharded PyTorch files. It persists input IDs, loss masks, target hidden states, and auxiliary features in shards while maintaining a unified schema and shard manifest. The collection side supports step-based flushing and shard-capacity controls. The training side provides read-only loading, sample shuffling, cyclic iteration, and strict schema validation, helping prevent failures caused by missing fields, shape drift, or configuration mismatches. The release also adds the `verl-speco-inspect-features` command, which samples and validates stored features before training begins. Its strict exit codes make it suitable for automated pipelines, allowing users to verify data quality before committing training resources. ### 3.3 Distributed Training verl-SpeCo adds the `verl-speco-draft-train` command and the `draft_trainer.yaml` top-level configuration. The launcher normalizes single-node multi-GPU and multi-node parameters, and automatically selects a single-node distributed torchrun configuration when appropriate. The offline training loop initializes the distributed environment, selects a Drafter Backend, reads from the Feature Store, performs gradient accumulation, saves checkpoints at configured intervals, and writes the final checkpoint. Standalone training uses the same Drafter Backend interface as online co-training. The project currently includes training backends for EAGLE-1, EAGLE-2, EAGLE3, DFlash, DSpark, Domino, and P-EAGLE. For observability, the training loop records loss, vloss, ploss, the current learning rate, and cumulative optimizer steps. Block-wise algorithms such as DFlash, DSpark, and Domino additionally expose CE/L1 loss, Top-1/Top-5 accuracy, valid token counts, sampled vocabulary size, loss and accuracy by block position, and timing for batch preparation, forward computation, loss reduction, backward propagation, and optimizer updates. ### 3.4 Getting Started verl-SpeCo includes the `run_qwen3-8b_drafter_separate_training.sh` example script to demonstrate the standalone workflow in two stages. In the first stage, rollout runs in `collect_only` mode and writes features to a specified directory. In the second stage, the standalone trainer runs in `offline` mode, reads from the same Feature Store, and trains the draft model. To get started, users only need to replace the target model, draft model, dataset, feature directory, and checkpoint directory. *Figure: Standalone training data loop* ## 4. Broader Support Across Algorithms and Runtimes ### 4.1 Seven Draft Model Training Backends In addition to DSpark, verl-SpeCo 0.1.0 supports EAGLE-1, EAGLE-2, EAGLE3, DFlash, Domino, and P-EAGLE. EAGLE-1 and EAGLE-2 reuse vLLM's native EAGLE draft method, with EAGLE-2 adding dynamic tree decoding. Domino is trained as a DFlash projector sub-mode and deployed through a DFlash runtime that supports the Domino projector. ### 4.2 vLLM, SGLang, GPU, and NPU Examples On the inference side, vLLM support covers EAGLE-1, EAGLE-2, EAGLE3, DFlash, and DSpark, while SGLang support covers EAGLE3 and DFlash. The project provides launch examples for both GPUs and NPUs. Graph modes, draft-model parameters, and version requirements remain explicit in the configuration, making each hardware setup easier to verify and reproduce. | Algorithm | Online Co-Training | Standalone Training | vLLM | SGLang | | --- | :---: | :---: | :---: | :---: | | EAGLE-1 | Supported | Supported | Supported | — | | EAGLE-2 | Supported | Supported | Supported | — | | EAGLE3 | Supported | Supported | Supported | Supported | | DFlash | Supported | Supported | Supported | Supported | | DSpark | Supported | Supported | Supported | — | | Domino | Supported | Supported | Compatibility mode | — | | P-EAGLE | Supported | Supported | — | — | *Table: Training, standalone training, and rollout runtime support across seven algorithms* ## 5. A Unified, Non-Intrusive Architecture Beyond the two headline capabilities, verl-SpeCo 0.1.0 establishes a foundation for continued expansion. It is compatible with verl `release/v0.8.0` and ships as a separate package that provides layered Hydra configuration, training Workers, and rollout adapters without modifying the installed verl source code. The inference side supports vLLM and SGLang, with examples for GPUs and NPUs. The training side uses a unified FSDP backend and extends support for draft models through pluggable backends. At runtime, the framework records drafter training time, weight publication time, and vLLM speculative decoding acceptance metrics, including `mean_acceptance_length`. This gives users visibility into both draft model learning and end-to-end rollout performance. ## 6. What's Next verl-SpeCo 0.1.0 addresses the challenge of integrating draft models reliably into reinforcement learning workflows. Native DSpark integration broadens support for block-wise speculative decoding, while standalone training provides more flexible ways to organize feature collection and model optimization resources. Future work will focus on end-to-end training and runtime support for additional draft algorithms, multi-hardware validation for standalone training, performance optimization, stronger feature storage and data governance, and more systematic benchmarking and observability. --- ## File: content/posts/2026-08-14-verl-on-amd-rocm/index.md --- title: "verl on AMD: production-ready RL post-training on ROCm" date: 2026-08-14 authors: - "Fuwei Yang" - "Zhaodong Bing" - "Mingjie Lu" - "Xiaohong Kou" - "Yuhan Yang" - "Wei Cai" - "Liz Li" - "Yuankai Chen" - "Yao Fu" - "Dong Li" - "Zhenyu Gu" summary: "RL post-training on AMD Instinct GPUs is here: a turnkey ROCm container, AITER-accelerated vLLM and SGLang rollout, and accuracy validated on both MI300 and MI350 series." tags: - amd - rocm - performance - vllm - sglang - megatron - fsdp image: "cover.png" math: false toc: true --- Reinforcement learning post-training on AMD Instinct GPUs is here — with a turnkey container, AITER-accelerated vLLM and SGLang rollout, and accuracy validated on both MI300 and MI350 series. [verl](https://github.com/verl-project/verl) — the open-source implementation of the RL controller — has quickly become one of the most widely adopted RL post-training frameworks, powering PPO, GRPO and DAPO pipelines with a flexible mix of rollout and training engines. Today, that same power runs on AMD Instinct GPUs. Whether you're on MI300 or the latest MI350 series, you can stand up verl on ROCm in minutes, scale it across your rollout and training engines, and trust the numbers. Here's what makes AMD a first-class home for verl. ## Zero-friction setup: one container, no dependency headaches Getting a modern RL stack running usually means fighting a maze of dependencies — matching CUDA/ROCm versions, building attention kernels from source, and praying the wheels line up. On AMD, that pain is gone. AMD ships a turnkey ROCm container, `verlai/verl:rocm7.14_torch2.12_release_0724`, built from [`docker/rocm/Dockerfile.rocm`](https://github.com/verl-project/verl/blob/main/docker/rocm/Dockerfile.rocm), so you can go from zero to training without hand-assembling a single dependency. The image bundles the entire runtime stack — ROCm, PyTorch, Triton, vLLM, SGLang, AITER, TransformerEngine and Megatron-core — all pinned to a set of versions verified to work together. Pull it, mount your data, and run: RL post-training on AMD is a `docker run` away, so you can spend your time on reward design and experiments rather than on environment setup. - **No dependency installation.** Everything RL post-training needs is baked in — no extra source builds, no extra version detective work. There's an [end-to-end AMD tutorial](https://verl.readthedocs.io/en/latest/amd_tutorial/amd_quick_start.html) covering build, run, and example PPO/GRPO commands. - **Built for multiple architectures.** The image targets `gfx942` (MI300 series — MI300X / MI308X / MI325X) and `gfx950` (MI350 series — MI350X / MI355X) out of the box. - **Customizable for your setup.** Teams that want to tailor the image can do so — target a specific GPU architecture, pin component versions, and tune the build to their environment — all documented in the [AMD support section of the verl README](https://github.com/verl-project/verl/tree/main#amd-support-rocm-kernel). This is all backed by CI on real AMD hardware: end-to-end tests now run on a community-hosted runner, so changes to the verl repository are validated on AMD hardware before merge. Coverage is a work in progress and actively expanding toward parity with the other platforms. ## Two rollout backends — vLLM and SGLang, both AITER-accelerated Rollout is the key component of RL post-training, and on AMD you're not locked into a single engine: verl supports both vLLM and SGLang on ROCm. Both are accelerated by [AITER](https://github.com/ROCm/aiter), AMD's AI Tensor Engine for ROCm kernel library — powering attention, RMSNorm / RoPE, mixture-of-experts (MoE), and quantization — with everything fully overridable if you want to experiment. - **vLLM** — the default, battle-tested rollout engine; enable the AITER path with a few environment variables. - **SGLang** — a first-class alternative that runs on ROCm with AITER kernels enabled by default. Both engines work across both of verl's integration modes — colocated (rollout and training share GPUs with fast weight transfer) and fully async (rollout and training run concurrently for maximum GPU utilization, with live parameter sync back to the rollout workers). Whichever engine and mode fit your workload, AMD-accelerated rollout is ready to go. ## Validated accuracy — MI300 and MI350 series, colocated and fully-async Enablement is only half the story. What ultimately matters is whether the model you train on AMD is as good as the model you'd train anywhere else. It is — and we've measured it. Training accuracy has been validated on both AMD MI300 and MI350 series, across both colocated and fully-async modes. RL post-training on Instinct GPUs converges to the quality you expect, so you can move real workloads onto AMD with confidence rather than crossing your fingers. The MI350-series runs below were measured on MI350. MI350X and MI355X are the same `gfx950` silicon and differ in board power and cooling rather than in numerics, so the accuracy results carry over between them unchanged — only throughput tracks the power envelope. ### Case study: Qwen3-8B GRPO with SGLang rollout on MI350 Start with the SGLang rollout backend — the first-class alternative to vLLM introduced above — to show it delivers the same accuracy on AMD. Key run parameters: | Parameter | Value | | --- | --- | | Algorithm | GRPO (`use_kl_loss=True`, `kl_loss_coef=0.001`) | | Training backend | FSDP (PyTorch Fully Sharded Data Parallel) | | Rollout engine | SGLang async rollout server, AITER-accelerated, group size n = 5 | | Task | GSM8K + MATH (train and validate on both) | | Max prompt / response length | 1,024 / 2,048 tokens | | Train batch / PPO mini-batch | 1,024 / 256 | | Learning rate | 1e-6 | | Execution mode | Colocated, single node — 8 GPUs | Note that "async" here refers to SGLang's async rollout server, which serves generation requests concurrently within a rollout step. It is not verl's fully-async execution mode: rollout and training still share the same 8 GPUs and alternate, which is what "colocated" in the table means. The fully-async mode, where rollout and training run on disjoint GPUs at the same time, is covered in the next two case studies. The training curves show textbook convergence: {{< figure src="qwen3_8b.jpg" alt="Four panels over 200 training steps: GSM8K and MATH validation accuracy, mean training reward, rollout-to-actor KL, and wall-clock time per step" caption="Qwen3-8B GRPO on GSM8K and MATH. SGLang async rollout server with AITER, FSDP trainer, colocated on 8xMI350." >}} 1. **Strong accuracy on both benchmarks.** GSM8K validation accuracy climbs from 0.82 to ~0.955 (peak 0.956) within the first ~30 steps and holds there, while the harder MATH benchmark rises steadily from 0.37 to ~0.835 (peak 0.839). Mean training reward tracks them in lockstep, from 0.62 to ~0.90 — no reward-hacking, no collapse. An 8B model reaching ~95% on GSM8K and ~84% on MATH is exactly the quality bar you'd expect from a well-behaved GRPO run. 2. **SGLang and the FSDP trainer stay numerically locked.** The rollout↔actor mismatch is negligible: `actor/ppo_kl` hovers around 1e-4 during the entire run. In other words, the tokens that the AITER-accelerated SGLang samples during rollout are assigned essentially identical probabilities when the FSDP actor recomputes them. ### Case study: Qwen2.5-Math-7B DAPO, fully-async on MI300 We reproduced verl's fully-async experiments on AMD MI300: DAPO post-training of Qwen2.5-Math-7B on the MATH task. Key parameters are defined as follows: | Parameter | Value | | --- | --- | | Rollout engine | vLLM, AITER enabled, group size n = 16 | | Training backend | FSDP2 | | Max prompt / response length | 2,048 / 28,672 (28k) tokens | | PPO mini-batch size | 32 | | Learning rate | 1e-6 | | Execution mode | Fully-async — 16 rollout GPUs + 16 training GPUs (32 total) | | `staleness_threshold` | 0.5 | | `trigger_parameter_sync_step` | 4 | | `partial_rollout` | True | The training curves tell the story: {{< figure src="qwen2.5_7b.jpg" alt="Four panels over 400 logged steps: MATH validation accuracy, mean training reward, wall-clock time per step, and rollout-to-actor KL" caption="Qwen2.5-Math-7B DAPO on MATH. vLLM with AITER, FSDP2 trainer, fully-async on 32 MI300 GPUs (16 rollout + 16 training)." >}} 1. **Convergence is clean, and the final score matches the reference.** Validation accuracy climbs from 0.13 to a peak of 0.34 and holds around 0.32, while mean training reward rises from −0.98 to +0.17 in lockstep — no reward-hacking, no collapse. Critically, these numbers land right on top of verl's published reference for the same configuration (`staleness_threshold = 0.5`, partial rollout): the upstream [fully-async-policy results](https://github.com/verl-project/verl/tree/main/verl/experimental/fully_async_policy#experiments) report max accuracy 0.3302 for the same 16+16 GPU allocation on H20, while on AMD we reproduce the result with the same training settings and measure 0.34. 2. **The KL curve confirms stable off-policy training.** Rollout↔actor KL (`actor/ppo_kl`) sits at ~0.0007 for the first ~100 logged steps, then rises and settles onto a controlled 0.15–0.25 plateau once staleness kicks in. The run is stable end to end, with no explosive divergence despite training on deliberately stale samples. 3. **Fully-async is dramatically faster per training step.** Against a colocated DAPO run of the same model on the same 32 GPUs, we compare wall-clock time per training step over the first 64 steps: | | Fully-async | Colocated | | --- | --- | --- | | Step time (median) | ~219 s | ~394 s | | Step time (mean) | ~249 s | ~400 s | | Speed-up per step | ~1.8× | 1.0× (baseline) | A training step of fully-async is ~1.8× faster than the colocated baseline. In this experiment we verified on the AMD platform that fully-async mode achieves the expected accuracy and performance speedup that were originally reported on H20. {{< note type="info" title="One honest caveat" >}} In the step-time panel above, per-step time is flat at ~190–200 s for the first ~260 steps, then climbs into the 600–1,200 s range toward the end of training. This isn't a regression or instability — it's the expected response-length growth of reasoning RL: as model capacity saturates, the model only learns to increase the response length without improving accuracy. We observe its average response roughly double (from ~890 to ~1,850 tokens) during the late stage of the experiment. The same pattern shows up in the reported H20 experiment data. {{< /note >}} ### Case study: Qwen3.5-35B-A3B MoE on MI350 Now a more complex case: GRPO post-training of Qwen3.5-35B-A3B on the geometry3k reasoning task on MI350, with the Megatron backend, validated in both colocated and fully-async modes. | | Colocated | Fully-async | | --- | --- | --- | | Rollout / training placement | Shared 8 GPUs | 4 GPUs rollout + 4 GPUs training | | Megatron parallelism | TP 2, expert-parallel 8 | TP 2, expert-parallel 4 | | Async knobs | N/A | `staleness_threshold=0.5`, sync every 4 steps | | Rollout engine | vLLM + AITER, group size n = 5 | vLLM + AITER, group size n = 5 | | Max response length | 2,048 tokens | 2,048 tokens | Both modes deliver clean, stable convergence: {{< figure src="qwen3.5_35b_a3b.jpg" alt="Four panels comparing colocated and fully-async over 150 training steps: geometry3k validation accuracy, mean training reward, rollout-to-actor KL, and per-GPU throughput" caption="Qwen3.5-35B-A3B GRPO on geometry3k. vLLM with AITER, Megatron backend, 8xMI350, colocated versus fully-async." >}} - **Validation accuracy climbs steadily.** Both modes rise from a ~0.37–0.48 start to a peak of ~0.77–0.78 within the first ~75 steps and hold there. - **Training reward rises in lockstep**, from ~0.34 to ~0.82 in both modes, tracking the validation gains rather than reward-hacking. - **Rollout↔actor KL stays tiny** — ≈0.001 for fully-async and ≈0 for colocated — meaning the AITER-accelerated vLLM rollout and the Megatron training policy stay numerically almost identical. - **Fully-async sustains ~310 tok/s/GPU versus colocated's ~216** — roughly 1.4× higher per-GPU throughput. That's exactly the payoff of decoupling. So a 35B-parameter MoE model trained end-to-end with GRPO on AMD MI350 converges smoothly to strong accuracy in both colocated and fully-async modes. ## The bottom line RL post-training on AMD Instinct is production-ready: 1. **Effortless setup** — a turnkey ROCm container with the full stack baked in, so there are no dependencies to wrangle, and end-to-end CI on real MI300 hardware keeping it solid (with coverage still expanding). 2. **Two rollout backends** — both vLLM and SGLang, AITER-accelerated out of the box, in both colocated and fully-async modes. 3. **Validated accuracy** — confirmed on MI300 and MI350 series, in both execution modes, with AITER-accelerated rollout delivering speed without sacrificing convergence. Looking ahead, AMD will continue working closely with the verl community to support emerging models and capabilities, including architectures such as DeepSeek-V4, agentic RL workloads, and integration with platforms such as VeOmni. We will also improve rollout efficiency through technologies such as speculative decoding and FP4 inference. Our goal is simple: make the latest verl models and features work out of the box on AMD Instinct GPUs, with strong performance, validated accuracy, and a reliable experience for customers. --- ## File: content/posts/2026-08-17-verl-omni-v0-2-0/index.md --- title: "VeRL-Omni v0.2.0: Faster Diffusion RL and Stable Omni Training" date: 2026-08-17 authors: - "Yongxiang Huang" - "Didan Deng" - "Kawai Cheung" - "Nguyen Long" - "Jingan Zhou" - "Yingshu Chen" - "Ziyang Hong" summary: "A release focused on higher-throughput diffusion rollout, reusable omni adapters, and broader recipe coverage." image: "cover.png" tags: - release - performance - recipes - training - rollout math: false toc: true --- VeRL-Omni `v0.2.0` establishes a stronger foundation for production-grade omni-modal reinforcement learning. This release improves the training stack across rollout performance, model integration, reward support, hardware coverage, and documentation, with two changes carrying the most impact: - Faster diffusion RL, centered on higher-throughput Qwen-Image FlowGRPO rollout and V1 trainer support. - Stable omni training, built around the omni V1 trainer, reusable model adapters, FSDP2, and vLLM-Omni rollout. {{< figure src="verl_omni_v0_2_0_blog_overview.png" alt="VeRL-Omni v0.2.0 release overview" caption="VeRL-Omni v0.2.0 release overview" width="100%" >}} ## 1. Faster Diffusion RL Diffusion RL is expensive, but not in the same way as autoregressive language-model RL. A single rollout carries many denoising steps, large latent tensors, prompt embeddings, optional classifier-free guidance, reward-model scoring, old-log-prob recomputation, and policy-weight synchronization. For Qwen-Image FlowGRPO, there is no single villain in the profile. Step time is shaped by rollout generation, old-log-prob computation, reward scoring, actor update, and LoRA weight sync together. ### Key Features The faster diffusion RL work has two main features. - Request-level batching leads the rollout side. For supported diffusion adapters, it becomes the default batching path. Instead of sending diffusion generations through a serial loop, the engine now acts more like a traffic controller, with explicit concurrency knobs for scheduling rollout work. - The trainer path matters just as much. Diffusion now has a V1 trainer path, bringing diffusion RL closer to the modern trainer architecture used elsewhere in VeRL-Omni and laying the groundwork for decoupled rollout and training execution. Faster rollout only matters if the generated trajectories and log-probs still describe the same policy. This release fixes several correctness-sensitive areas: request-batched diffusion log-probs, async rollout semantics, rank-local LoRA weight-update routes, and the hooks used by optional rollout-correction recipes. ### Newly Support The [rollout batching guide](https://verl-omni.readthedocs.io/en/latest/start/rollout_batching.html) explains both diffusion batching modes, how to enable them, and when to choose each mode. Current faster diffusion RL support is organized around these recipes: | Model x Algorithm | Acceleration / support | Script | W&B run | | --- | --- | --- | --- | | Qwen-Image x FlowGRPO LoRA | **request-level batching** | [script](https://github.com/verl-project/verl-omni/blob/main/examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora.sh) | [w&b run](https://wandb.ai/mikecheung/flow_grpo/runs/1vsrnhbd) | | Qwen-Image x FlowGRPO full model | step-wise continuous batching | [script](https://github.com/verl-project/verl-omni/blob/main/examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr.sh) | [w&b run](https://wandb.ai/andyzhou/VeRL-Omni-demo/runs/8p8y9olb) | | SD3.5 Medium x FlowGRPO LoRA, **V1 trainer** | **request-level batching**, sync mode | [script](https://github.com/verl-project/verl-omni/blob/main/examples/flowgrpo_trainer/sd35/run_sd35_medium_ocr_lora_v1.sh) | [w&b run](https://wandb.ai/mikecheung/flow_grpo/runs/h04p15jr) | | SD3.5 Medium x FlowGRPO LoRA, **V1 trainer** | **request-level batching**, `separate_async` | [script](https://github.com/verl-project/verl-omni/blob/main/examples/flowgrpo_trainer/sd35/run_sd35_medium_ocr_lora_v1_separate_async.sh) | [w&b run](https://api.wandb.ai/links/didan/kk5uxbmh) | A full diffusion post training support table in VeRL-Omni is available at [README.md](https://github.com/verl-project/verl-omni#model-and-algorithm-support-). ### Recipe and Benchmark The Qwen-Image LoRA OCR recipe is a good place to see the change. In the v0.1 line, rollout was the core bottleneck: each request effectively ran as serial `B≈1` DiT forwards, with 10 denoising steps and True-CFG doubling each step into two forwards. GPU utilization hovered around `80%`, not because the model was small, but because the engine could not keep enough diffusion work packed together. In `v0.2.0`, request-level packing changes that shape. Multiple complete requests are packed into one transformer forward, GPU utilization rises to about `100%`, and isolated generation time drops from `226s` to `108s`, a `52%` reduction. The same story shows up in per-image generation latency, which falls with the packed rollout path. Reference runs: [Qwen-Image OCR LoRA v0.1](https://wandb.ai/mikecheung/flow_grpo/runs/o7x44yrr) and [Qwen-Image OCR LoRA v0.2](https://wandb.ai/mikecheung/flow_grpo/runs/1vsrnhbd). In the charts below, the blue curve is `v0.1` and the green curve is `v0.2`. {{< figure src="qwen-image-gpu-utilization.svg" alt="Qwen-Image FlowGRPO GPU utilization" caption="GPU utilization rises after request-level packing. Blue: v0.1; green: v0.2." >}} {{< figure src="qwen-image-timing-gen.svg" alt="Qwen-Image FlowGRPO generation time" caption="Generation time drops from the v0.1 path to the v0.2 path. Blue: v0.1; green: v0.2." >}} {{< figure src="qwen-image-timing-step.svg" alt="Qwen-Image FlowGRPO step time" caption="Step time follows the same trend. Blue: v0.1; green: v0.2." >}} The production-style Qwen-Image FlowGRPO recipes enable request-level batching by default. The main entry points are `run_qwen_image_ocr_lora.sh` for the baseline OCR reward setup. The request-level rollout batching is enabled by default: ```bash actor_rollout_ref.rollout.step_execution=false ++actor_rollout_ref.rollout.engine_kwargs.vllm_omni.max_num_seqs=32 ``` For Qwen-Image LoRA with True-CFG at 512 px, a practical tuning range is `max_num_seqs=8` to `32`; larger values can run into HBM pressure. SD3.5 has a lighter request-level memory shape and can use `max_num_seqs=256`. The recipe-level step-time numbers line up with that story: the baseline Qwen-Image FlowGRPO LoRA run is about `420s` per step on 4 × H800, while the async reward variant reaches about `360s` per step on 5 GPUs. ## 2. Stable Omni Training The other half of the release is stable omni training. Omni models are not just bigger language models; they are small systems with processors, modality-specific towers, trainable stages, and rollout-time behavior that has to stay aligned with the actor. `v0.2.0` moves the project from model-specific integrations toward a reusable omni training stack, so multimodal autoregressive training fits more naturally into VeRL-Omni's trainer, adapter, rollout, and recipe structure. ### Key Features Here, the release pulls on two levers. One lever is the `verl` V1 trainer architecture. Omni recipes get clearer worker orchestration, standard configuration overrides, and better alignment with vLLM-Omni rollout. The other is the reusable omni model adapter layer. Instead of wiring each architecture as a one-off path, the trainer can rely on a shared interface for model setup, processor setup, trainable-stage selection, FSDP preparation, and rollout alignment. The repository-level call flow is roughly: {{< figure src="omni-ppo-adapter-flow.svg" alt="Omni PPO trainer and OmniModelBase adapter call flow" caption="Omni PPO trainer and OmniModelBase adapter call flow." >}} The module boundary is intentionally narrow. `main_omni.py` only decides that an online omni job should enter the verl PPO V1 path. The PPO trainer then owns the generic RL loop: rollout scheduling, advantage computation, and policy updates. When the actor model is built, the FSDP omni engine loads the Hugging Face model and asks `OmniModelBase` to resolve the adapter for the configured architecture and stage. That adapter is where model-specific work lives. For Qwen3-Omni thinker training, `Qwen3OmniThinkerAdapter` strips inactive modules, redirects `forward` to the thinker component, and prepares the processor and rollout alignment hooks before control returns to the PPO loop. ### Newly Support The current Qwen3-Omni adapter supports thinker-only training by redirecting training to the target component, stripping unused modules such as Talker and codec-related components, and working with FSDP/FSDP2 wrapping. Current stable omni training support is organized around these recipes: | Model x Algorithm | Modality / dataset | Support | Script | W&B run | | --- | --- | --- | --- | --- | | Qwen3-Omni Thinker x GSPO | text -> text / GSM8K | **V1 trainer**, reusable omni adapter, FSDP2, vLLM-Omni rollout | [script](https://github.com/verl-project/verl-omni/blob/main/examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_v1.sh) | [w&b run](https://wandb.ai/mikecheung/gspo/runs/j5mro1tn) | | Qwen3-Omni Thinker x GSPO | image -> text / MMK12 | **V1 trainer**, multimodal data, actor-rollout consistency signals | [script](https://github.com/verl-project/verl-omni/blob/main/examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh) | [w&b run](https://wandb.ai/mikecheung/gspo/runs/2j8hxr36) | | Qwen3-Omni Thinker x GSPO | text + image + audio -> text / AVQA-R1-6K | **V1 trainer**, NPU recipe, multimodal inputs | [script](https://github.com/verl-project/verl-omni/blob/main/examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_npu_avqa_v1.sh) | - | | Qwen3-Omni Thinker x DPO | multimodal -> preference / Omni-Preference | `OmniDPOLoss`, modality-grouped batches | [script](https://github.com/verl-project/verl-omni/blob/main/examples/dpo_trainer/qwen3_omni/qwen3_omni/run_qwen3_omni_omni_preference_lora.sh) | [w&b report](https://api.wandb.ai/links/didan/iumxl2zr) | A full omni post training support table in VeRL-Omni is available at [README.md](https://github.com/verl-project/verl-omni#model-and-algorithm-support-). ### Recipe and Benchmark The best single recipe to highlight is **MMK12**. It exercises the new stable Qwen3-Omni path with real multimodal input: image plus text prompt, text answer, GSPO optimization, FSDP actor training, and vLLM-Omni rollout. **MMK12 anchor recipe.** `run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh` trains Qwen3-Omni on K12 visual math reasoning (`image -> text`) with GSPO, LoRA rank 32, and colocated actor-rollout workers on 4 × H800 80GB. The rollout shape is 128 prompts × 16 responses, or 2048 samples per rollout. After training, the run reaches `0.833` validation reward, `0.998` actor-rollout Pearson correlation, and about `59 GB` GPU memory usage. See some training results in the reference run: [`MMK12 (wandb)`](https://wandb.ai/mikecheung/gspo/runs/2j8hxr36). {{< figure src="mmk12_training_rewards.svg" alt="MMK12 training rewards mean scores" caption="MMK12 training rewards mean scores." >}} {{< figure src="mmk12_val_rewards.svg" alt="MMK12 validation rewards mean scores" caption="MMK12 validation rewards mean scores." >}} The MMK12 data pipeline converts raw MMK12 parquet shards into the verl RL parquet layout. Each row carries the image bytes inline and uses a prompt format that asks the model to produce a structured answer. The reward combines `math_verify` accuracy with a progressive format reward on the `...\boxed{}...` template. To run the recipe: ```bash python examples/gspo_trainer/data_process/mmk12.py \ --local_dataset_path /path/to/mmk12/ \ --local_save_dir ~/data/mmk12 TRAIN_FILE=$HOME/data/mmk12/train.parquet \ VAL_FILE=$HOME/data/mmk12/test.parquet \ bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh ``` This anchors the `v0.2.0` stability story: Qwen3-Omni training is no longer just a model-specific launch path. It is a V1 trainer recipe with a reusable omni adapter, multimodal data handling, actor-rollout consistency metrics, and a documented image-to-text benchmark. ## Model and Algorithm Extensions The release also expands the broader VeRL-Omni model and algorithm surface: | Model / family | Category | Modality | Algorithm / recipe | Update | |---|---|---|---|---| | [LTX2.3](https://github.com/verl-project/verl-omni/blob/main/examples/flowgrpo_trainer/ltx2/README.md) | Diffusion generator | Text -> Video + Audio | FlowGRPO | Adds text-to-video+audio training with CLAP and ImageBind rewards. | | [Qwen-Image-Edit](https://github.com/verl-project/verl-omni/blob/main/examples/flowgrpo_trainer/qwen_image_edit/README.md) | Diffusion image editor | Text + Image -> Image | FlowGRPO | Adds image-editing data preparation and a general edit-training interface. | | [BAGEL](https://github.com/verl-project/verl-omni/blob/main/examples/flowgrpo_trainer/bagel/README.md) | Unified understand + generation model | Text + Image | FlowGRPO | Adds full-parameter and LoRA recipes with OCR and PickScore rewards. | | [SD3.5 + DiNa-LRM](https://verl-omni.readthedocs.io/en/latest/examples/flowgrpo_trainer_sd35_drm.html) | Diffusion generator | Text -> Image | FlowGRPO with latent reward model | Scores clean diffusion latents directly, avoiding VAE decode during reward scoring. | | [Flow-DPPO](https://verl-omni.readthedocs.io/en/latest/algo/flowdppo.html) | Diffusion generator algorithm | Text/Image -> Image | Flow-DPPO | Adds an alternative policy-optimization recipe for Qwen-Image style diffusion RL. | | [Wan2.2](https://github.com/verl-project/verl-omni/blob/main/examples/dancegrpo_trainer/README.md) | Diffusion video generator | Text -> Video | DanceGRPO | Adds video-generation RL recipe coverage. | Outside the model-algorithm matrix, `v0.2.0` also adds Ascend NPU Dockerfiles and install guidance. ## Future Plan - Optimize omni-modal models via fully async training. - Extend new models and algorithms, such as MiniMax-H3, MiniCPM-o models, and OPD/M-OPD trainers. - Make video diffusion model training more efficient via batching, TQ, and the V1 trainer. - Harden diffusion and omni-modal rollout code for async training. - Support agentic RL with multi-stage and multi-turn generation. --- ## File: content/posts/_index.md --- title: "All posts" summary: "All posts." --- Everything published on this blog, newest first. Posts are written by verl maintainers and contributors; if you have shipped something worth explaining, [here is how to write it up](/contribute/). --- ## File: content/contribute.md --- title: "Write for the verl blog" summary: "How to propose, write and ship a post about work you have done in verl." --- If you have landed something in verl and want to explain how it works, this blog is the place to write it up. {{< note type="tip" title="Talk to us before you write" >}} Open an issue with your idea before you start drafting. A maintainer can tell you early whether the topic fits and whether somebody else is already writing it, which is a lot cheaper than finding out once the draft is done. {{< /note >}} ## What we publish Posts that help someone else understand verl better. In practice that tends to be one of: - **Feature deep dives.** You landed something non-trivial and want to explain the design, the trade-offs you rejected, and how to use it. - **Performance work.** A change with numbers attached. Say what hardware, what model, what batch shape, and what you measured against. - **Recipes and case studies.** An end-to-end account of training something real, including the parts that did not work at first. - **Release notes with substance.** What actually changed and why it matters. - **… and anything else worth explaining.** Treat the list as a starting point rather than a set of boxes to fit into. If you are unsure whether an idea fits, open an issue and ask. One thing this blog is not is a substitute for the [documentation](https://verl.readthedocs.io/en/latest/). Docs describe how the current version behaves and get updated as it changes; a post captures the reasoning behind a change at the time it was made. Reference material belongs in the docs — and both are good places to contribute. ## The short version 1. Open an issue on the [blog repository](https://github.com/verl-project/verl-project.github.io/issues). 2. Fork the repository and create a directory under `content/posts/`, named `YYYY-MM-DD-short-slug`. 3. Write `index.md` inside it. Put images in the same directory. 4. Open a pull request. CI checks the front matter and builds the site. 5. A maintainer reviews for accuracy and clarity. Once it is merged, the site redeploys automatically and your post is live within a couple of minutes. ## Writing the post Every post is a directory, not a loose file, so that its images travel with it: ```text content/posts/2026-08-04-your-post-slug/ ├── index.md ├── cover.png └── throughput-comparison.png ``` The front matter at the top of `index.md` looks like this: ```yaml --- title: "Async rollout in verl" date: 2026-08-04 authors: - "Jane Doe" summary: "Async rollout overlaps generation with training and cuts idle GPU time on long-horizon tasks." image: "cover.png" tags: - performance math: false toc: true --- ``` `title`, `date`, `authors` and `summary` are required; CI will reject the pull request without them. A few notes on the ones that trip people up: - **`summary`** must be 240 characters or fewer and cannot contain line breaks. It is the card blurb, the meta description and the social card text all at once. Write what the post concludes, not what it is "about". - **`authors`** is a list. Use real names, GitHub handles, or a team name — whatever you want readers to see. Each value gets an archive page. - **`image`** is a filename relative to the post directory. Hugo resizes it and converts it to WebP at build time, so commit the original rather than a hand-shrunk copy. Aim for something around 1200×630 for the cover. - **`date`** in the future keeps the post unpublished until that date. This is the mechanism to use when a post has to land with a release. If you would rather not write the front matter by hand, scaffold it: ```bash hugo new content posts/2026-08-04-your-post-slug/index.md ``` ### Things the theme gives you Fenced code blocks are syntax highlighted. Line numbers are off by default, since they get in the way of copying shell commands — add `{linenos=true}` after the language when you want to refer to a specific line. Plain markdown images are turned into captioned figures automatically, with the alt text becoming the caption. For anything more specific there are two shortcodes: ```markdown {{}} {{}} This API changed in v0.8. On earlier versions, use `foo` instead. {{}} ``` Set `math: true` in the front matter to load KaTeX, then write `$$...$$` for display equations and `\(...\)` inline. ### House style We are not precious about this, but a few things make review faster: - **Lead with the outcome.** State the result in the first paragraph. Save the background for later — readers who need it will keep reading. - **Numbers need context.** Hardware, model, sequence length, batch size, baseline. A speedup with no baseline is not a result. - **Link to the code.** Pull requests, issues and file paths let readers verify what you are describing and go deeper. - **Write for someone who knows RL but not your subsystem.** Expand an acronym the first time you use it. - **English is the default.** ## Previewing your work You need [Hugo](https://gohugo.io/installation/) v0.153.0 or newer. It is a single binary with no runtime dependencies: ```bash git clone https://github.com/verl-project/verl-project.github.io.git cd verl-project.github.io hugo server --buildDrafts ``` The site is then at `http://localhost:1313` and reloads as you save. Drafts and future-dated posts only appear with `--buildDrafts` and `--buildFuture`. ## Copyright You keep the copyright in your post. Opening a pull request grants verl-project the right to publish it here under your byline, and that is all it grants: you stay free to put the same piece on your own site or anywhere else. The post remains attributed to you indefinitely.