vllm-omni Documentation

1. Project Overview & Quickstart (vllm-project/vllm-omni) Copy Chapter

File: README.md Copy Chapter

Easy, fast, and cheap omni-modality model serving for everyone

Latest News 🔥

• [2026/08] VeRL-Omni v0.2.0 is released: faster diffusion RL powered by vLLM-Omni (request-level/step-wise batching with FA3), rebuilt Qwen3-Omni multimodal training (DPO & GSPO), plus LTX-2.3, Qwen-Image-Edit support and more. See the release notes.
• [2026/08] We released 0.26.0 - aligned with the vLLM 0.26 release line, featuring MiniMax H3 joint video/audio generation, an experimental full-duplex realtime runtime for MiniCPM-o 4.5, distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
• [2026/07] We released 0.24.0 - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
• [2026/06] Starting with 0.14.0, vLLM-Omni publishes a stable release aligned with every even-numbered upstream vLLM minor version. 0.16.0, 0.18.0, 0.20.0, and 0.22.0 continued this cadence, expanding omni and world-model support with NVIDIA Cosmos3 and DreamZero, adding models such as MiniCPM-o 4.5, MOSS-TTS, and Lance, and advancing TTS, diffusion, distributed execution, quantization, RL integration through VeRL-Omni, and CUDA/ROCm/MUSA/NPU/XPU coverage.
• [2026/03] Check out our first public project deepdive at the vLLM Hong Kong Meetup!
• [2025/11] vLLM community officially released vllm-project/vllm-omni in order to support omni-modality models serving.

About Copy Chapter

vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:

• Omni-modality: Text, image, audio, video, and action data processing
• Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
• Heterogeneous outputs: from traditional text generation to multimodal and action outputs

vLLM-Omni is fast with:

• State-of-the-art AR support by leveraging efficient KV cache management from vLLM
• Pipelined stage execution overlapping for high throughput performance
• Fully disaggregation based on OmniConnector and dynamic resource allocation across stages

vLLM-Omni is flexible and easy to use with:

• Heterogeneous pipeline abstraction to manage complex model workflows
• Seamless integration with popular Hugging Face models
• Tensor, pipeline, data and expert parallelism support for distributed inference
• Streaming outputs
• OpenAI-compatible API server
• Full-duplex realtime serving with streaming audio input and output (experimental)

vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:

• Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
• TTS models (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
• Diffusion models — image, video, and audio generation (e.g. MiniMax H3, Qwen-Image, Wan2.2, FLUX)
• Robot-policy and action models (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)

Getting Started Copy Chapter

Visit our documentation to learn more.

• Installation
• Quickstart
• List of Supported Models
• Deployment Recipes for vLLM-Omni model serving

Contributing Copy Chapter

We welcome and value any contributions and collaborations.
Please check out Contributing to vLLM-Omni for how to get involved.

Citation Copy Chapter

If you use vLLM-Omni for your research, please cite our paper:

@article{yin2026vllmomni,
  title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
  author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu},
  journal={arXiv preprint arXiv:2602.02204},
  year={2026}
}

Join the Community Copy Chapter

Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in #sig-omni slack channel at slack.vllm.ai or vLLM user forum at discuss.vllm.ai.

Star History Copy Chapter

License Copy Chapter

Apache License 2.0, as found in the LICENSE file.

File: .claude/skills/readme.md Copy Chapter

Repository Skills for vLLM-Omni

This directory contains repository-scale skills maintained for vllm-omni.
They capture repeatable workflows for common contributor and maintainer tasks
such as model integration, CI-aligned testing, performance optimization, and
pull request review.

Directory Structure Copy Chapter

Each skill lives in its own directory under .claude/skills/. A skill may
include:

• SKILL.md: the main workflow and operating instructions
• references/: focused reference material used by the skill
• scripts/: small helper scripts used by the skill

Using the Skills Copy Chapter

Coding agents that discover repository skills can invoke the relevant skill by
name. For other agents, point them to the skill's SKILL.md and ask them to
read it before changing code. Combine a domain skill with vllm-omni-test when
tests or CI coverage change, then use precheck-pr for the contributor's final
self-check.

Available Skills Copy Chapter

• add-diffusion-model: guides integration of
  a new diffusion model into vllm-omni
• add-tts-model: covers integration of new TTS
  models and related serving workflows
• diffusion-perf-opt: guides diffusion model
  performance optimization, including profiling traces, parallel strategies,
  stage timing analysis, and benchmark-driven tuning
• precheck-pr: self-checks a branch before creating a
  PR by validating title format, dead code, accuracy and performance claims,
  and merge readiness
• quantization: guides quantization method selection,
  model integration, checkpoint loading, and quality/performance validation
  for vLLM-Omni
• review-pr: provides a frozen-snapshot, contract-aware
  workflow for maintainers and reviewers; contributors should use
  precheck-pr for self-review
• vllm-omni-npu-model-runner-upgrade:
  upgrades NPU model runners to align with the latest vllm-ascend
  NPUModelRunner
• vllm-omni-test: guides generation and execution
  of CI-aligned tests (L1–L4), pytest marker selection (core_model /
  advanced_model / full_model, omni / tts / diffusion), Buildkite
  wiring (test-ready.yml, test-merge.yml, test-nightly.yml,
  test-weekly.yml), and copy-paste local plus CI-like pytest commands; see
  references/test-routing.md for level-to-command mapping

Maintenance Guidelines Copy Chapter

• Keep skill names short and task-oriented.
• Prefer repository-local paths, commands, and examples.
• Avoid hardcoding fast-changing support matrices unless the skill is actively
  maintained alongside those changes.
• Treat skills as contributor tooling: optimize for clarity, actionability, and
  low maintenance overhead.

File: apps/lvsa-omni-showcase/README.md Copy Chapter

LVSA Long-Video Showcase

Training-free block-sparse attention for long-video diffusion, running on
vLLM-Omni via the public plugin entry points (no core changes). LVSA speeds up
extended-horizon generation and prevents the freeze/loop failure mode dense
attention exhibits beyond a model's training length — with no fine-tuning and no
weight changes.

Upstream: LongVideoSparseAttention
(algorithm, all model adapters, full docs).

Results Copy Chapter

Wan 2.1 1.3B, single A100 80 GB, mean over 5 prompts (Dense vs LVSA-FlashInfer):

The speedup grows with length (LVSA is the dense regime at 1× by design, so
the win starts at 2×), and quality improves at extension — LVSA's rotating
keyframes prevent the looping/static output dense produces beyond its horizon
(VQeval loop-quality +27 to +34 at ≥3×). At long horizons dense OOMs on 80 GB
(HunyuanVideo ≥2×, Wan2.1-14B and Cosmos at 6×) where LVSA still generates.

→ Full sweep (5 models × 6 horizons, VQeval + VBench-Long) and the
quality-metric methodology:
benchmarks/.

Quickstart Copy Chapter

# 1. vLLM-Omni 0.22.0 (stable) + matching vLLM
pip install "vllm==0.22.0"
pip install --no-build-isolation \
  "vllm-omni @ git+https://github.com/vllm-project/[email protected]"

# 2. LVSA core + plugin
git clone https://github.com/JiusiServe/LongVideoSparseAttention.git
( cd LongVideoSparseAttention && pip install -e . && pip install -e lvsa-vllm-omni/ )

# 3. this showcase's client dep
pip install requests

FlashInfer (used by the default flashinfer backend) ships with vLLM — step 1
already pulls the pinned flashinfer-python + flashinfer-cubin, so the fused
backend works with no extra install. Two optional extras:

# (optional) faster startup — prebuilt JIT cache, matched to your CUDA + flashinfer
CU=cu128   # match your CUDA: cu121 / cu124 / cu128 / …
FIVER=$(python -c "import importlib.metadata as m; print(m.version('flashinfer-python'))")
pip install --extra-index-url "https://flashinfer.ai/whl/${CU}" \
  "flashinfer-jit-cache==${FIVER}+${CU}"

# (fallback) no FlashInfer in your environment? run the SDPA backend instead:
export LVSA_BACKEND=sdpa

# Serve an LVSA-enabled endpoint (Wan 2.1 1.3B at 2× horizon):
MODEL=/path/to/Wan2.1-T2V-1.3B-Diffusers MODEL_FAMILY=wan FRAMES=161 \
  bash serve_lvsa.sh

# In another shell — generate:
python generate.py --frames 161 --height 480 --width 832 \
  --prompt "A dog running through a sunlit forest." --out wan_lvsa.mp4

serve_lvsa.sh sets the LVSA env vars for the chosen MODEL_FAMILY
(wan / hunyuan). Wan and HunyuanVideo run through the LVSA attention
backend by default — it stays sparse even under sequence-parallel, unlike the
monkey-patch hooks.

Configuration Copy Chapter

The one knob you must get right is the model's training horizon
(LVSA_REFERENCE_LATENT_FRAMES: Wan = 21, HunyuanVideo = 33); serve_lvsa.sh
sets it per family. For the full environment-variable reference, the multi-GPU
support matrix, and tuning (sparsity_scale, rotating keyframes, non-standard
resolutions), see the upstream docs:

• lvsa-vllm-omni plugin reference — every LVSA_* env var
• docs/parallelism.md — TP / Ulysses / HSDP support per model
• docs/tuning.md — picking knobs for your model and horizon

File: apps/ComfyUI-vLLM-Omni/README.md Copy Chapter

vLLM-Omni

vLLM-Omni offers a ComfyUI integration on top of its online serving API.
It can send model inference requests to either a locally running vLLM-Omni service or a remote one.

Requirement Copy Chapter

• Python 3.12 or above
• ComfyUI installed
• vLLM-Omni installed on either the same device or another device discoverable via the internet.
• No need to install additional packages apart from those already required by ComfyUI.

[!TIP]
If you run both ComfyUI and vLLM-Omni on the same device, you can create separate virtual environments and use different Python versions for them.

Installation Copy Chapter

Copy this folder to the custom_nodes subfolder of your ComfyUI installation. Your directory should look like ComfyUI/custom_nodes/ComfyUI-vLLM-Omni.

If you are running ComfyUI during copying, you should restart ComfyUI to load this extension.

[!TIP]
You can use utility websites such as https://download-directory.github.io/ to download a subdirectory of a repo. Also checkout community discussions (e.g., https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repository) for more info.

On the device and virtual environment you run ComfyUI, launch ComfyUI with

cd ComfyUI

# The regular way
python main.py

# If you are mainly using this node, launch it faster with
python main.py --cpu

On the device and virtual environment you run vLLM-Omni, start a model service with

vllm serve The_Model_ID_to_Serve --omni --port 8000

Check ComfyUI's sidebar -> Node Library. There should be a new folder named vLLM-Omni.
If no, check your shell running the ComfyUI process. There may be some error messages before the line Import times for custom nodes: and the line To see the GUI go to: http://127.0.0.1:8188.

Quickstart Copy Chapter

This extension offers the following nodes based on the output modalities (at ComfyUI sidebar -> Node Library):

• Generate Image for text-to-image and image-to-image tasks
• Generate Video for text-to-video, first-frame/image-to-video, and reference-conditioned video
• Multimodality Understanding for multimodality-to-text and multimodality-to-audio tasks
• TTS and TTS Voice Clone for TTS tasks

This extension also offers example workflows (at ComfyUI sidebar -> Templates -> vLLM-Omni)

[!NOTE]
The node UI and feature designs are intended to match vLLM-Omni online serving interfaces. It cannot offer more than what the interfaces support.

To build a simple workflow yourself,

• Drag a generation node onto the canvas.
• Depending on your need, grab built-in multimedia file loader nodes, such as image->Load Image, image->video->Load Video, audio->Load Audio
• Depending on your need, grab built-in multimedia file preview nodes, such as image->Preview Image, image->video->Save Video, audio->Preview Audio, utils->Preview as Text.
• If you want to tune sampling parameters, grab corresponding nodes from vLLM-Omni-> Sampling Params.
  • For multi-stage models, you can connect multiple AR Sampling Params and Diffusion Sampling Params nodes to a Multi-Stage Sampling Params List node, and connect this node to the generation node.
  • For some multi-stage models like BAGEL, only one stage's sampling parameters are exposed and tunable via vLLM-Omni's online serving API. Thus, these models are treated as single-stage ones. Please check the vLLM-Omni documentation on how to correctly set each model's sampling parameters.
  • For multi-stage models where all stages are either autoregression or diffusion, you can also connect only a single Sampling Params node, indicating that this set of sampling parameters will be used for all stages.

Screenshots and Examples Copy Chapter

Multimodal understanding (e.g., Qwen Omni series, BAGEL) Copy Section

(Also available at ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Multimodal Understanding)

[!TIP]
Although this node enables all-modality input, you should check whether the specific model you host and request for supports the modalities you connect to the node.

You can configure per-stage sampling parameters for multi-stage models.

Text-to-image and image-to-image generation (e.g., Z-Image-Turbo, Qwen-Image-Edit, BAGEL) Copy Section

(Also available at ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Image Generation)

[!TIP]
The node automatically choose text-to-image or image-to-image API endpoints depending on whether you connect an image input or not.

Text-to-video and image-to-video generation (e.g., Wan, MiniMax-H3) Copy Section

(Also available at ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Video Generation)

[!TIP]
Connect a frame image for first-frame / image-to-video (e.g., Wan I2V, MiniMax-H3 FL2VA).

For reference-conditioned generation (MiniMax-H3 Ref2VA), connect a Video References node instead.

Do not use frame and references together. Task routing is automatic from which inputs you connect.

TTS (e.g., Qwen TTS series) Copy Section

(Also available at ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni TTS)

[!TIP]
There is a dedicated node for VoiceClone tasks with reference audio input. Other simple text-to-speech tasks should use the regular TTS node.

Chaining multiple model services Copy Section

(Also available at ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Chaining Services)

Develop Copy Chapter

Follow the development convention and rules of vLLM-Omni.

Limitation and Non-Goals Copy Chapter

• Single server mode only. No automatic load balancing or failover.
• Features set is bounded to vLLM-Omni's online service capability, including
  • The types of models supported in online mode,
  • The types of sampling parameters supported in the online mode,
  • The ways to send files (primarily through full-length base64 in JSON payload),
  • Figuring out errors in the payload (such as unsupported fields by a specific model) if the endpoint does not explicitly return an error,
  • (The lack of) Authentication
  • (The lack of) Progress indicator

Support Copy Chapter

If you are new to ComfyUI, please check out its documentation for usage instructions.

If you are new to vLLM-Omni, please also check out its documentation for usage instructions.

Whenever you find an issue or problem, please

• First find out if this is an upstream limitation of vLLM-Omni's online serving mode, by checking their documentation.
• Open an issue that clearly describes this ComfyUI or online service problem.

Acknowledgements Copy Chapter

Features

• https://github.com/dougbtv/comfyui-vllm-omni/ The official reference implementation for ComfyUI integration with vLLM-Omni's DALL-E compatible image generation API.
• https://github.com/Comfy-Org/ComfyUI/tree/master/comfy_extras ComfyUI's built-in node implementations.

UI/UX design references

• https://github.com/sgl-project/sglang/pull/15271 SGLang Diffusion's official ComfyUI integration for image and video generation.
• https://github.com/SXQBW/ComfyUI-Qwen-Omni A third party ComfyUI integration for Qwen Omni series.
• https://github.com/flybirdxx/ComfyUI-Qwen-TTS https://github.com/DarioFT/ComfyUI-Qwen3-TTS Third party ComfyUI integrations for Qwen TTS series.

File: benchmarks/accuracy/image_to_image/README.md Copy Chapter

GEdit-Bench on vLLM-Omni

This integration adapts the upstream stepfun-ai/Step1X-Edit/GEdit-Bench
evaluation flow into vllm-omni/benchmarks/accuracy/image_to_image.

Upstream mapping:

• run_gedit_score.py -> run_gedit_bench.py evaluate
• calculate_statistics.py -> run_gedit_bench.py summarize
• upstream output layout under results/<method>/fullset/... is preserved

What changed:

• The upstream repo mainly ships evaluation scripts. This integration adds a
  generation runner that uses the local vllm-omni OpenAI-compatible
  /v1/images/edits endpoint to produce benchmark outputs in the expected
  directory structure.
• The evaluator keeps the same VIEScore-style decomposition:
  • semantics_score
  • quality_score
  • overall_score = sqrt(semantics_score * quality_score)
• Judge calls are routed to a local OpenAI-compatible model served by
  vllm-omni, not a remote provider.

Dataset:

• Default --dataset-ref is stepfun-ai/GEdit-Bench
• You can also pass a local dataset directory previously saved with
  Hugging Face datasets

Example usage:

python benchmarks/accuracy/image_to_image/run_gedit_bench.py generate \
  --output-root benchmarks/accuracy/image_to_image/results \
  --base-url http://127.0.0.1:8000 \
  --model Qwen/Qwen-Image-Edit \
  --model-name qwen_image_edit \
  --dataset-ref stepfun-ai/GEdit-Bench \
  --task-type all \
  --instruction-language en

python benchmarks/accuracy/image_to_image/run_gedit_bench.py evaluate \
  --output-root benchmarks/accuracy/image_to_image/results \
  --model-name qwen_image_edit \
  --save-dir benchmarks/accuracy/image_to_image/scores \
  --dataset-ref stepfun-ai/GEdit-Bench \
  --judge-base-url http://127.0.0.1:8000 \
  --judge-model Qwen/Qwen2.5-VL-7B-Instruct \
  --judge-api-key EMPTY

python benchmarks/accuracy/image_to_image/run_gedit_bench.py summarize \
  --csv-path benchmarks/accuracy/image_to_image/scores/qwen_image_edit_all_all_vie_score.csv \
  --language en

Example summary output:

{
  "language": "all",
  "languages": {
    "en": {
      "overall": {"count": 110, "Q_SC": 6.58, "Q_PQ": 5.89, "Q_O": 5.86},
      "intersection": {"count": 78, "Q_SC": 6.50, "Q_PQ": 5.66, "Q_O": 5.65}
    },
    "cn": {
      "overall": {"count": 110, "Q_SC": 6.90, "Q_PQ": 5.78, "Q_O": 6.11},
      "intersection": {"count": 63, "Q_SC": 7.22, "Q_PQ": 5.59, "Q_O": 6.28}
    }
  }
}

Example generated images to inspect:

• benchmarks/accuracy/image_to_image/results/qwen_image_edit/fullset/background_change/en/<sample>.png
• benchmarks/accuracy/image_to_image/results/qwen_image_edit/fullset/text_change/en/<sample>.png
• benchmarks/accuracy/image_to_image/results/qwen_image_edit/fullset/subject-replace/cn/<sample>.png

Example score artifacts to inspect together with the images:

• benchmarks/accuracy/image_to_image/scores/qwen_image_edit_all_all_vie_score.csv
• benchmarks/accuracy/image_to_image/scores/qwen_image_edit_all_all_summary.json

What to expect:

• Q_SC measures instruction following and content preservation.
• Q_PQ measures image naturalness and artifact quality.
• Q_O is the combined overall score; higher is better.
• overall.count is the number of evaluated samples for that language, while intersection.count is the subset with intersection_exist == True.

Notes:

• This flow requires the optional Hugging Face datasets package.
• generate writes generation_manifest.json with local output coverage.
• The current repo marker set exposes L4 but not L5, so if you promote an
  end-to-end smoke test into CI, use the full_model, benchmark,
  and L4 markers for nightly (or advanced_model for merge) or introduce a new repo-wide marker explicitly first.

File: benchmarks/accuracy/text_to_image/README.md Copy Chapter

GEBench on vLLM-Omni

This integration adapts the upstream stepfun-ai/GEBench scripts into
vllm-omni/benchmarks/accuracy/text_to_image.

Upstream mapping:

• scripts/generate.py -> run_gebench.py generate
• scripts/evaluate.py -> run_gebench.py evaluate
• upstream prompt / judge logic -> gbench.py

What changed:

• Generation calls the local OpenAI-compatible vllm-omni endpoints:
  • /v1/images/generations for text-only frame generation
  • /v1/images/edits for image-conditioned GUI transition generation
• Evaluation still keeps the GEBench scoring dimensions:
  • goal
  • logic
  • cons
  • ui
  • qual
• Judge calls are also routed to a local OpenAI-compatible model served by
  vllm-omni instead of a remote service.

Expected dataset layout:

• Clone the benchmark dataset from Hugging Face into a local directory:

git clone https://huggingface.co/datasets/stepfun-ai/GEBench /path/to/GEBench

Example usage:

python benchmarks/accuracy/text_to_image/run_gebench.py generate \
  --dataset-root /path/to/GEBench \
  --output-root benchmarks/accuracy/text_to_image/outputs \
  --base-url http://127.0.0.1:8000 \
  --model Tongyi-MAI/Z-Image-Turbo \
  --data-type type3

python benchmarks/accuracy/text_to_image/run_gebench.py evaluate \
  --dataset-root /path/to/GEBench \
  --output-root benchmarks/accuracy/text_to_image/outputs \
  --data-type type3 \
  --judge-base-url http://127.0.0.1:8000 \
  --judge-model Qwen/Qwen2.5-VL-7B-Instruct \
  --judge-api-key EMPTY

python benchmarks/accuracy/text_to_image/run_gebench.py summarize \
  --output-root benchmarks/accuracy/text_to_image/outputs

Example summary output:

{
  "generation": {
    "count": 20,
    "by_type": {
      "type3": {"count": 10},
      "type4": {"count": 10}
    }
  },
  "evaluation": {
    "count": 20,
    "overall_mean": 0.52,
    "by_type": {
      "type3": {"count": 10, "overall_mean": 0.50, "overall_mean_100": 50.0},
      "type4": {"count": 10, "overall_mean": 0.54, "overall_mean_100": 54.0}
    }
  }
}

Example generated images to inspect:

• benchmarks/accuracy/text_to_image/outputs/03_trajectory_text_fictionalapp/<lang_device>/<sample>/frame0.png
• benchmarks/accuracy/text_to_image/outputs/03_trajectory_text_fictionalapp/<lang_device>/<sample>/frame5.png
• benchmarks/accuracy/text_to_image/outputs/04_trajectory_text_realapp/<lang_device>/<sample>/frame0.png
• benchmarks/accuracy/text_to_image/outputs/04_trajectory_text_realapp/<lang_device>/<sample>/frame5.png

What to expect:

• overall_mean is normalized to 0.0 ~ 1.0; higher is better.
• frame0.png is the initial GUI frame; frame5.png is the final trajectory frame most often used for quick inspection.
• For full debugging, inspect the whole frame0.png ... frame5.png sequence for one sample directory.

Notes:

• GEBench upstream leaves type3/type4 generation unfinished. This integration
  fills that gap with a trajectory runner that generates frame0.png followed
  by frame1.png ... frame5.png.
• Type1/2/5 require an image-edit capable model exposed through
  vllm-omni serve.
• summarize will report both generated coverage and any existing evaluation
  summary files.

File: benchmarks/accuracy/README.md Copy Chapter

Accuracy Benchmarks

This directory hosts accuracy benchmark integrations that run entirely through a
local vllm-omni serve deployment.

Current integrations:

• text_to_image/: GEBench generation + local judge scoring flow.
• image_to_image/: GEdit-Bench generation + local VIEScore-style scoring flow.

Design notes:

• Generation is executed through the OpenAI-compatible endpoints exposed by
  vllm-omni serve.
• Evaluation is also executed through a local OpenAI-compatible judge model
  served by vllm-omni.
• Both generation and judge requests accept either http://host:port or
  http://host:port/v1.
• Output directory layout intentionally stays close to the upstream repos.

Test guidance:

• Local static/self-checks live in tests/benchmarks/test_accuracy_bench_utils.py.
• End-to-end generation/evaluation should be validated in a remote GPU
  environment. In the current repo marker system there is L4 but no L5
  marker, so benchmark smoke tests should be wired as full_model + benchmark + L4 for nightly when GPU capacity is available.

File: benchmarks/diffusion/README.md Copy Chapter

Diffusion Serving Benchmark (Image/Video)

This folder contains an online-serving benchmark script for diffusion models.
It sends requests to a vLLM OpenAI-compatible endpoint and reports throughput,
latency percentiles, and optional SLO attainment.

The main entrypoint is:

• benchmarks/diffusion/diffusion_benchmark_serving.py

1. Quick Start Copy Chapter

1. Start the server:

vllm serve Qwen/Qwen-Image --omni --port 8099

2. Run a minimal benchmark:

python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
	--base-url http://localhost:8099 \
	--model Qwen/Qwen-Image \
	--task t2i \
	--dataset vbench \
	--num-prompts 5

Notes

• By default, image tasks talk to http://<host>:/v1/chat/completions; video tasks talk to /v1/videos.
• If you run the server on another host or port, pass --base-url accordingly.

2. Supported Datasets Copy Chapter

The benchmark supports three dataset modes via --dataset:

• vbench: Built-in prompt/data loader.
• trace: Heterogeneous request traces (each request can have different resolution/frames/steps).
• random: Synthetic prompts for quick smoke tests.

VBench dataset Copy Section

vbench only provides prompt data (and image paths for i2v/i2i); it does not carry
per-request generation fields. In this mode, all requests share CLI values:
--width --height --num-frames --fps --num-inference-steps
(pass --width and --height together).

Example (t2v):

python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
	--base-url http://localhost:8099 \
	--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
	--task t2v \
	--dataset vbench \
	--num-prompts 50 \
	--width 640 --height 480 \
	--num-frames 81 --fps 16 \
	--num-inference-steps 40

Note: vbench can also be used for other tasks such as t2i / i2v (and i2i). For t2i, the loader reuses VBench t2v text prompts; for i2v / i2i, it loads the VBench i2v dataset (with image paths).

If you use i2v/i2i bench datasets and need auto-download support, you may need:

uv pip install gdown

Trace dataset Copy Section

Use --dataset trace to replay a trace file. The trace can specify per-request fields such as:

• width, height
• num_frames (video)
• num_inference_steps
• seed, fps
• optional slo_ms (per-request SLO target)

By default (when --dataset-path is not provided), the script downloads a default trace from
the HuggingFace dataset repo asukaqaqzz/Dit_Trace. The default filename can depend on --task
(e.g., t2v uses a video trace).

Current defaults:

• --task t2i -> sd3_trace.txt
• --task t2v -> cogvideox_trace.txt

You can point to your own trace using --dataset-path.

3. Benchmark Parameters Copy Chapter

Basic flags Copy Section

• --base-url: Server address; --endpoint selects the path appended to this base URL.
• --model: The OpenAI-compatible model field.
• --endpoint: API endpoint path. Leading / is optional, e.g. /v1/videos or v1/videos.
• --task: Task type (e.g., t2i, t2v, i2i, i2v).
• --dataset: Dataset mode (vbench / trace / random).
• --num-prompts: Number of requests to send.

Common optional flags:

• --output-file: Write metrics to a JSON file.
• --disable-tqdm: Disable the progress bar.

Resolution / frames / steps: CLI defaults vs dataset fields Copy Section

Related flags: --width, --height, --num-frames, --fps, --num-inference-steps.

• For vbench / random: these CLI flags act as global defaults for all generated requests.
• For trace: requests can carry their own fields (e.g., width/height/num_frames/num_inference_steps), with overrides/fallbacks as below.

Precedence rules for trace (i.e., what actually gets sent):

• width/height: if either --width or --height is explicitly set, it overrides per-request values from the trace; otherwise per-request values are used when present.
• num_frames: per-request num_frames takes precedence; otherwise fall back to --num-frames.
• num_inference_steps: per-request num_inference_steps takes precedence; otherwise fall back to --num-inference-steps.

SLO, warmup, and max concurrency Copy Section

Enable SLO evaluation with --slo.

• If a request in the trace already has slo_ms, that value is used.
• Otherwise, the script runs warmup requests to infer a base unit time, estimates expected_ms by linearly scaling with area/frames/steps, and then sets slo_ms = expected_ms * --slo-scale.

Warmup flags:

• --warmup-requests: Number of warmup requests.
• --warmup-num-inference-steps: Steps used during warmup.
• --warmup-concurrency: Maximum concurrent warmup requests. Use this to warm
  the same batch shape as the measured run instead of warming only batch=1.
• For --task t2v: warmup requests are forced to use num_frames=1 to make warmup faster and less noisy.

Traffic / concurrency flags:

• --request-rate: Target request rate (requests/second). If set to inf, the script sends all requests immediately.
• --max-concurrency: Max number of in-flight requests (default: 1). This can hard-cap the achieved QPS: if it is too small, requests will queue behind the semaphore, and both achieved throughput and observed SLO attainment can be skewed.

Batched warmup note Copy Section

For batched serving runs, warm the same in-flight shape you plan to measure.
For example, a run with --max-concurrency 8 should usually also use
--warmup-requests 8 --warmup-concurrency 8; otherwise the first measured
batch may still pay compile or CUDA-graph capture cost.

For a Qwen-Image continuous-batching replay example, see
performance_dashboard/qwen_image_serving_performance.md.

File: benchmarks/distributed/omni_connectors/README.md Copy Chapter

RDMA Test Configuration Guide

This document explains how to configure the RDMA environment and run tests for MooncakeTransferEngineConnector.

Table of Contents Copy Chapter

• Docker Container Permissions
• Single-Node Testing
• Multi-Node Testing
• Running Tests
• Cross-Node Testing
• Troubleshooting

Docker Container Permissions Copy Chapter

RDMA tests require access to InfiniBand/RoCE devices and system topology. Add the following permissions when running docker run.

Option 1: Minimal Permissions (Recommended) Copy Section

docker run -it \
    --cap-add=SYS_PTRACE \
    --cap-add=IPC_LOCK \
    --security-opt seccomp=unconfined \
    --network=host \
    --device=/dev/infiniband \
    -v /sys/class/infiniband:/sys/class/infiniband:ro \
    your-image:tag

Parameter explanation:

• --cap-add=SYS_PTRACE: Allow reading system topology information
• --cap-add=IPC_LOCK: Allow memory locking (required for RDMA memory registration)
• --security-opt seccomp=unconfined: Disable seccomp restrictions
• --network=host: Use host network (required for RDMA)
• --device=/dev/infiniband: Mount InfiniBand devices
• -v /sys/class/infiniband: Mount IB device info (read-only)

Option 2: Full Permissions (Quick but not recommended for production) Copy Section

docker run -it \
    --privileged \
    --network=host \
    your-image:tag

--privileged grants full host permissions. Suitable for quick testing but not recommended for production.

Single-Node Testing Copy Chapter

When running single-node tests (producer and consumer on the same machine), ensure they use the same RDMA device.

Problem Background Copy Section

InfiniBand devices use LID (Local Identifier) for routing. Different devices have different LIDs and cannot communicate directly. If no device is specified, Mooncake may assign different devices to connectors, causing handshake failures.

Common error:

[Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Invalid argument [22]

Solution Copy Section

Method 1: Set Environment Variable (Recommended)

# List available RDMA devices
ibstat

# Select a device (e.g., mlx5_0)
export RDMA_DEVICE_NAME='mlx5_0'

# Run tests
pytest test_mooncake_transfer_engine_rdma.py -v -s

Method 2: Use RoCE Devices

If the system has RoCE devices (using IPv4 routing), the test code will automatically detect and prefer them. RoCE device GIDs start with 00:00:00:00:00:00:00:00:00:00:ff:ff (IPv4-mapped).

Method 3: Ensure MTU Consistency

Make sure both endpoints use the same MTU:

# Check device MTU
ibstatus mlx5_0

Multi-Node Testing Copy Chapter

For multi-node tests, producer and consumer run on different machines connected via InfiniBand switch.

Prerequisites Copy Section

1. Both machines have Mooncake and RDMA drivers installed
2. Both machines are in the same InfiniBand subnet
3. Switch is properly configured

Configuration Copy Section

Machine A (Producer):

# Set RDMA host IP (InfiniBand interface IP)
export RDMA_TEST_HOST='10.0.0.1'

# Optional: Specify device
export RDMA_DEVICE_NAME='mlx5_0'

Machine B (Consumer):

# Set RDMA host IP
export RDMA_TEST_HOST='10.0.0.2'

# Optional: Specify device
export RDMA_DEVICE_NAME='mlx5_0'

Verify Connectivity Copy Section

# Ping IB interface
ping 10.0.0.2

# Test RDMA connectivity with ibping
# On Machine B (server)
ibping -S

# On Machine A (client)
ibping -G <Machine_B_GID>

Running Tests Copy Chapter

Run All RDMA Tests (Single-Node, fast suite) Copy Section

Slow tests (large payloads, stress, concurrency integrity) are marked @pytest.mark.slow. Use -m "not slow" to skip them in quick CI or local fast iteration.

cd tests/distributed/omni_connectors

# Fast suite only (excludes slow/stress tests)
pytest test_mooncake_transfer_engine_rdma.py test_mooncake_transfer_engine_buffer.py -v -s -m "not slow"

Run Including Slow Tests Copy Section

# Run ALL tests including slow/stress tests
pytest test_mooncake_transfer_engine_rdma.py test_mooncake_transfer_engine_buffer.py -v -s

# Run ONLY the slow/stress tests
pytest test_mooncake_transfer_engine_rdma.py test_mooncake_transfer_engine_buffer.py -v -s -m slow

Run Buffer Management Tests Copy Section

# Fast only
pytest test_mooncake_transfer_engine_buffer.py -v -s -m "not slow"

# Including allocator invariant tests (double-free, overlap, merge)
pytest test_mooncake_transfer_engine_buffer.py -v -s

Run Specific Test Classes Copy Section

# Basic connector tests
pytest test_mooncake_transfer_engine_rdma.py::TestBasicConnector -v -s

# End-to-end RDMA transfer tests
pytest test_mooncake_transfer_engine_rdma.py::TestEndToEnd -v -s

# Lifecycle & resource management tests
pytest test_mooncake_transfer_engine_rdma.py::TestLifecycle -v -s

# GPU memory pool tests (requires CUDA)
pytest test_mooncake_transfer_engine_rdma.py::TestGPUPool -v -s

# Stress / correctness tests (slow)
pytest test_mooncake_transfer_engine_rdma.py::TestStressCorrectness -v -s

RDMA Environment Diagnostics Copy Section

For quick diagnostics (device status, Mooncake availability, env vars, etc.),
see the Troubleshooting section
in the connector documentation.

Cross-Node Testing Copy Chapter

The cross_node_mooncake_transfer_engine.py script enables testing RDMA transfers between two separate physical machines. This script is not auto-discovered by pytest (it does not start with test_) — it must be run manually on each node.

Prerequisites Copy Section

1. Both machines have Mooncake installed
2. Both machines are connected via InfiniBand/RoCE switch
3. Firewall allows ZMQ ports (default: 15500, 15501)
4. Same RDMA device name on both nodes (if multiple devices exist)

Running Cross-Node Tests Copy Section

On Machine A (Producer) — start first:

cd benchmarks/distributed/omni_connectors/

# Optional: specify device if multiple exist
export RDMA_DEVICE_NAME='mlx5_0'

python cross_node_mooncake_transfer_engine.py \
    --role producer \
    --local-host  \
    --remote-host <CONSUMER_IP> \
    --tensor-size-mb 100 \
    --num-transfers 3

On Machine B (Consumer) — start after producer:

cd benchmarks/distributed/omni_connectors/

export RDMA_DEVICE_NAME='mlx5_0'

python cross_node_mooncake_transfer_engine.py \
    --role consumer \
    --local-host <CONSUMER_IP> \
    --remote-host  \
    --tensor-size-mb 100 \
    --num-transfers 3

Transfer Modes Copy Section

Benchmark Mode Copy Section

Skip MD5 verification and measure pure RDMA throughput:

# Producer
python cross_node_mooncake_transfer_engine.py \
    --role producer \
    --local-host  \
    --remote-host <CONSUMER_IP> \
    --tensor-size-mb 1024 \
    --num-transfers 20 \
    --benchmark

# Consumer
python cross_node_mooncake_transfer_engine.py \
    --role consumer \
    --local-host <CONSUMER_IP> \
    --remote-host  \
    --tensor-size-mb 1024 \
    --num-transfers 20 \
    --benchmark

Cross-Node Test Options Copy Section

Troubleshooting Copy Chapter

1. "Failed to modify QP to RTR" Error Copy Section

Cause: QP handshake failed, usually due to device configuration mismatch.

Solution:

# Force using the same device
export RDMA_DEVICE_NAME='mlx5_0'

2. "Mooncake TransferEngine is not available" Copy Section

Cause: Mooncake not installed or import failed.

Solution:

# Check Mooncake installation
python -c "from mooncake.engine import TransferEngine; print('OK')"

# Reinstall if needed
pip install mooncake-transfer-engine
# Or using uv
uv pip install mooncake-transfer-engine

3. "Permission denied" accessing /dev/infiniband Copy Section

Cause: Container lacks IB device access permissions.

Solution:

docker run --device=/dev/infiniband --cap-add=IPC_LOCK ...

4. Test Timeout Copy Section

Cause: RDMA connection establishment failed or network latency.

Solution:

# Check network status
ibstat
ibstatus

5. GPU Test Failed "CUDA is not available" Copy Section

Cause: CUDA environment not configured or GPU unavailable.

Solution:

# Check CUDA
python -c "import torch; print(torch.cuda.is_available())"

# Docker needs NVIDIA runtime
docker run --gpus all ...

Environment Variables Reference Copy Chapter

Test Files Overview Copy Chapter

test_mooncake_transfer_engine_rdma.py — Test Classes Copy Section

test_mooncake_transfer_engine_buffer.py — Test Classes Copy Section

File: benchmarks/glm_image/README.md Copy Chapter

GLM-Image Benchmarks

Benchmark GLM-Image T2I (text-to-image) and I2I (image-to-image) performance across three backends: HuggingFace baseline, vLLM-Omni offline, and vLLM-Omni online serving.

Benchmarks Copy Chapter

HuggingFace Baseline Copy Chapter

Single-request sequential inference using the reference HuggingFace pipeline.

# T2I
CUDA_VISIBLE_DEVICES=0 python benchmarks/glm_image/huggingface/inference.py \
    --model-path /path/to/GLM-Image --mode t2i --num-prompts 10

# I2I
CUDA_VISIBLE_DEVICES=0 python benchmarks/glm_image/huggingface/inference.py \
    --model-path /path/to/GLM-Image --mode i2i --num-prompts 10

Options Copy Section

vLLM-Omni Offline Copy Chapter

Multi-GPU offline inference with pipeline parallelism and continuous batching.

# T2I
CUDA_VISIBLE_DEVICES=0,1 python benchmarks/glm_image/vllm-omni/inference.py \
    --model-path /path/to/GLM-Image --mode t2i --num-prompts 10

# I2I
CUDA_VISIBLE_DEVICES=0,1 python benchmarks/glm_image/vllm-omni/inference.py \
    --model-path /path/to/GLM-Image --mode i2i --num-prompts 10

Options Copy Section

Latency Computation Copy Section

In offline mode all requests are submitted simultaneously and processed with continuous batching. The per-request latency is computed by summing the actual per-stage times (with stage_0_gen_ms diffed against the previous request to remove accumulated queue/scheduling wait).

vLLM-Omni Online Serving Copy Chapter

Start the server Copy Section

CUDA_VISIBLE_DEVICES=0,1 vllm serve /path/to/GLM-Image \
    --omni --port 8091 --host 0.0.0.0 \
    --served-model-name glm-image

Run the benchmark Copy Section

# T2I
python benchmarks/glm_image/benchmark_glm_image.py \
    --mode t2i --num-prompts 10 --model glm-image

# I2I
python benchmarks/glm_image/benchmark_glm_image.py \
    --mode i2i --num-prompts 10 --model glm-image

# Custom dataset
python benchmarks/glm_image/benchmark_glm_image.py \
    --mode i2i --dataset custom \
    --dataset-path prompts.json --num-prompts 5

Options Copy Section

Dataset Copy Chapter

The default dataset is hosted on HuggingFace (prompt.json). It is automatically downloaded and cached to prompt/prompt.json on first run. No manual setup needed.

Each entry contains:

• t2i_prompt: Text prompt for text-to-image generation
• i2i_prompt: Text prompt for image-to-image editing
• image_url: Source image URL for I2I (downloaded and cached on first use)

Custom datasets use the same JSON format and can be provided via --dataset-path.

Pipeline Timings Copy Chapter

All three benchmarks report per-stage pipeline timings (in milliseconds):

The stages are ordered by execution: preprocess → stage_0 (AR) → ar2diffusion → stage_1 (Diffusion).

Sample Results Copy Chapter

Tested on 2x GPU with 10 prompts, 1024x1024, 50 denoising steps:

2. Official Technical Reference & Guides (vllm-project/vllm-project.github.io) Copy Chapter

vLLM blog source

Local development Copy Chapter

1. Install jekyll and bundler by running gem install jekyll bundler. Ask ChatGPT for help if you encounter any issues.
2. bundle install to install the necessary gems.
3. rm ./Gemfile.lock if you meet gem error.
4. bundle exec jekyll server to compile and start the server.

To add a new blogpost, please refer to _posts/2023-06-20-vllm.md as an example. Some notes:

• Google Doc can be saved as markdown format which will make your life easier.
• Note that the blogpost will only show when its date is in the past.
• Add summary front matter for SEO. Keep it concise, specific, and 240 characters or fewer.
• Put figures under assets/figures/yourblogname/.
• Make a pull request.

The blog is automatically built and deployed by GitHub Actions when main is pushed to.

LaTeX Math Copy Chapter

The blog supports LaTeX math via MathJax.

It can be enabled by adding math: true to the document frontmatter. It has been configured to support the standard LaTeX style math notation, i.e.:

$ inline math $

$$
math block
$$

GitHub Flavored Admonitions Copy Chapter

The blog supports GitHub flavored admonitions via jekyll-gfm-admonition. It supports the following syntax:

> [!NOTE]
> Highlights information that users should take into account, even when skimming.
> And supports multi-line text.

> [!TIP]
> Optional information to help a user be more successful.

> [!IMPORTANT]
> Crucial information necessary for users to succeed.

> [!WARNING]
> Critical content demanding immediate
> user attention due to potential risks.

> [!CAUTION]
> Negative potential consequences of an action.
> Opportunity to provide more context.

Theme customization Copy Chapter

The theme we are using is Minima. If you need to customise anything from this theme, see Overriding theme defaults.