## 1. Project Overview & Quickstart (vllm-project/vllm-omni) ## File: README.md

Easy, fast, and cheap omni-modality model serving for everyone

--- *Latest News* πŸ”₯ - [2026/08] [VeRL-Omni](https://github.com/verl-project/verl-omni) `v0.2.0` is released: faster diffusion RL powered by vLLM-Omni (request-level/step-wise batching with FA3), rebuilt Qwen3-Omni multimodal training (DPO & GSPO), plus LTX-2.3, Qwen-Image-Edit support and more. See the [release notes](https://github.com/verl-project/verl-omni/releases/tag/v0.2.0). - [2026/08] We released [0.26.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.26.0) - aligned with the vLLM 0.26 release line, featuring [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) joint video/audio generation, an experimental full-duplex realtime runtime for [MiniCPM-o 4.5](recipes/OpenBMB/MiniCPM-o-4_5.md), distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support. - [2026/07] We released [0.24.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.24.0) - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support. - [2026/06] Starting with [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0), vLLM-Omni publishes a stable release aligned with every even-numbered upstream vLLM minor version. [0.16.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.16.0), [0.18.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.18.0), [0.20.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.20.0), and [0.22.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.22.0) continued this cadence, expanding omni and world-model support with [NVIDIA Cosmos3](recipes/cosmos3/Cosmos3-Nano.md) and DreamZero, adding models such as MiniCPM-o 4.5, MOSS-TTS, and Lance, and advancing TTS, diffusion, distributed execution, quantization, RL integration through [VeRL-Omni](https://github.com/verl-project/verl-omni), and CUDA/ROCm/MUSA/NPU/XPU coverage. - [2026/03] Check out our first public [project deepdive](https://youtu.be/sgwNfsNnR9I) at the vLLM Hong Kong Meetup! - [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving. --- ## About [vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving: - **Omni-modality**: Text, image, audio, video, and action data processing - **Non-autoregressive Architectures**: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models - **Heterogeneous outputs**: from traditional text generation to multimodal and action outputs vLLM-Omni is fast with: - State-of-the-art AR support by leveraging efficient KV cache management from vLLM - Pipelined stage execution overlapping for high throughput performance - Fully disaggregation based on OmniConnector and dynamic resource allocation across stages vLLM-Omni is flexible and easy to use with: - Heterogeneous pipeline abstraction to manage complex model workflows - Seamless integration with popular Hugging Face models - Tensor, pipeline, data and expert parallelism support for distributed inference - Streaming outputs - OpenAI-compatible API server - Full-duplex realtime serving with streaming audio input and output (experimental) vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including: - **Omni-modality models** (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL) - **TTS models** (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3) - **Diffusion models** β€” image, video, and audio generation (e.g. MiniMax H3, Qwen-Image, Wan2.2, FLUX) - **Robot-policy and action models** (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy) ## Getting Started Visit our [documentation](https://vllm-omni.readthedocs.io/en/latest/) to learn more. - [Installation](https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/) - [Quickstart](https://vllm-omni.readthedocs.io/en/latest/getting_started/quickstart/) - [List of Supported Models](https://vllm-omni.readthedocs.io/en/latest/models/supported_models/) - [Deployment Recipes](https://recipes.vllm.ai) for vLLM-Omni model serving ## Contributing We welcome and value any contributions and collaborations. Please check out [Contributing to vLLM-Omni](https://vllm-omni.readthedocs.io/en/latest/contributing/) for how to get involved. ## Citation If you use vLLM-Omni for your research, please cite our [paper](https://arxiv.org/abs/2602.02204): ```bibtex @article{yin2026vllmomni, title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models}, author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu}, journal={arXiv preprint arXiv:2602.02204}, year={2026} } ``` ## Join the Community Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in `#sig-omni` slack channel at [slack.vllm.ai](https://slack.vllm.ai) or vLLM user forum at [discuss.vllm.ai](https://discuss.vllm.ai). ## Star History [](https://www.star-history.com/#vllm-project/vllm-omni&type=date&legend=top-left) ## License Apache License 2.0, as found in the [LICENSE](./LICENSE) file. --- ## File: .claude/skills/readme.md # Repository Skills for vLLM-Omni This directory contains repository-scale skills maintained for `vllm-omni`. They capture repeatable workflows for common contributor and maintainer tasks such as model integration, CI-aligned testing, performance optimization, and pull request review. ## Directory Structure Each skill lives in its own directory under `.claude/skills/`. A skill may include: - `SKILL.md`: the main workflow and operating instructions - `references/`: focused reference material used by the skill - `scripts/`: small helper scripts used by the skill ## Using the Skills Coding agents that discover repository skills can invoke the relevant skill by name. For other agents, point them to the skill's `SKILL.md` and ask them to read it before changing code. Combine a domain skill with `vllm-omni-test` when tests or CI coverage change, then use `precheck-pr` for the contributor's final self-check. ## Available Skills - [`add-diffusion-model`](add-diffusion-model/SKILL.md): guides integration of a new diffusion model into `vllm-omni` - [`add-tts-model`](add-tts-model/SKILL.md): covers integration of new TTS models and related serving workflows - [`diffusion-perf-opt`](diffusion-perf-opt/SKILL.md): guides diffusion model performance optimization, including profiling traces, parallel strategies, stage timing analysis, and benchmark-driven tuning - [`precheck-pr`](precheck-pr/SKILL.md): self-checks a branch before creating a PR by validating title format, dead code, accuracy and performance claims, and merge readiness - [`quantization`](quantization/SKILL.md): guides quantization method selection, model integration, checkpoint loading, and quality/performance validation for vLLM-Omni - [`review-pr`](review-pr/SKILL.md): provides a frozen-snapshot, contract-aware workflow for maintainers and reviewers; contributors should use `precheck-pr` for self-review - [`vllm-omni-npu-model-runner-upgrade`](vllm-omni-npu-upgrade/SKILL.md): upgrades NPU model runners to align with the latest vllm-ascend `NPUModelRunner` - [`vllm-omni-test`](vllm-omni-test/SKILL.md): guides generation and execution of CI-aligned tests (L1–L4), pytest marker selection (`core_model` / `advanced_model` / `full_model`, `omni` / `tts` / `diffusion`), Buildkite wiring (`test-ready.yml`, `test-merge.yml`, `test-nightly.yml`, `test-weekly.yml`), and copy-paste local plus CI-like `pytest` commands; see `references/test-routing.md` for level-to-command mapping ## Maintenance Guidelines - Keep skill names short and task-oriented. - Prefer repository-local paths, commands, and examples. - Avoid hardcoding fast-changing support matrices unless the skill is actively maintained alongside those changes. - Treat skills as contributor tooling: optimize for clarity, actionability, and low maintenance overhead. --- ## File: apps/lvsa-omni-showcase/README.md # LVSA Long-Video Showcase **Training-free block-sparse attention for long-video diffusion**, running on vLLM-Omni via the public plugin entry points (no core changes). LVSA speeds up extended-horizon generation and prevents the freeze/loop failure mode dense attention exhibits beyond a model's training length β€” with no fine-tuning and no weight changes. Upstream: **[LongVideoSparseAttention](https://github.com/JiusiServe/LongVideoSparseAttention)** (algorithm, all model adapters, full docs). ## Results Wan 2.1 1.3B, single A100 80 GB, mean over 5 prompts (Dense vs LVSA-FlashInfer): | Horizon | 1Γ— | 2Γ— | 3Γ— | 4Γ— | 5Γ— | 6Γ— | |---|---|---|---|---|---|---| | **Speedup** | 0.9Γ— | 1.4Γ— | 1.9Γ— | 2.5Γ— | 3.0Γ— | **3.5Γ—** | | **VQeval composite** (Denseβ†’LVSA) | 63β†’63 | 59β†’63 | 61β†’63 | 61β†’64 | 59β†’64 | 59β†’63 | The speedup **grows with length** (LVSA is the dense regime at 1Γ— by design, so the win starts at 2Γ—), and quality *improves* at extension β€” LVSA's rotating keyframes prevent the looping/static output dense produces beyond its horizon (VQeval loop-quality +27 to +34 at β‰₯3Γ—). At long horizons dense **OOMs on 80 GB** (HunyuanVideo β‰₯2Γ—, Wan2.1-14B and Cosmos at 6Γ—) where LVSA still generates. β†’ Full sweep (5 models Γ— 6 horizons, VQeval + VBench-Long) and the quality-metric methodology: **[benchmarks/](https://github.com/JiusiServe/LongVideoSparseAttention/tree/main/benchmarks)**. ## Quickstart ```bash # 1. vLLM-Omni 0.22.0 (stable) + matching vLLM pip install "vllm==0.22.0" pip install --no-build-isolation \ "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@v0.22.0" # 2. LVSA core + plugin git clone https://github.com/JiusiServe/LongVideoSparseAttention.git ( cd LongVideoSparseAttention && pip install -e . && pip install -e lvsa-vllm-omni/ ) # 3. this showcase's client dep pip install requests ``` FlashInfer (used by the default `flashinfer` backend) **ships with vLLM** β€” step 1 already pulls the pinned `flashinfer-python` + `flashinfer-cubin`, so the fused backend works with no extra install. Two optional extras: ```bash # (optional) faster startup β€” prebuilt JIT cache, matched to your CUDA + flashinfer CU=cu128 # match your CUDA: cu121 / cu124 / cu128 / … FIVER=$(python -c "import importlib.metadata as m; print(m.version('flashinfer-python'))") pip install --extra-index-url "https://flashinfer.ai/whl/${CU}" \ "flashinfer-jit-cache==${FIVER}+${CU}" # (fallback) no FlashInfer in your environment? run the SDPA backend instead: export LVSA_BACKEND=sdpa ``` ```bash # Serve an LVSA-enabled endpoint (Wan 2.1 1.3B at 2Γ— horizon): MODEL=/path/to/Wan2.1-T2V-1.3B-Diffusers MODEL_FAMILY=wan FRAMES=161 \ bash serve_lvsa.sh # In another shell β€” generate: python generate.py --frames 161 --height 480 --width 832 \ --prompt "A dog running through a sunlit forest." --out wan_lvsa.mp4 ``` `serve_lvsa.sh` sets the LVSA env vars for the chosen `MODEL_FAMILY` (`wan` / `hunyuan`). Wan and HunyuanVideo run through the LVSA **attention backend** by default β€” it stays sparse even under sequence-parallel, unlike the monkey-patch hooks. ## Configuration The one knob you must get right is the model's training horizon (`LVSA_REFERENCE_LATENT_FRAMES`: Wan = 21, HunyuanVideo = 33); `serve_lvsa.sh` sets it per family. For the full environment-variable reference, the multi-GPU support matrix, and tuning (`sparsity_scale`, rotating keyframes, non-standard resolutions), see the upstream docs: - **[lvsa-vllm-omni plugin reference](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/lvsa-vllm-omni/README.md)** β€” every `LVSA_*` env var - **[docs/parallelism.md](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/docs/parallelism.md)** β€” TP / Ulysses / HSDP support per model - **[docs/tuning.md](https://github.com/JiusiServe/LongVideoSparseAttention/blob/main/docs/tuning.md)** β€” picking knobs for your model and horizon --- ## File: apps/ComfyUI-vLLM-Omni/README.md # vLLM-Omni vLLM-Omni offers a ComfyUI integration on top of its online serving API. It can send model inference requests to either a locally running vLLM-Omni service or a remote one. ## Requirement - Python 3.12 or above - [ComfyUI installed](https://docs.comfy.org/installation/system_requirements) - [vLLM-Omni installed](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/) on either the same device or another device discoverable via the internet. - No need to install additional packages apart from those already required by ComfyUI. > [!TIP] > If you run both ComfyUI and vLLM-Omni on the same device, you can create separate virtual environments and use different Python versions for them. ## Installation Copy this folder to the `custom_nodes` subfolder of your ComfyUI installation. Your directory should look like `ComfyUI/custom_nodes/ComfyUI-vLLM-Omni`. If you are running ComfyUI during copying, you should restart ComfyUI to load this extension. > [!TIP] > You can use utility websites such as https://download-directory.github.io/ to download a subdirectory of a repo. Also checkout community discussions (e.g., https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repository) for more info. On the device and virtual environment you run ComfyUI, launch ComfyUI with ```bash cd ComfyUI # The regular way python main.py # If you are mainly using this node, launch it faster with python main.py --cpu ``` On the device and virtual environment you run vLLM-Omni, start a model service with ```bash vllm serve The_Model_ID_to_Serve --omni --port 8000 ``` Check **ComfyUI's sidebar -> Node Library**. There should be a new folder named **vLLM-Omni**. If no, check your shell running the ComfyUI process. There may be some error messages before the line `Import times for custom nodes:` and the line `To see the GUI go to: http://127.0.0.1:8188`. ## Quickstart This extension offers the following nodes based on the output modalities (at **ComfyUI sidebar -> Node Library**): - **Generate Image** for text-to-image and image-to-image tasks - **Generate Video** for text-to-video, first-frame/image-to-video, and reference-conditioned video - **Multimodality Understanding** for multimodality-to-text and multimodality-to-audio tasks - **TTS** and **TTS Voice Clone** for TTS tasks This extension also offers example workflows (at **ComfyUI sidebar -> Templates -> vLLM-Omni**) > [!NOTE] > The node UI and feature designs are intended to match vLLM-Omni online serving interfaces. It cannot offer more than what the interfaces support. To build a simple workflow yourself, - Drag a generation node onto the canvas. - Depending on your need, grab built-in multimedia file loader nodes, such as **image->Load Image**, **image->video->Load Video**, **audio->Load Audio** - Depending on your need, grab built-in multimedia file preview nodes, such as **image->Preview Image**, **image->video->Save Video**, **audio->Preview Audio**, **utils->Preview as Text**. - If you want to tune sampling parameters, grab corresponding nodes from **vLLM-Omni-> Sampling Params**. - For multi-stage models, you can connect multiple **AR Sampling Params** and **Diffusion Sampling Params** nodes to a **Multi-Stage Sampling Params List** node, and connect this node to the generation node. - For some multi-stage models like BAGEL, [only one stage's sampling parameters are exposed and tunable via vLLM-Omni's online serving API](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/online_serving/bagel/). Thus, these models are treated as single-stage ones. Please check the vLLM-Omni documentation on how to correctly set each model's sampling parameters. - For multi-stage models where all stages are either autoregression or diffusion, you can also connect only a single Sampling Params node, indicating that this set of sampling parameters will be used for all stages. ## Screenshots and Examples ### Multimodal understanding (e.g., Qwen Omni series, BAGEL) (Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Multimodal Understanding**) > [!TIP] > Although this node enables all-modality input, you should check whether the specific model you host and request for supports the modalities you connect to the node. You can configure per-stage sampling parameters for multi-stage models. ### Text-to-image and image-to-image generation (e.g., Z-Image-Turbo, Qwen-Image-Edit, BAGEL) (Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Image Generation**) > [!TIP] > The node automatically choose text-to-image or image-to-image API endpoints depending on whether you connect an image input or not. ### Text-to-video and image-to-video generation (e.g., Wan, MiniMax-H3) (Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Video Generation**) > [!TIP] > Connect a **frame** image for first-frame / image-to-video (e.g., Wan I2V, MiniMax-H3 FL2VA). > > For reference-conditioned generation (MiniMax-H3 Ref2VA), connect a **Video References** node instead. > > Do not use `frame` and `references` together. Task routing is automatic from which inputs you connect. ### TTS (e.g., Qwen TTS series) (Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni TTS**) > [!TIP] > There is a dedicated node for VoiceClone tasks with reference audio input. Other simple text-to-speech tasks should use the regular TTS node. ### Chaining multiple model services (Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Chaining Services**) ## Develop Follow the [development convention and rules of vLLM-Omni](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/). ## Limitation and Non-Goals - Single server mode only. No automatic load balancing or failover. - Features set is bounded to vLLM-Omni's online service capability, including - The types of models supported in online mode, - The types of sampling parameters supported in the online mode, - The ways to send files (primarily through full-length base64 in JSON payload), - Figuring out errors in the payload (such as unsupported fields by a specific model) if the endpoint does not explicitly return an error, - (The lack of) Authentication - (The lack of) Progress indicator ## Support If you are new to ComfyUI, please check out [its documentation](https://docs.comfy.org/) for usage instructions. If you are new to vLLM-Omni, please also check out [its documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/) for usage instructions. Whenever you find an issue or problem, please - First find out if this is an upstream limitation of vLLM-Omni's online serving mode, by [checking their documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/examples/). - [Open an issue](https://github.com/vllm-project/vllm-omni/issues) that clearly describes this ComfyUI or online service problem. ## Acknowledgements Features - https://github.com/dougbtv/comfyui-vllm-omni/ The official reference implementation for ComfyUI integration with vLLM-Omni's DALL-E compatible image generation API. - https://github.com/Comfy-Org/ComfyUI/tree/master/comfy_extras ComfyUI's built-in node implementations. UI/UX design references - https://github.com/sgl-project/sglang/pull/15271 SGLang Diffusion's official ComfyUI integration for image and video generation. - https://github.com/SXQBW/ComfyUI-Qwen-Omni A third party ComfyUI integration for Qwen Omni series. - https://github.com/flybirdxx/ComfyUI-Qwen-TTS https://github.com/DarioFT/ComfyUI-Qwen3-TTS Third party ComfyUI integrations for Qwen TTS series. --- ## File: benchmarks/accuracy/image_to_image/README.md # GEdit-Bench on vLLM-Omni This integration adapts the upstream `stepfun-ai/Step1X-Edit/GEdit-Bench` evaluation flow into `vllm-omni/benchmarks/accuracy/image_to_image`. Upstream mapping: - `run_gedit_score.py` -> `run_gedit_bench.py evaluate` - `calculate_statistics.py` -> `run_gedit_bench.py summarize` - upstream output layout under `results//fullset/...` is preserved What changed: - The upstream repo mainly ships evaluation scripts. This integration adds a generation runner that uses the local `vllm-omni` OpenAI-compatible `/v1/images/edits` endpoint to produce benchmark outputs in the expected directory structure. - The evaluator keeps the same VIEScore-style decomposition: - `semantics_score` - `quality_score` - `overall_score = sqrt(semantics_score * quality_score)` - Judge calls are routed to a local OpenAI-compatible model served by `vllm-omni`, not a remote provider. Dataset: - Default `--dataset-ref` is `stepfun-ai/GEdit-Bench` - You can also pass a local dataset directory previously saved with Hugging Face `datasets` Example usage: ```bash python benchmarks/accuracy/image_to_image/run_gedit_bench.py generate \ --output-root benchmarks/accuracy/image_to_image/results \ --base-url http://127.0.0.1:8000 \ --model Qwen/Qwen-Image-Edit \ --model-name qwen_image_edit \ --dataset-ref stepfun-ai/GEdit-Bench \ --task-type all \ --instruction-language en ``` ```bash python benchmarks/accuracy/image_to_image/run_gedit_bench.py evaluate \ --output-root benchmarks/accuracy/image_to_image/results \ --model-name qwen_image_edit \ --save-dir benchmarks/accuracy/image_to_image/scores \ --dataset-ref stepfun-ai/GEdit-Bench \ --judge-base-url http://127.0.0.1:8000 \ --judge-model Qwen/Qwen2.5-VL-7B-Instruct \ --judge-api-key EMPTY ``` ```bash python benchmarks/accuracy/image_to_image/run_gedit_bench.py summarize \ --csv-path benchmarks/accuracy/image_to_image/scores/qwen_image_edit_all_all_vie_score.csv \ --language en ``` Example summary output: ```json { "language": "all", "languages": { "en": { "overall": {"count": 110, "Q_SC": 6.58, "Q_PQ": 5.89, "Q_O": 5.86}, "intersection": {"count": 78, "Q_SC": 6.50, "Q_PQ": 5.66, "Q_O": 5.65} }, "cn": { "overall": {"count": 110, "Q_SC": 6.90, "Q_PQ": 5.78, "Q_O": 6.11}, "intersection": {"count": 63, "Q_SC": 7.22, "Q_PQ": 5.59, "Q_O": 6.28} } } } ``` Example generated images to inspect: - `benchmarks/accuracy/image_to_image/results/qwen_image_edit/fullset/background_change/en/.png` - `benchmarks/accuracy/image_to_image/results/qwen_image_edit/fullset/text_change/en/.png` - `benchmarks/accuracy/image_to_image/results/qwen_image_edit/fullset/subject-replace/cn/.png` Example score artifacts to inspect together with the images: - `benchmarks/accuracy/image_to_image/scores/qwen_image_edit_all_all_vie_score.csv` - `benchmarks/accuracy/image_to_image/scores/qwen_image_edit_all_all_summary.json` What to expect: - `Q_SC` measures instruction following and content preservation. - `Q_PQ` measures image naturalness and artifact quality. - `Q_O` is the combined overall score; higher is better. - `overall.count` is the number of evaluated samples for that language, while `intersection.count` is the subset with `intersection_exist == True`. Notes: - This flow requires the optional Hugging Face `datasets` package. - `generate` writes `generation_manifest.json` with local output coverage. - The current repo marker set exposes `L4` but not `L5`, so if you promote an end-to-end smoke test into CI, use the `full_model`, `benchmark`, and `L4` markers for nightly (or `advanced_model` for merge) or introduce a new repo-wide marker explicitly first. --- ## File: benchmarks/accuracy/text_to_image/README.md # GEBench on vLLM-Omni This integration adapts the upstream `stepfun-ai/GEBench` scripts into `vllm-omni/benchmarks/accuracy/text_to_image`. Upstream mapping: - `scripts/generate.py` -> `run_gebench.py generate` - `scripts/evaluate.py` -> `run_gebench.py evaluate` - upstream prompt / judge logic -> `gbench.py` What changed: - Generation calls the local OpenAI-compatible `vllm-omni` endpoints: - `/v1/images/generations` for text-only frame generation - `/v1/images/edits` for image-conditioned GUI transition generation - Evaluation still keeps the GEBench scoring dimensions: - `goal` - `logic` - `cons` - `ui` - `qual` - Judge calls are also routed to a local OpenAI-compatible model served by `vllm-omni` instead of a remote service. Expected dataset layout: - Clone the benchmark dataset from Hugging Face into a local directory: ```bash git clone https://huggingface.co/datasets/stepfun-ai/GEBench /path/to/GEBench ``` Example usage: ```bash python benchmarks/accuracy/text_to_image/run_gebench.py generate \ --dataset-root /path/to/GEBench \ --output-root benchmarks/accuracy/text_to_image/outputs \ --base-url http://127.0.0.1:8000 \ --model Tongyi-MAI/Z-Image-Turbo \ --data-type type3 ``` ```bash python benchmarks/accuracy/text_to_image/run_gebench.py evaluate \ --dataset-root /path/to/GEBench \ --output-root benchmarks/accuracy/text_to_image/outputs \ --data-type type3 \ --judge-base-url http://127.0.0.1:8000 \ --judge-model Qwen/Qwen2.5-VL-7B-Instruct \ --judge-api-key EMPTY ``` ```bash python benchmarks/accuracy/text_to_image/run_gebench.py summarize \ --output-root benchmarks/accuracy/text_to_image/outputs ``` Example summary output: ```json { "generation": { "count": 20, "by_type": { "type3": {"count": 10}, "type4": {"count": 10} } }, "evaluation": { "count": 20, "overall_mean": 0.52, "by_type": { "type3": {"count": 10, "overall_mean": 0.50, "overall_mean_100": 50.0}, "type4": {"count": 10, "overall_mean": 0.54, "overall_mean_100": 54.0} } } } ``` Example generated images to inspect: - `benchmarks/accuracy/text_to_image/outputs/03_trajectory_text_fictionalapp///frame0.png` - `benchmarks/accuracy/text_to_image/outputs/03_trajectory_text_fictionalapp///frame5.png` - `benchmarks/accuracy/text_to_image/outputs/04_trajectory_text_realapp///frame0.png` - `benchmarks/accuracy/text_to_image/outputs/04_trajectory_text_realapp///frame5.png` What to expect: - `overall_mean` is normalized to `0.0 ~ 1.0`; higher is better. - `frame0.png` is the initial GUI frame; `frame5.png` is the final trajectory frame most often used for quick inspection. - For full debugging, inspect the whole `frame0.png` ... `frame5.png` sequence for one sample directory. Notes: - GEBench upstream leaves type3/type4 generation unfinished. This integration fills that gap with a trajectory runner that generates `frame0.png` followed by `frame1.png` ... `frame5.png`. - Type1/2/5 require an image-edit capable model exposed through `vllm-omni serve`. - `summarize` will report both generated coverage and any existing evaluation summary files. --- ## File: benchmarks/accuracy/README.md # Accuracy Benchmarks This directory hosts accuracy benchmark integrations that run entirely through a local `vllm-omni serve` deployment. Current integrations: - `text_to_image/`: GEBench generation + local judge scoring flow. - `image_to_image/`: GEdit-Bench generation + local VIEScore-style scoring flow. Design notes: - Generation is executed through the OpenAI-compatible endpoints exposed by `vllm-omni serve`. - Evaluation is also executed through a local OpenAI-compatible judge model served by `vllm-omni`. - Both generation and judge requests accept either `http://host:port` or `http://host:port/v1`. - Output directory layout intentionally stays close to the upstream repos. Test guidance: - Local static/self-checks live in `tests/benchmarks/test_accuracy_bench_utils.py`. - End-to-end generation/evaluation should be validated in a remote GPU environment. In the current repo marker system there is `L4` but no `L5` marker, so benchmark smoke tests should be wired as `full_model + benchmark + L4` for nightly when GPU capacity is available. --- ## File: benchmarks/diffusion/README.md # Diffusion Serving Benchmark (Image/Video) This folder contains an online-serving benchmark script for diffusion models. It sends requests to a vLLM OpenAI-compatible endpoint and reports throughput, latency percentiles, and optional SLO attainment. The main entrypoint is: - `benchmarks/diffusion/diffusion_benchmark_serving.py` ## 1. Quick Start 1. Start the server: ```bash vllm serve Qwen/Qwen-Image --omni --port 8099 ``` 2. Run a minimal benchmark: ```bash python3 benchmarks/diffusion/diffusion_benchmark_serving.py \ --base-url http://localhost:8099 \ --model Qwen/Qwen-Image \ --task t2i \ --dataset vbench \ --num-prompts 5 ``` **Notes** - By default, image tasks talk to `http://:/v1/chat/completions`; video tasks talk to `/v1/videos`. - If you run the server on another host or port, pass `--base-url` accordingly. ## 2. Supported Datasets The benchmark supports three dataset modes via `--dataset`: - `vbench`: Built-in prompt/data loader. - `trace`: Heterogeneous request traces (each request can have different resolution/frames/steps). - `random`: Synthetic prompts for quick smoke tests. ### VBench dataset `vbench` only provides prompt data (and image paths for i2v/i2i); it does not carry per-request generation fields. In this mode, all requests share CLI values: `--width --height --num-frames --fps --num-inference-steps` (pass `--width` and `--height` together). Example (`t2v`): ```bash python3 benchmarks/diffusion/diffusion_benchmark_serving.py \ --base-url http://localhost:8099 \ --model Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --task t2v \ --dataset vbench \ --num-prompts 50 \ --width 640 --height 480 \ --num-frames 81 --fps 16 \ --num-inference-steps 40 ``` Note: `vbench` can also be used for other tasks such as `t2i` / `i2v` (and `i2i`). For `t2i`, the loader reuses VBench t2v text prompts; for `i2v` / `i2i`, it loads the VBench i2v dataset (with image paths). If you use i2v/i2i bench datasets and need auto-download support, you may need: ```bash uv pip install gdown ``` ### Trace dataset Use `--dataset trace` to replay a trace file. The trace can specify per-request fields such as: - `width`, `height` - `num_frames` (video) - `num_inference_steps` - `seed`, `fps` - optional `slo_ms` (per-request SLO target) By default (when `--dataset-path` is not provided), the script downloads a default trace from the HuggingFace dataset repo `asukaqaqzz/Dit_Trace`. The default filename can depend on `--task` (e.g., `t2v` uses a video trace). Current defaults: - `--task t2i` -> `sd3_trace.txt` - `--task t2v` -> `cogvideox_trace.txt` You can point to your own trace using `--dataset-path`. ## 3. Benchmark Parameters ### Basic flags - `--base-url`: Server address; `--endpoint` selects the path appended to this base URL. - `--model`: The OpenAI-compatible `model` field. - `--endpoint`: API endpoint path. Leading `/` is optional, e.g. `/v1/videos` or `v1/videos`. - `--task`: Task type (e.g., `t2i`, `t2v`, `i2i`, `i2v`). - `--dataset`: Dataset mode (`vbench` / `trace` / `random`). - `--num-prompts`: Number of requests to send. Common optional flags: - `--output-file`: Write metrics to a JSON file. - `--disable-tqdm`: Disable the progress bar. ### Resolution / frames / steps: CLI defaults vs dataset fields Related flags: `--width`, `--height`, `--num-frames`, `--fps`, `--num-inference-steps`. - For `vbench` / `random`: these CLI flags act as global defaults for all generated requests. - For `trace`: requests can carry their own fields (e.g., `width/height/num_frames/num_inference_steps`), with overrides/fallbacks as below. Precedence rules for `trace` (i.e., what actually gets sent): - `width/height`: if either `--width` or `--height` is explicitly set, it overrides per-request values from the trace; otherwise per-request values are used when present. - `num_frames`: per-request `num_frames` takes precedence; otherwise fall back to `--num-frames`. - `num_inference_steps`: per-request `num_inference_steps` takes precedence; otherwise fall back to `--num-inference-steps`. ### SLO, warmup, and max concurrency Enable SLO evaluation with `--slo`. - If a request in the trace already has `slo_ms`, that value is used. - Otherwise, the script runs warmup requests to infer a base unit time, estimates `expected_ms` by linearly scaling with area/frames/steps, and then sets `slo_ms = expected_ms * --slo-scale`. Warmup flags: - `--warmup-requests`: Number of warmup requests. - `--warmup-num-inference-steps`: Steps used during warmup. - `--warmup-concurrency`: Maximum concurrent warmup requests. Use this to warm the same batch shape as the measured run instead of warming only batch=`1`. - For `--task t2v`: warmup requests are forced to use `num_frames=1` to make warmup faster and less noisy. Traffic / concurrency flags: - `--request-rate`: Target request rate (requests/second). If set to `inf`, the script sends all requests immediately. - `--max-concurrency`: Max number of in-flight requests (default: `1`). This can hard-cap the achieved QPS: if it is too small, requests will queue behind the semaphore, and both achieved throughput and observed SLO attainment can be skewed. ### Batched warmup note For batched serving runs, warm the same in-flight shape you plan to measure. For example, a run with `--max-concurrency 8` should usually also use `--warmup-requests 8 --warmup-concurrency 8`; otherwise the first measured batch may still pay compile or CUDA-graph capture cost. For a Qwen-Image continuous-batching replay example, see [`performance_dashboard/qwen_image_serving_performance.md`](./performance_dashboard/qwen_image_serving_performance.md). --- ## File: benchmarks/distributed/omni_connectors/README.md # RDMA Test Configuration Guide This document explains how to configure the RDMA environment and run tests for `MooncakeTransferEngineConnector`. ## Table of Contents - [Docker Container Permissions](#docker-container-permissions) - [Single-Node Testing](#single-node-testing) - [Multi-Node Testing](#multi-node-testing) - [Running Tests](#running-tests) - [Cross-Node Testing](#cross-node-testing) - [Troubleshooting](#troubleshooting) --- ## Docker Container Permissions RDMA tests require access to InfiniBand/RoCE devices and system topology. Add the following permissions when running `docker run`. ### Option 1: Minimal Permissions (Recommended) ```bash docker run -it \ --cap-add=SYS_PTRACE \ --cap-add=IPC_LOCK \ --security-opt seccomp=unconfined \ --network=host \ --device=/dev/infiniband \ -v /sys/class/infiniband:/sys/class/infiniband:ro \ your-image:tag ``` Parameter explanation: - `--cap-add=SYS_PTRACE`: Allow reading system topology information - `--cap-add=IPC_LOCK`: Allow memory locking (required for RDMA memory registration) - `--security-opt seccomp=unconfined`: Disable seccomp restrictions - `--network=host`: Use host network (required for RDMA) - `--device=/dev/infiniband`: Mount InfiniBand devices - `-v /sys/class/infiniband`: Mount IB device info (read-only) ### Option 2: Full Permissions (Quick but not recommended for production) ```bash docker run -it \ --privileged \ --network=host \ your-image:tag ``` `--privileged` grants full host permissions. Suitable for quick testing but not recommended for production. --- ## Single-Node Testing When running single-node tests (producer and consumer on the same machine), ensure they use the **same RDMA device**. ### Problem Background InfiniBand devices use LID (Local Identifier) for routing. Different devices have different LIDs and cannot communicate directly. If no device is specified, Mooncake may assign different devices to connectors, causing handshake failures. Common error: ``` [Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Invalid argument [22] ``` ### Solution **Method 1: Set Environment Variable (Recommended)** ```bash # List available RDMA devices ibstat # Select a device (e.g., mlx5_0) export RDMA_DEVICE_NAME='mlx5_0' # Run tests pytest test_mooncake_transfer_engine_rdma.py -v -s ``` **Method 2: Use RoCE Devices** If the system has RoCE devices (using IPv4 routing), the test code will automatically detect and prefer them. RoCE device GIDs start with `00:00:00:00:00:00:00:00:00:00:ff:ff` (IPv4-mapped). **Method 3: Ensure MTU Consistency** Make sure both endpoints use the same MTU: ```bash # Check device MTU ibstatus mlx5_0 ``` --- ## Multi-Node Testing For multi-node tests, producer and consumer run on different machines connected via InfiniBand switch. ### Prerequisites 1. Both machines have Mooncake and RDMA drivers installed 2. Both machines are in the same InfiniBand subnet 3. Switch is properly configured ### Configuration **Machine A (Producer):** ```bash # Set RDMA host IP (InfiniBand interface IP) export RDMA_TEST_HOST='10.0.0.1' # Optional: Specify device export RDMA_DEVICE_NAME='mlx5_0' ``` **Machine B (Consumer):** ```bash # Set RDMA host IP export RDMA_TEST_HOST='10.0.0.2' # Optional: Specify device export RDMA_DEVICE_NAME='mlx5_0' ``` ### Verify Connectivity ```bash # Ping IB interface ping 10.0.0.2 # Test RDMA connectivity with ibping # On Machine B (server) ibping -S # On Machine A (client) ibping -G ``` --- ## Running Tests ### Run All RDMA Tests (Single-Node, fast suite) Slow tests (large payloads, stress, concurrency integrity) are marked `@pytest.mark.slow`. Use `-m "not slow"` to skip them in quick CI or local fast iteration. ```bash cd tests/distributed/omni_connectors # Fast suite only (excludes slow/stress tests) pytest test_mooncake_transfer_engine_rdma.py test_mooncake_transfer_engine_buffer.py -v -s -m "not slow" ``` ### Run Including Slow Tests ```bash # Run ALL tests including slow/stress tests pytest test_mooncake_transfer_engine_rdma.py test_mooncake_transfer_engine_buffer.py -v -s # Run ONLY the slow/stress tests pytest test_mooncake_transfer_engine_rdma.py test_mooncake_transfer_engine_buffer.py -v -s -m slow ``` ### Run Buffer Management Tests ```bash # Fast only pytest test_mooncake_transfer_engine_buffer.py -v -s -m "not slow" # Including allocator invariant tests (double-free, overlap, merge) pytest test_mooncake_transfer_engine_buffer.py -v -s ``` ### Run Specific Test Classes ```bash # Basic connector tests pytest test_mooncake_transfer_engine_rdma.py::TestBasicConnector -v -s # End-to-end RDMA transfer tests pytest test_mooncake_transfer_engine_rdma.py::TestEndToEnd -v -s # Lifecycle & resource management tests pytest test_mooncake_transfer_engine_rdma.py::TestLifecycle -v -s # GPU memory pool tests (requires CUDA) pytest test_mooncake_transfer_engine_rdma.py::TestGPUPool -v -s # Stress / correctness tests (slow) pytest test_mooncake_transfer_engine_rdma.py::TestStressCorrectness -v -s ``` ### RDMA Environment Diagnostics For quick diagnostics (device status, Mooncake availability, env vars, etc.), see the [Troubleshooting section](../../../docs/design/feature/omni_connectors/mooncake_transfer_engine_connector.md#troubleshooting) in the connector documentation. --- ## Cross-Node Testing The `cross_node_mooncake_transfer_engine.py` script enables testing RDMA transfers between two separate physical machines. This script is **not** auto-discovered by `pytest` (it does not start with `test_`) β€” it must be run manually on each node. ### Prerequisites 1. Both machines have Mooncake installed 2. Both machines are connected via InfiniBand/RoCE switch 3. Firewall allows ZMQ ports (default: 15500, 15501) 4. Same RDMA device name on both nodes (if multiple devices exist) ### Running Cross-Node Tests **On Machine A (Producer) β€” start first:** ```bash cd benchmarks/distributed/omni_connectors/ # Optional: specify device if multiple exist export RDMA_DEVICE_NAME='mlx5_0' python cross_node_mooncake_transfer_engine.py \ --role producer \ --local-host \ --remote-host \ --tensor-size-mb 100 \ --num-transfers 3 ``` **On Machine B (Consumer) β€” start after producer:** ```bash cd benchmarks/distributed/omni_connectors/ export RDMA_DEVICE_NAME='mlx5_0' python cross_node_mooncake_transfer_engine.py \ --role consumer \ --local-host \ --remote-host \ --tensor-size-mb 100 \ --num-transfers 3 ``` ### Transfer Modes | Mode | Description | Example | |------|-------------|---------| | `copy` | Normal path β€” tensor copied to RDMA pool (default) | `--mode copy` | | `zerocopy` | Zero-copy path β€” data created directly in RDMA pool | `--mode zerocopy` | | `gpu` | GPU transfer β€” RDMA pool on GPU, uses GPUDirect | `--mode gpu --gpu-id 0` | ### Benchmark Mode Skip MD5 verification and measure pure RDMA throughput: ```bash # Producer python cross_node_mooncake_transfer_engine.py \ --role producer \ --local-host \ --remote-host \ --tensor-size-mb 1024 \ --num-transfers 20 \ --benchmark # Consumer python cross_node_mooncake_transfer_engine.py \ --role consumer \ --local-host \ --remote-host \ --tensor-size-mb 1024 \ --num-transfers 20 \ --benchmark ``` ### Cross-Node Test Options | Option | Description | Default | |--------|-------------|---------| | `--role` | `producer` or `consumer` | Required | | `--local-host` | Local RDMA IP address | Required | | `--remote-host` | Remote RDMA IP address | Required | | `--local-port` | Local ZMQ port for RDMA data | 15500 | | `--remote-port` | Remote ZMQ port for RDMA data | 15500 | | `--ctrl-port` | Control channel port | 15501 | | `--tensor-size-mb` | Tensor size in MB | 100 | | `--num-transfers` | Number of transfers | 3 | | `--mode` | `copy`, `zerocopy`, or `gpu` | `copy` | | `--gpu-id` | GPU ID for GPU mode | 0 | | `--benchmark` | Skip MD5, pure performance test | off | --- ## Troubleshooting ### 1. "Failed to modify QP to RTR" Error **Cause**: QP handshake failed, usually due to device configuration mismatch. **Solution**: ```bash # Force using the same device export RDMA_DEVICE_NAME='mlx5_0' ``` ### 2. "Mooncake TransferEngine is not available" **Cause**: Mooncake not installed or import failed. **Solution**: ```bash # Check Mooncake installation python -c "from mooncake.engine import TransferEngine; print('OK')" # Reinstall if needed pip install mooncake-transfer-engine # Or using uv uv pip install mooncake-transfer-engine ``` ### 3. "Permission denied" accessing /dev/infiniband **Cause**: Container lacks IB device access permissions. **Solution**: ```bash docker run --device=/dev/infiniband --cap-add=IPC_LOCK ... ``` ### 4. Test Timeout **Cause**: RDMA connection establishment failed or network latency. **Solution**: ```bash # Check network status ibstat ibstatus ``` ### 5. GPU Test Failed "CUDA is not available" **Cause**: CUDA environment not configured or GPU unavailable. **Solution**: ```bash # Check CUDA python -c "import torch; print(torch.cuda.is_available())" # Docker needs NVIDIA runtime docker run --gpus all ... ``` --- ## Environment Variables Reference | Variable | Description | Example | |----------|-------------|---------| | `RDMA_DEVICE_NAME` | Specify RDMA device name | `mlx5_0` | | `RDMA_TEST_HOST` | Specify test host IP | `10.0.0.1` | | `MC_TE_METRIC` | Enable Mooncake metrics | `1` | | `MC_IB_PCI_RELAXED_ORDERING` | Enable PCIe relaxed ordering | `1` | --- ## Test Files Overview | File | Description | Auto-discovered by pytest | |------|-------------|--------------------------| | `test_mooncake_transfer_engine_rdma.py` | Integration tests for MooncakeTransferEngineConnector (basic, E2E, lifecycle, GPU) | Yes | | `test_mooncake_transfer_engine_buffer.py` | Memory pool and buffer management unit tests | Yes | | `cross_node_mooncake_transfer_engine.py` | Cross-node (multi-machine) testing script β€” run manually | No (filename does not start with `test_`) | ### test_mooncake_transfer_engine_rdma.py β€” Test Classes | Test Class | Memory Pool | Marker | Description | |------------|-------------|--------|-------------| | `TestBasicConnector` | CPU | β€” | Initialization, put tensor/bytes/object, cleanup, pool exhaustion | | `TestEndToEnd` | CPU | β€” | E2E RDMA transfer: tensor, bytes, object, zero-copy, large payload (100MB), mixed types, concurrency | | `TestLifecycle` | CPU | β€” | Close, context manager, double-close safety | | `TestGPUPool` | GPU | β€” | GPU pool init, put CPU/GPU tensor, GPU E2E transfer | | `TestStressCorrectness` | CPU | `slow` | Concurrent put+get with MD5 integrity, bidirectional concurrency, edge cases (1-element tensor, empty bytes), 500MB payload, rapid alloc/free cycles | ### test_mooncake_transfer_engine_buffer.py β€” Test Classes | Test Class | Marker | Description | |------------|--------|-------------| | `TestBufferAllocator` | β€” | Basic alloc/free, alignment, exhaustion/recovery, thread safety | | `TestAllocatorInvariants` | `slow` | Double-free safety, overlap corruption detection, adjacent-block merging, fragmentation/defrag | | `TestManagedBuffer` | β€” | Tensor views, context manager | --- ## File: benchmarks/glm_image/README.md # GLM-Image Benchmarks Benchmark GLM-Image T2I (text-to-image) and I2I (image-to-image) performance across three backends: HuggingFace baseline, vLLM-Omni offline, and vLLM-Omni online serving. ## Benchmarks | Benchmark | Script | Description | |-----------|--------|-------------| | HuggingFace Baseline | `huggingface/inference.py` | Single-GPU transformers + diffusers pipeline | | vLLM-Omni Offline | `vllm-omni/inference.py` | Offline inference with continuous batching | | vLLM-Omni Online | `benchmark_glm_image.py` | Online serving via `/v1/chat/completions` | ## HuggingFace Baseline Single-request sequential inference using the reference HuggingFace pipeline. ```bash # T2I CUDA_VISIBLE_DEVICES=0 python benchmarks/glm_image/huggingface/inference.py \ --model-path /path/to/GLM-Image --mode t2i --num-prompts 10 # I2I CUDA_VISIBLE_DEVICES=0 python benchmarks/glm_image/huggingface/inference.py \ --model-path /path/to/GLM-Image --mode i2i --num-prompts 10 ``` ### Options | Flag | Default | Description | |------|---------|-------------| | `--model-path` | `zai-org/GLM-Image` | Model path | | `--mode` | `t2i` | `t2i` or `i2i` | | `--dataset-path` | `prompt/prompt.json` | Path to prompt.json | | `--num-prompts` | `10` | Number of images to generate | | `--width` / `--height` | `1024` | Output image size | | `--num-inference-steps` | `50` | Diffusion denoising steps | | `--output-dir` | `benchmarks/glm_image/huggingface/outputs` | Output directory | | `--output-file` | - | JSON file for metrics | ## vLLM-Omni Offline Multi-GPU offline inference with pipeline parallelism and continuous batching. ```bash # T2I CUDA_VISIBLE_DEVICES=0,1 python benchmarks/glm_image/vllm-omni/inference.py \ --model-path /path/to/GLM-Image --mode t2i --num-prompts 10 # I2I CUDA_VISIBLE_DEVICES=0,1 python benchmarks/glm_image/vllm-omni/inference.py \ --model-path /path/to/GLM-Image --mode i2i --num-prompts 10 ``` ### Options | Flag | Default | Description | |------|---------|-------------| | `--model-path` | `zai-org/GLM-Image` | Model path | | `--deploy-config` | - | Deploy config YAML | | `--mode` | `t2i` | `t2i` or `i2i` | | `--dataset-path` | `prompt/prompt.json` | Path to prompt.json | | `--num-prompts` | `10` | Number of images to generate | | `--width` / `--height` | `1024` | Output image size | | `--num-inference-steps` | `50` | Diffusion denoising steps | | `--output-dir` | `benchmarks/glm_image/vllm-omni/outputs` | Output directory | | `--output-file` | - | JSON file for metrics | | `--stage-init-timeout` | `600` | Stage initialization timeout (s) | ### Latency Computation In offline mode all requests are submitted simultaneously and processed with continuous batching. The per-request latency is computed by summing the actual per-stage times (with `stage_0_gen_ms` diffed against the previous request to remove accumulated queue/scheduling wait). ## vLLM-Omni Online Serving ### Start the server ```bash CUDA_VISIBLE_DEVICES=0,1 vllm serve /path/to/GLM-Image \ --omni --port 8091 --host 0.0.0.0 \ --served-model-name glm-image ``` ### Run the benchmark ```bash # T2I python benchmarks/glm_image/benchmark_glm_image.py \ --mode t2i --num-prompts 10 --model glm-image # I2I python benchmarks/glm_image/benchmark_glm_image.py \ --mode i2i --num-prompts 10 --model glm-image # Custom dataset python benchmarks/glm_image/benchmark_glm_image.py \ --mode i2i --dataset custom \ --dataset-path prompts.json --num-prompts 5 ``` ### Options | Flag | Default | Description | |------|---------|-------------| | `--mode` | `t2i` | `t2i` or `i2i` | | `--dataset` | `prompt` | `prompt`, `random`, or `custom` | | `--dataset-path` | - | JSON file path (required for `custom`) | | `--num-prompts` | `10` | Number of benchmark requests | | `--max-concurrency` | `1` | Max concurrent requests | | `--request-rate` | `inf` | Requests per second (Poisson arrival) | | `--warmup-requests` | `1` | Warmup requests before measurement | | `--width` / `--height` | `1024` | Output image size | | `--num-inference-steps` | `50` | Diffusion denoising steps | | `--seed` | - | Random seed | | `--model` | `default` | Model name (must match `--served-model-name`) | | `--host` | `localhost` | Server host | | `--port` | `8091` | Server port | | `--output-file` | - | JSON output file for metrics | | `--num-input-images` | `1` | Number of input images for random I2I | ## Dataset The default dataset is hosted on [HuggingFace](https://huggingface.co/datasets/JaredforReal/glm-image-bench) (`prompt.json`). It is automatically downloaded and cached to `prompt/prompt.json` on first run. No manual setup needed. Each entry contains: - `t2i_prompt`: Text prompt for text-to-image generation - `i2i_prompt`: Text prompt for image-to-image editing - `image_url`: Source image URL for I2I (downloaded and cached on first use) Custom datasets use the same JSON format and can be provided via `--dataset-path`. ## Pipeline Timings All three benchmarks report per-stage pipeline timings (in milliseconds): | Key | Description | |-----|-------------| | `preprocess_ms` | Input preprocessing (tokenization, multimodal encoding) | | `stage_0_gen_ms` | AR (autoregressive) model generation time | | `ar2diffusion_ms` | AR output to diffusion input conversion | | `stage_1_gen_ms` | Diffusion model denoising time | | `queue_wait_ms` | Queue wait time before processing | The stages are ordered by execution: `preprocess β†’ stage_0 (AR) β†’ ar2diffusion β†’ stage_1 (Diffusion)`. ## Sample Results Tested on 2x GPU with 10 prompts, 1024x1024, 50 denoising steps: | Backend | Mode | Latency Mean (s) | Throughput (img/s) | |---------|------|-------------------|--------------------| | HuggingFace | T2I | 72.6 | 0.014 | | HuggingFace | I2I | 70.9 | 0.014 | | vLLM-Omni Offline | T2I | 35.0 | 0.044 | | vLLM-Omni Offline | I2I | 31.0 | 0.053 | | vLLM-Omni Online | T2I | 38.8 | 0.026 | | vLLM-Omni Online | I2I | 34.7 | 0.029 | ## 2. Official Technical Reference & Guides (vllm-project/vllm-project.github.io) # vLLM blog source ## Local development 1. Install `jekyll` and `bundler` by running `gem install jekyll bundler`. Ask ChatGPT for help if you encounter any issues. 2. `bundle install` to install the necessary gems. 3. `rm ./Gemfile.lock` if you meet gem error. 5. `bundle exec jekyll server` to compile and start the server. To add a new blogpost, please refer to `_posts/2023-06-20-vllm.md` as an example. Some notes: - Google Doc can be saved as markdown format which will make your life easier. - Note that the blogpost will only show when its date is in the past. - Add `summary` front matter for SEO. Keep it concise, specific, and 240 characters or fewer. - Put figures under `assets/figures/yourblogname/`. - Make a pull request. The blog is automatically built and deployed by GitHub Actions when `main` is pushed to. ## LaTeX Math The blog supports LaTeX math via [MathJax](https://docs.mathjax.org/en/latest/index.html). It can be enabled by adding `math: true` to the document frontmatter. It has been configured to support the standard LaTeX style math notation, i.e.: ```latex $ inline math $ ``` ```latex $$ math block $$ ``` ## GitHub Flavored Admonitions The blog supports GitHub flavored admonitions via [jekyll-gfm-admonition](https://github.com/Helveg/jekyll-gfm-admonitions). It supports the following syntax: ```markdown > [!NOTE] > Highlights information that users should take into account, even when skimming. > And supports multi-line text. > [!TIP] > Optional information to help a user be more successful. > [!IMPORTANT] > Crucial information necessary for users to succeed. > [!WARNING] > Critical content demanding immediate > user attention due to potential risks. > [!CAUTION] > Negative potential consequences of an action. > Opportunity to provide more context. ``` ## Theme customization The theme we are using is [Minima](https://github.com/jekyll/minima). If you need to customise anything from this theme, see [Overriding theme defaults](https://jekyllrb.com/docs/themes/#overriding-theme-defaults).