## 1. Project Overview & Quickstart (vllm-project/vllm)
## File: README.md
Easy, fast, and cheap LLM serving for everyone
🔥 We have built a vLLM website to help you get started with vLLM. Please visit [vllm.ai](https://vllm.ai) to learn more.
For events, please visit [vllm.ai/events](https://vllm.ai/events) to join us.
---
## About
vLLM is a fast and easy-to-use library for LLM inference and serving.
Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.
vLLM is fast with:
- State-of-the-art serving throughput
- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
- Continuous batching of incoming requests, chunked prefill, prefix caching
- Fast and flexible model execution with piecewise and full CUDA/HIP graphs
- Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and [more](https://docs.vllm.ai/en/latest/features/quantization/index.html)
- Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton
- Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL
- Speculative decoding including n-gram, suffix, EAGLE, DFlash
- Automatic kernel generation and graph-level transformations using torch.compile
- Disaggregated prefill, decode, and encode
vLLM is flexible and easy to use with:
- Seamless integration with popular Hugging Face models
- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
- Tensor, pipeline, data, expert, and context parallelism for distributed inference
- Streaming outputs
- Generation of structured outputs using xgrammar or guidance
- Tool calling and reasoning parsers
- OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
- Efficient multi-LoRA support for dense and MoE layers
- Support for NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.
vLLM seamlessly supports 200+ model architectures on Hugging Face, including:
- Decoder-only LLMs (e.g., Llama, Qwen, Gemma)
- Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)
- Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)
- Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)
- Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)
- Reward and classification models (e.g., Qwen-Math)
Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
## Getting Started
Install vLLM with [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`:
```bash
uv pip install vllm
```
Or [build from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for development.
Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.
- [Installation](https://docs.vllm.ai/en/latest/getting_started/installation.html)
- [Quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart.html)
- [List of Supported Models](https://docs.vllm.ai/en/latest/models/supported_models.html)
## Contributing
We welcome and value any contributions and collaborations.
Please check out [Contributing to vLLM](https://docs.vllm.ai/en/latest/contributing/index.html) for how to get involved.
## Citation
If you use vLLM for your research, please cite our [paper](https://arxiv.org/abs/2309.06180):
```bibtex
@inproceedings{kwon2023efficient,
title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
year={2023}
}
```
## Contact Us
- For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm/issues)
- For discussing with fellow users, please use the [vLLM Forum](https://discuss.vllm.ai)
- For coordinating contributions and development, please use [Slack](https://slack.vllm.ai)
- For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm/security/advisories) feature
- For collaborations and partnerships, please contact us at [collaboration@vllm.ai](mailto:collaboration@vllm.ai)
## Media Kit
- If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit)
---
## File: benchmarks/kernels/deepgemm/README.md
# DeepSeek DeepGEMM Kernels Benchmark
This directory includes benchmarks between DeepSeek's DeepGEMM block fp8 kernels against vLLM's existing triton and CUTLASS-based kernels.
Currently, this just includes dense GEMMs and only works on Hopper GPUs.
## Setup
You need to install vLLM in your usual fashion, then install DeepGEMM from source in its own directory:
```bash
git clone --recursive https://github.com/deepseek-ai/DeepGEMM
cd DeepGEMM
python setup.py install
uv pip install -e .
```
## Usage
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## File: benchmarks/README.md
# Benchmarks
This directory used to contain vLLM's benchmark scripts and utilities for performance testing and evaluation.
## Contents
- **Serving benchmarks**: Scripts for testing online inference performance (latency, throughput)
- **Throughput benchmarks**: Scripts for testing offline batch inference performance
- **Specialized benchmarks**: Tools for testing specific features like structured output, prefix caching, long document QA, request prioritization, and multi-modal inference
- **Dataset utilities**: Framework for loading and sampling from various benchmark datasets (ShareGPT, HuggingFace datasets, synthetic data, etc.)
## Usage
For detailed usage instructions, examples, and dataset information, see the [Benchmark CLI documentation](https://docs.vllm.ai/en/latest/benchmarking/cli/#benchmark-cli).
For full CLI reference see:
-
-
-
## 2. Official Technical Reference & Guides (vllm-project/vllm-docs)
# vllm-docs-site
Independent rendering layer for **docs.vllm.ai**. Consumes markdown from
`vllm-project/vllm` (vendored as a git submodule at `external/vllm`) and
renders it through Next.js on Vercel.
> Design principle: source markdown stays in the main repo and is never
> modified. Site-side `content/content-map.yaml` provides the overlay; the
> renderer infers templates from path + title.
## Layout
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Local development
```bash
# 1. Install
pnpm install
# 2. Pull the vllm submodule (or set VLLM_REPO_PATH to a local clone)
git submodule update --init --recursive
# offline alternative:
# export VLLM_REPO_PATH=/path/to/local/vllm
# 3. Build the content bundle
pnpm bundle:latest
# 4. Dev server
pnpm dev # http://localhost:3030
```
Iteration loop:
- After a `content/content-map.yaml` edit or a `apps/docs/` edit, Next.js HMR
picks it up — no rebuild needed.
- After a source-markdown change (i.e. `external/vllm/docs/**`), rerun
`pnpm bundle:latest`.
```bash
pnpm typecheck # tsc --noEmit across the workspace
pnpm lint # next lint
pnpm build # bundle:latest + next build (production)
```
## Versions
The renderer is single-version today (the loader reads `bundle/latest/`).
The `build-bundle.sh` script and `pnpm bundle:{stable,nightly}` scripts
still exist and accept `VLLM_REF=v0.x.y` overrides — the multi-version
path is preserved for when stable / nightly channels come back.
```bash
VLLM_REF=v0.x.y pnpm bundle:stable
VLLM_REF=main pnpm bundle:nightly
```
Legacy ReadTheDocs URLs (`/en/stable/...`, `/latest/...`, etc.) are
redirected back to root by `next.config.mjs` and `vercel.json`.
## What is intentionally NOT here
- **Markdown frontmatter requirements.** Source `.md` files are consumed
as-is. Per-page customization lives in `content-map.yaml`, not on the
page.
- **A copy of vllm content.** We always read from the submodule.
- **Bidirectional sync.** The bundle is a build-time artifact only.
## Site-authored heroes
Most architecture / flow / topology pages carry an SVG hero infographic.
The 33 site-authored SVGs live under `apps/docs/public/_vllm-assets/_site/`
and are tracked in git. The build script clears the rest of
`_vllm-assets/` on each run (it's a mirror of upstream
`external/vllm/docs/assets/`) but preserves `_site/`.
To wire a new hero:
1. Drop the SVG at `apps/docs/public/_vllm-assets/_site//.svg`
2. Reference it from `content-map.yaml` `pages..hero`:
```yaml
hero:
type: svg
src: ../assets/_site//.svg
alt: ...
caption: ...
aspect: 960/400
```
3. For pages inside a sub-directory (`features/speculative_decoding/README.md`),
the src needs an extra `../` step: `../../assets/_site/...`
The Hero component resolves doc-relative `../assets/` paths to the
`/_vllm-assets/` mount automatically.
## Status
What's wired:
- single-version bundle pipeline (`bundle/latest/bundle.json`)
- 6 templates, 6 sections, content-map overlay
- markdown → HTML pipeline with callouts, math, code-example collapsibles,
autoref, asset-paths, gh-* url schemes
- Python extractor scaffold (argparse / api / metrics / examples dumps)
- header search, theme toggle (light / dark / system), favicon
- 33 site-authored hero SVGs across high-traffic pages
- Vercel deploy via `scripts/vercel-install.sh` + `pnpm build`
What's deferred:
- Bringing `stable` and `nightly` back as live channels (the bundle
pipeline already supports `VLLM_REF` and the CI workflow is multi-leg).
- Search ranking tuning beyond the MiniSearch defaults.
- The Choose / decision-flow surface (intentionally absent — see
`CLAUDE.md`).