{"owner":"vllm-project","repo":"aibrix","hasSkills":true,"totalSkillsCount":13,"totalTokensCount":43489,"categories":["anthropic-skill","root-instruction","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["pkg/plugins/gateway/algorithms/multi_router_readme.md","pkg/plugins/gateway/algorithms/prefix_cache_readme.md","AGENTS.md","pkg/plugins/gateway/algorithms/pd_readme.md","pkg/plugins/gateway/ENV_VARS.md","pkg/plugins/gateway/algorithms/vtc_readme.md","pkg/plugins/gateway/ratelimiter/README.md","pkg/plugins/gateway/statesync/README.md","pkg/plugins/gateway/algorithms/external/testdata/denied.json","pkg/plugins/gateway/testdata/envoy_gateway_v1.2.8_xds.json","pkg/plugins/gateway/algorithms/external/testdata/no-decision.json","pkg/plugins/gateway/algorithms/load_balance.md","pkg/plugins/gateway/algorithms/external/testdata/selected.json"],"skills":{"pkg/plugins/gateway/algorithms/multi_router_readme.md":"# Multi-Strategy Routing (Batch Soft Scoring)\n\nAIBrix Gateway supports multi-strategy routing, allowing you to combine multiple routing algorithms with different weights. Instead of making hard decisions (where one strategy completely filters out pods or dictates the single winner), the multi-strategy router uses a **Batch Soft Scoring** mechanism to evaluate pods across all specified strategies.\n\n## How it works\n\n1. **Batch Scoring:** Each configured strategy evaluates all ready pods in a single batch operation (`ScoreAll`), returning raw scores and a boolean array indicating whether each pod could be scored (e.g., handling missing metrics).\n2. **Normalization:** The raw scores from each strategy are normalized into a `[0, 1]` range:\n * **Higher is better** (e.g., `throughput`): `(score - min) / (max - min)`\n * **Lower is better** (e.g., `least-request`): `(max - score) / (max - min)`\n * If a pod lacks metrics (`scored=false`), its normalized score is `0.0` (acting as a penalty).\n3. **Weighted Aggregation:** The normalized scores are multiplied by their respective configured weights and summed up for each pod.\n4. **Top-1 Selection:** The pod with the highest total aggregated score is selected to serve the request.\n\n## Configuration Format\n\nYou can configure multi-strategy routing by setting the `AIBRIX_ROUTING_ALGORITHM` environment variable using a comma-separated list of `strategy:weight` pairs.\n\n* **Format:** `strategy1:weight1,strategy2:weight2,...`\n* If `:weight` is omitted, it defaults to `1`.\n* If a weight is `0`, the strategy is completely ignored.\n* If an invalid strategy is specified, the Gateway rejects the request with **400 Bad Request** instead of silently falling back to `random`.\n* `session-affinity` is treated as a soft-scoring strategy in multi-strategy mode: it boosts the matching pod, but the final weighted winner may still be a different pod. The response session header is updated to the final selected pod.\n With Redis-backed pinning, background sync applies its result only if the local entry has not changed since the read began. A delayed result cannot overwrite or evict a pin committed by a concurrent request.\n* A caller may send an opaque application session identifier in `x-aibrix-session-key`. When no valid gateway-issued `x-session-id` target is ready, rendezvous hashing selects a stable pod for that key. This is useful for concurrent Agent turns that start before a response cookie is available. The opaque key is never returned to the client. The `pd` router's `token_load` and `hybrid_cache_load` prefill policies read the same key to charge a multi-turn conversation only for the tokens added since its previous turn.\n\n## Examples\n\nHere are some real-world examples demonstrating how the weighted aggregation works under different scenarios.\n\nAssume we have two pods serving a model:\n* **Pod A**: High load (10 requests) but High throughput (100 tok/s).\n* **Pod B**: Low load (2 requests) but Low throughput (10 tok/s).\n\n### Example 1: `least-request` Dominates\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:10,throughput:1\"\n```\n* `least-request` (weight 10): Pod B is much better (2 vs 10). Normalized: Pod A = 0.0, Pod B = 1.0. Weighted Score: A = 0.0 * 10/11 = 0.0, B = 1.0 * 10/11 ≈ 0.91.\n* `throughput` (weight 1): Pod A is much better (100 vs 10). Normalized: Pod A = 1.0, Pod B = 0.0. Weighted Score: A = 1.0 * 1/11 ≈ 0.09, B = 0.0 * 1/11 = 0.0.\n* **Total Score**: Pod A ≈ 0.09, Pod B ≈ 0.91. **Pod B wins!**\n\n### Example 2: `throughput` Dominates\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,throughput:10\"\n```\n* `least-request` (weight 1): Weighted Score: A = 0.0 * 1/11 = 0.0, B = 1.0 * 1/11 ≈ 0.09.\n* `throughput` (weight 10): Weighted Score: A = 1.0 * 10/11 ≈ 0.91, B = 0.0 * 10/11 = 0.0.\n* **Total Score**: Pod A ≈ 0.91, Pod B ≈ 0.09. **Pod A wins!**\n\n### Example 3: Complex Scenario with Corner Cases (3 Strategies)\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request,throughput:2,least-latency:0\"\n```\n* `least-request`: No weight specified, **defaults to 1**. Weighted Score: A = 0.0 * 1/3 = 0.0, B = 1.0 * 1/3 ≈ 0.33.\n* `throughput`: Explicitly set to weight **2**. Weighted Score: A = 1.0 * 2/3 ≈ 0.67, B = 0.0 * 2/3 = 0.0.\n* `least-latency`: Weight set to **0**, so this strategy is **ignored** and does not participate in scoring.\n* **Total Score**: Pod A ≈ 0.67, Pod B ≈ 0.33. **Pod A wins!**\n\n### Example 4: Invalid Configuration Fallback\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,invalid-strategy:1\"\n```\n* Because `invalid-strategy` is not a registered algorithm, the multi-strategy parser will fail. To preserve the API contract, the Gateway will reject the request with a **400 Bad Request** error rather than silently falling back to a random router.\n\n### Example 5: Multi-Strategy Finding the Sweet Spot\nIn reality, a pod with the absolute minimum active requests might have just recovered from being severely overloaded. Multi-strategy routing prevents routing to extreme outliers by considering multiple dimensions.\n\nAssume three pods with the following metrics:\n* **Pod A**: Very low active requests (1 req), but historically massively overloaded (1500 tokens).\n* **Pod B**: Historically barely used (15 tokens), but currently slammed with active requests (50 req).\n* **Pod C**: The \"Sweet Spot\" - Medium active requests (10 req), and medium historical load (150 tokens).\n\n**Single Strategy Flaws:**\n* `AIBRIX_ROUTING_ALGORITHM=\"least-request\"`: Completely ignores historical load and picks **Pod A** (which is exhausted).\n* `AIBRIX_ROUTING_ALGORITHM=\"throughput\"`: Completely ignores current spikes and picks **Pod B** (which is currently slammed).\n\n**Multi-Strategy Solution:**\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,throughput:1\"\n```\n* `least-request` (weight 1): Normalizes active requests [1, 50]. \n * Pod A: 1.0\n * Pod B: 0.0\n * Pod C: 0.816\n* `throughput` (weight 1): Normalizes historical load [15, 1500].\n * Pod A: 0.0\n * Pod B: 1.0\n * Pod C: 0.909\n* **Total Score (Weighted)**: Pod A = 0.5, Pod B = 0.5, Pod C ≈ **0.86**. \n* **Result**: **Pod C wins!** The router correctly identifies the most balanced pod, avoiding the exhausted Pod A and the currently slammed Pod B.\n","pkg/plugins/gateway/algorithms/prefix_cache_readme.md":"# Prefix Cache Aware Routing\n\nPrefix Cache Aware routing improves KV cache reuse across GPU pods by steering requests whose\nprompt shares a common prefix to the pod that already has those KV blocks cached. Pod selection\namong prefix-matched candidates is still load-aware (see Step 2 below), and this router itself\nis purely about prefix caching — it does not gate or restrict candidates based on cluster-wide\nload imbalance. That concern is instead handled centrally by the gateway: `selectTargetPod` in\n`gateway.go` applies the [load-balance router](load_balance.go)'s `ApplyLoadImbalanceGate` once,\nahead of whichever strategy actually routes the request (prefix-cache included), narrowing\n`readyPods` before this router ever sees them. This replaced an older prefix-cache-specific gate\nthat lived in this file; see [ENV_VARS.md](../ENV_VARS.md#variable-dependency-notes) for the\nmigration note. The gate applies the same way when combining strategies via multi-strategy\nsoft-scoring (`AIBRIX_ROUTING_ALGORITHM=prefix-cache:W1,load-balance:W2`, see\n[multi_router_readme.md](multi_router_readme.md)): `selectTargetPod` narrows `readyPods` once,\nbefore resolving and calling into the multi-strategy router, so every sub-strategy's `ScoreAll`\n— this one included — only ever sees the already-gated pod list. It is skipped only for\nexclusive strategies (`pd`, `slo*`), which manage their own pod subsets.\n\nTwo routing modes are supported, selected automatically at startup:\n\n| Mode | When active | Prefix index |\n|------|-------------|--------------|\n| **Standard** | `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=false` (default) | Local hash table updated by the router after each request |\n| **KV Sync** | `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true` | Distributed sync indexer fed by real-time block-stored/removed events from vLLM pods |\n\nBoth modes share the same stddev-based pod selection logic described below.\n\n## Routing Flow\n\n```\n Incoming Request\n │\n ▼\n┌─────────────────────────────┐\n│ Tokenize & hash prompt │ splits prompt into fixed-size blocks,\n│ into prefix blocks │ computes hash per block\n└────────────┬────────────────┘\n │\n ▼\n┌─────────────────────────────┐\n│ Match prefix hashes │\n│ against ready pods │\n└────────────┬────────────────┘\n │\n ┌────────┴────────┐\n │ match_pods │ NO match_pods\n │ found? │──────────────────────────────────────────────┐\n └────────┬────────┘ │\n │ YES │\n┌─────────────────────────────┐ │\n│ Sort match_pods by: │ │\n│ 1. prefix_match% DESC │ │\n│ 2. running_requests ASC │ │\n└────────────┬────────────────┘ │\n │ │\n┌─────────────────────────────┐ NO pod within threshold │\n│ Select first pod where: │─────────────────────────────────────►│\n│ running_req <= │ │\n│ mean + load_factor * σ │ │\n└────────────┬────────────────┘ │\n │ │\n │ ▼\n │ Least-request fallback among all ready pods\n │ │\n ▼ ▼\n Target Pod Selected\n```\n\n## Algorithm Details\n\n### Step 1 — Tokenize & Hash\n\nThe prompt is tokenized and split into fixed-size blocks. A hash is computed for each block.\nTwo tokenizer backends are supported:\n\n| Tokenizer | How it works |\n|------------|-------------------------------------------------------|\n| `character`| Splits raw text into individual characters (default) |\n| `tiktoken` | Uses the OpenAI [tiktoken](https://github.com/openai/tiktoken) BPE tokenizer |\n\nBlock hashes are used as cache keys — a pod that has processed a request with an identical\nprefix will have those KV blocks hot in GPU memory.\n\n### Step 2 — Prefix Match & Pod Selection\n\nEach candidate pod is scored by how many of the request's prefix blocks it already holds\n(`prefix_match_percent`). Matched pods are sorted:\n\n1. **Descending** by `prefix_match_percent` — prefer higher cache hit rate.\n2. **Ascending** by `running_requests` — break ties by choosing the less loaded pod.\n\nThe first pod in this sorted list that satisfies the load threshold is selected:\n\n```\npod.running_requests <= mean_running_requests + load_factor × std_dev\npod.running_requests <= max(load_cap_factor, 1) × (mean_running_requests + 1)\n```\n\n`load_factor` (= `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR`) controls how aggressively\nthe router skips overloaded cache-holding pods. If no matched pod is within the threshold, the\nrouter falls back to the global least-loaded pod across all ready pods. `load_cap_factor`\n(= `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR`) bounds load directly: no value in n pods sits more than\n(n−1)/√n standard deviations above the mean, so at `load_factor` 2 the first check cannot skip\nany pod in a pool of five or fewer.\n\n## Configuration\n\n| Environment Variable | Description | Default |\n|---|---|---|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | Tokenizer used to split the prompt. Options: `character`, `tiktoken`. | `character` |\n| `AIBRIX_PREFIX_CACHE_BLOCK_SIZE` | Number of tokens per prefix block. Smaller blocks = finer-grained matching but more hash overhead. | `128` |\n| `AIBRIX_PREFIX_CACHE_BLOCK_NUMBER` | Maximum number of prefix cache blocks tracked per pod. | `200000` |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `load_factor` in `mean + load_factor × σ`. Higher value tolerates more load on a high-cache-hit pod. | `1` |\n| `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR` | `load_cap_factor` in `max(load_cap_factor, 1) × (mean + 1)`. Higher value tolerates more load on a high-cache-hit pod; set a large value such as `100` to effectively disable it. Applies only when `prefix-cache` routes on its own, with the auto-blend off (`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT=0`). | `1.5` |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED` | Enable KV sync routing mode (requires remote tokenizer). | `false` |\n| `AIBRIX_PREFIX_CACHE_INCLUDE_TOOLS` | Prepend the canonicalized chat request `tools` to the prompt text before hashing, so requests that share messages but carry different tools do not look like a full prefix match. Token estimates keep using the messages only. KV-sync mode tokenizing through the remote chat template forwards the raw `tools` to the tokenizer regardless of this flag, so the template places them. Set to `false` to hash the messages only. | `true` |\n\nSee [ENV_VARS.md](../ENV_VARS.md#load-balance-router-algorithmsload_balancego) for the\nload-imbalance gate's `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` /\n`AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` variables — they belong to the load-balance router, not\nthis one.\n\n### Tuning guidance\n\n- **Block size**: use `128` for `character` tokenizer and `16` for `tiktoken`. Larger blocks\n reduce hash overhead but require longer identical prefixes for a match.\n- **Standard deviation factor**: `1` (default) keeps all pods within one stddev of mean load.\n Increase to `2` to allow higher-loaded pods to still be selected for cache-hit requests.\n","AGENTS.md":"# AIBrix\n\nAIBrix is a Kubernetes-native platform for building scalable GenAI inference\ninfrastructure. This repository contains Go controllers, gateway plugins,\ncustom resources, deployment manifests, tests, and Python runtime components.\nThis file gives coding agents the repository-specific context and safety rules\nneeded to make focused changes.\n\n## Scope and repository layout\n\n| Path | Role |\n|------|------|\n| `api/` | Kubernetes API types and CRD definitions |\n| `pkg/controller/` | Kubernetes controllers and reconciliation logic |\n| `pkg/plugins/` | Gateway plugins and request processing |\n| `pkg/` | Shared Go libraries, clients, caches, metrics, and utilities |\n| `cmd/` | Go binary entrypoints |\n| `config/` | Kustomize configuration, CRDs, RBAC, and deployment manifests |\n| `test/integration/` | Ginkgo-based integration tests |\n| `test/e2e/` | Kind and cluster-level end-to-end tests |\n| `hack/` | Code generation, verification, and CI scripts |\n| `python/aibrix/` | Python runtime, downloader, batch, metadata, and optimizer |\n| `python/aibrix_kvcache/` | Python distributed KV-cache components |\n| `apps/` | Application services and the console |\n| `docs/` | Project documentation and generated documentation assets |\n| `samples/` | Example deployments and feature samples |\n\nRules in a deeper `AGENTS.md` take precedence for files under that directory.\nThe Python runtime has additional guidance in\n[`python/aibrix/AGENTS.md`](python/aibrix/AGENTS.md).\n\n## Working rules\n\n- State the intended scope and success criteria before making a non-trivial\n change.\n- Read the closest existing implementation and its tests before introducing a\n new pattern.\n- Keep changes focused. Do not combine unrelated refactors, formatting, or\n generated output with a feature or bug fix.\n- Verify behavior from code and tests rather than inferring it from filenames.\n- Preserve unrelated user changes in the working tree.\n- Do not push branches, rewrite history, modify remote GitHub state, or send\n external messages without explicit authorization for that action.\n- Ask before upgrading dependencies or changing `.github/` workflow policy.\n\n## API and compatibility surfaces\n\nTreat the following as compatibility surfaces:\n\n- Kubernetes API types, CRD schemas, JSON/YAML tags, defaults, validation,\n list semantics, subresources, and printer columns.\n- Public HTTP endpoints, request and response fields, headers, status codes,\n and gateway plugin contracts.\n- CLI flags, configuration keys, environment variables, labels, and\n annotations used by deployment or controller logic.\n\nDo not rename, remove, or repurpose an existing field or key without an\nexplicit migration and compatibility plan. Preserve pointer and `omitempty`\nchoices when absent, zero, and explicit values have different meanings.\nKeep status fields observational; status transitions must remain consistent\nwith every controller that writes or consumes them.\n\nKeep API types declarative. Admission behavior belongs in webhooks,\nreconciliation policy belongs in controllers, and transport compatibility\nbelongs in the relevant API or gateway layer.\n\n## Build and verification\n\nRun commands from the repository root. Start with the narrowest relevant check\nand run broader checks for changes that cross package or language boundaries.\n\n| Command | Purpose |\n|---------|---------|\n| `make fmt` | Format Go code |\n| `make vet` | Run `go vet ./...` |\n| `make generate` | Regenerate Go and Kubernetes generated artifacts |\n| `make manifests` | Regenerate webhook, RBAC, and CRD manifests |\n| `make verify` | Verify generated code and CRD synchronization |\n| `make lint` | Run Go linting |\n| `make lint-all` | Run license and Go lint checks |\n| `make test` | Run Go unit tests, excluding integration and E2E packages |\n| `make test-race-condition` | Run Go unit tests with the race detector |\n| `make test-integration` | Run integration tests |\n| `make python-ci` | Run Python Ruff, mypy, and tests for `python/aibrix` |\n| `make test-e2e` | Run Kind-based end-to-end tests |\n\nFor a Go package, a focused check such as `go test ./pkg//...` is\nappropriate while iterating. For cluster-level behavior, use the existing\nhelpers and test organization under `test/e2e/`; do not replace polling with\narbitrary sleeps.\n\nWhen API types, controller-gen markers, or CRDs change, run the applicable\ngeneration and verification targets and include the resulting generated files\nin the change. Do not hand-edit generated files unless the generator explicitly\nrequires it.\n\n## Coding conventions\n\n- Follow existing Go package structure and standard Go style; use `gofmt` or\n `make fmt`.\n- Put reusable implementation in `pkg/`; keep `cmd/` entrypoints thin.\n- Use table-driven tests where they match the surrounding package style.\n- Add or update tests for behavior changes, including failure and compatibility\n cases when applicable.\n- Comments should explain non-obvious reasons or invariants, not restate code.\n- Preserve existing logging, error handling, retry, timeout, and context\n propagation patterns unless the change specifically addresses them.\n- For Python changes, follow the more specific rules in\n [`python/aibrix/AGENTS.md`](python/aibrix/AGENTS.md) and run checks from that\n subtree as documented there.\n\n## Pull requests and documentation\n\n- Keep a PR focused on one issue or coherent behavior change.\n- Use the repository [PR template](.github/PULL_REQUEST_TEMPLATE.md), link the\n relevant issue, and describe the tests actually run.\n- Update user or developer documentation when behavior, configuration, API,\n deployment, or CLI usage changes.\n- When changing any `docs/source/**/*.rst` file, update the corresponding\n `docs/source/locale/zh_CN/LC_MESSAGES/**/*.po` catalog in the same change,\n including added, modified, and removed content. Do not leave new or changed\n documentation falling back to English on the Chinese site.\n- Before submitting an RST documentation change, run `make html-all` from\n `docs/` and confirm that both the English and Simplified Chinese HTML builds\n succeed.\n- Large design changes should be discussed before implementation and recorded\n in the appropriate documentation area.\n- Review every generated or AI-assisted change yourself; do not submit code\n that you cannot explain and verify.\n\n## Further reading\n\n- [Contributing guide](CONTRIBUTING.md)\n- [Development guide](development/README.md)\n- [Test guide](test/README.md)\n- [Project documentation](docs/README.md)\n","pkg/plugins/gateway/algorithms/pd_readme.md":"# PD Disaggregation Router\n\n`pd/router/router.go` implements the **prefill-decode (PD) disaggregated routing algorithm** (`RouterPD = \"pd\"`). **This file (`pd_readme.md`) is the canonical documentation** for PD: request flow, scoring, engine behavior, metrics, and environment variables. PD-only settings are under **Environment Variables**; variables shared with the prefix-cache router (tokenizer, stddev factor, and related `AIBRIX_PREFIX_CACHE_*` knobs) are listed in **Inherited from Prefix Cache Router** in the same section.\n\nIn PD disaggregation, inference is split across two specialized pod roles:\n\n- **Prefill pod** — processes the prompt (context), builds the KV-cache, then transfers it.\n- **Decode pod** — generates tokens using the KV-cache transferred from the prefill pod.\n\nThe router is responsible for selecting one prefill pod and one decode pod per request, executing the prefill HTTP request synchronously (vLLM/TRT-LLM context-first) or asynchronously (SGLang/TRT-LLM generation-first), and routing the decode request through Envoy to the selected decode pod.\n\n---\n\n## High-Level Request Flow\n\n```\nClient Request\n │\n ▼\nRoute(ctx, readyPodList)\n │\n ├─► ValidateAndGetLLMEngine()\n │ Ensure all pods use the same engine (vllm / sglang / trtllm)\n │\n ├─► filterPrefillDecodePods()\n │ ┌─────────────────────────────────────────────────────┐\n │ │ collectAndBucketPods() │\n │ │ Phase 1: group by roleset (prefill / decode) │\n │ │ Phase 2: apply prompt-length bucketing filter │\n │ └─────────────────────────────────────────────────────┘\n │ │\n │ ├─► [Prompt-length bucketing ON]\n │ │ → combined pod path (see Combined Pod section)\n │ │\n │ ├─► loadImbalanceSelectPrefillPod() (fast path)\n │ ├─► loadImbalanceSelectDecodePod() (fast path)\n │ ├─► scorePrefillPods()\n │ ├─► scoreDecodePods()\n │ └─► finalPDScore() → (prefillPod, decodePod)\n │\n ├─► AddPendingDecode() + defer RemovePendingDecode() (all paths)\n │\n ├─► [prefillPod != nil]\n │ AddPrefillRequest()\n │ doPrefillRequest(ctx, prefillPod, handler)\n │ ├─ SGLang → async goroutine (bootstrap handshake; Route does not wait for completion)\n │ ├─ vLLM → sync, extract kv_transfer_params from response\n │ └─ TRT-LLM → context_first: sync, merge response params\n │ generation_first: prepare both bodies, async prefill\n │\n └─► ctx.SetTargetPod(decodePod)\n return decodePod address\n```\n\n---\n\n## Pod Classification\n\nPods are grouped using Kubernetes labels:\n\n| Label | Values | Purpose |\n|-------|--------|---------|\n| `roleset-name` | any string | Groups prefill+decode pairs into a \"roleset\" |\n| `role-name` | `prefill` / `decode` / `encode` / other | Pod's role in PD disaggregation. `encode` pods are optional and only take the encode stage (see [vLLM encode stage (EPD)](#vllm-encode-stage-epd)) |\n| `stormservice.orchestration.aibrix.ai/pod-group-index` | `\"0\"` or absent | Only pods with index `\"0\"` (or no label) host the HTTP server (multi-node TP) |\n| `stormservice.orchestration.aibrix.ai/role-replica-index` | any string | Distinct replica index within a role; used by `roleReplicaCardinality` to count unique replicas across multi-node TP groups. When absent, each routable pod counts as its own replica. |\n| `model.aibrix.ai/engine` | `vllm` / `sglang` / `trtllm` | LLM engine type |\n\nOnly rolesets with **both** at least one prefill replica and one decode replica are eligible. Incomplete rolesets are excluded. See [Roleset Eligibility Utilities](#roleset-eligibility-utilities) for diagnostic functions.\n\n---\n\n## Roleset Eligibility Utilities\n\n`pd/router/roleset.go` provides helper functions used by both the router and the gateway's pod-readiness logic.\n\n### Replica cardinality\n\nA roleset's replica count is the number of **distinct** values of `stormservice.orchestration.aibrix.ai/role-replica-index`. When that label is absent on all pods of a role, each routable pod counts as its own replica. This correctly handles multi-node tensor-parallel groups where multiple pods share one logical replica slot.\n\n```\nroleReplicaCardinality(pods):\n if any pod has role-replica-index label:\n count distinct index values\n else:\n count routable pods\n```\n\n### Eligibility checks\n\nA roleset is **eligible** when it has ≥1 prefill replica **and** ≥1 decode replica.\n\n| Function | Use |\n|----------|-----|\n| `HasEligibleRoleset(readyPods)` | Gateway check: at least one roleset is fully ready for PD routing |\n| `eligiblePDRolesets(readyPods)` | Internal: returns only complete rolesets for scoring |\n| `DescribePDRolesetEligibility(readyPods)` | Diagnostics: per-roleset replica counts and eligibility flag |\n\n`ValidateAndGetLLMEngine` (called from the gateway before `Route`) checks that all ready pods for the model share the same `model.aibrix.ai/engine` label.\n\n---\n\n## Pod Selection Flow\n\n```\ncollectAndBucketPods()\n │\n ▼\n┌───────────────────────────────┐\n│ prefillPods / decodePods │\n│ (grouped by eligible │\n│ rolesets only) │\n└───────────────────────────────┘\n │\n ▼\nloadImbalanceSelectPrefillPod()\n ┌─────────────────────────────────────────────────────────┐\n │ max(outstanding_prefill_reqs) - min > MIN_SPREAD? │\n │ → YES: narrow to single least-loaded prefill pod │\n │ align decodePods to selected pod's roleset │\n │ (continues to decode fast path below) │\n │ → NO: continue with all prefill pods │\n └─────────────────────────────────────────────────────────┘\n │\n │ NOTE: the two fast paths are independent. Both can fire on the\n │ same request if both sides are imbalanced: prefill narrows first,\n │ then decode narrows within the already-aligned decode set.\n │\n ▼\nloadImbalanceSelectDecodePod() (3 ordered checks, metrics-bearing pods only)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Effective request count = RealtimeNumRequestsRunning │\n │ + PendingDecodeTracker count │\n │ Pending counts bridge the gap between pod selection and the │\n │ first metric update, preventing thundering-herd on a newly │\n │ selected pod whose running count has not yet ticked up. │\n │ │\n │ Check 1 – Request count spread │\n │ Only pods with RealtimeNumRequestsRunning metric participate │\n │ (pods without the metric are excluded from spread; their │\n │ pending count is still stored for scoring — prevents │\n │ thundering-herd on freshly restarted pods) │\n │ max(observed running+pending) - min >= DECODE_LOAD_SPREAD? │\n │ → YES: return least-loaded observed pod; │\n │ align prefillPods to selected pod's roleset │\n │ │\n │ Check 2 – Throughput spread │\n │ Only pods with AvgGenerationThroughputToksPerS participate │\n │ max(observed throughput) - min > THROUGHPUT_SPREAD? │\n │ → YES: return lowest-throughput observed pod │\n │ │\n │ Check 3 – Drain rate score (soft path) │\n │ all pods have drain_rate > 0? │\n │ score = effective_running_reqs / drain_rate │\n │ max_score / min_score > DECODE_SCORE_RATIO? │\n │ → YES: return pod with lowest drain-rate score │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nscorePrefillPods() (per roleset, best pod wins)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Skip pods with req_count > mean + N * stddev │\n │ (N = AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR) │\n │ Skip pods with req_count > max(c, 1) * (mean + 1) │\n │ (c = AIBRIX_PREFILL_LOAD_CAP_FACTOR) │\n │ When all pods in a roleset are filtered, that roleset has no │\n │ entry in the output map and finalPDScore cannot select it. │\n │ │\n │ Policy: prefix_cache (default) │\n │ score = (100 - match_pct) * 0.1 + req_cnt / max_req_cnt │\n │ │\n │ Policy: least_request │\n │ score = req_cnt │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nscoreDecodePods() → pd.DecodeScoreRun (per roleset best pod + MaxScore + policy metadata)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Cold-start (if some pods have metrics but this pod does not): │\n │ score = 1.0 + pending_decode_count │\n │ (neutral idle-warm score; pod competes but is not favored) │\n │ │\n │ Policy: load_balancing (default) │\n │ norm_reqs = running_reqs / max_running_reqs │\n │ norm_thru = 1 - throughput / max_throughput │\n │ norm_free_gpu = free_gpu_pct / max_free_gpu_pct │\n │ score = (w_run*norm_reqs + w_thru*norm_thru) / norm_free_gpu │\n │ (w_run, w_thru from AIBRIX_DECODE_LB_WEIGHT_*, default 1) │\n │ NaN scores: skip; non-NaN Inf possible if norm_free_gpu = 0 │\n │ │\n │ Policy: least_request │\n │ score = running_reqs (with pending) │\n │ │\n │ If primary policy yields NaN, one load_balancing retry per pod │\n │ lower is better │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nfinalPDScore()\n ┌──────────────────────────────────────────────────────────────────┐\n │ For each roleset with both prefill and decode scores: │\n │ final = norm_prefill_score + norm_decode_score │\n │ Pick roleset with minimum final score │\n │ → selectedPrefillPod, selectedDecodePod │\n │ │\n │ Errors: │\n │ • prefillScores nil → Prepare failed in scorer │\n │ • prefillScores empty → all prefill pods filtered by │\n │ stddev or cap (no candidate) │\n │ • no roleset intersection → load-imbalance alignment left │\n │ prefill/decode on different sets │\n └──────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Prefill Score Policies\n\nControlled by `AIBRIX_PREFILL_SCORE_POLICY`, or overridden per request via the model **`routingConfig`** in `model.aibrix.ai/config` (see [Config profile overrides](#config-profile-overrides-for-pd-score-policies)).\n\n### `prefix_cache` (default)\n\nUses the shared `PrefixHashTable` to find how much of the request's token prefixes are already cached on each prefill pod.\n\n```\ntokens = tokenize(request.message)\nmatchedPods, hashes = prefixCacheIndexer.MatchPrefix(tokens, model, readyPodsMap)\n\nscore = (100 - match_percent) * 0.1 + req_count / max_req_count\n```\n\n- Lower score = better (more cache hits, fewer outstanding requests).\n- After routing, the prefix hashes are enqueued to `prefixUpdateCh` for asynchronous indexer update.\n- Tokenizer is selected by `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` (same as the prefix-cache router): `tiktoken` or `character` (default).\n\n### `least_request`\n\nNo prefix cache consultation. Score equals the raw running request count on the pod.\n\n```\nscore = req_count\n```\n\n---\n\n## Decode Score Policies\n\nImplementation lives in `algorithms/pd/decode_scorer.go` (`package pd`). Policies are selected by **`pd.ResolveDecodePolicy`** from `AIBRIX_DECODE_SCORE_POLICY`. Extra names can be registered at init time with **`pd.RegisterDecodePolicy`**. Per-pass results use **`pd.DecodeScoreRun`** (map of roleset → best pod/score, `MaxScore` for normalization, optional `Err`, `FallbackUsed`, `Policy`).\n\nMetrics are passed per pod via **`pd.DecodePodInput`** (running/throughput/free GPU and batch maxima). The router guarantees positive maxima from `loadImbalanceSelectDecodePod` as today.\n\nControlled by `AIBRIX_DECODE_SCORE_POLICY`, or overridden per request via **`routingConfig`** in the model config profile (see [Config profile overrides](#config-profile-overrides-for-pd-score-policies)).\n\nMetrics are pulled from the cache per pod (same maps as `loadImbalanceSelectDecodePod`):\n- `RealtimeNumRequestsRunning` + `PendingDecodeTracker` count\n- `AvgGenerationThroughputToksPerS` (per model)\n- `KVCacheUsagePerc` (free = 100 − usage%)\n\n**Cold-start gate**: if at least one pod has `RealtimeNumRequestsRunning`, pods that lack it are given a neutral **cold-start score** (`1.0 + pending_decode_count`) rather than full policy scoring. When *no* pod has that metric yet, all pods are scored normally.\n\n### `load_balancing` (default)\n\nBalances normalized queue depth, inverse throughput, and free GPU headroom:\n\n```\nnorm_running_reqs = running_reqs_with_pending / max_running_reqs\nnorm_throughput = 1 - avg_gen_throughput_tok_s / max_throughput\nnorm_free_gpu = free_gpu_percent / max_free_gpu_percent\n\ndecode_score = (w_run * norm_running_reqs + w_thru * norm_throughput) / norm_free_gpu\n```\n\n`w_run` and `w_thru` default to `1` (historical equal weighting). Lower score is better (fewer outstanding requests, higher throughput, more free GPU).\n\n### `least_request`\n\nNo throughput or GPU terms. Score equals raw running decode request count (including pending decode), same units as `RealtimeNumRequestsRunning` + pending.\n\n```\ndecode_score = running_reqs_with_pending\n```\n\n### `token_load`\n\nScores from the gateway's decode ledger in `pd.TokenLoadTracker` rather than from request counts or scraped KV usage. Each request is charged its `prompt_tokens` to the selected decode pod, under `selectMu` together with the other selection bookkeeping, and released on request completion or prefill failure (`releaseTokenLoad`, or the prefill executor for a terminal async failure). The whole prompt is charged because the decode pod receives all of its KV; there is no fixed per-request cost (`AIBRIX_TOKEN_LOAD_REQUEST_COST` is prefill-only).\n\n```\ndecode_score = decode_tokens + generated_tokens\ngenerated_tokens = rate × Σ over outstanding charges (now − routed_at)\nrate = AvgGenerationThroughputToksPerS / running requests (mean of the other pods if unknown)\n```\n\nWhen the prefill response carries `usage.prompt_tokens`, the prefill executor replaces the decode charge with it (`TokenLoadTracker.CorrectDecodeCost`), since the body-size estimate is far off for requests with media. `generated_tokens` estimates the output the pod's outstanding requests have produced so far (it sits in the decode KV too); `pd.TokenLoadTracker.DecodeGrowth` computes it in O(1) per pod from the count and the summed charge times of the outstanding charges. `AIBRIX_DECODE_TOKEN_LOAD_OUTPUT_GROWTH=false` scores prompt tokens only. The cold-start score does not apply to `token_load`: a pod without metrics is scored from the ledger like any other. The decode load-imbalance fast path is skipped under `token_load` (it still runs to fill the metric maps): it picks by request count, throughput or drain rate, and the pod holding one long prompt has the fewest requests. With Redis configured, the decode ledger is shared across gateway replicas (`AIBRIX_TOKEN_LOAD_SHARED_LEDGER`, default on): each replica publishes its per-pod charges, count and summed charge times under its instance ID (`cache.Store.PublishDecodeLedger`), and `scoreDecodePods` adds the other live replicas' values, read with the running-request counts (`GetPodsRunningRequestsAndDecodeLedger`). Without Redis the ledger is local to each replica. Each decode pod's `token_load` score is then multiplied by the candidates' mean KV capacity over its own (`decodeKVCapacities`, `AIBRIX_DECODE_TOKEN_LOAD_CAPACITY_AWARE`, default on), with capacities from the scraped `kv_cache_num_gpu_blocks` × `kv_cache_block_size` (vLLM `vllm:cache_config_info` labels) or `kv_cache_capacity_tokens` (SGLang); unknown capacities count as the mean.\n\n### Config profile overrides for PD score policies\n\nWhen the gateway resolves a model config profile (`routingCtx.ConfigProfile` from `model.aibrix.ai/config` and the `config-profile` header), the **`routingConfig`** JSON may include:\n\n| Field | Values | Effect |\n|-------|--------|--------|\n| `prefillScorePolicy` | `prefix_cache`, `least_request` | Overrides `AIBRIX_PREFILL_SCORE_POLICY` for that request |\n| `decodeScorePolicy` | `load_balancing`, `least_request`, `conductor`, `token_load` (or any name registered via `pd.RegisterDecodePolicy`) | Overrides `AIBRIX_DECODE_SCORE_POLICY` for that request |\n\nExample fragment inside a profile:\n\n```json\n\"routingConfig\": {\n \"promptLenBucketMinLength\": 0,\n \"promptLenBucketMaxLength\": 8192,\n \"prefillScorePolicy\": \"prefix_cache\",\n \"decodeScorePolicy\": \"least_request\"\n}\n```\n\nOmitted fields keep the gateway defaults from environment variables.\n\n---\n\n\n## Engine-Specific Prefill Behaviour\n\n### vLLM encode stage (EPD)\n\nWhen the selected prefill pod's roleset has `role-name: encode` pods and a vLLM `/v1/chat/completions` request carries media items (`image_url`, `audio_url`, `input_audio`, `video_url`), `Route` runs an encode stage before the prefill call:\n\n```\nGateway ──POST /v1/chat/completions {messages:[{role:user, content:[item i]}]}──► Encode Pod (one per item, in parallel)\n ◄── 200 ──────────────────────────────────────────────────────────────── Encode Pod (output stored by the EC connector)\n\nGateway ──prefill as below, request body unchanged──────────────────────────────► Prefill Pod (finds the output by the item's hash)\n```\n\nItems are spread round-robin over the roleset's encode pods. The body is not rewritten: the encoder and the prefill pod hash the same item bytes, so the prefill pod's EC connector finds the output, and encodes the item itself when it can't. A failed encode is therefore logged and counted, not returned. Each call uses the prefill request timeout. Connectors that need transfer handles passed through the request (vLLM's `*_embeds` rewrite in `disagg_epd_proxy.py`) are not supported.\n\nWhen bucketing routes the request to a combined pod instead, `Route` runs the same stage on the combined pod's roleset. For E+PD rolesets, where encode pods sit in front of pods that serve the whole request, the gateway runs it outside the PD router: with any other routing strategy, `selectTargetPod` drops encode pods from the candidates and encodes on the chosen pod's roleset before returning it.\n\n### vLLM (`AIBRIX_KV_CONNECTOR_TYPE=shfs`, default)\n\n```\nGateway ──POST /v1/... (max_tokens=1, kv_transfer_params={do_remote_decode:true})──► Prefill Pod\n ◄── response {kv_transfer_params: {remote_block_ids, ...}} ──────────────── Prefill Pod\n\nGateway updates ReqBody:\n kv_transfer_params.remote_host = prefill_pod_ip\n\nGateway ──POST /v1/... (with kv_transfer_params)──────────────────────────────────► Decode Pod\n```\n\n### vLLM (`AIBRIX_KV_CONNECTOR_TYPE=nixl`, Neuron)\n\n```\nGateway ──POST /v1/... (max_tokens=1)──────────────────────────────────────────────► Prefill Pod\n ◄── response { entire prefill response } ──────────────────────────────── Prefill Pod\n\nGateway updates ReqBody:\n disagg_prefill_resp = \n\nGateway ──POST /v1/... (with disagg_prefill_resp)──────────────────────────────────► Decode Pod\n```\n\n### SGLang (async)\n\n```\nGateway modifies ReqBody:\n bootstrap_host = prefill_pod_ip\n bootstrap_port = pod annotation / default 8998\n bootstrap_room = random int63\n rid = -<16 hex nonce> (identical on both legs)\n\nGateway ──POST (async goroutine)──────────────────────────────────────────────────► Prefill Pod\n (no wait; prefill/decode coordinate via bootstrap protocol)\n\nGateway ──POST /v1/... (updated body)─────────────────────────────────────────────► Decode Pod\n```\n\nFor SGLang, `Route()` returning a decode pod means the gateway has dispatched\nthe asynchronous prefill worker, not that the prefill HTTP request has already\ncompleted. Prefill success/failure metrics are emitted by the async worker after\nthe prefill HTTP request returns.\n\n`rid` is the engine-visible request id of the attempt. SGLang reads it from the\nrequest body (the `X-Request-Id` header is ignored) and its `/abort_request`\nendpoint matches live requests by rid alone, so injecting the same gateway-owned\nrid into both legs is what makes the decode leg cancellable. The rid always ends\nwith a fixed-width nonce so that no gateway rid can be a proper prefix of\nanother - SGLang aborts by rid prefix - and so that a client retrying with the\nsame request id (or the same traceparent) still gets a distinct rid per attempt.\nSee [PD Prefill Fail-Fast](#pd-prefill-fail-fast).\n\n### TensorRT-LLM\n\n`AIBRIX_TRT_SCHEDULE_STYLE=context_first` (default) preserves the sequential flow:\n\n```\nGateway adds disaggregated_params to prefill request:\n { request_type: \"context_only\", disagg_request_id: }\n\nGateway ──POST (sync)──────────────────────────────────────────────────────────────► Prefill Pod\n ◄── response { disaggregated_params: {...}, prompt_token_ids: [...] } ── Prefill Pod\n\nGateway updates ReqBody:\n disaggregated_params.request_type = \"generation_only\"\n prompt / prompt_token_ids from prefill response\n\nGateway ──POST /v1/... (with disaggregated_params)─────────────────────────────────► Decode Pod\n```\n\nTRT-LLM uses a Snowflake-style `disagg_request_id` (63-bit) to correlate prefill and decode:\n\n```\n[41-bit timestamp ms since 2023-01-01] [10-bit machine_id] [12-bit counter]\n```\n\nMachine ID is set via `AIBRIX_TRT_MACHINE_ID` (must be in `[0, 1024)`). Assign different machine IDs to gateway processes sharing the same TRT workers.\n\n#### Generation-first (parallel)\n\nSet `AIBRIX_TRT_SCHEDULE_STYLE=generation_first` **on the gateway plugin**, not on\nCTX/GEN workers. The setting is fixed when the PD router is constructed; an\nunknown value is logged and leaves the router on `context_first`. It does not\naffect other engines or combined pods. No automatic fallback or retry occurs after\neither leg has been dispatched.\n\nPrerequisites (TRT-LLM `1.3.0rc8`, the Python KV-cache transceiver on both roles,\nmatching model/tokenizer/chat-template) are listed in\n`docs/source/features/gateway-routing/pd-disaggregation.rst`.\n\n```\nGateway selects CTX + GEN\n ├─ GET CTX /server_info (cached)\n ├─ prepare both bodies with the same integer disagg_request_id\n ├─ goroutine: POST context_only, schedule_style=1, stream=false, max_tokens=1\n └─ return GEN address and generation_only body to Envoy immediately\n disagg_request_id = ctx_request_id = shared ID\n ctx_info_endpoint, ctx_dp_rank, optional encoded_opaque_state = CTX metadata\n schedule_style = 1 (TRT-LLM GENERATION_FIRST wire enum)\n original prompt/messages, sampling settings and stream mode preserved\n\nCTX <── engine-managed coordination / KV transfer ──> GEN\nEnvoy <──────────────────── response / SSE ────────── GEN\n```\n\nThere is no second Go-issued decode POST. GEN can initialize while CTX is still\nworking; it cannot generate without the required KV. The CTX response is not merged\nback into the already-dispatched GEN request. Both endpoints must therefore use\nmatching model/tokenizer/chat-template configuration, since GEN cannot reuse\n`prompt_token_ids` from the CTX HTTP response in this mode.\n\n**Worker discovery:** each router loads the selected CTX worker's `/server_info`\nlazily before either leg is dispatched, so newly discovered or autoscaled pods work\nwithout a gateway restart. Concurrent misses for one incarnation are coalesced;\nlookups have a 3-second timeout and a 1-minute TTL, and responses are capped at\n1 MiB. Pod UID, address and restart count identify an incarnation. A missing or\ninvalid `/server_info` fails routing before dispatch, and failed lookups are not\ncached.\n\nThe expected HTTP response follows TRT-LLM's `1.3.0rc8` OpenAI server schema:\n\n```json\n{\"disaggregated_params\":{\"ctx_info_endpoint\":\"tcp://:\",\"ctx_dp_rank\":0}}\n```\n\n`ctx_info_endpoint` must be a single string, which is what the Python transceiver\nreturns; any other JSON shape fails the lookup.\n\n`encoded_opaque_state` is also propagated when present. Only these fields are copied\nfrom `/server_info`: other keys are ignored so worker metadata can never overwrite\nthe gateway-owned `request_type`, IDs or `schedule_style`. A field the handshake\nrequires but this allowlist does not carry must be added to the handler explicitly.\nThe endpoint must be reachable from GEN. `ctx_dp_rank` must be explicitly present and nonnegative; it is **attention\nDP rank**, not TP rank, replica index, or Pod number. Rank zero is valid but is never\nsubstituted for missing data. Metadata is taken from the exact selected Pod. A\nfront-end that distributes requests across multiple DP ranks without stable\nworker/rank affinity is not supported by this Pod-level cache: expose rank-affine\nworkers or retain `context_first`. Validate nonzero-rank deployments against the\nactual engine before enabling this mode.\n\n**Failure handling:** the CTX HTTP call retains client cancellation and the existing\nprefill timeout. Failure wakes the gateway through the per-incarnation PD leg state,\nnever through a recycled RoutingContext. TRT-LLM has no SGLang `rid`/`/abort_request`\ncontract; the gateway instead fails/resets the Envoy stream, relying on TRT-LLM's\nHTTP-disconnect cancellation. Keep Envoy `failure_mode_allow=false`. TRT's SSE headers\ncan precede KV arrival, so a terminal CTX failure after response headers resets the\nstream rather than being ignored. This can also truncate an already-generating\nresponse; it never attempts to rewrite headers already sent to the client.\n\nStart validation with text-only 1P1D using the sample under\n`samples/quickstart/tensorrt/`; its README carries the smoke test and the rollback\ncommand.\n\n---\n\n## PD Prefill Fail-Fast\n\nOnly asynchronous modes (SGLang and TRT generation-first) can fail their prefill leg after `Route()`\nhas already returned a decode pod. Without fail-fast the client waits for the\ndecode leg to give up on a KV transfer that will never arrive - 300s in SGLang -\nand the decode pod holds its pre-allocated KV pages for that whole window.\n\n```\nPrefill leg fails (async worker)\n │\n ├─► record failure on the RoutingContext's PD leg state (first failure wins)\n │\n ├─► SGLang: POST /abort_request {\"rid\": ...} to decode (goroutine, twice)\n │ TRT: no abort POST; reset Envoy upstream on failure\n │\n └─► wake the ext_proc stream ──► 5xx ImmediateResponse to the client\n header x-error-pd-prefill: true\n```\n\nFailure classes, as reported in the metrics and in the client error message:\n\n| Class | Meaning | Terminal |\n|-------|---------|----------|\n| `request_setup` | The prefill HTTP request could not be built or the body could not be prepared | yes |\n| `timeout` | `AIBRIX_PREFILL_REQUEST_TIMEOUT` elapsed | yes |\n| `canceled` | The client or the gateway canceled the prefill request | yes |\n| `transport` | Connection refused, reset, DNS failure, ... | yes |\n| `http_status` | The prefill engine answered a non-2xx status | yes |\n| `bad_response` | The prefill engine answered 2xx with a body the gateway could not parse | engine-dependent |\n\n`bad_response` is terminal only for engines whose KV transfer is out of band\n(TRT generation-first): there the HTTP body is not the KV handshake, so an\nunparseable 2xx leaves the gateway unable to tell whether context processing\nsucceeded. For SGLang, whose handshake completes in the body, it is not terminal:\nthe prefill leg did run, so the decode leg is left alone and the client keeps\nwhatever the decode leg produces.\n\nThe client status code is the prefill engine's own status for `http_status`, and\n`503` for every other terminal class, since those have no upstream status. The\nresponse body is an OpenAI-shaped error carrying the class and a truncated\nupstream message, and it is sent even while the gateway is still waiting for the\ndecode leg's response headers. For SGLang, once the decode leg has started answering the client, the failure is\nonly recorded and logged. TRT generation-first instead resets the stream even\nafter headers, because TRT can send SSE headers before KV is available; it does not\nattempt to replace the already-started response.\n\n### PD Decode Watchdog\n\nFail-fast covers a prefill leg that dies. The opposite case is a prefill leg\nthat *succeeds* and a decode pod that then never answers: in SGLang the prefill\ncall only returns once the decode pod has taken the KV transfer, so from that\npoint the decode pod owes the client a response. If its scheduler dies or wedges,\nthe client would otherwise hang until Envoy's route timeout.\n\n```\nPrefill leg succeeds (async worker)\n │\n └─► mark the PD leg state (MarkPrefillSucceeded) ──► wake the ext_proc stream\n │\n arm a timer: FIRST_RESPONSE (stream) / RESPONSE (non-stream)\n │\n response headers ──► no change (sent before the first token)\n first body chunk ──► switch to STREAM_IDLE │ (streaming only)\n every later chunk ──► re-arm STREAM_IDLE │\n ▼ timer fires\n before any headers: 504 ImmediateResponse, header x-error-pd-decode: true\n + gRPC DeadlineExceeded close\n after the headers: gRPC DeadlineExceeded close only (Envoy resets\n the response the client already started reading)\n always: POST /abort_request {\"rid\": ...} to decode (goroutine, once)\n```\n\nThe watchdog runs only for requests with a gateway-owned `rid` (SGLang). Its\ntimeouts are `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` (streaming, default `60`) and\n`AIBRIX_DECODE_RESPONSE_TIMEOUT` (non-streaming, default `0`, i.e. off, since the\ndecode pod only answers a non-streaming request once the whole generation is\ndone). Both run until the first response body chunk: SGLang's streaming\nendpoint sends its headers before the first token, so the headers prove nothing.\nFrom the first chunk of a streaming response on,\n`AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` (default `120`) bounds the gap between two\nchunks.\n\n---\n\n## Request Trackers\n\n### PrefillRequestTracker\n\nTracks active **prefill** request counts per pod using `sync.Map` and `atomic.Int32`. Used by both prefill load-imbalance detection and `scorePrefillPods` (mean/stddev filter and load cap).\n\nBoth request trackers are shared by every model the router serves, so pods are identified by their pod key, `namespace/name` (`utils.GeneratePodKey(pod.Namespace, pod.Name)`), not by bare pod name.\n\n```\nAddPrefillRequest(requestID, podKey) // in Route(), after PD pod selection, before prefill HTTP\nRemovePrefillRequest(requestID) // on prefill end (executor defer) or Route error\nGetPrefillRequestCountsForPods(pods) // for scoring\n```\n\n### PendingDecodeTracker\n\nBridges the gap between **decode pod selection** and the actual decode request starting. Without this, concurrent requests could all route to the same decode pod (since `RealtimeNumRequestsRunning` hasn't updated yet).\n\n```\nAddPendingDecode(requestID, podKey) // in Route(), immediately after pod selection\nRemovePendingDecode(requestID) // deferred in Route()\nGetPendingDecodeCount(podKey) // added to running reqs in decode scoring\n```\n\nTimeline:\n```\nRoute() called\n │\n ├─ filterPrefillDecodePods() → (prefillPod, decodePod)\n │ └─ [combined path] returns prefillPod=nil\n │\n ├─ AddPendingDecode(requestID, decodePodKey)\n ├─ defer RemovePendingDecode(requestID)\n │\n ├─ [PD path] AddPrefillRequest(requestID, prefillPodKey)\n ├─ [PD path] doPrefillRequest() (may take seconds)\n │ └─ on error: RemovePrefillRequest(requestID) called immediately\n ├─ ctx.SetTargetPod(decodePod)\n └─ return address to caller\n```\n\n---\n\n## Prompt Length Bucketing\n\nEnabled by `AIBRIX_PROMPT_LENGTH_BUCKETING=true`.\n\nEach pod can declare a prompt-length range via its config profile (`promptLenBucketMinLength` / `promptLenBucketMaxLength`). A pod is eligible only when the request's prompt length falls within its range.\n\n```\ncollectAndBucketPods()\n ├─ Combined pass: collect combined-role pods (role-name is not prefill/decode)\n │ with combined=true and a prompt-length match\n ├─ Eligible rolesets: group by roleset-name (prefill + decode both present)\n └─ Bucket pass: per roleset, keep pods whose bucket covers promptLength\n only when BOTH prefill and decode sides match (half-pairs excluded)\n\nIf both bucket-filtered lists are non-empty, they replace the unfiltered prefill/decode lists atomically so the sides stay roleset-aligned.\n```\n\n### No-match guard and combined fallback\n\nWhen `AIBRIX_PROMPT_LENGTH_BUCKETING=true`, a roleset enters the bucket-filtered lists only when **both** its prefill and decode pods match the request's prompt length. A roleset with only one matching side (misconfigured bucket or a missing decode pod) is excluded from the bucket lists.\n\nIf no complete bucket-matched prefill/decode pair exists for the request — including when the prompt length is outside every declared range, or when the matching roleset is incomplete (e.g. decode pod is down) — the router falls back to a combined pod when one is available. Otherwise it returns an error rather than cross-routing to a different bucket's PD pair:\n\n```\nif AIBRIX_PROMPT_LENGTH_BUCKETING &&\n (bucketPrefills == 0 || bucketDecodes == 0):\n if combined pods available for this prompt length → route to combined (random pick)\n else → error: \"no prompt-length bucket matches prompt length N and no combined pods available\"\n```\n\nPods without a `model.aibrix.ai/config` annotation are treated as suitable for any prompt length. With bucketing enabled globally, models that do not declare explicit bucket ranges still participate using those default bounds.\n\n### Combined Pods\n\nWhen bucketing is enabled, pods with `combined=true` in their config profile can act as both prefill and decode. Combined pods use a `role-name` other than `prefill` or `decode` (for example `all`). The router falls back to a combined pod when:\n\n1. No complete bucket-matched prefill/decode pair exists for the request's prompt length (prompt outside all buckets, incomplete roleset, or decode pod unavailable), **or**\n2. `shouldPickCombined()` returns `true`:\n - At least one combined pod has queue-drain score < `0.25` (low load), **and**\n - At least one prefill pod **or** decode pod has queue-drain score > `1.0` (high load).\n\nQueue-drain score is `(waiting_reqs + prealloc_queues) / drain_rate_1m` (see `calculatePodScoreBasedOffRequestRate`).\n\n```\nshouldPickCombined():\n combinedLowLoad = any combined pod score < 0.25\n prefillHighLoad = any prefill pod score > 1.0\n decodeHighLoad = any decode pod score > 1.0 (skipped if prefillHighLoad)\n return (prefillHighLoad OR decodeHighLoad) AND combinedLowLoad\n```\n\nWhen a combined pod is selected for load imbalance, `scoreCombinedPods()` picks the lowest-score combined pod. The no-match path picks a combined pod at random.\n\nWhen a combined pod is selected, `prefillPod` is `nil` (no prefill HTTP call) and `decodePod` is the selected combined pod.\n\n### Adaptive Bucket Serving\n\nEnabled by `AIBRIX_BUCKET_SERVE=true`, which requires `AIBRIX_PROMPT_LENGTH_BUCKETING=true` as well: the plan only re-orders the rolesets that bucketing already filtered to.\n\nPlain bucketing keeps every roleset whose range covers the request as a candidate, and the scoring decides among them. Adaptive bucket serving narrows that choice. The gateway keeps a per-model picture of the prompt lengths it routes, charges every roleset the traffic only it can serve, and splits each range that several rolesets still need into one band per roleset, drawing each band against one replica-proportional share so a roleset carrying several overlaps does not draw a fresh share from each. The cut points sit at quantiles of the observed traffic. The picture is an EWMA with a 30s half-life, a model's plan is recomputed at most once every 5s, and one plan holds at most 16 affinity bands. A shared range needs a meaningful share of the model's traffic before it is split at all, and a range that cannot afford one band per needing roleset splits into as many bands as the budget left allows; an interval that is not split keeps every covering roleset a candidate.\n\n```\nbucketServeBand() (before selectMu, one call per request)\n ├─ observe promptLength for the model\n ├─ groups: one per eligible roleset, with the range its profile declares and its prefill replica count\n ├─ plan: shared ranges split at mode quantiles in replica proportion, light ones kept whole\n └─ band for promptLength → banded roleset and band upper bound\n\nfilterPrefillDecodePods()\n ├─ steps 1-5: bucket filtering and the load-imbalance fast paths\n ├─ banded roleset still has prefill and decode candidates\n │ → narrow both sides to that roleset (step 6)\n └─ otherwise keep the load fast paths' choice\n```\n\n`AIBRIX_BUCKET_SERVE_MODE` picks the unit the cut points are measured in: `throughput` (default) balances prompt-token mass, while `rps` balances request counts. Within a shared range the bands divide the observed traffic in proportion to the replica share each roleset still needs, counted in that unit. Both are per-model: a profile that selects another mode re-plans only the models it routes.\n\nThe plan is advisory. It never widens a choice, only prefers one of the rolesets that already cover the request's length. Wherever the plan holds no band, the scoring considers every covering roleset, and when the banded roleset has no candidate left after the load fast paths, the request routes exactly as it would without the plan. The steady state does narrow the choice, which is the feature, and a stale or wrong plan costs balance, not correctness. With the feature off the planner records and publishes nothing.\n\nMetrics: `pd_bucket_serve_band_total` and `pd_bucket_serve_prompt_tokens_total` count the requests and prompt tokens each banded roleset carried (label `roleset`), so a counter joins the pods that received the band, and the `pd_bucket_serve_band_max` gauge holds the highest upper prompt-length bound among the bands a roleset currently holds (a roleset can hold two bands when its range overlaps two others), deleted when no live plan holds the roleset.\n\n---\n\n## Environment Variables\n\n### Core Behaviour\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFILL_SCORE_POLICY` | `prefix_cache` | Prefill pod scoring: `prefix_cache` or `least_request`. Any other value logs a warning and falls back to `prefix_cache`. |\n| `AIBRIX_DECODE_SCORE_POLICY` | `load_balancing` | Decode pod scoring for `finalPDScore`: `load_balancing`, `least_request`, `conductor` or `token_load`. Any other value logs a warning and falls back to `load_balancing`. |\n| `AIBRIX_KV_CONNECTOR_TYPE` | `shfs` | KV transfer backend: `shfs` (GPU/SHFS), `nixl` (Neuron/NIXL), or `mooncake` (Mooncake) |\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | `30` | Prefill HTTP request timeout in seconds. Exceeding it is a terminal `timeout` prefill failure. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | `false` | Enable prompt-length-based pod bucketing |\n| `AIBRIX_BUCKET_SERVE` | `false` | Enable adaptive bucket serving on top of prompt-length bucketing |\n| `AIBRIX_BUCKET_SERVE_MODE` | `throughput` | The unit the adaptive cut points are measured in: `throughput` (prompt token mass) or `rps` (request counts) |\n\n### Prefill Load Balancing\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | `16` | Min `(max − min)` outstanding prefill request count to trigger imbalance routing |\n| `AIBRIX_PREFILL_LOAD_CAP_FACTOR` | `1.5` | Factor `c`: prefill pods with `req_count > max(c, 1) * (mean + 1)` are skipped from per-roleset selection. Unlike the stddev skip, it fires in pools of five or fewer pods. Environment-only |\n\n### Decode Load Balancing\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | `1.0` | Multiplier on normalized running-request term in `load_balancing` numerator |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | `1.0` | Multiplier on normalized inverse-throughput term in `load_balancing` numerator |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | `16.0` | Min `(max − min)` running request count spread to trigger decode imbalance routing |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | `2048.0` | Min `(max − min)` token throughput spread (tok/s) to trigger throughput-based routing |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | `1.5` | `max_drain_score / min_drain_score` ratio threshold to trigger drain-rate routing |\n\n### PD Prefill Fail-Fast\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | `3` | Per-attempt timeout in seconds of the `/abort_request` call to the decode pod. `0` disables decode aborts; the client is still failed fast |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | `2` | Delay in seconds before the second abort attempt. `0` sends a single attempt |\n\n### PD Decode Watchdog\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | `60` | Seconds a streaming SGLang request waits, after its prefill leg succeeded, for the first response body chunk (first token) from the decode pod. `0` disables |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | `0` | Same for a non-streaming request, where the first body chunk is the finished answer. `0` (default) disables |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | `120` | Longest gap allowed between two body chunks of a streaming response once the first one arrived. `0` disables |\n\n### TensorRT-LLM\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_TRT_MACHINE_ID` | `0` | 10-bit machine ID used in Snowflake disagg request ID generation (range: `[0, 1024)`); distinct per gateway process sharing TRT workers |\n| `AIBRIX_TRT_SCHEDULE_STYLE` | `context_first` | TRT-only dispatch mode: `context_first` (sequential) or `generation_first` (parallel). Read at router initialization; unknown values are logged and fall back to `context_first` |\n\n### Inherited from Prefix Cache Router\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | `character` | Tokenizer for prefix-cache prefill scoring: `tiktoken` or `character` |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `1` | Multiplier N: prefill pods with `req_count > mean + N*stddev` are skipped from per-roleset selection |\n\n---\n\n## SGLang Bootstrap Port\n\nSGLang uses a bootstrap mechanism for prefill/decode coordination. The port is resolved in order:\n\n1. Pod annotation `model.aibrix.ai/sglang-bootstrap-port`\n2. Default: `8998`\n\n---\n\n## Metrics Emitted\n\n| Metric | When |\n|--------|------|\n| `GatewayPrefillRequestFailTotal` | Engine validation fail, pod filter fail, prefill HTTP error |\n| `GatewayPrefillRequestSuccessTotal` | Prefill HTTP succeeded. For SGLang and TRT generation-first, emitted asynchronously after the background prefill request completes, not when `Route()` returns. |\n| `gateway_pd_encode_request_total{status,status_code}` | One media item sent to an encode pod. `status` is `success`, or the failure class (`http_error`, `deadline_exceeded`, ...) |\n| `PDSelectedPrefillPodTotal` | Prefill pod selected (per pod label) |\n| `PDSelectedDecodePodTotal` | Decode pod selected (per pod label) |\n| `gateway_pd_prefill_failure_total{class,stage}` | A terminal prefill failure reached the client-facing handler. `stage` is `before_response` (the client was failed fast) or `after_response` (the decode leg had already started answering) |\n| `gateway_pd_decode_abort_total{prefill_failure_class,result}` | One decode abort attempt. `result` is `ok`, `error`, `skipped_streaming`, `skipped_disabled`, `skipped_no_rid` or `skipped_no_target`. `prefill_failure_class` is `watchdog_first_response` or `watchdog_stream_idle` for an abort sent by the decode watchdog |\n| `gateway_pd_decode_watchdog_total{phase}` | The decode watchdog failed a request whose decode pod stopped responding. `phase` is `first_response` or `stream_idle` |\n\n---\n\n## Response Headers Set\n\n| Header | Value |\n|--------|-------|\n| `prefill-target-pod` | Name of the selected prefill pod |\n| `prefill-target-pod-ip` | IP of the selected prefill pod |\n| `x-error-pd-prefill` | `true`, on the error response the gateway generates when the PD prefill leg failed |\n| `x-error-pd-decode` | `true`, on the error response the gateway generates when the decode watchdog fired |\n\n---\n\n## Prefix Cache Update Pipeline\n\nAfter a prefill pod is selected, its prefix hashes are enqueued asynchronously to avoid blocking the routing path:\n\n```\nfinalPDScore()\n └─ enqueuePrefixUpdate(hashes, model, pod)\n │\n ▼ (non-blocking send; dropped if channel full)\n prefixUpdateCh (buffered, capacity 1024)\n │\n ▼ (single background goroutine)\n prefixCacheIndexer.AddPrefix(hashes, model, pod)\n```\n\n---\n\n## Initialization\n\n```go\nNewPDRouter()\n │\n ├─ cache.Get() // shared metrics cache\n ├─ prefixcacheindexer.GetSharedPrefixHashTable()\n ├─ prefillScorePolicy from AIBRIX_PREFILL_SCORE_POLICY:\n │ least_request → LeastRequestPrefillPolicy\n │ prefix_cache → PrefixCachePrefillPolicy (shared tokenizer + PrefixHashTable)\n │ (other) → log warning, same as prefix_cache\n ├─ decode score policy (package algorithms/pd, decode_scorer.go) from AIBRIX_DECODE_SCORE_POLICY:\n │ least_request → pd.LeastRequestDecodePolicy\n │ load_balancing → pd.LoadBalancingDecodePolicy\n │ (other) → log warning, same as load_balancing\n ├─ create HTTP client with connection pool\n │ MaxIdleConns=100, MaxIdleConnsPerHost=10, IdleConnTimeout=90s\n ├─ TRT handler from AIBRIX_TRT_SCHEDULE_STYLE (immutable, router-scoped)\n │ owns a lazily populated /server_info cache for generation_first\n ├─ NewPrefillRequestTracker()\n ├─ NewPendingDecodeTracker()\n └─ startPrefixUpdater() // background goroutine\n\n// pd/router/roleset.go (no per-router state; pure functions called on each request)\nHasEligibleRoleset(readyPods) // gateway: skip PD when no complete roleset\neligiblePDRolesets(readyPods) // internal: filter to complete rolesets only\nDescribePDRolesetEligibility(readyPods) // diagnostics: per-roleset replica counts\n```\n","pkg/plugins/gateway/ENV_VARS.md":"# Gateway Plugin Environment Variables\n\nThis document covers all environment variables used in the `pkg/plugins/gateway` package and its sub-packages.\n\nVariables of type `duration` are parsed with Go's [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration), so they need a unit: `30s`, `5m`, `1h30m`. A bare number such as `30`, or a zero or negative duration such as `0s` or `-1s`, is rejected with a warning and the default is used instead.\n\n---\n\n## General Gateway\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_DISABLE_RATE_LIMITING` | bool | `false` | Disable AIBrix user RPM/TPM and model RPS quota enforcement. Redis and `requestsInflight` remain active. | [cmd/plugins/main.go](../../../cmd/plugins/main.go), [gateway.go](gateway.go) |\n| `POD_NAME` | string | `\"\"` | Kubernetes pod name. Used for logging and metric label tagging. | [gateway.go](gateway.go), [util.go](util.go) |\n| `ROUTING_ALGORITHM` | string | _(none)_ | Default routing algorithm when no per-request override is set. | [types.go](types.go), [util.go](util.go) |\n| `AIBRIX_PRIORITY_TIER_ENABLED` | bool | `false` | Forward the priority tier declared by the `x-aibrix-priority-tier` request header as the upstream vLLM request priority. | [cmd/plugins/main.go](../../../cmd/plugins/main.go), [gateway_req_priority.go](gateway_req_priority.go) |\n\nWhen `AIBRIX_PRIORITY_TIER_ENABLED=true`, the gateway maps the `x-aibrix-priority-tier`\nrequest header to the `priority` field of the backend request body: `batch` becomes `100` and\n`background` becomes `1000`. The engine serves smaller values first, so the mapping only\nde-prioritizes a request and never pulls one ahead of another; the values sit below the engine\ndefault of `0`. Requests without the header, or with a tier outside the table, are forwarded\nunchanged, and a `priority` the caller already set in the body is kept. Tier names are matched\ncase-insensitively. The deployment has to run a priority-aware vLLM scheduler\n(`--scheduling-policy=priority`). With the default `false`, the header is ignored and request\nbodies are forwarded byte for byte.\n\nWhen `AIBRIX_DISABLE_RATE_LIMITING=true`, the gateway skips AIBrix user lookup, ignores the\n`user` header as routing identity, and does not write user or model quota counters. The header\nis still forwarded unchanged. Keep Redis configured when session affinity, asynchronous jobs,\nstate synchronization, or other gateway features require it. The per-replica\n`requestsInflight` guard remains active. Because routing user identity remains empty, asynchronous\nvideo jobs use the shared ownership scope; the unauthenticated `user` header does not isolate jobs\nbetween callers in this mode.\n\n---\n\n## Response Processing\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_TTFT_THRESHOLD_S` | int (seconds) | `1` | Time-to-first-token threshold in seconds. Requests exceeding this are flagged in response processing. | [gateway_rsp_body.go](gateway_rsp_body.go) |\n\n`AIBRIX_TTFT_THRESHOLD_S` can also be set per model with the top-level `ttftThresholdS` profile field; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Redis Sync (`statesync/`)\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_STATESYNC_ENABLED` | bool | `false` | Enable cross-replica state sync via Redis. Must be `true` to activate the statesync manager. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `AIBRIX_STATESYNC_SYNC_PERIOD` | duration | `10s` | Interval at which gateway state is synced to Redis across replicas. | [statesync/redissync.go](statesync/redissync.go) |\n\n---\n\n## Prefix Cache Router (`algorithms/prefixcache/router.go`)\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | string | `\"character\"` | Tokenizer type for prefix cache hashing. Options: `character`, `tiktoken`, `remote`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | int | `1` | Factor multiplied by the standard deviation of pod loads when selecting among prefix-matched pods (`pod.req ≤ mean + factor × σ`). | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR` | float64 | `1.5` | Prefix-matched pods with more than `max(c, 1) × (mean + 1)` running requests are skipped, where `c` is this factor. Applies only when `prefix-cache` routes on its own, with the auto-blend off (`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT=0`). Unlike the `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` check, which cannot skip any pod in a pool of five or fewer at a factor of `2`, it applies to pools of any size. Values between `0` and `1` are treated as `1`; `0`, negative or unparsable values use the default `1.5`. To effectively disable the cap, set a large value such as `100`. Environment-only. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_INCLUDE_TOOLS` | bool | `true` | Prepend a canonical rendering of the chat request `tools` (keys sorted, compact) to the text used for prefix matching, since chat templates commonly render tools ahead of the messages. For templates that render tools later in the prompt (e.g. near the last user turn), requests sharing tools but not messages can see a partial match the engine does not have; such deployments can set this to `false`. Requests without tools (absent, `null`, `[]` or not an array) are unaffected. Set to `false` to match on messages only. The tools text goes into a separate prefix-match text (`RoutingContext.PrefixText()`), which feeds prefix matching in `prefix-cache`, `prefix-cache-preble` and the PD prefill policies `prefix_cache`, `conductor` (including its matched/unmatched token estimate) and `hybrid_cache_load`. The routing message (`routingCtx.Message`) stays messages-only, so token estimates (VTC, `PromptTokens`/`PromptLength`, config-profile `promptTokensGte`/`promptTokensLt`) are unchanged. In KV-sync mode, chat requests tokenized through the remote chat template always forward the raw `tools` to the tokenizer, independent of this flag, since the template itself decides where tools go. | [prefix_tools.go](prefix_tools.go) |\n| `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER` | bool | `false` | Use a remote HTTP tokenizer service instead of the local tokenizer. Requires `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED` | bool | `false` | Enable KV cache event synchronization across gateway replicas. When `true`, also requires `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_REMOTE_TOKENIZER_ENDPOINT` | string | `\"\"` | Remote tokenizer service endpoint URL. Required when `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true`. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_PUBLISH_ADDR` | string | `\"\"` | ZMQ publish address for KV cache events. Used when KV event sync is enabled. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SUBSCRIBE_ADDRS` | string | `\"\"` | ZMQ subscribe addresses for KV cache events (comma-separated). Used when KV event sync is enabled. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n\n### Remote Tokenizer Pool (`algorithms/prefixcache/router.go`)\n\nThese configure the pool of remote tokenizer connections used when `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_VLLM_TOKENIZER_ENDPOINT_TEMPLATE` | string | `\"http://%s:8000\"` | HTTP endpoint template for tokenizer pods. `%s` is replaced with the pod name. |\n| `AIBRIX_TOKENIZER_HEALTH_CHECK_PERIOD` | duration | `30s` | How often to health-check tokenizer pool members. |\n| `AIBRIX_TOKENIZER_TTL` | duration | `300s` | TTL for cached tokenizer connections in the pool. |\n| `AIBRIX_MAX_TOKENIZERS_PER_POOL` | int | `100` | Maximum number of tokenizer connections in the pool. |\n| `AIBRIX_TOKENIZER_REQUEST_TIMEOUT` | duration | `5s` | Timeout for individual remote tokenizer requests. |\n\n`AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` can also be set per request by the model config profile (`routingConfig.prefixCache.standardDeviationFactor`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Load Balance Router (`algorithms/load_balance.go`)\n\nThe load-imbalance gate below is applied centrally by the gateway (`selectTargetPod` in\n`gateway.go`), once per request, ahead of whichever strategy actually routes it — not just when\n`load-balance` is selected. It restricts candidates to the least-loaded pods before that\nstrategy runs when load is severely skewed, and it **is** applied to the Prefix Cache and\nPreble routers (this replaces the prefix-cache-specific gate that used to live in\n`prefixcache/router.go`; see the deprecation note under [Variable Dependency\nNotes](#variable-dependency-notes)). It is exempted only for exclusive strategies (`pd`,\n`slo*`), which manage their own pod subsets and would have that management disrupted by a\nblanket running-request-count filter applied ahead of time. It does not participate in\nmulti-strategy soft-scoring (`ScoreAll`) for strategies blended alongside `load-balance` — see\n`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` below for how `load-balance` itself gets\nsilently blended into other strategies to compensate.\n\n**TODO / known limitation:** the gate compares raw running-request count, with no notion of a\npod's capacity. In a heterogeneous pool this conflates \"carrying a lot of work\" with\n\"overloaded\" — a replica that's simply *faster* than its peers is expected to carry more\nconcurrent requests without being more loaded (the same capacity signal `load-balance`'s own\nscore uses), but this gate can still flag it as a hotspot and exclude it from routing before any\nstrategy's capacity-aware scoring gets a chance to run. See `getTargetPodListOnLoadImbalance` in\n[algorithms/load_balance.go](algorithms/load_balance.go).\n\nWhen `load-balance` is the (sole, non-blended) strategy that routes the request and multiple\npods tie on the lowest effective-load score, `Route()` breaks the tie using least combined\nGPU+CPU KV-cache usage (falling back to a random pick if cache metrics are unavailable for the\ntied pods) — the same secondary-signal pattern the Prefix Cache router uses to break ties in\nprefix-match percentage via request count.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` | float64 | `2.0` | Gate multiplier for 3+ pods: gate fires when `max_req > factor × (mean_req + 1)`. Ignored for 2-pod clusters (the relative check never holds there). | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` | int | `8` | Minimum absolute gap (`max_req − min_req`) required to trigger the load-imbalance gate. For 2 pods this is the sole trigger; for 3+ pods it is required alongside the factor check. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` | float64 | `0` | Weight λ of engine-queued requests (`num_requests_waiting`) added to the running count in the score: `(running + λ·queued) / capacity × …`. The default `0` scores on running requests only, since the gateway's running count already includes requests queued inside the engine; raise it only if you want queued work to count extra. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` | float64 | `2.0` | Strength α of the KV-pressure penalty: the score is multiplied by `1 + α·(1 − kv_free)²`. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` | float64 | `0.10` | Free-KV-cache fraction below which a pod scores `+Inf` from `load-balance`. When `load-balance` is the sole routing strategy this excludes the pod outright (falling back to the pod with the most KV headroom if every pod is below the threshold, instead of failing the request). When `load-balance` is blended with other strategies (the default for most requests — see \"Router Selection / Auto-Blend\" below), a `+Inf` score is only a strong penalty, not an exclusion: another strategy can still select the pod. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n\nThe five `AIBRIX_LOAD_BALANCE_*` gate and score variables can also be set per request by the model config profile (`routingConfig.loadBalance.*`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Least-Request Top-K Router (`algorithms/least/request_top_k.go`)\n\nStrategy name `least-request-top-k`. Plain `least-request` always resolves to the single\nleast-loaded pod, so when several gateway replicas read the same Redis-backed running-request\ncounter and decide at the same moment, they can all send their next request to that pod. This\nrouter instead picks uniformly at random among the ready pods that are both among the `K`\nleast-loaded and within `epsilon` requests of the least-loaded pod's count. It is also a scorer\n(`ScoreAll`), so it can be blended with other strategies; it is the load-count scorer that the\n[auto-blend](#router-selection--auto-blend-algorithmsroutergo) appends by default, so the variables below\napply to every auto-blended request, not only to requests that name this strategy.\n\nA pod that has just become eligible reads as count `0` and would otherwise be the sole eligible\ncandidate until it catches up. To prevent that, a pod's effective count is raised by a penalty\nthat starts at the lowest count among the already-warm pods and decays linearly to `0` over the\nramp window. The window is measured from when this gateway first sees the pod routable (Ready,\nwith an IP, not terminating or draining), not from when the pod was created, so time spent\nPending or loading a model does not use it up. Ramp state is local to each gateway; a gateway\nthat has just restarted sees every pod as new, so with no warm pod to compare against it applies\nno penalty until some pods age out of the window. The clock is kept, not restarted, when a pod\nflaps NotReady or is re-added with the same IP within 60 seconds; a new IP, or a longer gap, starts\nit over.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K` | int | `5` | `K`: how many of the least-loaded ready pods are eligible. Should be at least the number of gateway replicas deciding concurrently. Must be positive; any other value falls back to the default. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K_EPSILON` | int | `4` | A pod is eligible only if its count is within this many requests of the least-loaded pod's. Combined with `K` as an AND. Must be positive; any other value falls back to the default. The default is a starting point, not a validated one: tune it to the deployment's typical running-request count. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K_RAMP_WINDOW` | duration | `300s` | Window over which a newly routable pod's ramp penalty decays to `0`, counted from the first time the gateway sees the pod routable. `0` disables the ramp. Any other non-positive or unparseable value falls back to the default. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n\nThese are read once at startup and are environment-only (no `routingConfig` override).\n\n---\n\n## Router Selection / Auto-Blend (`algorithms/router.go`)\n\n`RouterManager.Select` silently blends `load-balance` and [`least-request-top-k`](#least-request-top-k-router-algorithmsleastrequest_top_kgo)\nbehind whatever strategy a caller actually asked for, so no single strategy can keep steering\ntraffic at an already-hot pod even without going through the central load-imbalance gate above.\nThe caller never sees this: `ctx.Algorithm`, response headers, and `Validate()` all continue to\nreflect exactly what was requested. The blend is skipped entirely for exclusive strategies\n(`pd`, `slo*`), for an explicit, standalone `load-balance` selection, and for a bare\n`session-affinity` selection, all of which must keep running their own dedicated routing logic.\n`session-affinity`'s own scoring is binary (1 for its resolved pod, 0 for everything else), so\nany load-balance weight below its own could never change the outcome anyway — blending it in\nwould be dead weight at best. A bare `prefix-cache` request, whose scoring is graded rather than\nbinary, still uses a 5:4 (1.25:1) ratio against `load-balance` instead of the flat weight-1\nappend, and does not receive `least-request-top-k`, so the cache-affinity signal wins an exact-tie\ndisagreement without masking real load imbalance.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` | int | `1` | Weight coefficient for the silently-appended `load-balance` scorer. Set to `0` to disable the whole auto-blend feature. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_LEAST_REQUEST_WEIGHT` | int | `1` | Weight coefficient for the silently-appended `least-request-top-k` scorer. It is added only when neither `least-request` nor `least-request-top-k` is already in the strategy (an explicit weight of `0` on either also suppresses it), and is needed so multi-port/data-parallel pod routing keeps working under the blend. Plain `least-request` is not used here because every gateway replica would score the same single least-loaded pod best. The variable keeps its `LEAST_REQUEST` name for compatibility. Set to `0` to omit it from the blend. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` | int | `5` | Primary weight used when auto-blending a bare `prefix-cache` request. With the matching load-balance weight this is a 5:4 (1.25:1) lean toward cache affinity. `0` is rejected with a warning and the default is used. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_LOAD_BALANCE_WEIGHT` | int | `4` | `load-balance` weight paired with `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` for a bare `prefix-cache` request. `0` leaves those requests with prefix-cache scoring alone. | [algorithms/router.go](algorithms/router.go) |\n\nThe four `AIBRIX_ROUTING_AUTO_BLEND_*` weights can also be set per request by the model config profile (`routingConfig.autoBlend.*`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## External Replica Router (`algorithms/external*.go`)\n\nThe optional `external` strategy sends the already-filtered candidate snapshot to one\noperator-configured HTTP endpoint. An unset endpoint disables the strategy. Invalid enabled\nconfiguration fails Gateway startup. Candidate metrics and labels are opt-in allowlists; prompts,\nrequest bodies, credentials, arbitrary client headers, and pod IPs are never sent.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_EXTERNAL_ROUTER_ENDPOINT` | URL | _(unset)_ | Complete `http` or `https` operation URL. Userinfo and fragments are rejected. |\n| `AIBRIX_EXTERNAL_ROUTER_POLICY_MODE` | enum | _(required)_ | `Advisory` or `Authoritative`. |\n| `AIBRIX_EXTERNAL_ROUTER_FAILURE_MODE` | enum | _(required)_ | `FailOpen` or `FailClosed`; Authoritative requires FailClosed. |\n| `AIBRIX_EXTERNAL_ROUTER_FALLBACK` | string | _(none)_ | Registered non-external, non-exclusive local router; required for Advisory and FailOpen. `pd` and `slo*` are rejected because they require dedicated Gateway preprocessing. |\n| `AIBRIX_EXTERNAL_ROUTER_TIMEOUT` | duration | `10ms` | Deadline for one decision exchange. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_INFLIGHT` | int | `256` | Non-blocking per-process bulkhead capacity. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_REQUEST_BYTES` | bytes | `256KiB` | Maximum encoded request size. Accepts bytes, `KiB`, or `MiB`. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_RESPONSE_BYTES` | bytes | `64KiB` | Maximum response body size. Accepts bytes, `KiB`, or `MiB`. |\n| `AIBRIX_EXTERNAL_ROUTER_FAILURE_THRESHOLD` | int | `5` | Consecutive attempted-exchange system failures before opening the circuit. |\n| `AIBRIX_EXTERNAL_ROUTER_OPEN_DURATION` | duration | `1s` | Circuit-open duration before one half-open probe. |\n| `AIBRIX_EXTERNAL_ROUTER_AUTH_TOKEN_FILE` | path | _(unset)_ | Optional non-empty bearer token file loaded at startup. |\n| `AIBRIX_EXTERNAL_ROUTER_CANDIDATE_ATTRIBUTES` | CSV | _(empty)_ | Pod-label keys allowed into candidate attributes. |\n| `AIBRIX_EXTERNAL_ROUTER_CANDIDATE_METRICS` | CSV | _(empty)_ | Any of `runningRequests`, `engineUtilization`, `kvCacheUsage`. |\n| `AIBRIX_EXTERNAL_ROUTER_POLICY_ATTRIBUTES` | CSV | _(empty)_ | Downstream extension-point allowlist for trusted RoutingContext policy attributes. Upstream AIBrix has no built-in producer; integrators must call `SetTrustedPolicyAttribute`. Raw headers are never copied automatically. |\n\nSee the [External Replica Routing guide](../../../docs/source/features/gateway-routing/external-replica-routing.rst)\nand the published OpenAPI contract for policy and protocol details.\n\n---\n\n## Preble (Prefix Cache with Histogram) Router (`algorithms/prefixcache/preble.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_PREBLE_TARGET_GPU` | string | `\"V100\"` | GPU model used for hardware-specific latency estimates in the Preble algorithm. |\n| `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` | int | `45` | Expected decode sequence length used for cache allocation decisions. |\n| `AIBRIX_ROUTER_PREBLE_SLIDING_WINDOW_PERIOD` | int (minutes) | `3` | Sliding window length in minutes for histogram metrics collection. |\n| `AIBRIX_ROUTER_PREBLE_EVICTION_LOOP_INTERVAL` | int (ms) | `1000` | Interval in milliseconds between cache eviction loop executions. |\n\n`AIBRIX_ROUTER_PREBLE_TARGET_GPU` and `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` can also be set per request by the model config profile (`routingConfig.preble.*`). The window and eviction variables stay environment-only; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## VTC (Virtual Token Counter) Router\n\n### Token Tracker (`algorithms/vtc/token_tracker.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | int | `5` | Sliding window size (in `TIME_UNIT` units) for token usage tracking. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | string | `\"minutes\"` | Time unit for the sliding window. Options: `minutes`, `seconds`, `milliseconds`. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | float64 | `1000.0` | Minimum token count threshold for adaptive load normalization. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | float64 | `8000.0` | Maximum token count threshold for adaptive load normalization. |\n\n### VTC Basic Scorer (`algorithms/vtc/vtc_basic.go`)\n\nScoring formula: `score = (fairnessWeight * normFairness + utilizationWeight * normUtilization) / normFreeGPU`\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | float64 | `100.0` | Load value at which a pod is considered fully saturated. |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | float64 | `1.0` | Weight applied to input tokens when computing pod load. |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | float64 | `2.0` | Weight applied to output tokens when computing pod load (typically higher than input). |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | float64 | `1.0` | Weight of the fairness component in the routing score. |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | float64 | `1.0` | Weight of the utilization component in the routing score. |\n\n`AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD`, `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` and `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` can also be set per request by the model config profile (`routingConfig.vtc.*`). The token tracker variables stay environment-only; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## PD (Prefill-Decode) Disaggregation Router (`algorithms/pd/router/router.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | int (seconds) | `30` | HTTP request timeout for prefill pod calls. |\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | int32 | `16` | Minimum (max − min) running-request spread across prefill pods to trigger load-imbalance routing. |\n| `AIBRIX_PREFILL_LOAD_CAP_FACTOR` | float64 | `1.5` | Prefill pods with more than `c × (mean + 1)` in-flight prefill requests are skipped, where `c` is this factor. Unlike the `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` check, it applies to pools of any size. Values between `0` and `1` are treated as `1`; `0`, negative or unparsable values use the default `1.5`. To effectively disable the cap, set a large value such as `100`. Lower values give up more prefix-cache affinity: with random (Poisson) load and no hot pod, `1.5` skips at least one pod on up to about 10% of requests with 4 prefill pods and about 20% with 6. `2` skips less often but cannot fire in a two-pod pool. |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | float64 | `16.0` | Minimum (max − min) running-request spread across decode pods to trigger load-imbalance routing. |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | float64 | `2048.0` | Minimum (max − min) token-throughput spread (tokens/s) across decode pods to trigger throughput-imbalance routing. |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | float64 | `1.5` | Max/min drain-rate score ratio above which the slowest decode pod is excluded from selection. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | bool | `false` | Route requests to prefill pods whose prompt-length bucket matches the request length. |\n| `AIBRIX_BUCKET_SERVE` | bool | `false` | Adaptive bucket serving: band the prompt-length range that several rolesets declare in common and prefer the roleset a request length is banded to. Requires `AIBRIX_PROMPT_LENGTH_BUCKETING=true`. |\n| `AIBRIX_BUCKET_SERVE_MODE` | string | `\"throughput\"` | The unit the adaptive cut points are measured in: `throughput` (prompt token mass) or `rps` (request counts). An unknown value keeps the default. |\n| `AIBRIX_KV_CONNECTOR_TYPE` | string | `\"shfs\"` | KV cache transfer backend. Options: `shfs` (GPU shared memory), `nixl` (Neuron). |\n| `AIBRIX_PREFILL_SCORE_POLICY` | string | `\"prefix_cache\"` | Strategy for selecting the prefill pod. Options: `prefix_cache`, `least_request`. |\n| `AIBRIX_DECODE_SCORE_POLICY` | string | `\"load_balancing\"` | Strategy for selecting the decode pod. Options: `load_balancing`, `least_request`, `conductor`, `token_load`. |\n| `AIBRIX_DECODE_TOKEN_LOAD_OUTPUT_GROWTH` | bool | `true` | `token_load` decode policy only: add an estimate of the output generated so far (per-request decode rate × time since routing) to the decode score. `false` scores prompt tokens only. |\n| `AIBRIX_DECODE_TOKEN_LOAD_CAPACITY_AWARE` | bool | `true` | `token_load` decode policy only: multiply each decode pod's score by the candidates' mean KV capacity divided by the pod's own (`num_gpu_blocks` × `block_size` from vLLM's `vllm:cache_config_info`, or SGLang's `sglang:max_total_num_tokens`), so pods of different sizes are compared by how full they are. Unknown capacities count as the mean; with none known, or `false`, scores are token counts. |\n| `AIBRIX_TOKEN_LOAD_SHARED_LEDGER` | bool | `true` | `token_load` decode policy only, with Redis configured: share the decode ledger across gateway replicas, so each replica adds the other live replicas' charges to its decode score. `false` keeps the ledger local to each replica. |\n\nThe prefill/decode routing thresholds (`AIBRIX_PREFILL_*`, `AIBRIX_DECODE_*`, `AIBRIX_TOKEN_LOAD_*`, `AIBRIX_HYBRID_CACHE_LOAD_FACTOR`, `AIBRIX_MIN_MATCH_PCT`, `AIBRIX_PROMPT_LENGTH_BUCKETING`, `AIBRIX_BUCKET_SERVE`,\n`AIBRIX_BUCKET_SERVE_MODE`), except `AIBRIX_PREFILL_LOAD_CAP_FACTOR`, can also be set per request by the model config profile\n(`routingConfig.pd.*`, `routingConfig.promptLengthBucketing` and `routingConfig.bucketServe*`); see [Model Config Profile Overrides](#model-config-profile-overrides). `AIBRIX_PREFILL_LOAD_CAP_FACTOR` is environment-only.\n\n### PD Prefill Fail-Fast (`algorithms/pd/abort.go`)\n\nWhen the prefill leg of a PD request fails, the decode pod is left waiting for a\nKV transfer that will never arrive until its own bootstrap timeout expires\n(300s in SGLang), holding its pre-allocated KV pages. The gateway injects its own\nrequest id (`rid`) into both legs, so on a prefill failure it can post\n`{\"rid\": ...}` to the decode pod's `/abort_request` and release those pages\nimmediately. The abort is best-effort and asynchronous: nothing waits on it, and\nit is skipped once the decode pod has started answering the client - a call\nalready on the wire when that happens is cancelled rather than left to run.\n\nThe abort is sent twice, because the first attempt can overtake the decode\nrequest itself on the pod (the engine answers `200` either way, so the gateway\ncannot tell a matched abort from a dropped one). Repeating an abort the engine\nalready applied, or never matched, is a no-op on both sides. The wait between\nthe two attempts is interruptible, so a decode leg that starts answering in the\nmeantime ends the retry immediately instead of at the end of the delay.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | int (seconds) | `3` | Per-attempt timeout of the `/abort_request` call to the decode pod. `0` disables decode aborts entirely; the prefill failure is still recorded, logged and reported to the client. |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | int (seconds) | `2` | Delay before the second abort attempt. `0` sends a single attempt. |\n\nMetrics: `gateway_pd_decode_abort_total{prefill_failure_class, result}`, where\n`result` is one of `ok`, `error`, `skipped_streaming`, `skipped_disabled`,\n`skipped_no_rid`, `skipped_no_target`.\n\n### PD Decode Watchdog (`gateway_decode_watchdog.go`)\n\nIn SGLang disaggregation the prefill leg only returns once the decode pod has\ntaken the KV transfer, so a successful prefill means the decode pod owes the\nclient a response. If the decode pod then stops answering (its scheduler died or\nwedged), nothing else notices and the client hangs until Envoy's route timeout.\nThe watchdog bounds that wait: once the prefill leg succeeded and until the\nfirst response body chunk comes back from the decode pod, it fails the request\nafter the first-response timeout and posts one best-effort `/abort_request` for\nthe request's `rid` to the decode pod. Response headers do not end this phase:\nSGLang sends the headers of a streaming response before its first token, so a\ndecode pod that wedges before the first token still delivers them. How the\nclient is failed depends on whether those headers already went out:\n\n- No headers yet: the client gets a `504` carrying the `x-error-pd-decode`\n header, and the ext_proc stream is closed with `DeadlineExceeded`.\n- Headers sent: no error response can replace a response whose status line is\n already on its way, so the ext_proc stream is only closed with\n `DeadlineExceeded` and Envoy resets the client connection. The client sees a\n `200` with a truncated (here: empty) body and no `x-error-pd-decode` header.\n\nFrom the first chunk of a streaming response on, the watchdog watches the gap\nbetween chunks instead: a stream that goes silent for\n`AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` is cut the same way, and the decode pod\ngets the same abort. It is armed only for requests that carry a gateway-owned\n`rid`, i.e. SGLang PD requests.\n\nA prefill leg that answered `200` with a body the gateway could not parse\n(`bad_response`) counts as succeeded for the watchdog: the KV transfer completed\nall the same, so fail-fast leaves the request alone and the watchdog times the\ndecode pod from that answer.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | int (seconds) | `60` | **Streaming requests only** (`\"stream\": true`). Time to first token: how long the decode pod is given to send its first response body chunk, measured from the moment the prefill leg returned successfully. Response headers alone do not count. On expiry the client gets a `504` (`did not start responding within …`), or a cut stream if the headers already went out. `0` disables the watchdog for streaming requests. |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | int (seconds) | `0` (disabled) | **Non-streaming requests only**. Same arming point, different meaning: the decode pod sends the body of a non-streaming response only once generation is finished, so this budget bounds the *whole* decode, not the time to first token. It is off by default because the safe value depends on the caller's `max_tokens` and the pod's throughput; operators who know both can set one. On expiry the client gets a `504` (`did not finish responding within …`). |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | int (seconds) | `120` | **Streaming requests only**. Longest gap allowed between two response body chunks once the first one arrived. Every chunk re-arms it. On expiry the stream is cut short. `0` disables it. |\n\nThe abort uses `AIBRIX_DECODE_ABORT_TIMEOUT` as its timeout and is counted in\n`gateway_pd_decode_abort_total` with `prefill_failure_class=\"watchdog_first_response\"`\nor `\"watchdog_stream_idle\"`.\n\nMetrics: `gateway_pd_decode_watchdog_total{phase}`, where `phase` is\n`first_response` or `stream_idle`. A request failed by the watchdog is also counted in\n`gateway_request_model_fail_total` with `status=\"pd_decode_watchdog\"` and\n`status_code=\"504\"`, including the cut streams whose client saw a `200`: watch\nfor those through this counter and the client's stream error, not through an\nerror body.\n\n### Decode Load Balancer Scorer (`algorithms/pd/decode_scorer.go`)\n\nScoring formula: `score = (wRun × normRunning + wThroughput × normInvThroughput) / normFreeGPU`\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | float64 | `1.0` | Weight for the normalized running-request term in the decode LB score. |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | float64 | `1.0` | Weight for the normalized inverse-throughput term in the decode LB score. |\n\n---\n\n## Utilities (`algorithms/pd/engine/trtllm.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_TRT_MACHINE_ID` | int64 | `0` | 10-bit machine ID (0–1023) used in Snowflake-style disaggregation request ID generation: `[timestamp:41b][machineID:10b][counter:12b]`. Panics on init if out of range. |\n\n---\n\n## Model Config Profile Overrides\n\nA model's `model.aibrix.ai/config` annotation can carry per-profile routing knobs\n(`routingConfig`) that override the matching variable above for that profile's requests only.\nAn unset knob keeps the environment default, and a value the variable would reject (a negative\nfactor, a percentage outside its range, an unknown GPU name) is ignored, so a profile can only\nnarrow or sharpen routing behavior. The values are resolved once per request and applied to that\nrequest alone; a request whose profile sets none behaves exactly as before. The user-facing\ndescription and examples live in the Config Profiles section of the gateway plugin guide\n([docs/source/features/gateway-routing/gateway-plugins.rst](../../../docs/source/features/gateway-routing/gateway-plugins.rst)).\n\n| Variable | Profile field | Notes |\n|---|---|---|\n| `AIBRIX_TTFT_THRESHOLD_S` | `ttftThresholdS` | Top-level profile field, not inside `routingConfig`. `0` counts as unset and keeps the environment default, so a profile can only change the threshold to another positive value, never to `0`. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | `routingConfig.promptLengthBucketing` | Turns bucketing on or off for the profile's requests. |\n| `AIBRIX_BUCKET_SERVE` | `routingConfig.bucketServe` | Turns the adaptive plan on or off for the profile's requests. |\n| `AIBRIX_BUCKET_SERVE_MODE` | `routingConfig.bucketServeMode` | `throughput` or `rps`; a name the planner does not know keeps the environment default. |\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | `routingConfig.pd.decodeAbortTimeout` | `0` sends the abort without waiting. |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | `routingConfig.pd.decodeAbortRetryDelay` | `0` repeats the abort immediately. |\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | `routingConfig.pd.decodeFirstResponseTimeout` | Seconds; `0` disables the decode watchdog for the profile's streaming requests. |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | `routingConfig.pd.decodeResponseTimeout` | Seconds; `0` disables the decode watchdog for the profile's non-streaming requests. |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | `routingConfig.pd.decodeStreamIdleTimeout` | Seconds; `0` disables the stream-idle phase for the profile. |\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.prefillLoadImbalanceMinSpread` | |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.decodeLoadImbalanceMinSpread` | |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.decodeThroughputImbalanceMinSpread` | |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | `routingConfig.pd.decodeScoreRatioThreshold` | |\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | `routingConfig.pd.decodeLBWeightRunning` | |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | `routingConfig.pd.decodeLBWeightThroughput` | |\n| `AIBRIX_TOKEN_LOAD_KV_WEIGHT` | `routingConfig.pd.tokenLoadKVWeight` | |\n| `AIBRIX_TOKEN_LOAD_REQUEST_COST` | `routingConfig.pd.tokenLoadRequestCost` | |\n| `AIBRIX_TOKEN_LOAD_TTL_SECONDS` | `routingConfig.pd.tokenLoadTTLSeconds` | `0` disables the sweep for the profile's requests. |\n| `AIBRIX_TOKEN_LOAD_SESSION_TTL_SECONDS` | `routingConfig.pd.tokenLoadSessionTTLSeconds` | `0` disables the session delta. |\n| `AIBRIX_HYBRID_CACHE_LOAD_FACTOR` | `routingConfig.pd.hybridCacheLoadFactor` | Accepted range 0 to 1. |\n| `AIBRIX_MIN_MATCH_PCT` | `routingConfig.pd.minMatchPct` | Accepted range 0 to 100; `0` disables the minimum-match clamp. |\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | `routingConfig.pd.prefillRequestTimeout` | Seconds. |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` | `routingConfig.loadBalance.imbalanceFactor` | |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` | `routingConfig.loadBalance.imbalanceMinGap` | |\n| `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` | `routingConfig.loadBalance.queuedWeight` | `0` is the V1 formula, running requests only. |\n| `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` | `routingConfig.loadBalance.kvPressureAlpha` | `0` drops the penalty. |\n| `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` | `routingConfig.loadBalance.kvCriticalFree` | Accepted range 0 to 1; `0` disables the guardrail. |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `routingConfig.prefixCache.standardDeviationFactor` | Also read by the PD prefill candidacy filter. |\n| `AIBRIX_ROUTER_PREBLE_TARGET_GPU` | `routingConfig.preble.targetGPU` | Known values `A6000` and `V100`; anything else is ignored. |\n| `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` | `routingConfig.preble.decodingLength` | |\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | `routingConfig.vtc.maxPodLoad` | |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | `routingConfig.vtc.fairnessWeight` | `0` drops the fairness term. |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | `routingConfig.vtc.utilizationWeight` | `0` drops the utilization term. |\n| `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` | `routingConfig.autoBlend.loadBalanceWeight` | `0` disables the auto-blend for the profile's requests. |\n| `AIBRIX_ROUTING_AUTO_BLEND_LEAST_REQUEST_WEIGHT` | `routingConfig.autoBlend.leastRequestWeight` | |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` | `routingConfig.autoBlend.prefixCacheWeight` | `0` is rejected with a warning and the environment value is kept; it would drop the caller's own strategy from the blend. |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_LOAD_BALANCE_WEIGHT` | `routingConfig.autoBlend.prefixCacheLoadBalanceWeight` | `0` leaves those requests with prefix-cache scoring alone. |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | `routingConfig.vtc.inputTokenWeight` | Also scopes the VTC token tracker: this weight is baked into the tracker, so the profile's requests get one of their own. |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | `routingConfig.vtc.outputTokenWeight` | Scopes the tracker the same way as the input weight. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | `routingConfig.vtc.tokenTrackerWindowSize` | Sliding window of the profile's tracker, in `tokenTrackerTimeUnit` units. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | `routingConfig.vtc.tokenTrackerTimeUnit` | Bucket size of that window: `minutes`, `seconds` or `milliseconds`. An unknown name is ignored instead of being normalized. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | `routingConfig.vtc.tokenTrackerMinTokens` | Floor the profile's tracker reports while its window holds little activity. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | `routingConfig.vtc.tokenTrackerMaxTokens` | Ceiling the profile's tracker reports while its window holds little activity. |\n\n### Scoped state behind the profile overrides\n\nThe VTC token tracker is constructed once from its window, time unit, token floors and the two\ntoken weights, so a profile that overrides any of the six gets a tracker of its own instead of\nretuning the shared one: trackers are keyed by the resolved values, so profiles that agree share\none, a profile that sets none keeps the shared tracker exactly as before, and the number of\ntrackers is bounded (16 per process; a profile past the bound keeps the shared tracker rather\nthan failing its requests).\n\n### Environment-only routing variables\n\nThese configure state shared by every model of one gateway process, so a profile cannot override\nthem without splitting or resizing that state for everyone:\n\n- `AIBRIX_ROUTER_PREBLE_SLIDING_WINDOW_PERIOD` and `AIBRIX_ROUTER_PREBLE_EVICTION_LOOP_INTERVAL`: the preble histogram window and its eviction loop are process-wide timers; scoping those needs a design of its own.\n- `AIBRIX_SESSION_AFFINITY_MAX_LOCAL_KEYS`: bounds the gateway-local session pin cache, whose entries are shared facts about a model and a session.\n- `AIBRIX_ROUTER_MAX_CACHED_ALGORITHM_STRINGS`: bounds the process-wide routing string caches.\n- `AIBRIX_TOKEN_LOAD_MAX_SESSIONS`: bounds the (model, session) table of the token-load tracker, which the gateway shares across models.\n\nVariables without a `routingConfig` field (tokenizer endpoints, Redis and statesync settings, the\nrate limiting switches, `AIBRIX_KV_CONNECTOR_TYPE`) stay environment-only as deployment-level\nsettings.\n\n---\n\n## Variable Dependency Notes\n\nThe following variables have interdependencies that must be satisfied together:\n\n- **KV event sync** requires all three to be set consistently:\n ```\n AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true\n AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true\n AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote\n AIBRIX_PREFIX_CACHE_REMOTE_TOKENIZER_ENDPOINT=\n ```\n\n- **Remote tokenizer pool** is initialized whenever `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`. The pool variables (`AIBRIX_VLLM_TOKENIZER_ENDPOINT_TEMPLATE`, `AIBRIX_TOKENIZER_*`, `AIBRIX_MAX_TOKENIZERS_PER_POOL`) all apply in that case.\n\n- **`AIBRIX_PREFIX_CACHE_POD_RUNNING_REQUEST_IMBALANCE_ABS_COUNT` is removed.** The\n prefix-cache-specific load-imbalance gate it used to tune was replaced by the centralized gate\n described under [Load Balance Router](#load-balance-router-algorithmsload_balancego), which\n now applies to prefix-cache (and every other non-exclusive strategy) via\n `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` / `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` instead. The\n trigger condition also changed: the old gate fired on absolute gap alone; the new one\n additionally requires a relative `factor × (mean + 1)` check for 3+ pod clusters, so it fires\n less often on larger fleets. Setting the old variable now only logs a startup warning.\n\n## OpenTelemetry\n\nThe gateway plugins feature built-in support for distributed tracing via OpenTelemetry (OTel), empowering you to monitor and trace end-to-end requests across the entire external processing pipeline.\n\n**Tracing is opt-in by default.** Telemetry components will only initialize if you explicitly configure either ``OTEL_EXPORTER_OTLP_ENDPOINT`` or ``OTEL_EXPORTER_OTLP_TRACES_ENDPOINT``. If both are omitted, all tracing capabilities remain disabled to conserve system resources.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|\n| `OTEL_EXPORTER_OTLP_PROTOCOL` | string | `grpc` | The transport protocol for OTLP data. Valid options are `grpc`, `http`, or `http/protobuf`. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | string | `\"\"` | Base URL for all OTLP signals. **⚠️ Note:** ensure your ENDPOINT URLs include the `http://` or `https://` prefix. The SDK automatically appends signal-specific paths to this URL (e.g., appending `/v1/traces` for HTTP exports). | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | string | `\"\"` | Target URL specifically for traces. Takes precedence over the global `ENDPOINT`. **⚠️ Note:** This URL is used exactly **as-is**. The SDK will **NOT** automatically append `/v1/traces` to it. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_HEADERS` | string | `\"\"` | Key-value pairs used as headers for OTLP requests (e.g., `key1=value1,key2=value2`). Useful for passing auth tokens to backends like Datadog or Honeycomb. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_TIMEOUT` | string | `10s` | Maximum time the OTLP exporter will wait for each batch export. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_INSECURE` | bool | `false` | Set to `true` to disable TLS/HTTPS for the exporter. Force downgrade to HTTP. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_INSECURE_SKIP_VERIFY` | bool | `false` | Keeps TLS active but skips server certificate validation (useful for self-signed certs). Set to \"true\" to enable. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n\n> **Note on OpenTelemetry Configuration:**\n> Aibrix supports the standard OpenTelemetry Protocol Exporter environment variables. For advanced OTLP configurations (such as `_CERTIFICATE`, `_CLIENT_KEY`, or specific `_COMPRESSION` settings), please refer to the [OpenTelemetry Protocol Exporter](https://opentelemetry.io/docs/specs/otel/protocol/exporter/)\n","pkg/plugins/gateway/algorithms/vtc_readme.md":"# Virtual Token Counter (VTC) Routing\n\nVirtual Token Counter (VTC) is a fair scheduling algorithm for LLM serving based on the paper\n[Fairness in Serving Large Language Models (Sheng et al., 2024)](https://arxiv.org/abs/2401.00588).\nVTC tracks the weighted token service each client has received and routes new requests to the pod\nthat gives the least-served client the best chance of progressing — balancing fairness with\ncluster utilization.\n\n## vtc-basic\n\n`vtc-basic` is a Kubernetes-native simplification of VTC. Rather than managing a single global\nqueue, it selects a target pod per request by combining two scores:\n\n- **Fairness score** — how much token service this client has received relative to all clients\n within a sliding time window. Clients who have received less service get a lower (better) score.\n- **Utilization score** — the pod's current load normalized against `MAX_POD_LOAD`.\n\nThe pod with the **lowest combined score** is selected.\n\nToken counts and the adaptive window extrema exclude expired usage, including\nclients that have stopped sending requests. The in-memory tracker reclaims that\nusage on reads and updates; it does not need a request from the departed client.\n\n### Scoring diagram\n\n```\nFor each ready pod:\n\n fairness_score = client_tokens_in_window / total_tokens_in_window\n (normalized, 0.0 → 1.0; lower = client received less service)\n\n utilization_score = pod_running_requests / MAX_POD_LOAD\n (normalized, 0.0 → 1.0; lower = pod is less loaded)\n\n combined_score = FAIRNESS_WEIGHT × fairness_score\n + UTILIZATION_WEIGHT × utilization_score\n\n ──────────────────────────────────────────────────────────────\n Select pod with minimum combined_score\n```\n\n### Request lifecycle\n\n```\nIncoming Request (with user identity)\n │\n ▼\n┌──────────────────────────────────┐\n│ Look up client token count │\n│ in sliding time window │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ Compute fairness_score │\n│ per ready pod │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ Compute utilization_score │\n│ per ready pod │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ combined_score = │\n│ w_f × fairness │\n│ + w_u × utilization │\n└───────────────┬──────────────────┘\n │\n ▼\n Select min-score pod\n │\n ▼\n┌──────────────────────────────────┐\n│ After response: update client │\n│ token count in window │\n│ (input × w_i + output × w_o) │\n└──────────────────────────────────┘\n```\n\nEnd-to-end tests: [vtc_routing_test.go](../../../../test/e2e/vtc_routing_test.go)\n\n## Configuration\n\n| Environment Variable | Description | Default |\n|-----------------------------------------------|------------------------------------------------------------------------------|----------------|\n| `AIBRIX_ROUTING_ALGORITHM` | Set to `vtc-basic` to enable this routing strategy. | `prefix-aware` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | Time unit for the sliding window. Options: `seconds`, `minutes`, `hours`. | `minutes` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | Number of time units in the sliding window. | `5` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | Minimum token floor for adaptive tracking — prevents division by near-zero. | `1000` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | Maximum token cap for adaptive tracking. | `8000` |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | Weight applied to input tokens when updating a client's service counter. | `1.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | Weight applied to output tokens when updating a client's service counter. | `2.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | Normalization denominator for pod running-request count. | `100.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | Multiplier for the fairness term in the combined score (`w_f`). | `1.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | Multiplier for the utilization term in the combined score (`w_u`). | `1.0` |\n\n### Tuning guidance\n\n- **Output token weight > input token weight** (default `2.0` vs `1.0`) reflects that output\n generation is more GPU-expensive than prompt processing.\n- **Increase `FAIRNESS_WEIGHT`** relative to `UTILIZATION_WEIGHT` when strict per-client\n fairness matters more than raw throughput.\n- **Increase `UTILIZATION_WEIGHT`** to prioritize routing to less-loaded pods, trading some\n fairness for better overall cluster efficiency.\n- **Window size** controls how long past token usage influences routing. A shorter window (e.g.\n `1 minute`) makes the router more reactive; a longer window (e.g. `10 minutes`) smooths out\n bursty clients.\n\n## Other VTC variants\n\nThe original VTC paper and reference implementation\n([slora/server/router](https://github.com/Ying1123/VTC-artifact/tree/main/slora/server/router))\ndescribe additional variants:\n\n| Variant | Description |\n|--------------|-----------------------------------------------------------------------------|\n| `vtc-fair` | Pure fairness — ignores utilization entirely. |\n| `vtc-max-fair` | Maximizes minimum service across all clients. |\n| `vtc-pred` | Uses output-length prediction to improve fairness estimates. |\n\nAdapting these variants to the AIBrix pod-selection model (rather than a single shared queue)\nis a candidate for future work.\n","pkg/plugins/gateway/ratelimiter/README.md":"# ratelimiter\n\nPackage `ratelimiter` provides a distributed, Redis-backed fixed-window rate limiter used by the Aibrix gateway to enforce per-user and per-model request limits across multiple gateway instances.\n\n## Interface\n\n```go\ntype RateLimiter interface {\n Get(ctx context.Context, key string) (int64, error)\n GetLimit(ctx context.Context, key string) (int64, error)\n Incr(ctx context.Context, key string, val int64, window ...time.Duration) (int64, error)\n}\n```\n\n| Method | Description |\n|------------|-------------|\n| `Get` | Returns the current counter value for a key in the active time window. |\n| `GetLimit` | Returns the configured maximum for a key (used for limit lookups, not window counters). |\n| `Incr` | Atomically increments the counter by `val` and returns the new value. The first write sets the key TTL to the window size; later writes do not extend it. |\n\n## Implementations\n\n### `redisRateLimiter` — `NewRedisAccountRateLimiter(name, client, windowSize)`\n\nA fixed-window counter backed by Redis. Suitable for cluster-wide enforcement when multiple gateway replicas share the same Redis instance.\n\n**Counter key scheme:** `{name}:{key}:{windowMilliseconds}:counter`. The key is stable for each configured window duration and separate from the static `{name}:{key}` key used by `GetLimit`. The counter expires `windowSize` after its first write, and the next write starts a new window. This window is independent of wall-clock bucket boundaries.\n\n- Minimum `windowSize` is 1 second (smaller values are clamped up).\n- `Incr` uses a Redis Lua script to increment and set the TTL atomically on the first write.\n- A missing key (`redis.Nil`) is treated as `0`, not an error.\n\n### `noopRateLimiter` — `NewNoopRateLimiter()`\n\nA no-op implementation that always allows requests. Used when Redis is unavailable (e.g., local development without a Redis sidecar). `GetLimit` returns `math.MaxInt64`.\n\n## Usage in the gateway\n\nTwo limiter instances are created at server startup:\n\n| Instance | Constructor arg `name` | `windowSize` | Purpose |\n|--------------------|------------------------|--------------|----------------------------------|\n| `ratelimiter` | `\"aibrix\"` | 1 minute | Per-user RPM and TPM enforcement |\n| `modelRateLimiter` | `\"aibrix_model\"` | 1 second | Per-model RPS enforcement |\n\nThe separate `name` prefix ensures the two limiters' keys never collide in Redis.\n\n### Per-user limits (RPM / TPM)\n\nCounter keys follow the pattern `aibrix:{username}_{RPM|TPM}_CURRENT:60000:counter`. Limits are read from the user record resolved during request header processing.\n\n### Per-model RPS\n\nCounter keys follow the pattern `aibrix_model:{modelName}_MODEL_RPS_CURRENT:{windowMilliseconds}:counter`. The RPS limit is configured per model via the `requestsPerSecond` field in a config profile:\n\n```json\n{\n \"profiles\": {\n \"rps-limited\": {\n \"routingStrategy\": \"least-request\",\n \"requestsPerSecond\": 1\n }\n }\n}\n```\n\nThe active profile is selected by passing the `config-profile` request header. If no profile is active or `requestsPerSecond` is unset, the RPS check is skipped entirely.\n\n#### Enforcement flow\n\nPer-model RPS enforcement is handled in `HandleRequestBody` via `enforceModelRPS` and a deferred compensation step:\n\n1. **Pre-routing gate (`enforceModelRPS`)** — called before routing. It:\n - atomically increments the counter with `Incr(..., +1)` and receives the new value\n - rejects with HTTP 429 and rolls back with `Incr(..., -1)` if `newVal > limit`\n - allows the request through when `newVal <= limit`\n2. **Deferred compensation (`decrModelRPS`)** — armed immediately after a successful pre-charge. If routing later fails, the deferred call refunds quota with `Incr(..., -1)`.\n3. **Success path** — after routing succeeds and request accounting is attached, compensation is disabled, so the pre-charge remains counted.\n\nUsing incr-then-check (rather than check-then-increment) means the `INCRBY` is the sole gate. Because Redis processes `INCRBY` atomically, each concurrent caller receives a unique sequential result — eliminating the TOCTOU race that would otherwise allow over-admission during the check→increment window.\n\n### Upgrading from wall-clock bucket keys\n\nThe counter key format changed from `{name}:{key}:{timebin}` to `{name}:{key}:{windowMilliseconds}:counter`. Existing bucket keys expire on their own TTL, but the new limiter does not read their counts. A deployment therefore starts a new counter window for each key. During a rolling upgrade, old and new gateway instances count requests separately and may collectively admit more than the configured limit. Deployments that require uninterrupted strict limits should avoid serving rate-limited traffic from both versions at the same time; a cutover to the new version still starts fresh counters.\n","pkg/plugins/gateway/statesync/README.md":"# statesync\n\nGeneric Redis-backed state sync for multiple gateway (or other) replicas. Uses **per-entity keys** with `SETEX` (per-record TTL), `MGET` (batched), and `SCAN` (enumeration). Keys are namespaced via a hash tag (`aibrix:{namespace}:e:{id}`). No PUB/SUB.\n\n- **Periodic sync**: Each replica runs **Pull first** (load from Redis), then **Push** (delta or full) every sync period.\n- **Optional write-through**: After local updates you may call `Put`/`Delete` directly. The default gateway wiring uses periodic delta push only.\n- **Deletions/tombstones**: If a Syncable implements `TombstoneSupport`, deletions are written as tombstone payloads so peers can remove the local entry on the next pull; otherwise `DEL` is used.\n- **Jitter and backoff**: Random initial startup delay (up to half `syncPeriod`); per-cycle jitter is roughly ±10% of `syncPeriod` (see `syncLoop` in `redissync.go`); exponential backoff (capped at 1 min) on errors.\n- **Lifecycle**: Register all Syncables **before** `Start()`; registrations after `Start()` are silently ignored.\n\n**Per-record TTL:** Each entity is stored as its own key with `SETEX` and a default TTL of 2 minutes. A record expires if it is not rewritten within that window. The TTL of one record does not affect others.\n\nRedis key format: `aibrix:{namespace}:e:{entityId}` — `{namespace}` is the hash tag. Namespace must be non-empty.\n\n### Prefix cache (gateway default wiring)\n\nWith `AIBRIX_STATESYNC_ENABLED=true`, `cmd/plugins/main.go`:\n\n1. Calls `table.EnableDeltaSync()` to activate dirty tracking on `PrefixHashTable` (no-op cost when disabled).\n2. Registers `PrefixHashTableSyncable` with the manager.\n3. Relies on **periodic delta push + pull** — `Put` is not called on every `AddPrefix`.\n\nThis gives eventual consistency with staleness bounded by the sync interval and per-key TTL. For faster cross-replica visibility you can call `syncManager.Put(ctx, \"prefixcache\", blockIDStr, data)` on the `*statesync.RedisSync` using bytes from `EncodeBlockForSync` after each `AddPrefix` (higher Redis write load trade-off).\n\n### Delta sync and LRU eviction\n\n`EnableDeltaSync()` must be called on `PrefixHashTable` before it is registered with `*statesync.RedisSync`; without it, dirty tracking is inactive and `GetDeltaForSync` always returns an empty delta. If a block is marked dirty then **evicted locally** before the next successful push, `GetDeltaForSync` skips it silently. The corresponding Redis key remains until the per-entity TTL expires. Strong delete propagation is not guaranteed for evicted entries.\n\n## Diagrams\n\n### Technical Flow\n\n```mermaid\nsequenceDiagram\n participant SyncLoop\n participant Redis\n participant Syncable\n Note over SyncLoop: Pull first, then Push every syncPeriod\n SyncLoop->>Redis: SCAN pattern (keys for namespace)\n Redis-->>SyncLoop: keys[]\n SyncLoop->>Redis: MGET keys (batched)\n Redis-->>SyncLoop: values[]\n SyncLoop->>Syncable: ApplyRemote(id, bytes) for each\n SyncLoop->>Syncable: GetDelta or GetSnapshot\n Syncable-->>SyncLoop: updated, deleted\n SyncLoop->>Redis: SETEX key val TTL per entity, DEL deleted\n SyncLoop->>Syncable: OptionalHooks (if implemented)\n```\n\n---\n\n## Adopting statesync in a new component\n\nChanges live **only in the package that owns the state**. The `statesync` package stays generic and never imports component types.\n\n### 1. Implement `syncable.Syncable`\n\nDepend on `github.com/vllm-project/aibrix/pkg/utils/syncable` and implement:\n\n| Method | Purpose |\n|--------|---------|\n| `Namespace() string` | Stable name for this state (e.g. `\"mytracker\"`). Used in the Redis key. |\n| `GetSnapshot(ctx) (map[string][]byte, error)` | Return current local state as **entity id → serialized bytes**. |\n| `ApplyRemote(ctx, id string, data []byte) error` | Apply one entity from Redis into local state (merge or overwrite). |\n\nOptional interfaces:\n\n| Interface | Purpose |\n|-----------|---------|\n| `DeltaSyncable` | Add `GetDelta` / `ClearDirty` to push only changed entities instead of the full snapshot. |\n| `OptionalHooks` | `OnSyncStart` / `OnSyncEnd` called around each **Pull** for that Syncable (not around Push). |\n| `StalePolicy` | `IsStale` — if true, the sync layer deletes the remote key and skips applying. |\n| `TombstoneSupport` | `MakeTombstone` / `IsTombstone` / `DeleteLocal` — propagate deletes as payloads instead of DEL. |\n\n### 2. Add serialization\n\n- **Encode**: turn your struct into `[]byte` (e.g. JSON). Use a stable format all replicas can decode.\n- **Decode**: in `ApplyRemote`, unmarshal `data` and merge into or replace local state.\n\n### 3. Expose a constructor returning `syncable.Syncable`\n\nReturn an adapter that holds a pointer to your state and implements the interface. Callers pass it to `(*statesync.RedisSync).Register(...)`.\n\n### 4. Wire in the process entry point\n\n```go\nrs := statesync.New(redisClient,\n statesync.WithSyncPeriod(30*time.Second), // optional; default 10s\n statesync.WithOpTimeout(15*time.Second), // optional; default 30s\n statesync.WithKeyPrefix(\"myapp\"), // optional; default \"aibrix\"\n)\n// Register all Syncables before Start.\nrs.Register(mypkg.NewMyStateSyncable(myState))\nrs.Start()\n\n// On shutdown:\nrs.Stop() // blocks until the sync loop exits or stop-wait timeout (default 90s)\n```\n\nAvailable `statesync.New` options:\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `WithSyncPeriod(d)` | 10s | Interval between sync cycles (jitter applied). |\n| `WithOpTimeout(d)` | 30s | Per-operation context timeout for Pull/Push. |\n| `WithKeyPrefix(s)` | `\"aibrix\"` | Prefix for all Redis keys. |\n| `WithRecordTTL(d)` | 2m | Per-entity `SETEX` TTL; values ≤0 are ignored. |\n| `WithStopWaitTimeout(d)` | 90s | Max time `Stop()` waits for the background loop to exit. |\n| `WithSetexChunkSize(n)` | 25 | Pipeline chunk size for SETEX writes. |\n| `WithMGetBatchSize(n)` | 200 | Batch size for MGET during Pull. |\n\n### Redis client\n\nThe Redis client is provided by the caller (e.g. `utils.GetRedisClient()`). Configure TLS and auth in the client options for production (see `pkg/utils/redis.go`).\n\n---\n\n## Example: a new component \"TokenTracker\"\n\nA component that keeps `tokenID → lastUsedTime` and should sync across gateway replicas.\n\n### Step 1 – Implement Syncable in the component package\n\n```go\n// pkg/plugins/gateway/algorithms/vtc/token_tracker_sync.go\n\nimport (\n \"context\"\n \"encoding/json\"\n \"strconv\"\n \"time\"\n\n \"github.com/vllm-project/aibrix/pkg/utils/syncable\"\n)\n\nconst tokenTrackerNamespace = \"token_tracker\"\n\n// TokenTrackerSyncable adapts TokenTracker for statesync.\ntype TokenTrackerSyncable struct {\n Tracker *TokenTracker\n}\n\nfunc NewTokenTrackerSyncable(t *TokenTracker) syncable.Syncable {\n return &TokenTrackerSyncable{Tracker: t}\n}\n\nfunc (s *TokenTrackerSyncable) Namespace() string { return tokenTrackerNamespace }\n\nfunc (s *TokenTrackerSyncable) GetSnapshot(ctx context.Context) (map[string][]byte, error) {\n s.Tracker.mu.RLock()\n defer s.Tracker.mu.RUnlock()\n out := make(map[string][]byte, len(s.Tracker.entries))\n for id, t := range s.Tracker.entries {\n b, _ := json.Marshal(t.UnixNano())\n out[strconv.FormatInt(id, 10)] = b\n }\n return out, nil\n}\n\nfunc (s *TokenTrackerSyncable) ApplyRemote(ctx context.Context, id string, data []byte) error {\n var nano int64\n if err := json.Unmarshal(data, &nano); err != nil {\n return err\n }\n parsed, _ := strconv.ParseInt(id, 10, 64)\n remote := time.Unix(0, nano)\n s.Tracker.mu.Lock()\n defer s.Tracker.mu.Unlock()\n if existing, ok := s.Tracker.entries[parsed]; !ok || remote.After(existing) {\n s.Tracker.entries[parsed] = remote\n }\n return nil\n}\n```\n\n### Step 2 – Wire in the entry point\n\n```go\nrs := statesync.New(redisClient, statesync.WithSyncPeriod(30*time.Second))\nrs.Register(prefixcacheindexer.NewPrefixHashTableSyncable(prefixHashTable))\nrs.Register(vtc.NewTokenTrackerSyncable(tokenTracker))\nrs.Start()\ndefer rs.Stop()\n```\n\n### Step 3 – Optional write-through\n\nIf you want other replicas to see a change before the next periodic push:\n\n```go\nfunc (t *TokenTracker) RecordUse(ctx context.Context, id int64, rs *statesync.RedisSync) {\n t.mu.Lock()\n t.entries[id] = time.Now()\n t.mu.Unlock()\n if rs != nil {\n data, _ := json.Marshal(time.Now().UnixNano())\n _ = rs.Put(ctx, tokenTrackerNamespace, strconv.FormatInt(id, 10), data)\n }\n}\n```\n\n---\n\n## Reference implementation\n\nSee `pkg/utils/prefixcacheindexer/sync.go`:\n- `PrefixHashTableSyncable` and `NewPrefixHashTableSyncable(table)` — full `DeltaSyncable` implementation.\n- `GetSnapshotForSync` / `GetDeltaForSync` / `ClearDirtyForSync` / `ApplyRemoteForSync` / `EncodeBlockForSync` on `PrefixHashTable`.\n- Remember to call `table.EnableDeltaSync()` before registering if you want delta push (not required for snapshot-only sync).\n","pkg/plugins/gateway/algorithms/external/testdata/denied.json":"{\n \"apiVersion\": \"routing.aibrix.ai/v1alpha1\",\n \"kind\": \"ReplicaSelectionResponse\",\n \"metadata\": {\"requestId\": \"req-123\", \"decisionId\": \"decision-789\"},\n \"status\": {\"decision\": \"Denied\", \"reason\": \"CompliancePolicy\"}\n}\n","pkg/plugins/gateway/testdata/envoy_gateway_v1.2.8_xds.json":"{\n \"xds\": {\n \"aibrix-system/aibrix-eg\": {\n \"configs\": [\n {\n \"@type\": \"type.googleapis.com/envoy.admin.v3.ListenersConfigDump\",\n \"dynamicListeners\": [\n {\n \"activeState\": {\n \"listener\": {\n \"@type\": \"type.googleapis.com/envoy.config.listener.v3.Listener\",\n \"accessLog\": [\n {\n \"filter\": {\n \"responseFlagFilter\": {\n \"flags\": [\n \"NR\"\n ]\n }\n },\n \"name\": \"envoy.access_loggers.file\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\",\n \"logFormat\": {\n \"jsonFormat\": {\n \":authority\": \"%REQ(:AUTHORITY)%\",\n \"bytes_received\": \"%BYTES_RECEIVED%\",\n \"bytes_sent\": \"%BYTES_SENT%\",\n \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n \"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\n \"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\n \"duration\": \"%DURATION%\",\n \"method\": \"%REQ(:METHOD)%\",\n \"protocol\": \"%PROTOCOL%\",\n \"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\n \"response_code\": \"%RESPONSE_CODE%\",\n \"response_code_details\": \"%RESPONSE_CODE_DETAILS%\",\n \"response_flags\": \"%RESPONSE_FLAGS%\",\n \"route_name\": \"%ROUTE_NAME%\",\n \"start_time\": \"%START_TIME%\",\n \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n \"upstream_host\": \"%UPSTREAM_HOST%\",\n \"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\n \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\n \"user-agent\": \"%REQ(USER-AGENT)%\",\n \"x-envoy-origin-path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n \"x-envoy-upstream-service-time\": \"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\n \"x-forwarded-for\": \"%REQ(X-FORWARDED-FOR)%\",\n \"x-request-id\": \"%REQ(X-REQUEST-ID)%\"\n }\n },\n \"path\": \"/dev/stdout\"\n }\n }\n ],\n \"address\": {\n \"socketAddress\": {\n \"address\": \"0.0.0.0\",\n \"portValue\": 10080\n }\n },\n \"defaultFilterChain\": {\n \"filters\": [\n {\n \"name\": \"envoy.filters.network.http_connection_manager\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\",\n \"accessLog\": [\n {\n \"name\": \"envoy.access_loggers.file\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\",\n \"logFormat\": {\n \"jsonFormat\": {\n \":authority\": \"%REQ(:AUTHORITY)%\",\n \"bytes_received\": \"%BYTES_RECEIVED%\",\n \"bytes_sent\": \"%BYTES_SENT%\",\n \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n \"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\n \"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\n \"duration\": \"%DURATION%\",\n \"method\": \"%REQ(:METHOD)%\",\n \"protocol\": \"%PROTOCOL%\",\n \"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\n \"response_code\": \"%RESPONSE_CODE%\",\n \"response_code_details\": \"%RESPONSE_CODE_DETAILS%\",\n \"response_flags\": \"%RESPONSE_FLAGS%\",\n \"route_name\": \"%ROUTE_NAME%\",\n \"start_time\": \"%START_TIME%\",\n \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n \"upstream_host\": \"%UPSTREAM_HOST%\",\n \"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\n \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\n \"user-agent\": \"%REQ(USER-AGENT)%\",\n \"x-envoy-origin-path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n \"x-envoy-upstream-service-time\": \"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\n \"x-forwarded-for\": \"%REQ(X-FORWARDED-FOR)%\",\n \"x-request-id\": \"%REQ(X-REQUEST-ID)%\"\n }\n },\n \"path\": \"/dev/stdout\"\n }\n }\n ],\n \"commonHttpProtocolOptions\": {\n \"headersWithUnderscoresAction\": \"REJECT_REQUEST\"\n },\n \"http2ProtocolOptions\": {\n \"initialConnectionWindowSize\": 1048576,\n \"initialStreamWindowSize\": 65536,\n \"maxConcurrentStreams\": 100\n },\n \"httpFilters\": [\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"STREAMED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"STREAMED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"BUFFERED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"name\": \"envoy.filters.http.router\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\",\n \"suppressEnvoyHeaders\": true\n }\n }\n ],\n \"mergeSlashes\": true,\n \"normalizePath\": true,\n \"pathWithEscapedSlashesAction\": \"UNESCAPE_AND_REDIRECT\",\n \"rds\": {\n \"configSource\": {\n \"ads\": {},\n \"resourceApiVersion\": \"V3\"\n },\n \"routeConfigName\": \"aibrix-system/aibrix-eg/http\"\n },\n \"serverHeaderTransformation\": \"PASS_THROUGH\",\n \"statPrefix\": \"http-10080\",\n \"useRemoteAddress\": true\n }\n }\n ],\n \"name\": \"aibrix-system/aibrix-eg/http\"\n },\n \"name\": \"aibrix-system/aibrix-eg/http\",\n \"perConnectionBufferLimitBytes\": 4194304\n }\n }\n }\n ]\n },\n {\n \"@type\": \"type.googleapis.com/envoy.admin.v3.RoutesConfigDump\",\n \"dynamicRouteConfigs\": [\n {\n \"routeConfig\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.RouteConfiguration\",\n \"ignorePortInHostMatching\": true,\n \"name\": \"aibrix-system/aibrix-eg/http\",\n \"virtualHosts\": [\n {\n \"domains\": [\n \"*\"\n ],\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"Gateway\",\n \"name\": \"aibrix-eg\",\n \"namespace\": \"aibrix-system\",\n \"sectionName\": \"http\"\n }\n ]\n }\n }\n },\n \"name\": \"aibrix-system/aibrix-eg/http/*\",\n \"routes\": [\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"safeRegex\": {\n \"regex\": \"^/v1/videos/sync$|^/v1/videos/[^/]+/content$\"\n }\n },\n \"name\": \"original_route_videos_streaming\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"safeRegex\": {\n \"regex\": \"^/v1/videos(/[^/]+)?$\"\n }\n },\n \"name\": \"original_route_videos\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"prefix\": \"/\"\n },\n \"name\": \"original_route\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"path\": \"/v1/videos/sync\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos-streaming\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"safeRegex\": {\n \"regex\": \"/v1/videos/[^/]+/content\"\n }\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos-streaming\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0/match/1/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/transcriptions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/15/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/images/generations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/13/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/translations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/16/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/video/generations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/19/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/chat/completions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/voices\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/18/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/images/edits\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/14/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/speech\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/17/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/completions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/3/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/embeddings\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/4/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/systemone\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/9/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/decisions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/8/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/responses\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/1/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/classify\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/6/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/messages\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/2/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"directResponse\": {\n \"status\": 500\n },\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/batches\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-metadata-endpoint\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-metadata-endpoint/rule/0/match/2/*\"\n },\n {\n \"directResponse\": {\n \"status\": 500\n },\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/models\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-metadata-endpoint\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-metadata-endpoint/rule/0/match/0/*\"\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/videos\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/rerank\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/5/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/tokenize\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/11/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy","pkg/plugins/gateway/algorithms/external/testdata/no-decision.json":"{\n \"apiVersion\": \"routing.aibrix.ai/v1alpha1\",\n \"kind\": \"ReplicaSelectionResponse\",\n \"metadata\": {\"requestId\": \"req-123\"},\n \"status\": {\"decision\": \"NoDecision\", \"reason\": \"NoApplicablePolicy\"}\n}\n","pkg/plugins/gateway/algorithms/load_balance.md":"# Load Balance Router\n\n`load-balance` routes each request to the replica with the lowest **effective load**: the work\nalready committed to it, divided by how fast it has been observed to drain work, inflated as its\nKV cache fills. It needs no hardware labels. A mixed B40 / A100 / H20 pool teaches the gateway\nits relative speeds from observed output-token throughput, and faster replicas receive\nproportionally more traffic.\n\nSource: [load_balance.go](load_balance.go). Environment variables:\n[ENV_VARS.md](../ENV_VARS.md#load-balance-router-algorithmsload_balancego).\n\n`load-balance` is also blended in behind other strategies by default (see\n`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` and [multi_router_readme.md](multi_router_readme.md)),\nso its `ScoreAll` runs on most requests, not only when it is selected explicitly.\n\n## Score\n\n```\n running + λ · queued\nscore = ───────────────────────────────────── × (1 + α · (1 − kvFree)²)\n EWMA(output tokens completed / sec)\n\nkvFree < kvCritical ⇒ score = +Inf\n```\n\nThe lowest score wins. The three parts answer three separate questions, and each is used for one\npurpose only:\n\n| Part | Question | Source |\n|---|---|---|\n| `EWMA(output tokens/sec)` | How fast does this replica drain work? (**capacity**) | `RealtimeOutputTokenRateEWMA`, derived in the cache from completed output tokens |\n| `running + λ·queued` | How much work is already committed to it? (**load**) | `GetPodsRunningRequests` (live cross-gateway count); `num_requests_waiting` when `λ > 0` |\n| `1 + α(1 − kvFree)²` | How close is it to memory pressure? (**headroom**) | `kv_cache_usage_perc` (`KVCacheUsagePerc`) |\n\nCapacity and load are kept apart on purpose. A saturated GPU has high token throughput *because*\nit is busy; if throughput were used as \"load\", the scheduler would send it even more traffic.\nHere throughput only sets the denominator, and load comes from the committed work.\n\n### Defaults\n\n| Symbol | Meaning | Default | Variable |\n|---|---|---|---|\n| `λ` | Weight of engine-queued requests | `0` | `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` |\n| `α` | KV-pressure penalty strength | `2.0` | `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` |\n| `kvCritical` | Free-KV fraction below which a pod scores `+Inf` from `load-balance` (excluded when `load-balance` routes alone; only strongly penalized in a blend — see \"Interaction with the rest of the gateway\" below) | `0.10` | `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` |\n\n`λ` defaults to `0` because the gateway's running count already includes requests that are\nqueued inside the engine; adding `num_requests_waiting` on top would count them twice. Values\n`<= 0` for these variables fall back to the default (the loader rejects non-positive numbers).\n\n### Worked example\n\n| Replica | Running | Tokens/s | KV free | Calculation | Score |\n|---|---|---|---|---|---|\n| B40 | 30 | 6000 | 50% | 30/6000 × (1 + 2·0.50²) | **0.0075** |\n| A100 | 20 | 4000 | 30% | 20/4000 × (1 + 2·0.70²) | 0.0099 |\n| H20 | 25 | 3000 | 60% | 25/3000 × (1 + 2·0.40²) | 0.0110 |\n\nThe B40 wins despite carrying the most requests, because it drains them fastest. This example is\nasserted in `TestLoadBalanceScoreAll_HeterogeneousExample`.\n\n## Where the capacity signal comes from\n\n```\ngateway: DoneRequestTrace(outputTokens) ──► Pod.completedOutputTokens (monotonic counter)\n │ every metrics refresh (default 1s)\n ▼\n calculateRate1m: tokens/sec over ~1 minute window\n │\n ▼\n EWMA, time constant 10s ──► RealtimeOutputTokenRateEWMA (pod-scoped)\n```\n\n- The counter is **gateway-tracked**, like `completedRequests`, so it works for every engine\n (vLLM, SGLang, TRT-LLM, xLLM) without depending on engine metric names. Code:\n `donePodStats` in `pkg/cache/cache_trace.go`, `updateRealtimeOutputTokenRateEWMA` in\n `pkg/cache/cache_metrics.go`.\n- Output tokens are the request's completion-token usage. A request that finishes through the\n count-only path (no usage available) adds one completion and **zero** tokens.\n- The counter survives a brief pod delete/re-add (informer flap) the same way `completedRequests`\n does.\n- A window with no completed tokens carries no information about speed (the replica was idle,\n not slow), so the previous estimate is **kept**, not decayed toward zero. Decaying would make a\n briefly idle replica look weak and starve it of the traffic that would re-measure it.\n\n## Behavior in edge cases\n\n| Situation | Behavior |\n|---|---|\n| Pod has no token-rate estimate yet (new pod, or no usage reported) | Scored at the **mean** estimate of the pods that have one, so it competes as an average replica. A fixed fallback such as `1.0` would be off by the tokens/sec scale (thousands) and lock the pod out of the traffic it needs to be measured. |\n| No pod has an estimate | Every pod gets capacity `1.0`; the score reduces to `running × KV penalty`. |\n| Engine reports no KV usage, or `NaN` | Treated as fully free: no penalty and no guardrail. |\n| `kvFree < kvCritical` | Score `+Inf` from `load-balance`. When `load-balance` routes alone this skips the pod entirely until it recovers. When `load-balance` is blended with other strategies (the default — see below), it is only a strong penalty, not a guaranteed exclusion. |\n| **Every** pod is below `kvCritical` | The request is **not** failed. Routing falls back to the tie-break over all pods, which picks the one with the least KV usage. This fallback is specific to `load-balance` routing alone; see below for the blended case. |\n| Pod is idle (`running = 0`) | Score `0` regardless of capacity, so idle pods are preferred. |\n| Several pods share the lowest score | Ties are broken by least combined GPU+CPU KV-cache usage (`least-kv-cache` scorer), falling back to random. |\n\nKV usage is clamped to `[0, 1]` before use, so an engine that reports a value above 1 is treated\nas full rather than producing a negative free fraction.\n\n## Interaction with the rest of the gateway\n\n- **Load-imbalance gate.** The gateway applies `ApplyLoadImbalanceGate` once, ahead of whichever\n strategy routes the request, so `Route` receives an already-narrowed pod list. If that narrowed\n set is entirely KV-critical, the all-critical fallback above applies. **TODO / known\n limitation:** the gate compares raw running-request count, with no notion of a pod's capacity —\n a replica that's simply faster than its peers is expected to carry more concurrent requests\n without being more loaded (the same capacity signal `load-balance`'s own score uses above), but\n this gate can still flag it as a hotspot and exclude it before `load-balance`'s own\n capacity-aware score ever gets to run. See `getTargetPodListOnLoadImbalance` in\n [load_balance.go](load_balance.go).\n- **Multi-strategy blending: the guardrail is soft here, not hard.** `kvCritical` is a hard\n exclusion only when `load-balance` routes alone. In a blend, `ScoreAll` still returns `+Inf` for\n a critical pod, but `multiStrategyRouter.normalizeScoresArray` treats a non-finite score the\n same as \"not scored\" and assigns it `0` — the worst possible score from `load-balance`, not an\n exclusion from routing. If another blended strategy scores that same pod highly enough, it can\n still be selected.\n\n Concretely: `load-balance` is blended in behind other strategies by default (see\n `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT`), so the blended path, not the standalone one, is\n what most requests actually take. With the default `prefix-cache:5,load-balance:4` blend, a\n KV-critical pod that has the best prefix-cache score still gets `5/9` of the combined weight\n versus `4/9` for a healthy pod that only wins on `load-balance`, so the critical pod wins the\n route. `kvCritical` should therefore be read as \"how hard `load-balance` argues against routing\n to this pod,\" not as a safety limit that blocks it from being chosen — it does not, on its own,\n prevent a KV-critical pod from receiving more traffic. Something that must never route to a\n KV-critical pod needs a filter ahead of scoring (comparable to `ApplyLoadImbalanceGate`), which\n does not currently exist for KV pressure.\n- **PD disaggregation.** The PD router's decode path keeps its own request drain-rate scoring\n (`RealtimeRunningRequestsDrainRate1m`) and is unchanged.\n\n## Known limitations\n\n- **Multiple gateway replicas.** The token counter is local to each gateway, while the running\n count is the cross-gateway total. Each gateway therefore measures roughly `1/N` of a pod's\n output. Relative ranking between pods is preserved as long as the gateways spread traffic\n similarly, but absolute score values are not comparable across deployments with different\n gateway counts.\n- **Estimate lag.** The estimate follows a ~1 minute window smoothed with a 10 second time\n constant, so a sudden change in a replica's speed (for example a workload shifting from\n decode-heavy to prefill-heavy) shows up within tens of seconds, not immediately.\n- **No prefill/decode split.** Capacity is a single output-token rate. Hardware whose relative\n performance differs between prompt-heavy and decode-heavy workloads is not modeled; a natural\n extension is `capacity = α·prefill_tokens/s + β·decode_tokens/s`.\n- **Deliberately not used:** GPU utilization / SM-active. They are useful for observability but\n make a generic scheduler unstable; the score uses only drain rate, committed work, and KV\n pressure.\n\n## Debugging\n\nAt `-v=4` the gateway logs `load_balance_score` once per pod (`running_requests`,\n`capacity`, `kv_free`, and the resulting `score`) and `load_balance_selected` (with the\nwinning `score`). `+Inf` scores are logged as strings so klog's JSON formatter does not\nfail. The cache logs `Updating output token rate metric` with the completed token count,\nthe 1-minute rate, and the EWMA for each pod. The estimate is stored on the pod as\n`realtime_output_token_rate_ewma`; it is not currently exported as a Prometheus series.\n","pkg/plugins/gateway/algorithms/external/testdata/selected.json":"# Testdata Skill\n\nPath: `pkg/plugins/gateway/algorithms/external/testdata/selected.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/external/testdata/selected.json)"},"files":{"pkg/plugins/gateway/algorithms/multi_router_readme.md":"# Multi-Strategy Routing (Batch Soft Scoring)\n\nAIBrix Gateway supports multi-strategy routing, allowing you to combine multiple routing algorithms with different weights. Instead of making hard decisions (where one strategy completely filters out pods or dictates the single winner), the multi-strategy router uses a **Batch Soft Scoring** mechanism to evaluate pods across all specified strategies.\n\n## How it works\n\n1. **Batch Scoring:** Each configured strategy evaluates all ready pods in a single batch operation (`ScoreAll`), returning raw scores and a boolean array indicating whether each pod could be scored (e.g., handling missing metrics).\n2. **Normalization:** The raw scores from each strategy are normalized into a `[0, 1]` range:\n * **Higher is better** (e.g., `throughput`): `(score - min) / (max - min)`\n * **Lower is better** (e.g., `least-request`): `(max - score) / (max - min)`\n * If a pod lacks metrics (`scored=false`), its normalized score is `0.0` (acting as a penalty).\n3. **Weighted Aggregation:** The normalized scores are multiplied by their respective configured weights and summed up for each pod.\n4. **Top-1 Selection:** The pod with the highest total aggregated score is selected to serve the request.\n\n## Configuration Format\n\nYou can configure multi-strategy routing by setting the `AIBRIX_ROUTING_ALGORITHM` environment variable using a comma-separated list of `strategy:weight` pairs.\n\n* **Format:** `strategy1:weight1,strategy2:weight2,...`\n* If `:weight` is omitted, it defaults to `1`.\n* If a weight is `0`, the strategy is completely ignored.\n* If an invalid strategy is specified, the Gateway rejects the request with **400 Bad Request** instead of silently falling back to `random`.\n* `session-affinity` is treated as a soft-scoring strategy in multi-strategy mode: it boosts the matching pod, but the final weighted winner may still be a different pod. The response session header is updated to the final selected pod.\n With Redis-backed pinning, background sync applies its result only if the local entry has not changed since the read began. A delayed result cannot overwrite or evict a pin committed by a concurrent request.\n* A caller may send an opaque application session identifier in `x-aibrix-session-key`. When no valid gateway-issued `x-session-id` target is ready, rendezvous hashing selects a stable pod for that key. This is useful for concurrent Agent turns that start before a response cookie is available. The opaque key is never returned to the client. The `pd` router's `token_load` and `hybrid_cache_load` prefill policies read the same key to charge a multi-turn conversation only for the tokens added since its previous turn.\n\n## Examples\n\nHere are some real-world examples demonstrating how the weighted aggregation works under different scenarios.\n\nAssume we have two pods serving a model:\n* **Pod A**: High load (10 requests) but High throughput (100 tok/s).\n* **Pod B**: Low load (2 requests) but Low throughput (10 tok/s).\n\n### Example 1: `least-request` Dominates\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:10,throughput:1\"\n```\n* `least-request` (weight 10): Pod B is much better (2 vs 10). Normalized: Pod A = 0.0, Pod B = 1.0. Weighted Score: A = 0.0 * 10/11 = 0.0, B = 1.0 * 10/11 ≈ 0.91.\n* `throughput` (weight 1): Pod A is much better (100 vs 10). Normalized: Pod A = 1.0, Pod B = 0.0. Weighted Score: A = 1.0 * 1/11 ≈ 0.09, B = 0.0 * 1/11 = 0.0.\n* **Total Score**: Pod A ≈ 0.09, Pod B ≈ 0.91. **Pod B wins!**\n\n### Example 2: `throughput` Dominates\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,throughput:10\"\n```\n* `least-request` (weight 1): Weighted Score: A = 0.0 * 1/11 = 0.0, B = 1.0 * 1/11 ≈ 0.09.\n* `throughput` (weight 10): Weighted Score: A = 1.0 * 10/11 ≈ 0.91, B = 0.0 * 10/11 = 0.0.\n* **Total Score**: Pod A ≈ 0.91, Pod B ≈ 0.09. **Pod A wins!**\n\n### Example 3: Complex Scenario with Corner Cases (3 Strategies)\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request,throughput:2,least-latency:0\"\n```\n* `least-request`: No weight specified, **defaults to 1**. Weighted Score: A = 0.0 * 1/3 = 0.0, B = 1.0 * 1/3 ≈ 0.33.\n* `throughput`: Explicitly set to weight **2**. Weighted Score: A = 1.0 * 2/3 ≈ 0.67, B = 0.0 * 2/3 = 0.0.\n* `least-latency`: Weight set to **0**, so this strategy is **ignored** and does not participate in scoring.\n* **Total Score**: Pod A ≈ 0.67, Pod B ≈ 0.33. **Pod A wins!**\n\n### Example 4: Invalid Configuration Fallback\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,invalid-strategy:1\"\n```\n* Because `invalid-strategy` is not a registered algorithm, the multi-strategy parser will fail. To preserve the API contract, the Gateway will reject the request with a **400 Bad Request** error rather than silently falling back to a random router.\n\n### Example 5: Multi-Strategy Finding the Sweet Spot\nIn reality, a pod with the absolute minimum active requests might have just recovered from being severely overloaded. Multi-strategy routing prevents routing to extreme outliers by considering multiple dimensions.\n\nAssume three pods with the following metrics:\n* **Pod A**: Very low active requests (1 req), but historically massively overloaded (1500 tokens).\n* **Pod B**: Historically barely used (15 tokens), but currently slammed with active requests (50 req).\n* **Pod C**: The \"Sweet Spot\" - Medium active requests (10 req), and medium historical load (150 tokens).\n\n**Single Strategy Flaws:**\n* `AIBRIX_ROUTING_ALGORITHM=\"least-request\"`: Completely ignores historical load and picks **Pod A** (which is exhausted).\n* `AIBRIX_ROUTING_ALGORITHM=\"throughput\"`: Completely ignores current spikes and picks **Pod B** (which is currently slammed).\n\n**Multi-Strategy Solution:**\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,throughput:1\"\n```\n* `least-request` (weight 1): Normalizes active requests [1, 50]. \n * Pod A: 1.0\n * Pod B: 0.0\n * Pod C: 0.816\n* `throughput` (weight 1): Normalizes historical load [15, 1500].\n * Pod A: 0.0\n * Pod B: 1.0\n * Pod C: 0.909\n* **Total Score (Weighted)**: Pod A = 0.5, Pod B = 0.5, Pod C ≈ **0.86**. \n* **Result**: **Pod C wins!** The router correctly identifies the most balanced pod, avoiding the exhausted Pod A and the currently slammed Pod B.\n","pkg/plugins/gateway/algorithms/prefix_cache_readme.md":"# Prefix Cache Aware Routing\n\nPrefix Cache Aware routing improves KV cache reuse across GPU pods by steering requests whose\nprompt shares a common prefix to the pod that already has those KV blocks cached. Pod selection\namong prefix-matched candidates is still load-aware (see Step 2 below), and this router itself\nis purely about prefix caching — it does not gate or restrict candidates based on cluster-wide\nload imbalance. That concern is instead handled centrally by the gateway: `selectTargetPod` in\n`gateway.go` applies the [load-balance router](load_balance.go)'s `ApplyLoadImbalanceGate` once,\nahead of whichever strategy actually routes the request (prefix-cache included), narrowing\n`readyPods` before this router ever sees them. This replaced an older prefix-cache-specific gate\nthat lived in this file; see [ENV_VARS.md](../ENV_VARS.md#variable-dependency-notes) for the\nmigration note. The gate applies the same way when combining strategies via multi-strategy\nsoft-scoring (`AIBRIX_ROUTING_ALGORITHM=prefix-cache:W1,load-balance:W2`, see\n[multi_router_readme.md](multi_router_readme.md)): `selectTargetPod` narrows `readyPods` once,\nbefore resolving and calling into the multi-strategy router, so every sub-strategy's `ScoreAll`\n— this one included — only ever sees the already-gated pod list. It is skipped only for\nexclusive strategies (`pd`, `slo*`), which manage their own pod subsets.\n\nTwo routing modes are supported, selected automatically at startup:\n\n| Mode | When active | Prefix index |\n|------|-------------|--------------|\n| **Standard** | `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=false` (default) | Local hash table updated by the router after each request |\n| **KV Sync** | `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true` | Distributed sync indexer fed by real-time block-stored/removed events from vLLM pods |\n\nBoth modes share the same stddev-based pod selection logic described below.\n\n## Routing Flow\n\n```\n Incoming Request\n │\n ▼\n┌─────────────────────────────┐\n│ Tokenize & hash prompt │ splits prompt into fixed-size blocks,\n│ into prefix blocks │ computes hash per block\n└────────────┬────────────────┘\n │\n ▼\n┌─────────────────────────────┐\n│ Match prefix hashes │\n│ against ready pods │\n└────────────┬────────────────┘\n │\n ┌────────┴────────┐\n │ match_pods │ NO match_pods\n │ found? │──────────────────────────────────────────────┐\n └────────┬────────┘ │\n │ YES │\n┌─────────────────────────────┐ │\n│ Sort match_pods by: │ │\n│ 1. prefix_match% DESC │ │\n│ 2. running_requests ASC │ │\n└────────────┬────────────────┘ │\n │ │\n┌─────────────────────────────┐ NO pod within threshold │\n│ Select first pod where: │─────────────────────────────────────►│\n│ running_req <= │ │\n│ mean + load_factor * σ │ │\n└────────────┬────────────────┘ │\n │ │\n │ ▼\n │ Least-request fallback among all ready pods\n │ │\n ▼ ▼\n Target Pod Selected\n```\n\n## Algorithm Details\n\n### Step 1 — Tokenize & Hash\n\nThe prompt is tokenized and split into fixed-size blocks. A hash is computed for each block.\nTwo tokenizer backends are supported:\n\n| Tokenizer | How it works |\n|------------|-------------------------------------------------------|\n| `character`| Splits raw text into individual characters (default) |\n| `tiktoken` | Uses the OpenAI [tiktoken](https://github.com/openai/tiktoken) BPE tokenizer |\n\nBlock hashes are used as cache keys — a pod that has processed a request with an identical\nprefix will have those KV blocks hot in GPU memory.\n\n### Step 2 — Prefix Match & Pod Selection\n\nEach candidate pod is scored by how many of the request's prefix blocks it already holds\n(`prefix_match_percent`). Matched pods are sorted:\n\n1. **Descending** by `prefix_match_percent` — prefer higher cache hit rate.\n2. **Ascending** by `running_requests` — break ties by choosing the less loaded pod.\n\nThe first pod in this sorted list that satisfies the load threshold is selected:\n\n```\npod.running_requests <= mean_running_requests + load_factor × std_dev\npod.running_requests <= max(load_cap_factor, 1) × (mean_running_requests + 1)\n```\n\n`load_factor` (= `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR`) controls how aggressively\nthe router skips overloaded cache-holding pods. If no matched pod is within the threshold, the\nrouter falls back to the global least-loaded pod across all ready pods. `load_cap_factor`\n(= `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR`) bounds load directly: no value in n pods sits more than\n(n−1)/√n standard deviations above the mean, so at `load_factor` 2 the first check cannot skip\nany pod in a pool of five or fewer.\n\n## Configuration\n\n| Environment Variable | Description | Default |\n|---|---|---|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | Tokenizer used to split the prompt. Options: `character`, `tiktoken`. | `character` |\n| `AIBRIX_PREFIX_CACHE_BLOCK_SIZE` | Number of tokens per prefix block. Smaller blocks = finer-grained matching but more hash overhead. | `128` |\n| `AIBRIX_PREFIX_CACHE_BLOCK_NUMBER` | Maximum number of prefix cache blocks tracked per pod. | `200000` |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `load_factor` in `mean + load_factor × σ`. Higher value tolerates more load on a high-cache-hit pod. | `1` |\n| `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR` | `load_cap_factor` in `max(load_cap_factor, 1) × (mean + 1)`. Higher value tolerates more load on a high-cache-hit pod; set a large value such as `100` to effectively disable it. Applies only when `prefix-cache` routes on its own, with the auto-blend off (`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT=0`). | `1.5` |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED` | Enable KV sync routing mode (requires remote tokenizer). | `false` |\n| `AIBRIX_PREFIX_CACHE_INCLUDE_TOOLS` | Prepend the canonicalized chat request `tools` to the prompt text before hashing, so requests that share messages but carry different tools do not look like a full prefix match. Token estimates keep using the messages only. KV-sync mode tokenizing through the remote chat template forwards the raw `tools` to the tokenizer regardless of this flag, so the template places them. Set to `false` to hash the messages only. | `true` |\n\nSee [ENV_VARS.md](../ENV_VARS.md#load-balance-router-algorithmsload_balancego) for the\nload-imbalance gate's `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` /\n`AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` variables — they belong to the load-balance router, not\nthis one.\n\n### Tuning guidance\n\n- **Block size**: use `128` for `character` tokenizer and `16` for `tiktoken`. Larger blocks\n reduce hash overhead but require longer identical prefixes for a match.\n- **Standard deviation factor**: `1` (default) keeps all pods within one stddev of mean load.\n Increase to `2` to allow higher-loaded pods to still be selected for cache-hit requests.\n","AGENTS.md":"# AIBrix\n\nAIBrix is a Kubernetes-native platform for building scalable GenAI inference\ninfrastructure. This repository contains Go controllers, gateway plugins,\ncustom resources, deployment manifests, tests, and Python runtime components.\nThis file gives coding agents the repository-specific context and safety rules\nneeded to make focused changes.\n\n## Scope and repository layout\n\n| Path | Role |\n|------|------|\n| `api/` | Kubernetes API types and CRD definitions |\n| `pkg/controller/` | Kubernetes controllers and reconciliation logic |\n| `pkg/plugins/` | Gateway plugins and request processing |\n| `pkg/` | Shared Go libraries, clients, caches, metrics, and utilities |\n| `cmd/` | Go binary entrypoints |\n| `config/` | Kustomize configuration, CRDs, RBAC, and deployment manifests |\n| `test/integration/` | Ginkgo-based integration tests |\n| `test/e2e/` | Kind and cluster-level end-to-end tests |\n| `hack/` | Code generation, verification, and CI scripts |\n| `python/aibrix/` | Python runtime, downloader, batch, metadata, and optimizer |\n| `python/aibrix_kvcache/` | Python distributed KV-cache components |\n| `apps/` | Application services and the console |\n| `docs/` | Project documentation and generated documentation assets |\n| `samples/` | Example deployments and feature samples |\n\nRules in a deeper `AGENTS.md` take precedence for files under that directory.\nThe Python runtime has additional guidance in\n[`python/aibrix/AGENTS.md`](python/aibrix/AGENTS.md).\n\n## Working rules\n\n- State the intended scope and success criteria before making a non-trivial\n change.\n- Read the closest existing implementation and its tests before introducing a\n new pattern.\n- Keep changes focused. Do not combine unrelated refactors, formatting, or\n generated output with a feature or bug fix.\n- Verify behavior from code and tests rather than inferring it from filenames.\n- Preserve unrelated user changes in the working tree.\n- Do not push branches, rewrite history, modify remote GitHub state, or send\n external messages without explicit authorization for that action.\n- Ask before upgrading dependencies or changing `.github/` workflow policy.\n\n## API and compatibility surfaces\n\nTreat the following as compatibility surfaces:\n\n- Kubernetes API types, CRD schemas, JSON/YAML tags, defaults, validation,\n list semantics, subresources, and printer columns.\n- Public HTTP endpoints, request and response fields, headers, status codes,\n and gateway plugin contracts.\n- CLI flags, configuration keys, environment variables, labels, and\n annotations used by deployment or controller logic.\n\nDo not rename, remove, or repurpose an existing field or key without an\nexplicit migration and compatibility plan. Preserve pointer and `omitempty`\nchoices when absent, zero, and explicit values have different meanings.\nKeep status fields observational; status transitions must remain consistent\nwith every controller that writes or consumes them.\n\nKeep API types declarative. Admission behavior belongs in webhooks,\nreconciliation policy belongs in controllers, and transport compatibility\nbelongs in the relevant API or gateway layer.\n\n## Build and verification\n\nRun commands from the repository root. Start with the narrowest relevant check\nand run broader checks for changes that cross package or language boundaries.\n\n| Command | Purpose |\n|---------|---------|\n| `make fmt` | Format Go code |\n| `make vet` | Run `go vet ./...` |\n| `make generate` | Regenerate Go and Kubernetes generated artifacts |\n| `make manifests` | Regenerate webhook, RBAC, and CRD manifests |\n| `make verify` | Verify generated code and CRD synchronization |\n| `make lint` | Run Go linting |\n| `make lint-all` | Run license and Go lint checks |\n| `make test` | Run Go unit tests, excluding integration and E2E packages |\n| `make test-race-condition` | Run Go unit tests with the race detector |\n| `make test-integration` | Run integration tests |\n| `make python-ci` | Run Python Ruff, mypy, and tests for `python/aibrix` |\n| `make test-e2e` | Run Kind-based end-to-end tests |\n\nFor a Go package, a focused check such as `go test ./pkg//...` is\nappropriate while iterating. For cluster-level behavior, use the existing\nhelpers and test organization under `test/e2e/`; do not replace polling with\narbitrary sleeps.\n\nWhen API types, controller-gen markers, or CRDs change, run the applicable\ngeneration and verification targets and include the resulting generated files\nin the change. Do not hand-edit generated files unless the generator explicitly\nrequires it.\n\n## Coding conventions\n\n- Follow existing Go package structure and standard Go style; use `gofmt` or\n `make fmt`.\n- Put reusable implementation in `pkg/`; keep `cmd/` entrypoints thin.\n- Use table-driven tests where they match the surrounding package style.\n- Add or update tests for behavior changes, including failure and compatibility\n cases when applicable.\n- Comments should explain non-obvious reasons or invariants, not restate code.\n- Preserve existing logging, error handling, retry, timeout, and context\n propagation patterns unless the change specifically addresses them.\n- For Python changes, follow the more specific rules in\n [`python/aibrix/AGENTS.md`](python/aibrix/AGENTS.md) and run checks from that\n subtree as documented there.\n\n## Pull requests and documentation\n\n- Keep a PR focused on one issue or coherent behavior change.\n- Use the repository [PR template](.github/PULL_REQUEST_TEMPLATE.md), link the\n relevant issue, and describe the tests actually run.\n- Update user or developer documentation when behavior, configuration, API,\n deployment, or CLI usage changes.\n- When changing any `docs/source/**/*.rst` file, update the corresponding\n `docs/source/locale/zh_CN/LC_MESSAGES/**/*.po` catalog in the same change,\n including added, modified, and removed content. Do not leave new or changed\n documentation falling back to English on the Chinese site.\n- Before submitting an RST documentation change, run `make html-all` from\n `docs/` and confirm that both the English and Simplified Chinese HTML builds\n succeed.\n- Large design changes should be discussed before implementation and recorded\n in the appropriate documentation area.\n- Review every generated or AI-assisted change yourself; do not submit code\n that you cannot explain and verify.\n\n## Further reading\n\n- [Contributing guide](CONTRIBUTING.md)\n- [Development guide](development/README.md)\n- [Test guide](test/README.md)\n- [Project documentation](docs/README.md)\n","pkg/plugins/gateway/algorithms/pd_readme.md":"# PD Disaggregation Router\n\n`pd/router/router.go` implements the **prefill-decode (PD) disaggregated routing algorithm** (`RouterPD = \"pd\"`). **This file (`pd_readme.md`) is the canonical documentation** for PD: request flow, scoring, engine behavior, metrics, and environment variables. PD-only settings are under **Environment Variables**; variables shared with the prefix-cache router (tokenizer, stddev factor, and related `AIBRIX_PREFIX_CACHE_*` knobs) are listed in **Inherited from Prefix Cache Router** in the same section.\n\nIn PD disaggregation, inference is split across two specialized pod roles:\n\n- **Prefill pod** — processes the prompt (context), builds the KV-cache, then transfers it.\n- **Decode pod** — generates tokens using the KV-cache transferred from the prefill pod.\n\nThe router is responsible for selecting one prefill pod and one decode pod per request, executing the prefill HTTP request synchronously (vLLM/TRT-LLM context-first) or asynchronously (SGLang/TRT-LLM generation-first), and routing the decode request through Envoy to the selected decode pod.\n\n---\n\n## High-Level Request Flow\n\n```\nClient Request\n │\n ▼\nRoute(ctx, readyPodList)\n │\n ├─► ValidateAndGetLLMEngine()\n │ Ensure all pods use the same engine (vllm / sglang / trtllm)\n │\n ├─► filterPrefillDecodePods()\n │ ┌─────────────────────────────────────────────────────┐\n │ │ collectAndBucketPods() │\n │ │ Phase 1: group by roleset (prefill / decode) │\n │ │ Phase 2: apply prompt-length bucketing filter │\n │ └─────────────────────────────────────────────────────┘\n │ │\n │ ├─► [Prompt-length bucketing ON]\n │ │ → combined pod path (see Combined Pod section)\n │ │\n │ ├─► loadImbalanceSelectPrefillPod() (fast path)\n │ ├─► loadImbalanceSelectDecodePod() (fast path)\n │ ├─► scorePrefillPods()\n │ ├─► scoreDecodePods()\n │ └─► finalPDScore() → (prefillPod, decodePod)\n │\n ├─► AddPendingDecode() + defer RemovePendingDecode() (all paths)\n │\n ├─► [prefillPod != nil]\n │ AddPrefillRequest()\n │ doPrefillRequest(ctx, prefillPod, handler)\n │ ├─ SGLang → async goroutine (bootstrap handshake; Route does not wait for completion)\n │ ├─ vLLM → sync, extract kv_transfer_params from response\n │ └─ TRT-LLM → context_first: sync, merge response params\n │ generation_first: prepare both bodies, async prefill\n │\n └─► ctx.SetTargetPod(decodePod)\n return decodePod address\n```\n\n---\n\n## Pod Classification\n\nPods are grouped using Kubernetes labels:\n\n| Label | Values | Purpose |\n|-------|--------|---------|\n| `roleset-name` | any string | Groups prefill+decode pairs into a \"roleset\" |\n| `role-name` | `prefill` / `decode` / `encode` / other | Pod's role in PD disaggregation. `encode` pods are optional and only take the encode stage (see [vLLM encode stage (EPD)](#vllm-encode-stage-epd)) |\n| `stormservice.orchestration.aibrix.ai/pod-group-index` | `\"0\"` or absent | Only pods with index `\"0\"` (or no label) host the HTTP server (multi-node TP) |\n| `stormservice.orchestration.aibrix.ai/role-replica-index` | any string | Distinct replica index within a role; used by `roleReplicaCardinality` to count unique replicas across multi-node TP groups. When absent, each routable pod counts as its own replica. |\n| `model.aibrix.ai/engine` | `vllm` / `sglang` / `trtllm` | LLM engine type |\n\nOnly rolesets with **both** at least one prefill replica and one decode replica are eligible. Incomplete rolesets are excluded. See [Roleset Eligibility Utilities](#roleset-eligibility-utilities) for diagnostic functions.\n\n---\n\n## Roleset Eligibility Utilities\n\n`pd/router/roleset.go` provides helper functions used by both the router and the gateway's pod-readiness logic.\n\n### Replica cardinality\n\nA roleset's replica count is the number of **distinct** values of `stormservice.orchestration.aibrix.ai/role-replica-index`. When that label is absent on all pods of a role, each routable pod counts as its own replica. This correctly handles multi-node tensor-parallel groups where multiple pods share one logical replica slot.\n\n```\nroleReplicaCardinality(pods):\n if any pod has role-replica-index label:\n count distinct index values\n else:\n count routable pods\n```\n\n### Eligibility checks\n\nA roleset is **eligible** when it has ≥1 prefill replica **and** ≥1 decode replica.\n\n| Function | Use |\n|----------|-----|\n| `HasEligibleRoleset(readyPods)` | Gateway check: at least one roleset is fully ready for PD routing |\n| `eligiblePDRolesets(readyPods)` | Internal: returns only complete rolesets for scoring |\n| `DescribePDRolesetEligibility(readyPods)` | Diagnostics: per-roleset replica counts and eligibility flag |\n\n`ValidateAndGetLLMEngine` (called from the gateway before `Route`) checks that all ready pods for the model share the same `model.aibrix.ai/engine` label.\n\n---\n\n## Pod Selection Flow\n\n```\ncollectAndBucketPods()\n │\n ▼\n┌───────────────────────────────┐\n│ prefillPods / decodePods │\n│ (grouped by eligible │\n│ rolesets only) │\n└───────────────────────────────┘\n │\n ▼\nloadImbalanceSelectPrefillPod()\n ┌─────────────────────────────────────────────────────────┐\n │ max(outstanding_prefill_reqs) - min > MIN_SPREAD? │\n │ → YES: narrow to single least-loaded prefill pod │\n │ align decodePods to selected pod's roleset │\n │ (continues to decode fast path below) │\n │ → NO: continue with all prefill pods │\n └─────────────────────────────────────────────────────────┘\n │\n │ NOTE: the two fast paths are independent. Both can fire on the\n │ same request if both sides are imbalanced: prefill narrows first,\n │ then decode narrows within the already-aligned decode set.\n │\n ▼\nloadImbalanceSelectDecodePod() (3 ordered checks, metrics-bearing pods only)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Effective request count = RealtimeNumRequestsRunning │\n │ + PendingDecodeTracker count │\n │ Pending counts bridge the gap between pod selection and the │\n │ first metric update, preventing thundering-herd on a newly │\n │ selected pod whose running count has not yet ticked up. │\n │ │\n │ Check 1 – Request count spread │\n │ Only pods with RealtimeNumRequestsRunning metric participate │\n │ (pods without the metric are excluded from spread; their │\n │ pending count is still stored for scoring — prevents │\n │ thundering-herd on freshly restarted pods) │\n │ max(observed running+pending) - min >= DECODE_LOAD_SPREAD? │\n │ → YES: return least-loaded observed pod; │\n │ align prefillPods to selected pod's roleset │\n │ │\n │ Check 2 – Throughput spread │\n │ Only pods with AvgGenerationThroughputToksPerS participate │\n │ max(observed throughput) - min > THROUGHPUT_SPREAD? │\n │ → YES: return lowest-throughput observed pod │\n │ │\n │ Check 3 – Drain rate score (soft path) │\n │ all pods have drain_rate > 0? │\n │ score = effective_running_reqs / drain_rate │\n │ max_score / min_score > DECODE_SCORE_RATIO? │\n │ → YES: return pod with lowest drain-rate score │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nscorePrefillPods() (per roleset, best pod wins)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Skip pods with req_count > mean + N * stddev │\n │ (N = AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR) │\n │ Skip pods with req_count > max(c, 1) * (mean + 1) │\n │ (c = AIBRIX_PREFILL_LOAD_CAP_FACTOR) │\n │ When all pods in a roleset are filtered, that roleset has no │\n │ entry in the output map and finalPDScore cannot select it. │\n │ │\n │ Policy: prefix_cache (default) │\n │ score = (100 - match_pct) * 0.1 + req_cnt / max_req_cnt │\n │ │\n │ Policy: least_request │\n │ score = req_cnt │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nscoreDecodePods() → pd.DecodeScoreRun (per roleset best pod + MaxScore + policy metadata)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Cold-start (if some pods have metrics but this pod does not): │\n │ score = 1.0 + pending_decode_count │\n │ (neutral idle-warm score; pod competes but is not favored) │\n │ │\n │ Policy: load_balancing (default) │\n │ norm_reqs = running_reqs / max_running_reqs │\n │ norm_thru = 1 - throughput / max_throughput │\n │ norm_free_gpu = free_gpu_pct / max_free_gpu_pct │\n │ score = (w_run*norm_reqs + w_thru*norm_thru) / norm_free_gpu │\n │ (w_run, w_thru from AIBRIX_DECODE_LB_WEIGHT_*, default 1) │\n │ NaN scores: skip; non-NaN Inf possible if norm_free_gpu = 0 │\n │ │\n │ Policy: least_request │\n │ score = running_reqs (with pending) │\n │ │\n │ If primary policy yields NaN, one load_balancing retry per pod │\n │ lower is better │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nfinalPDScore()\n ┌──────────────────────────────────────────────────────────────────┐\n │ For each roleset with both prefill and decode scores: │\n │ final = norm_prefill_score + norm_decode_score │\n │ Pick roleset with minimum final score │\n │ → selectedPrefillPod, selectedDecodePod │\n │ │\n │ Errors: │\n │ • prefillScores nil → Prepare failed in scorer │\n │ • prefillScores empty → all prefill pods filtered by │\n │ stddev or cap (no candidate) │\n │ • no roleset intersection → load-imbalance alignment left │\n │ prefill/decode on different sets │\n └──────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Prefill Score Policies\n\nControlled by `AIBRIX_PREFILL_SCORE_POLICY`, or overridden per request via the model **`routingConfig`** in `model.aibrix.ai/config` (see [Config profile overrides](#config-profile-overrides-for-pd-score-policies)).\n\n### `prefix_cache` (default)\n\nUses the shared `PrefixHashTable` to find how much of the request's token prefixes are already cached on each prefill pod.\n\n```\ntokens = tokenize(request.message)\nmatchedPods, hashes = prefixCacheIndexer.MatchPrefix(tokens, model, readyPodsMap)\n\nscore = (100 - match_percent) * 0.1 + req_count / max_req_count\n```\n\n- Lower score = better (more cache hits, fewer outstanding requests).\n- After routing, the prefix hashes are enqueued to `prefixUpdateCh` for asynchronous indexer update.\n- Tokenizer is selected by `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` (same as the prefix-cache router): `tiktoken` or `character` (default).\n\n### `least_request`\n\nNo prefix cache consultation. Score equals the raw running request count on the pod.\n\n```\nscore = req_count\n```\n\n---\n\n## Decode Score Policies\n\nImplementation lives in `algorithms/pd/decode_scorer.go` (`package pd`). Policies are selected by **`pd.ResolveDecodePolicy`** from `AIBRIX_DECODE_SCORE_POLICY`. Extra names can be registered at init time with **`pd.RegisterDecodePolicy`**. Per-pass results use **`pd.DecodeScoreRun`** (map of roleset → best pod/score, `MaxScore` for normalization, optional `Err`, `FallbackUsed`, `Policy`).\n\nMetrics are passed per pod via **`pd.DecodePodInput`** (running/throughput/free GPU and batch maxima). The router guarantees positive maxima from `loadImbalanceSelectDecodePod` as today.\n\nControlled by `AIBRIX_DECODE_SCORE_POLICY`, or overridden per request via **`routingConfig`** in the model config profile (see [Config profile overrides](#config-profile-overrides-for-pd-score-policies)).\n\nMetrics are pulled from the cache per pod (same maps as `loadImbalanceSelectDecodePod`):\n- `RealtimeNumRequestsRunning` + `PendingDecodeTracker` count\n- `AvgGenerationThroughputToksPerS` (per model)\n- `KVCacheUsagePerc` (free = 100 − usage%)\n\n**Cold-start gate**: if at least one pod has `RealtimeNumRequestsRunning`, pods that lack it are given a neutral **cold-start score** (`1.0 + pending_decode_count`) rather than full policy scoring. When *no* pod has that metric yet, all pods are scored normally.\n\n### `load_balancing` (default)\n\nBalances normalized queue depth, inverse throughput, and free GPU headroom:\n\n```\nnorm_running_reqs = running_reqs_with_pending / max_running_reqs\nnorm_throughput = 1 - avg_gen_throughput_tok_s / max_throughput\nnorm_free_gpu = free_gpu_percent / max_free_gpu_percent\n\ndecode_score = (w_run * norm_running_reqs + w_thru * norm_throughput) / norm_free_gpu\n```\n\n`w_run` and `w_thru` default to `1` (historical equal weighting). Lower score is better (fewer outstanding requests, higher throughput, more free GPU).\n\n### `least_request`\n\nNo throughput or GPU terms. Score equals raw running decode request count (including pending decode), same units as `RealtimeNumRequestsRunning` + pending.\n\n```\ndecode_score = running_reqs_with_pending\n```\n\n### `token_load`\n\nScores from the gateway's decode ledger in `pd.TokenLoadTracker` rather than from request counts or scraped KV usage. Each request is charged its `prompt_tokens` to the selected decode pod, under `selectMu` together with the other selection bookkeeping, and released on request completion or prefill failure (`releaseTokenLoad`, or the prefill executor for a terminal async failure). The whole prompt is charged because the decode pod receives all of its KV; there is no fixed per-request cost (`AIBRIX_TOKEN_LOAD_REQUEST_COST` is prefill-only).\n\n```\ndecode_score = decode_tokens + generated_tokens\ngenerated_tokens = rate × Σ over outstanding charges (now − routed_at)\nrate = AvgGenerationThroughputToksPerS / running requests (mean of the other pods if unknown)\n```\n\nWhen the prefill response carries `usage.prompt_tokens`, the prefill executor replaces the decode charge with it (`TokenLoadTracker.CorrectDecodeCost`), since the body-size estimate is far off for requests with media. `generated_tokens` estimates the output the pod's outstanding requests have produced so far (it sits in the decode KV too); `pd.TokenLoadTracker.DecodeGrowth` computes it in O(1) per pod from the count and the summed charge times of the outstanding charges. `AIBRIX_DECODE_TOKEN_LOAD_OUTPUT_GROWTH=false` scores prompt tokens only. The cold-start score does not apply to `token_load`: a pod without metrics is scored from the ledger like any other. The decode load-imbalance fast path is skipped under `token_load` (it still runs to fill the metric maps): it picks by request count, throughput or drain rate, and the pod holding one long prompt has the fewest requests. With Redis configured, the decode ledger is shared across gateway replicas (`AIBRIX_TOKEN_LOAD_SHARED_LEDGER`, default on): each replica publishes its per-pod charges, count and summed charge times under its instance ID (`cache.Store.PublishDecodeLedger`), and `scoreDecodePods` adds the other live replicas' values, read with the running-request counts (`GetPodsRunningRequestsAndDecodeLedger`). Without Redis the ledger is local to each replica. Each decode pod's `token_load` score is then multiplied by the candidates' mean KV capacity over its own (`decodeKVCapacities`, `AIBRIX_DECODE_TOKEN_LOAD_CAPACITY_AWARE`, default on), with capacities from the scraped `kv_cache_num_gpu_blocks` × `kv_cache_block_size` (vLLM `vllm:cache_config_info` labels) or `kv_cache_capacity_tokens` (SGLang); unknown capacities count as the mean.\n\n### Config profile overrides for PD score policies\n\nWhen the gateway resolves a model config profile (`routingCtx.ConfigProfile` from `model.aibrix.ai/config` and the `config-profile` header), the **`routingConfig`** JSON may include:\n\n| Field | Values | Effect |\n|-------|--------|--------|\n| `prefillScorePolicy` | `prefix_cache`, `least_request` | Overrides `AIBRIX_PREFILL_SCORE_POLICY` for that request |\n| `decodeScorePolicy` | `load_balancing`, `least_request`, `conductor`, `token_load` (or any name registered via `pd.RegisterDecodePolicy`) | Overrides `AIBRIX_DECODE_SCORE_POLICY` for that request |\n\nExample fragment inside a profile:\n\n```json\n\"routingConfig\": {\n \"promptLenBucketMinLength\": 0,\n \"promptLenBucketMaxLength\": 8192,\n \"prefillScorePolicy\": \"prefix_cache\",\n \"decodeScorePolicy\": \"least_request\"\n}\n```\n\nOmitted fields keep the gateway defaults from environment variables.\n\n---\n\n\n## Engine-Specific Prefill Behaviour\n\n### vLLM encode stage (EPD)\n\nWhen the selected prefill pod's roleset has `role-name: encode` pods and a vLLM `/v1/chat/completions` request carries media items (`image_url`, `audio_url`, `input_audio`, `video_url`), `Route` runs an encode stage before the prefill call:\n\n```\nGateway ──POST /v1/chat/completions {messages:[{role:user, content:[item i]}]}──► Encode Pod (one per item, in parallel)\n ◄── 200 ──────────────────────────────────────────────────────────────── Encode Pod (output stored by the EC connector)\n\nGateway ──prefill as below, request body unchanged──────────────────────────────► Prefill Pod (finds the output by the item's hash)\n```\n\nItems are spread round-robin over the roleset's encode pods. The body is not rewritten: the encoder and the prefill pod hash the same item bytes, so the prefill pod's EC connector finds the output, and encodes the item itself when it can't. A failed encode is therefore logged and counted, not returned. Each call uses the prefill request timeout. Connectors that need transfer handles passed through the request (vLLM's `*_embeds` rewrite in `disagg_epd_proxy.py`) are not supported.\n\nWhen bucketing routes the request to a combined pod instead, `Route` runs the same stage on the combined pod's roleset. For E+PD rolesets, where encode pods sit in front of pods that serve the whole request, the gateway runs it outside the PD router: with any other routing strategy, `selectTargetPod` drops encode pods from the candidates and encodes on the chosen pod's roleset before returning it.\n\n### vLLM (`AIBRIX_KV_CONNECTOR_TYPE=shfs`, default)\n\n```\nGateway ──POST /v1/... (max_tokens=1, kv_transfer_params={do_remote_decode:true})──► Prefill Pod\n ◄── response {kv_transfer_params: {remote_block_ids, ...}} ──────────────── Prefill Pod\n\nGateway updates ReqBody:\n kv_transfer_params.remote_host = prefill_pod_ip\n\nGateway ──POST /v1/... (with kv_transfer_params)──────────────────────────────────► Decode Pod\n```\n\n### vLLM (`AIBRIX_KV_CONNECTOR_TYPE=nixl`, Neuron)\n\n```\nGateway ──POST /v1/... (max_tokens=1)──────────────────────────────────────────────► Prefill Pod\n ◄── response { entire prefill response } ──────────────────────────────── Prefill Pod\n\nGateway updates ReqBody:\n disagg_prefill_resp = \n\nGateway ──POST /v1/... (with disagg_prefill_resp)──────────────────────────────────► Decode Pod\n```\n\n### SGLang (async)\n\n```\nGateway modifies ReqBody:\n bootstrap_host = prefill_pod_ip\n bootstrap_port = pod annotation / default 8998\n bootstrap_room = random int63\n rid = -<16 hex nonce> (identical on both legs)\n\nGateway ──POST (async goroutine)──────────────────────────────────────────────────► Prefill Pod\n (no wait; prefill/decode coordinate via bootstrap protocol)\n\nGateway ──POST /v1/... (updated body)─────────────────────────────────────────────► Decode Pod\n```\n\nFor SGLang, `Route()` returning a decode pod means the gateway has dispatched\nthe asynchronous prefill worker, not that the prefill HTTP request has already\ncompleted. Prefill success/failure metrics are emitted by the async worker after\nthe prefill HTTP request returns.\n\n`rid` is the engine-visible request id of the attempt. SGLang reads it from the\nrequest body (the `X-Request-Id` header is ignored) and its `/abort_request`\nendpoint matches live requests by rid alone, so injecting the same gateway-owned\nrid into both legs is what makes the decode leg cancellable. The rid always ends\nwith a fixed-width nonce so that no gateway rid can be a proper prefix of\nanother - SGLang aborts by rid prefix - and so that a client retrying with the\nsame request id (or the same traceparent) still gets a distinct rid per attempt.\nSee [PD Prefill Fail-Fast](#pd-prefill-fail-fast).\n\n### TensorRT-LLM\n\n`AIBRIX_TRT_SCHEDULE_STYLE=context_first` (default) preserves the sequential flow:\n\n```\nGateway adds disaggregated_params to prefill request:\n { request_type: \"context_only\", disagg_request_id: }\n\nGateway ──POST (sync)──────────────────────────────────────────────────────────────► Prefill Pod\n ◄── response { disaggregated_params: {...}, prompt_token_ids: [...] } ── Prefill Pod\n\nGateway updates ReqBody:\n disaggregated_params.request_type = \"generation_only\"\n prompt / prompt_token_ids from prefill response\n\nGateway ──POST /v1/... (with disaggregated_params)─────────────────────────────────► Decode Pod\n```\n\nTRT-LLM uses a Snowflake-style `disagg_request_id` (63-bit) to correlate prefill and decode:\n\n```\n[41-bit timestamp ms since 2023-01-01] [10-bit machine_id] [12-bit counter]\n```\n\nMachine ID is set via `AIBRIX_TRT_MACHINE_ID` (must be in `[0, 1024)`). Assign different machine IDs to gateway processes sharing the same TRT workers.\n\n#### Generation-first (parallel)\n\nSet `AIBRIX_TRT_SCHEDULE_STYLE=generation_first` **on the gateway plugin**, not on\nCTX/GEN workers. The setting is fixed when the PD router is constructed; an\nunknown value is logged and leaves the router on `context_first`. It does not\naffect other engines or combined pods. No automatic fallback or retry occurs after\neither leg has been dispatched.\n\nPrerequisites (TRT-LLM `1.3.0rc8`, the Python KV-cache transceiver on both roles,\nmatching model/tokenizer/chat-template) are listed in\n`docs/source/features/gateway-routing/pd-disaggregation.rst`.\n\n```\nGateway selects CTX + GEN\n ├─ GET CTX /server_info (cached)\n ├─ prepare both bodies with the same integer disagg_request_id\n ├─ goroutine: POST context_only, schedule_style=1, stream=false, max_tokens=1\n └─ return GEN address and generation_only body to Envoy immediately\n disagg_request_id = ctx_request_id = shared ID\n ctx_info_endpoint, ctx_dp_rank, optional encoded_opaque_state = CTX metadata\n schedule_style = 1 (TRT-LLM GENERATION_FIRST wire enum)\n original prompt/messages, sampling settings and stream mode preserved\n\nCTX <── engine-managed coordination / KV transfer ──> GEN\nEnvoy <──────────────────── response / SSE ────────── GEN\n```\n\nThere is no second Go-issued decode POST. GEN can initialize while CTX is still\nworking; it cannot generate without the required KV. The CTX response is not merged\nback into the already-dispatched GEN request. Both endpoints must therefore use\nmatching model/tokenizer/chat-template configuration, since GEN cannot reuse\n`prompt_token_ids` from the CTX HTTP response in this mode.\n\n**Worker discovery:** each router loads the selected CTX worker's `/server_info`\nlazily before either leg is dispatched, so newly discovered or autoscaled pods work\nwithout a gateway restart. Concurrent misses for one incarnation are coalesced;\nlookups have a 3-second timeout and a 1-minute TTL, and responses are capped at\n1 MiB. Pod UID, address and restart count identify an incarnation. A missing or\ninvalid `/server_info` fails routing before dispatch, and failed lookups are not\ncached.\n\nThe expected HTTP response follows TRT-LLM's `1.3.0rc8` OpenAI server schema:\n\n```json\n{\"disaggregated_params\":{\"ctx_info_endpoint\":\"tcp://:\",\"ctx_dp_rank\":0}}\n```\n\n`ctx_info_endpoint` must be a single string, which is what the Python transceiver\nreturns; any other JSON shape fails the lookup.\n\n`encoded_opaque_state` is also propagated when present. Only these fields are copied\nfrom `/server_info`: other keys are ignored so worker metadata can never overwrite\nthe gateway-owned `request_type`, IDs or `schedule_style`. A field the handshake\nrequires but this allowlist does not carry must be added to the handler explicitly.\nThe endpoint must be reachable from GEN. `ctx_dp_rank` must be explicitly present and nonnegative; it is **attention\nDP rank**, not TP rank, replica index, or Pod number. Rank zero is valid but is never\nsubstituted for missing data. Metadata is taken from the exact selected Pod. A\nfront-end that distributes requests across multiple DP ranks without stable\nworker/rank affinity is not supported by this Pod-level cache: expose rank-affine\nworkers or retain `context_first`. Validate nonzero-rank deployments against the\nactual engine before enabling this mode.\n\n**Failure handling:** the CTX HTTP call retains client cancellation and the existing\nprefill timeout. Failure wakes the gateway through the per-incarnation PD leg state,\nnever through a recycled RoutingContext. TRT-LLM has no SGLang `rid`/`/abort_request`\ncontract; the gateway instead fails/resets the Envoy stream, relying on TRT-LLM's\nHTTP-disconnect cancellation. Keep Envoy `failure_mode_allow=false`. TRT's SSE headers\ncan precede KV arrival, so a terminal CTX failure after response headers resets the\nstream rather than being ignored. This can also truncate an already-generating\nresponse; it never attempts to rewrite headers already sent to the client.\n\nStart validation with text-only 1P1D using the sample under\n`samples/quickstart/tensorrt/`; its README carries the smoke test and the rollback\ncommand.\n\n---\n\n## PD Prefill Fail-Fast\n\nOnly asynchronous modes (SGLang and TRT generation-first) can fail their prefill leg after `Route()`\nhas already returned a decode pod. Without fail-fast the client waits for the\ndecode leg to give up on a KV transfer that will never arrive - 300s in SGLang -\nand the decode pod holds its pre-allocated KV pages for that whole window.\n\n```\nPrefill leg fails (async worker)\n │\n ├─► record failure on the RoutingContext's PD leg state (first failure wins)\n │\n ├─► SGLang: POST /abort_request {\"rid\": ...} to decode (goroutine, twice)\n │ TRT: no abort POST; reset Envoy upstream on failure\n │\n └─► wake the ext_proc stream ──► 5xx ImmediateResponse to the client\n header x-error-pd-prefill: true\n```\n\nFailure classes, as reported in the metrics and in the client error message:\n\n| Class | Meaning | Terminal |\n|-------|---------|----------|\n| `request_setup` | The prefill HTTP request could not be built or the body could not be prepared | yes |\n| `timeout` | `AIBRIX_PREFILL_REQUEST_TIMEOUT` elapsed | yes |\n| `canceled` | The client or the gateway canceled the prefill request | yes |\n| `transport` | Connection refused, reset, DNS failure, ... | yes |\n| `http_status` | The prefill engine answered a non-2xx status | yes |\n| `bad_response` | The prefill engine answered 2xx with a body the gateway could not parse | engine-dependent |\n\n`bad_response` is terminal only for engines whose KV transfer is out of band\n(TRT generation-first): there the HTTP body is not the KV handshake, so an\nunparseable 2xx leaves the gateway unable to tell whether context processing\nsucceeded. For SGLang, whose handshake completes in the body, it is not terminal:\nthe prefill leg did run, so the decode leg is left alone and the client keeps\nwhatever the decode leg produces.\n\nThe client status code is the prefill engine's own status for `http_status`, and\n`503` for every other terminal class, since those have no upstream status. The\nresponse body is an OpenAI-shaped error carrying the class and a truncated\nupstream message, and it is sent even while the gateway is still waiting for the\ndecode leg's response headers. For SGLang, once the decode leg has started answering the client, the failure is\nonly recorded and logged. TRT generation-first instead resets the stream even\nafter headers, because TRT can send SSE headers before KV is available; it does not\nattempt to replace the already-started response.\n\n### PD Decode Watchdog\n\nFail-fast covers a prefill leg that dies. The opposite case is a prefill leg\nthat *succeeds* and a decode pod that then never answers: in SGLang the prefill\ncall only returns once the decode pod has taken the KV transfer, so from that\npoint the decode pod owes the client a response. If its scheduler dies or wedges,\nthe client would otherwise hang until Envoy's route timeout.\n\n```\nPrefill leg succeeds (async worker)\n │\n └─► mark the PD leg state (MarkPrefillSucceeded) ──► wake the ext_proc stream\n │\n arm a timer: FIRST_RESPONSE (stream) / RESPONSE (non-stream)\n │\n response headers ──► no change (sent before the first token)\n first body chunk ──► switch to STREAM_IDLE │ (streaming only)\n every later chunk ──► re-arm STREAM_IDLE │\n ▼ timer fires\n before any headers: 504 ImmediateResponse, header x-error-pd-decode: true\n + gRPC DeadlineExceeded close\n after the headers: gRPC DeadlineExceeded close only (Envoy resets\n the response the client already started reading)\n always: POST /abort_request {\"rid\": ...} to decode (goroutine, once)\n```\n\nThe watchdog runs only for requests with a gateway-owned `rid` (SGLang). Its\ntimeouts are `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` (streaming, default `60`) and\n`AIBRIX_DECODE_RESPONSE_TIMEOUT` (non-streaming, default `0`, i.e. off, since the\ndecode pod only answers a non-streaming request once the whole generation is\ndone). Both run until the first response body chunk: SGLang's streaming\nendpoint sends its headers before the first token, so the headers prove nothing.\nFrom the first chunk of a streaming response on,\n`AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` (default `120`) bounds the gap between two\nchunks.\n\n---\n\n## Request Trackers\n\n### PrefillRequestTracker\n\nTracks active **prefill** request counts per pod using `sync.Map` and `atomic.Int32`. Used by both prefill load-imbalance detection and `scorePrefillPods` (mean/stddev filter and load cap).\n\nBoth request trackers are shared by every model the router serves, so pods are identified by their pod key, `namespace/name` (`utils.GeneratePodKey(pod.Namespace, pod.Name)`), not by bare pod name.\n\n```\nAddPrefillRequest(requestID, podKey) // in Route(), after PD pod selection, before prefill HTTP\nRemovePrefillRequest(requestID) // on prefill end (executor defer) or Route error\nGetPrefillRequestCountsForPods(pods) // for scoring\n```\n\n### PendingDecodeTracker\n\nBridges the gap between **decode pod selection** and the actual decode request starting. Without this, concurrent requests could all route to the same decode pod (since `RealtimeNumRequestsRunning` hasn't updated yet).\n\n```\nAddPendingDecode(requestID, podKey) // in Route(), immediately after pod selection\nRemovePendingDecode(requestID) // deferred in Route()\nGetPendingDecodeCount(podKey) // added to running reqs in decode scoring\n```\n\nTimeline:\n```\nRoute() called\n │\n ├─ filterPrefillDecodePods() → (prefillPod, decodePod)\n │ └─ [combined path] returns prefillPod=nil\n │\n ├─ AddPendingDecode(requestID, decodePodKey)\n ├─ defer RemovePendingDecode(requestID)\n │\n ├─ [PD path] AddPrefillRequest(requestID, prefillPodKey)\n ├─ [PD path] doPrefillRequest() (may take seconds)\n │ └─ on error: RemovePrefillRequest(requestID) called immediately\n ├─ ctx.SetTargetPod(decodePod)\n └─ return address to caller\n```\n\n---\n\n## Prompt Length Bucketing\n\nEnabled by `AIBRIX_PROMPT_LENGTH_BUCKETING=true`.\n\nEach pod can declare a prompt-length range via its config profile (`promptLenBucketMinLength` / `promptLenBucketMaxLength`). A pod is eligible only when the request's prompt length falls within its range.\n\n```\ncollectAndBucketPods()\n ├─ Combined pass: collect combined-role pods (role-name is not prefill/decode)\n │ with combined=true and a prompt-length match\n ├─ Eligible rolesets: group by roleset-name (prefill + decode both present)\n └─ Bucket pass: per roleset, keep pods whose bucket covers promptLength\n only when BOTH prefill and decode sides match (half-pairs excluded)\n\nIf both bucket-filtered lists are non-empty, they replace the unfiltered prefill/decode lists atomically so the sides stay roleset-aligned.\n```\n\n### No-match guard and combined fallback\n\nWhen `AIBRIX_PROMPT_LENGTH_BUCKETING=true`, a roleset enters the bucket-filtered lists only when **both** its prefill and decode pods match the request's prompt length. A roleset with only one matching side (misconfigured bucket or a missing decode pod) is excluded from the bucket lists.\n\nIf no complete bucket-matched prefill/decode pair exists for the request — including when the prompt length is outside every declared range, or when the matching roleset is incomplete (e.g. decode pod is down) — the router falls back to a combined pod when one is available. Otherwise it returns an error rather than cross-routing to a different bucket's PD pair:\n\n```\nif AIBRIX_PROMPT_LENGTH_BUCKETING &&\n (bucketPrefills == 0 || bucketDecodes == 0):\n if combined pods available for this prompt length → route to combined (random pick)\n else → error: \"no prompt-length bucket matches prompt length N and no combined pods available\"\n```\n\nPods without a `model.aibrix.ai/config` annotation are treated as suitable for any prompt length. With bucketing enabled globally, models that do not declare explicit bucket ranges still participate using those default bounds.\n\n### Combined Pods\n\nWhen bucketing is enabled, pods with `combined=true` in their config profile can act as both prefill and decode. Combined pods use a `role-name` other than `prefill` or `decode` (for example `all`). The router falls back to a combined pod when:\n\n1. No complete bucket-matched prefill/decode pair exists for the request's prompt length (prompt outside all buckets, incomplete roleset, or decode pod unavailable), **or**\n2. `shouldPickCombined()` returns `true`:\n - At least one combined pod has queue-drain score < `0.25` (low load), **and**\n - At least one prefill pod **or** decode pod has queue-drain score > `1.0` (high load).\n\nQueue-drain score is `(waiting_reqs + prealloc_queues) / drain_rate_1m` (see `calculatePodScoreBasedOffRequestRate`).\n\n```\nshouldPickCombined():\n combinedLowLoad = any combined pod score < 0.25\n prefillHighLoad = any prefill pod score > 1.0\n decodeHighLoad = any decode pod score > 1.0 (skipped if prefillHighLoad)\n return (prefillHighLoad OR decodeHighLoad) AND combinedLowLoad\n```\n\nWhen a combined pod is selected for load imbalance, `scoreCombinedPods()` picks the lowest-score combined pod. The no-match path picks a combined pod at random.\n\nWhen a combined pod is selected, `prefillPod` is `nil` (no prefill HTTP call) and `decodePod` is the selected combined pod.\n\n### Adaptive Bucket Serving\n\nEnabled by `AIBRIX_BUCKET_SERVE=true`, which requires `AIBRIX_PROMPT_LENGTH_BUCKETING=true` as well: the plan only re-orders the rolesets that bucketing already filtered to.\n\nPlain bucketing keeps every roleset whose range covers the request as a candidate, and the scoring decides among them. Adaptive bucket serving narrows that choice. The gateway keeps a per-model picture of the prompt lengths it routes, charges every roleset the traffic only it can serve, and splits each range that several rolesets still need into one band per roleset, drawing each band against one replica-proportional share so a roleset carrying several overlaps does not draw a fresh share from each. The cut points sit at quantiles of the observed traffic. The picture is an EWMA with a 30s half-life, a model's plan is recomputed at most once every 5s, and one plan holds at most 16 affinity bands. A shared range needs a meaningful share of the model's traffic before it is split at all, and a range that cannot afford one band per needing roleset splits into as many bands as the budget left allows; an interval that is not split keeps every covering roleset a candidate.\n\n```\nbucketServeBand() (before selectMu, one call per request)\n ├─ observe promptLength for the model\n ├─ groups: one per eligible roleset, with the range its profile declares and its prefill replica count\n ├─ plan: shared ranges split at mode quantiles in replica proportion, light ones kept whole\n └─ band for promptLength → banded roleset and band upper bound\n\nfilterPrefillDecodePods()\n ├─ steps 1-5: bucket filtering and the load-imbalance fast paths\n ├─ banded roleset still has prefill and decode candidates\n │ → narrow both sides to that roleset (step 6)\n └─ otherwise keep the load fast paths' choice\n```\n\n`AIBRIX_BUCKET_SERVE_MODE` picks the unit the cut points are measured in: `throughput` (default) balances prompt-token mass, while `rps` balances request counts. Within a shared range the bands divide the observed traffic in proportion to the replica share each roleset still needs, counted in that unit. Both are per-model: a profile that selects another mode re-plans only the models it routes.\n\nThe plan is advisory. It never widens a choice, only prefers one of the rolesets that already cover the request's length. Wherever the plan holds no band, the scoring considers every covering roleset, and when the banded roleset has no candidate left after the load fast paths, the request routes exactly as it would without the plan. The steady state does narrow the choice, which is the feature, and a stale or wrong plan costs balance, not correctness. With the feature off the planner records and publishes nothing.\n\nMetrics: `pd_bucket_serve_band_total` and `pd_bucket_serve_prompt_tokens_total` count the requests and prompt tokens each banded roleset carried (label `roleset`), so a counter joins the pods that received the band, and the `pd_bucket_serve_band_max` gauge holds the highest upper prompt-length bound among the bands a roleset currently holds (a roleset can hold two bands when its range overlaps two others), deleted when no live plan holds the roleset.\n\n---\n\n## Environment Variables\n\n### Core Behaviour\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFILL_SCORE_POLICY` | `prefix_cache` | Prefill pod scoring: `prefix_cache` or `least_request`. Any other value logs a warning and falls back to `prefix_cache`. |\n| `AIBRIX_DECODE_SCORE_POLICY` | `load_balancing` | Decode pod scoring for `finalPDScore`: `load_balancing`, `least_request`, `conductor` or `token_load`. Any other value logs a warning and falls back to `load_balancing`. |\n| `AIBRIX_KV_CONNECTOR_TYPE` | `shfs` | KV transfer backend: `shfs` (GPU/SHFS), `nixl` (Neuron/NIXL), or `mooncake` (Mooncake) |\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | `30` | Prefill HTTP request timeout in seconds. Exceeding it is a terminal `timeout` prefill failure. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | `false` | Enable prompt-length-based pod bucketing |\n| `AIBRIX_BUCKET_SERVE` | `false` | Enable adaptive bucket serving on top of prompt-length bucketing |\n| `AIBRIX_BUCKET_SERVE_MODE` | `throughput` | The unit the adaptive cut points are measured in: `throughput` (prompt token mass) or `rps` (request counts) |\n\n### Prefill Load Balancing\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | `16` | Min `(max − min)` outstanding prefill request count to trigger imbalance routing |\n| `AIBRIX_PREFILL_LOAD_CAP_FACTOR` | `1.5` | Factor `c`: prefill pods with `req_count > max(c, 1) * (mean + 1)` are skipped from per-roleset selection. Unlike the stddev skip, it fires in pools of five or fewer pods. Environment-only |\n\n### Decode Load Balancing\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | `1.0` | Multiplier on normalized running-request term in `load_balancing` numerator |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | `1.0` | Multiplier on normalized inverse-throughput term in `load_balancing` numerator |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | `16.0` | Min `(max − min)` running request count spread to trigger decode imbalance routing |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | `2048.0` | Min `(max − min)` token throughput spread (tok/s) to trigger throughput-based routing |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | `1.5` | `max_drain_score / min_drain_score` ratio threshold to trigger drain-rate routing |\n\n### PD Prefill Fail-Fast\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | `3` | Per-attempt timeout in seconds of the `/abort_request` call to the decode pod. `0` disables decode aborts; the client is still failed fast |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | `2` | Delay in seconds before the second abort attempt. `0` sends a single attempt |\n\n### PD Decode Watchdog\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | `60` | Seconds a streaming SGLang request waits, after its prefill leg succeeded, for the first response body chunk (first token) from the decode pod. `0` disables |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | `0` | Same for a non-streaming request, where the first body chunk is the finished answer. `0` (default) disables |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | `120` | Longest gap allowed between two body chunks of a streaming response once the first one arrived. `0` disables |\n\n### TensorRT-LLM\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_TRT_MACHINE_ID` | `0` | 10-bit machine ID used in Snowflake disagg request ID generation (range: `[0, 1024)`); distinct per gateway process sharing TRT workers |\n| `AIBRIX_TRT_SCHEDULE_STYLE` | `context_first` | TRT-only dispatch mode: `context_first` (sequential) or `generation_first` (parallel). Read at router initialization; unknown values are logged and fall back to `context_first` |\n\n### Inherited from Prefix Cache Router\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | `character` | Tokenizer for prefix-cache prefill scoring: `tiktoken` or `character` |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `1` | Multiplier N: prefill pods with `req_count > mean + N*stddev` are skipped from per-roleset selection |\n\n---\n\n## SGLang Bootstrap Port\n\nSGLang uses a bootstrap mechanism for prefill/decode coordination. The port is resolved in order:\n\n1. Pod annotation `model.aibrix.ai/sglang-bootstrap-port`\n2. Default: `8998`\n\n---\n\n## Metrics Emitted\n\n| Metric | When |\n|--------|------|\n| `GatewayPrefillRequestFailTotal` | Engine validation fail, pod filter fail, prefill HTTP error |\n| `GatewayPrefillRequestSuccessTotal` | Prefill HTTP succeeded. For SGLang and TRT generation-first, emitted asynchronously after the background prefill request completes, not when `Route()` returns. |\n| `gateway_pd_encode_request_total{status,status_code}` | One media item sent to an encode pod. `status` is `success`, or the failure class (`http_error`, `deadline_exceeded`, ...) |\n| `PDSelectedPrefillPodTotal` | Prefill pod selected (per pod label) |\n| `PDSelectedDecodePodTotal` | Decode pod selected (per pod label) |\n| `gateway_pd_prefill_failure_total{class,stage}` | A terminal prefill failure reached the client-facing handler. `stage` is `before_response` (the client was failed fast) or `after_response` (the decode leg had already started answering) |\n| `gateway_pd_decode_abort_total{prefill_failure_class,result}` | One decode abort attempt. `result` is `ok`, `error`, `skipped_streaming`, `skipped_disabled`, `skipped_no_rid` or `skipped_no_target`. `prefill_failure_class` is `watchdog_first_response` or `watchdog_stream_idle` for an abort sent by the decode watchdog |\n| `gateway_pd_decode_watchdog_total{phase}` | The decode watchdog failed a request whose decode pod stopped responding. `phase` is `first_response` or `stream_idle` |\n\n---\n\n## Response Headers Set\n\n| Header | Value |\n|--------|-------|\n| `prefill-target-pod` | Name of the selected prefill pod |\n| `prefill-target-pod-ip` | IP of the selected prefill pod |\n| `x-error-pd-prefill` | `true`, on the error response the gateway generates when the PD prefill leg failed |\n| `x-error-pd-decode` | `true`, on the error response the gateway generates when the decode watchdog fired |\n\n---\n\n## Prefix Cache Update Pipeline\n\nAfter a prefill pod is selected, its prefix hashes are enqueued asynchronously to avoid blocking the routing path:\n\n```\nfinalPDScore()\n └─ enqueuePrefixUpdate(hashes, model, pod)\n │\n ▼ (non-blocking send; dropped if channel full)\n prefixUpdateCh (buffered, capacity 1024)\n │\n ▼ (single background goroutine)\n prefixCacheIndexer.AddPrefix(hashes, model, pod)\n```\n\n---\n\n## Initialization\n\n```go\nNewPDRouter()\n │\n ├─ cache.Get() // shared metrics cache\n ├─ prefixcacheindexer.GetSharedPrefixHashTable()\n ├─ prefillScorePolicy from AIBRIX_PREFILL_SCORE_POLICY:\n │ least_request → LeastRequestPrefillPolicy\n │ prefix_cache → PrefixCachePrefillPolicy (shared tokenizer + PrefixHashTable)\n │ (other) → log warning, same as prefix_cache\n ├─ decode score policy (package algorithms/pd, decode_scorer.go) from AIBRIX_DECODE_SCORE_POLICY:\n │ least_request → pd.LeastRequestDecodePolicy\n │ load_balancing → pd.LoadBalancingDecodePolicy\n │ (other) → log warning, same as load_balancing\n ├─ create HTTP client with connection pool\n │ MaxIdleConns=100, MaxIdleConnsPerHost=10, IdleConnTimeout=90s\n ├─ TRT handler from AIBRIX_TRT_SCHEDULE_STYLE (immutable, router-scoped)\n │ owns a lazily populated /server_info cache for generation_first\n ├─ NewPrefillRequestTracker()\n ├─ NewPendingDecodeTracker()\n └─ startPrefixUpdater() // background goroutine\n\n// pd/router/roleset.go (no per-router state; pure functions called on each request)\nHasEligibleRoleset(readyPods) // gateway: skip PD when no complete roleset\neligiblePDRolesets(readyPods) // internal: filter to complete rolesets only\nDescribePDRolesetEligibility(readyPods) // diagnostics: per-roleset replica counts\n```\n","pkg/plugins/gateway/ENV_VARS.md":"# Gateway Plugin Environment Variables\n\nThis document covers all environment variables used in the `pkg/plugins/gateway` package and its sub-packages.\n\nVariables of type `duration` are parsed with Go's [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration), so they need a unit: `30s`, `5m`, `1h30m`. A bare number such as `30`, or a zero or negative duration such as `0s` or `-1s`, is rejected with a warning and the default is used instead.\n\n---\n\n## General Gateway\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_DISABLE_RATE_LIMITING` | bool | `false` | Disable AIBrix user RPM/TPM and model RPS quota enforcement. Redis and `requestsInflight` remain active. | [cmd/plugins/main.go](../../../cmd/plugins/main.go), [gateway.go](gateway.go) |\n| `POD_NAME` | string | `\"\"` | Kubernetes pod name. Used for logging and metric label tagging. | [gateway.go](gateway.go), [util.go](util.go) |\n| `ROUTING_ALGORITHM` | string | _(none)_ | Default routing algorithm when no per-request override is set. | [types.go](types.go), [util.go](util.go) |\n| `AIBRIX_PRIORITY_TIER_ENABLED` | bool | `false` | Forward the priority tier declared by the `x-aibrix-priority-tier` request header as the upstream vLLM request priority. | [cmd/plugins/main.go](../../../cmd/plugins/main.go), [gateway_req_priority.go](gateway_req_priority.go) |\n\nWhen `AIBRIX_PRIORITY_TIER_ENABLED=true`, the gateway maps the `x-aibrix-priority-tier`\nrequest header to the `priority` field of the backend request body: `batch` becomes `100` and\n`background` becomes `1000`. The engine serves smaller values first, so the mapping only\nde-prioritizes a request and never pulls one ahead of another; the values sit below the engine\ndefault of `0`. Requests without the header, or with a tier outside the table, are forwarded\nunchanged, and a `priority` the caller already set in the body is kept. Tier names are matched\ncase-insensitively. The deployment has to run a priority-aware vLLM scheduler\n(`--scheduling-policy=priority`). With the default `false`, the header is ignored and request\nbodies are forwarded byte for byte.\n\nWhen `AIBRIX_DISABLE_RATE_LIMITING=true`, the gateway skips AIBrix user lookup, ignores the\n`user` header as routing identity, and does not write user or model quota counters. The header\nis still forwarded unchanged. Keep Redis configured when session affinity, asynchronous jobs,\nstate synchronization, or other gateway features require it. The per-replica\n`requestsInflight` guard remains active. Because routing user identity remains empty, asynchronous\nvideo jobs use the shared ownership scope; the unauthenticated `user` header does not isolate jobs\nbetween callers in this mode.\n\n---\n\n## Response Processing\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_TTFT_THRESHOLD_S` | int (seconds) | `1` | Time-to-first-token threshold in seconds. Requests exceeding this are flagged in response processing. | [gateway_rsp_body.go](gateway_rsp_body.go) |\n\n`AIBRIX_TTFT_THRESHOLD_S` can also be set per model with the top-level `ttftThresholdS` profile field; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Redis Sync (`statesync/`)\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_STATESYNC_ENABLED` | bool | `false` | Enable cross-replica state sync via Redis. Must be `true` to activate the statesync manager. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `AIBRIX_STATESYNC_SYNC_PERIOD` | duration | `10s` | Interval at which gateway state is synced to Redis across replicas. | [statesync/redissync.go](statesync/redissync.go) |\n\n---\n\n## Prefix Cache Router (`algorithms/prefixcache/router.go`)\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | string | `\"character\"` | Tokenizer type for prefix cache hashing. Options: `character`, `tiktoken`, `remote`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | int | `1` | Factor multiplied by the standard deviation of pod loads when selecting among prefix-matched pods (`pod.req ≤ mean + factor × σ`). | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR` | float64 | `1.5` | Prefix-matched pods with more than `max(c, 1) × (mean + 1)` running requests are skipped, where `c` is this factor. Applies only when `prefix-cache` routes on its own, with the auto-blend off (`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT=0`). Unlike the `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` check, which cannot skip any pod in a pool of five or fewer at a factor of `2`, it applies to pools of any size. Values between `0` and `1` are treated as `1`; `0`, negative or unparsable values use the default `1.5`. To effectively disable the cap, set a large value such as `100`. Environment-only. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_INCLUDE_TOOLS` | bool | `true` | Prepend a canonical rendering of the chat request `tools` (keys sorted, compact) to the text used for prefix matching, since chat templates commonly render tools ahead of the messages. For templates that render tools later in the prompt (e.g. near the last user turn), requests sharing tools but not messages can see a partial match the engine does not have; such deployments can set this to `false`. Requests without tools (absent, `null`, `[]` or not an array) are unaffected. Set to `false` to match on messages only. The tools text goes into a separate prefix-match text (`RoutingContext.PrefixText()`), which feeds prefix matching in `prefix-cache`, `prefix-cache-preble` and the PD prefill policies `prefix_cache`, `conductor` (including its matched/unmatched token estimate) and `hybrid_cache_load`. The routing message (`routingCtx.Message`) stays messages-only, so token estimates (VTC, `PromptTokens`/`PromptLength`, config-profile `promptTokensGte`/`promptTokensLt`) are unchanged. In KV-sync mode, chat requests tokenized through the remote chat template always forward the raw `tools` to the tokenizer, independent of this flag, since the template itself decides where tools go. | [prefix_tools.go](prefix_tools.go) |\n| `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER` | bool | `false` | Use a remote HTTP tokenizer service instead of the local tokenizer. Requires `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED` | bool | `false` | Enable KV cache event synchronization across gateway replicas. When `true`, also requires `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_REMOTE_TOKENIZER_ENDPOINT` | string | `\"\"` | Remote tokenizer service endpoint URL. Required when `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true`. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_PUBLISH_ADDR` | string | `\"\"` | ZMQ publish address for KV cache events. Used when KV event sync is enabled. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SUBSCRIBE_ADDRS` | string | `\"\"` | ZMQ subscribe addresses for KV cache events (comma-separated). Used when KV event sync is enabled. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n\n### Remote Tokenizer Pool (`algorithms/prefixcache/router.go`)\n\nThese configure the pool of remote tokenizer connections used when `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_VLLM_TOKENIZER_ENDPOINT_TEMPLATE` | string | `\"http://%s:8000\"` | HTTP endpoint template for tokenizer pods. `%s` is replaced with the pod name. |\n| `AIBRIX_TOKENIZER_HEALTH_CHECK_PERIOD` | duration | `30s` | How often to health-check tokenizer pool members. |\n| `AIBRIX_TOKENIZER_TTL` | duration | `300s` | TTL for cached tokenizer connections in the pool. |\n| `AIBRIX_MAX_TOKENIZERS_PER_POOL` | int | `100` | Maximum number of tokenizer connections in the pool. |\n| `AIBRIX_TOKENIZER_REQUEST_TIMEOUT` | duration | `5s` | Timeout for individual remote tokenizer requests. |\n\n`AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` can also be set per request by the model config profile (`routingConfig.prefixCache.standardDeviationFactor`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Load Balance Router (`algorithms/load_balance.go`)\n\nThe load-imbalance gate below is applied centrally by the gateway (`selectTargetPod` in\n`gateway.go`), once per request, ahead of whichever strategy actually routes it — not just when\n`load-balance` is selected. It restricts candidates to the least-loaded pods before that\nstrategy runs when load is severely skewed, and it **is** applied to the Prefix Cache and\nPreble routers (this replaces the prefix-cache-specific gate that used to live in\n`prefixcache/router.go`; see the deprecation note under [Variable Dependency\nNotes](#variable-dependency-notes)). It is exempted only for exclusive strategies (`pd`,\n`slo*`), which manage their own pod subsets and would have that management disrupted by a\nblanket running-request-count filter applied ahead of time. It does not participate in\nmulti-strategy soft-scoring (`ScoreAll`) for strategies blended alongside `load-balance` — see\n`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` below for how `load-balance` itself gets\nsilently blended into other strategies to compensate.\n\n**TODO / known limitation:** the gate compares raw running-request count, with no notion of a\npod's capacity. In a heterogeneous pool this conflates \"carrying a lot of work\" with\n\"overloaded\" — a replica that's simply *faster* than its peers is expected to carry more\nconcurrent requests without being more loaded (the same capacity signal `load-balance`'s own\nscore uses), but this gate can still flag it as a hotspot and exclude it from routing before any\nstrategy's capacity-aware scoring gets a chance to run. See `getTargetPodListOnLoadImbalance` in\n[algorithms/load_balance.go](algorithms/load_balance.go).\n\nWhen `load-balance` is the (sole, non-blended) strategy that routes the request and multiple\npods tie on the lowest effective-load score, `Route()` breaks the tie using least combined\nGPU+CPU KV-cache usage (falling back to a random pick if cache metrics are unavailable for the\ntied pods) — the same secondary-signal pattern the Prefix Cache router uses to break ties in\nprefix-match percentage via request count.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` | float64 | `2.0` | Gate multiplier for 3+ pods: gate fires when `max_req > factor × (mean_req + 1)`. Ignored for 2-pod clusters (the relative check never holds there). | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` | int | `8` | Minimum absolute gap (`max_req − min_req`) required to trigger the load-imbalance gate. For 2 pods this is the sole trigger; for 3+ pods it is required alongside the factor check. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` | float64 | `0` | Weight λ of engine-queued requests (`num_requests_waiting`) added to the running count in the score: `(running + λ·queued) / capacity × …`. The default `0` scores on running requests only, since the gateway's running count already includes requests queued inside the engine; raise it only if you want queued work to count extra. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` | float64 | `2.0` | Strength α of the KV-pressure penalty: the score is multiplied by `1 + α·(1 − kv_free)²`. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` | float64 | `0.10` | Free-KV-cache fraction below which a pod scores `+Inf` from `load-balance`. When `load-balance` is the sole routing strategy this excludes the pod outright (falling back to the pod with the most KV headroom if every pod is below the threshold, instead of failing the request). When `load-balance` is blended with other strategies (the default for most requests — see \"Router Selection / Auto-Blend\" below), a `+Inf` score is only a strong penalty, not an exclusion: another strategy can still select the pod. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n\nThe five `AIBRIX_LOAD_BALANCE_*` gate and score variables can also be set per request by the model config profile (`routingConfig.loadBalance.*`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Least-Request Top-K Router (`algorithms/least/request_top_k.go`)\n\nStrategy name `least-request-top-k`. Plain `least-request` always resolves to the single\nleast-loaded pod, so when several gateway replicas read the same Redis-backed running-request\ncounter and decide at the same moment, they can all send their next request to that pod. This\nrouter instead picks uniformly at random among the ready pods that are both among the `K`\nleast-loaded and within `epsilon` requests of the least-loaded pod's count. It is also a scorer\n(`ScoreAll`), so it can be blended with other strategies; it is the load-count scorer that the\n[auto-blend](#router-selection--auto-blend-algorithmsroutergo) appends by default, so the variables below\napply to every auto-blended request, not only to requests that name this strategy.\n\nA pod that has just become eligible reads as count `0` and would otherwise be the sole eligible\ncandidate until it catches up. To prevent that, a pod's effective count is raised by a penalty\nthat starts at the lowest count among the already-warm pods and decays linearly to `0` over the\nramp window. The window is measured from when this gateway first sees the pod routable (Ready,\nwith an IP, not terminating or draining), not from when the pod was created, so time spent\nPending or loading a model does not use it up. Ramp state is local to each gateway; a gateway\nthat has just restarted sees every pod as new, so with no warm pod to compare against it applies\nno penalty until some pods age out of the window. The clock is kept, not restarted, when a pod\nflaps NotReady or is re-added with the same IP within 60 seconds; a new IP, or a longer gap, starts\nit over.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K` | int | `5` | `K`: how many of the least-loaded ready pods are eligible. Should be at least the number of gateway replicas deciding concurrently. Must be positive; any other value falls back to the default. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K_EPSILON` | int | `4` | A pod is eligible only if its count is within this many requests of the least-loaded pod's. Combined with `K` as an AND. Must be positive; any other value falls back to the default. The default is a starting point, not a validated one: tune it to the deployment's typical running-request count. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K_RAMP_WINDOW` | duration | `300s` | Window over which a newly routable pod's ramp penalty decays to `0`, counted from the first time the gateway sees the pod routable. `0` disables the ramp. Any other non-positive or unparseable value falls back to the default. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n\nThese are read once at startup and are environment-only (no `routingConfig` override).\n\n---\n\n## Router Selection / Auto-Blend (`algorithms/router.go`)\n\n`RouterManager.Select` silently blends `load-balance` and [`least-request-top-k`](#least-request-top-k-router-algorithmsleastrequest_top_kgo)\nbehind whatever strategy a caller actually asked for, so no single strategy can keep steering\ntraffic at an already-hot pod even without going through the central load-imbalance gate above.\nThe caller never sees this: `ctx.Algorithm`, response headers, and `Validate()` all continue to\nreflect exactly what was requested. The blend is skipped entirely for exclusive strategies\n(`pd`, `slo*`), for an explicit, standalone `load-balance` selection, and for a bare\n`session-affinity` selection, all of which must keep running their own dedicated routing logic.\n`session-affinity`'s own scoring is binary (1 for its resolved pod, 0 for everything else), so\nany load-balance weight below its own could never change the outcome anyway — blending it in\nwould be dead weight at best. A bare `prefix-cache` request, whose scoring is graded rather than\nbinary, still uses a 5:4 (1.25:1) ratio against `load-balance` instead of the flat weight-1\nappend, and does not receive `least-request-top-k`, so the cache-affinity signal wins an exact-tie\ndisagreement without masking real load imbalance.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` | int | `1` | Weight coefficient for the silently-appended `load-balance` scorer. Set to `0` to disable the whole auto-blend feature. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_LEAST_REQUEST_WEIGHT` | int | `1` | Weight coefficient for the silently-appended `least-request-top-k` scorer. It is added only when neither `least-request` nor `least-request-top-k` is already in the strategy (an explicit weight of `0` on either also suppresses it), and is needed so multi-port/data-parallel pod routing keeps working under the blend. Plain `least-request` is not used here because every gateway replica would score the same single least-loaded pod best. The variable keeps its `LEAST_REQUEST` name for compatibility. Set to `0` to omit it from the blend. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` | int | `5` | Primary weight used when auto-blending a bare `prefix-cache` request. With the matching load-balance weight this is a 5:4 (1.25:1) lean toward cache affinity. `0` is rejected with a warning and the default is used. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_LOAD_BALANCE_WEIGHT` | int | `4` | `load-balance` weight paired with `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` for a bare `prefix-cache` request. `0` leaves those requests with prefix-cache scoring alone. | [algorithms/router.go](algorithms/router.go) |\n\nThe four `AIBRIX_ROUTING_AUTO_BLEND_*` weights can also be set per request by the model config profile (`routingConfig.autoBlend.*`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## External Replica Router (`algorithms/external*.go`)\n\nThe optional `external` strategy sends the already-filtered candidate snapshot to one\noperator-configured HTTP endpoint. An unset endpoint disables the strategy. Invalid enabled\nconfiguration fails Gateway startup. Candidate metrics and labels are opt-in allowlists; prompts,\nrequest bodies, credentials, arbitrary client headers, and pod IPs are never sent.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_EXTERNAL_ROUTER_ENDPOINT` | URL | _(unset)_ | Complete `http` or `https` operation URL. Userinfo and fragments are rejected. |\n| `AIBRIX_EXTERNAL_ROUTER_POLICY_MODE` | enum | _(required)_ | `Advisory` or `Authoritative`. |\n| `AIBRIX_EXTERNAL_ROUTER_FAILURE_MODE` | enum | _(required)_ | `FailOpen` or `FailClosed`; Authoritative requires FailClosed. |\n| `AIBRIX_EXTERNAL_ROUTER_FALLBACK` | string | _(none)_ | Registered non-external, non-exclusive local router; required for Advisory and FailOpen. `pd` and `slo*` are rejected because they require dedicated Gateway preprocessing. |\n| `AIBRIX_EXTERNAL_ROUTER_TIMEOUT` | duration | `10ms` | Deadline for one decision exchange. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_INFLIGHT` | int | `256` | Non-blocking per-process bulkhead capacity. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_REQUEST_BYTES` | bytes | `256KiB` | Maximum encoded request size. Accepts bytes, `KiB`, or `MiB`. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_RESPONSE_BYTES` | bytes | `64KiB` | Maximum response body size. Accepts bytes, `KiB`, or `MiB`. |\n| `AIBRIX_EXTERNAL_ROUTER_FAILURE_THRESHOLD` | int | `5` | Consecutive attempted-exchange system failures before opening the circuit. |\n| `AIBRIX_EXTERNAL_ROUTER_OPEN_DURATION` | duration | `1s` | Circuit-open duration before one half-open probe. |\n| `AIBRIX_EXTERNAL_ROUTER_AUTH_TOKEN_FILE` | path | _(unset)_ | Optional non-empty bearer token file loaded at startup. |\n| `AIBRIX_EXTERNAL_ROUTER_CANDIDATE_ATTRIBUTES` | CSV | _(empty)_ | Pod-label keys allowed into candidate attributes. |\n| `AIBRIX_EXTERNAL_ROUTER_CANDIDATE_METRICS` | CSV | _(empty)_ | Any of `runningRequests`, `engineUtilization`, `kvCacheUsage`. |\n| `AIBRIX_EXTERNAL_ROUTER_POLICY_ATTRIBUTES` | CSV | _(empty)_ | Downstream extension-point allowlist for trusted RoutingContext policy attributes. Upstream AIBrix has no built-in producer; integrators must call `SetTrustedPolicyAttribute`. Raw headers are never copied automatically. |\n\nSee the [External Replica Routing guide](../../../docs/source/features/gateway-routing/external-replica-routing.rst)\nand the published OpenAPI contract for policy and protocol details.\n\n---\n\n## Preble (Prefix Cache with Histogram) Router (`algorithms/prefixcache/preble.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_PREBLE_TARGET_GPU` | string | `\"V100\"` | GPU model used for hardware-specific latency estimates in the Preble algorithm. |\n| `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` | int | `45` | Expected decode sequence length used for cache allocation decisions. |\n| `AIBRIX_ROUTER_PREBLE_SLIDING_WINDOW_PERIOD` | int (minutes) | `3` | Sliding window length in minutes for histogram metrics collection. |\n| `AIBRIX_ROUTER_PREBLE_EVICTION_LOOP_INTERVAL` | int (ms) | `1000` | Interval in milliseconds between cache eviction loop executions. |\n\n`AIBRIX_ROUTER_PREBLE_TARGET_GPU` and `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` can also be set per request by the model config profile (`routingConfig.preble.*`). The window and eviction variables stay environment-only; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## VTC (Virtual Token Counter) Router\n\n### Token Tracker (`algorithms/vtc/token_tracker.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | int | `5` | Sliding window size (in `TIME_UNIT` units) for token usage tracking. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | string | `\"minutes\"` | Time unit for the sliding window. Options: `minutes`, `seconds`, `milliseconds`. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | float64 | `1000.0` | Minimum token count threshold for adaptive load normalization. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | float64 | `8000.0` | Maximum token count threshold for adaptive load normalization. |\n\n### VTC Basic Scorer (`algorithms/vtc/vtc_basic.go`)\n\nScoring formula: `score = (fairnessWeight * normFairness + utilizationWeight * normUtilization) / normFreeGPU`\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | float64 | `100.0` | Load value at which a pod is considered fully saturated. |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | float64 | `1.0` | Weight applied to input tokens when computing pod load. |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | float64 | `2.0` | Weight applied to output tokens when computing pod load (typically higher than input). |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | float64 | `1.0` | Weight of the fairness component in the routing score. |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | float64 | `1.0` | Weight of the utilization component in the routing score. |\n\n`AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD`, `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` and `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` can also be set per request by the model config profile (`routingConfig.vtc.*`). The token tracker variables stay environment-only; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## PD (Prefill-Decode) Disaggregation Router (`algorithms/pd/router/router.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | int (seconds) | `30` | HTTP request timeout for prefill pod calls. |\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | int32 | `16` | Minimum (max − min) running-request spread across prefill pods to trigger load-imbalance routing. |\n| `AIBRIX_PREFILL_LOAD_CAP_FACTOR` | float64 | `1.5` | Prefill pods with more than `c × (mean + 1)` in-flight prefill requests are skipped, where `c` is this factor. Unlike the `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` check, it applies to pools of any size. Values between `0` and `1` are treated as `1`; `0`, negative or unparsable values use the default `1.5`. To effectively disable the cap, set a large value such as `100`. Lower values give up more prefix-cache affinity: with random (Poisson) load and no hot pod, `1.5` skips at least one pod on up to about 10% of requests with 4 prefill pods and about 20% with 6. `2` skips less often but cannot fire in a two-pod pool. |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | float64 | `16.0` | Minimum (max − min) running-request spread across decode pods to trigger load-imbalance routing. |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | float64 | `2048.0` | Minimum (max − min) token-throughput spread (tokens/s) across decode pods to trigger throughput-imbalance routing. |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | float64 | `1.5` | Max/min drain-rate score ratio above which the slowest decode pod is excluded from selection. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | bool | `false` | Route requests to prefill pods whose prompt-length bucket matches the request length. |\n| `AIBRIX_BUCKET_SERVE` | bool | `false` | Adaptive bucket serving: band the prompt-length range that several rolesets declare in common and prefer the roleset a request length is banded to. Requires `AIBRIX_PROMPT_LENGTH_BUCKETING=true`. |\n| `AIBRIX_BUCKET_SERVE_MODE` | string | `\"throughput\"` | The unit the adaptive cut points are measured in: `throughput` (prompt token mass) or `rps` (request counts). An unknown value keeps the default. |\n| `AIBRIX_KV_CONNECTOR_TYPE` | string | `\"shfs\"` | KV cache transfer backend. Options: `shfs` (GPU shared memory), `nixl` (Neuron). |\n| `AIBRIX_PREFILL_SCORE_POLICY` | string | `\"prefix_cache\"` | Strategy for selecting the prefill pod. Options: `prefix_cache`, `least_request`. |\n| `AIBRIX_DECODE_SCORE_POLICY` | string | `\"load_balancing\"` | Strategy for selecting the decode pod. Options: `load_balancing`, `least_request`, `conductor`, `token_load`. |\n| `AIBRIX_DECODE_TOKEN_LOAD_OUTPUT_GROWTH` | bool | `true` | `token_load` decode policy only: add an estimate of the output generated so far (per-request decode rate × time since routing) to the decode score. `false` scores prompt tokens only. |\n| `AIBRIX_DECODE_TOKEN_LOAD_CAPACITY_AWARE` | bool | `true` | `token_load` decode policy only: multiply each decode pod's score by the candidates' mean KV capacity divided by the pod's own (`num_gpu_blocks` × `block_size` from vLLM's `vllm:cache_config_info`, or SGLang's `sglang:max_total_num_tokens`), so pods of different sizes are compared by how full they are. Unknown capacities count as the mean; with none known, or `false`, scores are token counts. |\n| `AIBRIX_TOKEN_LOAD_SHARED_LEDGER` | bool | `true` | `token_load` decode policy only, with Redis configured: share the decode ledger across gateway replicas, so each replica adds the other live replicas' charges to its decode score. `false` keeps the ledger local to each replica. |\n\nThe prefill/decode routing thresholds (`AIBRIX_PREFILL_*`, `AIBRIX_DECODE_*`, `AIBRIX_TOKEN_LOAD_*`, `AIBRIX_HYBRID_CACHE_LOAD_FACTOR`, `AIBRIX_MIN_MATCH_PCT`, `AIBRIX_PROMPT_LENGTH_BUCKETING`, `AIBRIX_BUCKET_SERVE`,\n`AIBRIX_BUCKET_SERVE_MODE`), except `AIBRIX_PREFILL_LOAD_CAP_FACTOR`, can also be set per request by the model config profile\n(`routingConfig.pd.*`, `routingConfig.promptLengthBucketing` and `routingConfig.bucketServe*`); see [Model Config Profile Overrides](#model-config-profile-overrides). `AIBRIX_PREFILL_LOAD_CAP_FACTOR` is environment-only.\n\n### PD Prefill Fail-Fast (`algorithms/pd/abort.go`)\n\nWhen the prefill leg of a PD request fails, the decode pod is left waiting for a\nKV transfer that will never arrive until its own bootstrap timeout expires\n(300s in SGLang), holding its pre-allocated KV pages. The gateway injects its own\nrequest id (`rid`) into both legs, so on a prefill failure it can post\n`{\"rid\": ...}` to the decode pod's `/abort_request` and release those pages\nimmediately. The abort is best-effort and asynchronous: nothing waits on it, and\nit is skipped once the decode pod has started answering the client - a call\nalready on the wire when that happens is cancelled rather than left to run.\n\nThe abort is sent twice, because the first attempt can overtake the decode\nrequest itself on the pod (the engine answers `200` either way, so the gateway\ncannot tell a matched abort from a dropped one). Repeating an abort the engine\nalready applied, or never matched, is a no-op on both sides. The wait between\nthe two attempts is interruptible, so a decode leg that starts answering in the\nmeantime ends the retry immediately instead of at the end of the delay.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | int (seconds) | `3` | Per-attempt timeout of the `/abort_request` call to the decode pod. `0` disables decode aborts entirely; the prefill failure is still recorded, logged and reported to the client. |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | int (seconds) | `2` | Delay before the second abort attempt. `0` sends a single attempt. |\n\nMetrics: `gateway_pd_decode_abort_total{prefill_failure_class, result}`, where\n`result` is one of `ok`, `error`, `skipped_streaming`, `skipped_disabled`,\n`skipped_no_rid`, `skipped_no_target`.\n\n### PD Decode Watchdog (`gateway_decode_watchdog.go`)\n\nIn SGLang disaggregation the prefill leg only returns once the decode pod has\ntaken the KV transfer, so a successful prefill means the decode pod owes the\nclient a response. If the decode pod then stops answering (its scheduler died or\nwedged), nothing else notices and the client hangs until Envoy's route timeout.\nThe watchdog bounds that wait: once the prefill leg succeeded and until the\nfirst response body chunk comes back from the decode pod, it fails the request\nafter the first-response timeout and posts one best-effort `/abort_request` for\nthe request's `rid` to the decode pod. Response headers do not end this phase:\nSGLang sends the headers of a streaming response before its first token, so a\ndecode pod that wedges before the first token still delivers them. How the\nclient is failed depends on whether those headers already went out:\n\n- No headers yet: the client gets a `504` carrying the `x-error-pd-decode`\n header, and the ext_proc stream is closed with `DeadlineExceeded`.\n- Headers sent: no error response can replace a response whose status line is\n already on its way, so the ext_proc stream is only closed with\n `DeadlineExceeded` and Envoy resets the client connection. The client sees a\n `200` with a truncated (here: empty) body and no `x-error-pd-decode` header.\n\nFrom the first chunk of a streaming response on, the watchdog watches the gap\nbetween chunks instead: a stream that goes silent for\n`AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` is cut the same way, and the decode pod\ngets the same abort. It is armed only for requests that carry a gateway-owned\n`rid`, i.e. SGLang PD requests.\n\nA prefill leg that answered `200` with a body the gateway could not parse\n(`bad_response`) counts as succeeded for the watchdog: the KV transfer completed\nall the same, so fail-fast leaves the request alone and the watchdog times the\ndecode pod from that answer.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | int (seconds) | `60` | **Streaming requests only** (`\"stream\": true`). Time to first token: how long the decode pod is given to send its first response body chunk, measured from the moment the prefill leg returned successfully. Response headers alone do not count. On expiry the client gets a `504` (`did not start responding within …`), or a cut stream if the headers already went out. `0` disables the watchdog for streaming requests. |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | int (seconds) | `0` (disabled) | **Non-streaming requests only**. Same arming point, different meaning: the decode pod sends the body of a non-streaming response only once generation is finished, so this budget bounds the *whole* decode, not the time to first token. It is off by default because the safe value depends on the caller's `max_tokens` and the pod's throughput; operators who know both can set one. On expiry the client gets a `504` (`did not finish responding within …`). |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | int (seconds) | `120` | **Streaming requests only**. Longest gap allowed between two response body chunks once the first one arrived. Every chunk re-arms it. On expiry the stream is cut short. `0` disables it. |\n\nThe abort uses `AIBRIX_DECODE_ABORT_TIMEOUT` as its timeout and is counted in\n`gateway_pd_decode_abort_total` with `prefill_failure_class=\"watchdog_first_response\"`\nor `\"watchdog_stream_idle\"`.\n\nMetrics: `gateway_pd_decode_watchdog_total{phase}`, where `phase` is\n`first_response` or `stream_idle`. A request failed by the watchdog is also counted in\n`gateway_request_model_fail_total` with `status=\"pd_decode_watchdog\"` and\n`status_code=\"504\"`, including the cut streams whose client saw a `200`: watch\nfor those through this counter and the client's stream error, not through an\nerror body.\n\n### Decode Load Balancer Scorer (`algorithms/pd/decode_scorer.go`)\n\nScoring formula: `score = (wRun × normRunning + wThroughput × normInvThroughput) / normFreeGPU`\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | float64 | `1.0` | Weight for the normalized running-request term in the decode LB score. |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | float64 | `1.0` | Weight for the normalized inverse-throughput term in the decode LB score. |\n\n---\n\n## Utilities (`algorithms/pd/engine/trtllm.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_TRT_MACHINE_ID` | int64 | `0` | 10-bit machine ID (0–1023) used in Snowflake-style disaggregation request ID generation: `[timestamp:41b][machineID:10b][counter:12b]`. Panics on init if out of range. |\n\n---\n\n## Model Config Profile Overrides\n\nA model's `model.aibrix.ai/config` annotation can carry per-profile routing knobs\n(`routingConfig`) that override the matching variable above for that profile's requests only.\nAn unset knob keeps the environment default, and a value the variable would reject (a negative\nfactor, a percentage outside its range, an unknown GPU name) is ignored, so a profile can only\nnarrow or sharpen routing behavior. The values are resolved once per request and applied to that\nrequest alone; a request whose profile sets none behaves exactly as before. The user-facing\ndescription and examples live in the Config Profiles section of the gateway plugin guide\n([docs/source/features/gateway-routing/gateway-plugins.rst](../../../docs/source/features/gateway-routing/gateway-plugins.rst)).\n\n| Variable | Profile field | Notes |\n|---|---|---|\n| `AIBRIX_TTFT_THRESHOLD_S` | `ttftThresholdS` | Top-level profile field, not inside `routingConfig`. `0` counts as unset and keeps the environment default, so a profile can only change the threshold to another positive value, never to `0`. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | `routingConfig.promptLengthBucketing` | Turns bucketing on or off for the profile's requests. |\n| `AIBRIX_BUCKET_SERVE` | `routingConfig.bucketServe` | Turns the adaptive plan on or off for the profile's requests. |\n| `AIBRIX_BUCKET_SERVE_MODE` | `routingConfig.bucketServeMode` | `throughput` or `rps`; a name the planner does not know keeps the environment default. |\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | `routingConfig.pd.decodeAbortTimeout` | `0` sends the abort without waiting. |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | `routingConfig.pd.decodeAbortRetryDelay` | `0` repeats the abort immediately. |\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | `routingConfig.pd.decodeFirstResponseTimeout` | Seconds; `0` disables the decode watchdog for the profile's streaming requests. |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | `routingConfig.pd.decodeResponseTimeout` | Seconds; `0` disables the decode watchdog for the profile's non-streaming requests. |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | `routingConfig.pd.decodeStreamIdleTimeout` | Seconds; `0` disables the stream-idle phase for the profile. |\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.prefillLoadImbalanceMinSpread` | |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.decodeLoadImbalanceMinSpread` | |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.decodeThroughputImbalanceMinSpread` | |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | `routingConfig.pd.decodeScoreRatioThreshold` | |\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | `routingConfig.pd.decodeLBWeightRunning` | |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | `routingConfig.pd.decodeLBWeightThroughput` | |\n| `AIBRIX_TOKEN_LOAD_KV_WEIGHT` | `routingConfig.pd.tokenLoadKVWeight` | |\n| `AIBRIX_TOKEN_LOAD_REQUEST_COST` | `routingConfig.pd.tokenLoadRequestCost` | |\n| `AIBRIX_TOKEN_LOAD_TTL_SECONDS` | `routingConfig.pd.tokenLoadTTLSeconds` | `0` disables the sweep for the profile's requests. |\n| `AIBRIX_TOKEN_LOAD_SESSION_TTL_SECONDS` | `routingConfig.pd.tokenLoadSessionTTLSeconds` | `0` disables the session delta. |\n| `AIBRIX_HYBRID_CACHE_LOAD_FACTOR` | `routingConfig.pd.hybridCacheLoadFactor` | Accepted range 0 to 1. |\n| `AIBRIX_MIN_MATCH_PCT` | `routingConfig.pd.minMatchPct` | Accepted range 0 to 100; `0` disables the minimum-match clamp. |\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | `routingConfig.pd.prefillRequestTimeout` | Seconds. |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` | `routingConfig.loadBalance.imbalanceFactor` | |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` | `routingConfig.loadBalance.imbalanceMinGap` | |\n| `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` | `routingConfig.loadBalance.queuedWeight` | `0` is the V1 formula, running requests only. |\n| `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` | `routingConfig.loadBalance.kvPressureAlpha` | `0` drops the penalty. |\n| `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` | `routingConfig.loadBalance.kvCriticalFree` | Accepted range 0 to 1; `0` disables the guardrail. |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `routingConfig.prefixCache.standardDeviationFactor` | Also read by the PD prefill candidacy filter. |\n| `AIBRIX_ROUTER_PREBLE_TARGET_GPU` | `routingConfig.preble.targetGPU` | Known values `A6000` and `V100`; anything else is ignored. |\n| `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` | `routingConfig.preble.decodingLength` | |\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | `routingConfig.vtc.maxPodLoad` | |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | `routingConfig.vtc.fairnessWeight` | `0` drops the fairness term. |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | `routingConfig.vtc.utilizationWeight` | `0` drops the utilization term. |\n| `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` | `routingConfig.autoBlend.loadBalanceWeight` | `0` disables the auto-blend for the profile's requests. |\n| `AIBRIX_ROUTING_AUTO_BLEND_LEAST_REQUEST_WEIGHT` | `routingConfig.autoBlend.leastRequestWeight` | |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` | `routingConfig.autoBlend.prefixCacheWeight` | `0` is rejected with a warning and the environment value is kept; it would drop the caller's own strategy from the blend. |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_LOAD_BALANCE_WEIGHT` | `routingConfig.autoBlend.prefixCacheLoadBalanceWeight` | `0` leaves those requests with prefix-cache scoring alone. |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | `routingConfig.vtc.inputTokenWeight` | Also scopes the VTC token tracker: this weight is baked into the tracker, so the profile's requests get one of their own. |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | `routingConfig.vtc.outputTokenWeight` | Scopes the tracker the same way as the input weight. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | `routingConfig.vtc.tokenTrackerWindowSize` | Sliding window of the profile's tracker, in `tokenTrackerTimeUnit` units. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | `routingConfig.vtc.tokenTrackerTimeUnit` | Bucket size of that window: `minutes`, `seconds` or `milliseconds`. An unknown name is ignored instead of being normalized. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | `routingConfig.vtc.tokenTrackerMinTokens` | Floor the profile's tracker reports while its window holds little activity. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | `routingConfig.vtc.tokenTrackerMaxTokens` | Ceiling the profile's tracker reports while its window holds little activity. |\n\n### Scoped state behind the profile overrides\n\nThe VTC token tracker is constructed once from its window, time unit, token floors and the two\ntoken weights, so a profile that overrides any of the six gets a tracker of its own instead of\nretuning the shared one: trackers are keyed by the resolved values, so profiles that agree share\none, a profile that sets none keeps the shared tracker exactly as before, and the number of\ntrackers is bounded (16 per process; a profile past the bound keeps the shared tracker rather\nthan failing its requests).\n\n### Environment-only routing variables\n\nThese configure state shared by every model of one gateway process, so a profile cannot override\nthem without splitting or resizing that state for everyone:\n\n- `AIBRIX_ROUTER_PREBLE_SLIDING_WINDOW_PERIOD` and `AIBRIX_ROUTER_PREBLE_EVICTION_LOOP_INTERVAL`: the preble histogram window and its eviction loop are process-wide timers; scoping those needs a design of its own.\n- `AIBRIX_SESSION_AFFINITY_MAX_LOCAL_KEYS`: bounds the gateway-local session pin cache, whose entries are shared facts about a model and a session.\n- `AIBRIX_ROUTER_MAX_CACHED_ALGORITHM_STRINGS`: bounds the process-wide routing string caches.\n- `AIBRIX_TOKEN_LOAD_MAX_SESSIONS`: bounds the (model, session) table of the token-load tracker, which the gateway shares across models.\n\nVariables without a `routingConfig` field (tokenizer endpoints, Redis and statesync settings, the\nrate limiting switches, `AIBRIX_KV_CONNECTOR_TYPE`) stay environment-only as deployment-level\nsettings.\n\n---\n\n## Variable Dependency Notes\n\nThe following variables have interdependencies that must be satisfied together:\n\n- **KV event sync** requires all three to be set consistently:\n ```\n AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true\n AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true\n AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote\n AIBRIX_PREFIX_CACHE_REMOTE_TOKENIZER_ENDPOINT=\n ```\n\n- **Remote tokenizer pool** is initialized whenever `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`. The pool variables (`AIBRIX_VLLM_TOKENIZER_ENDPOINT_TEMPLATE`, `AIBRIX_TOKENIZER_*`, `AIBRIX_MAX_TOKENIZERS_PER_POOL`) all apply in that case.\n\n- **`AIBRIX_PREFIX_CACHE_POD_RUNNING_REQUEST_IMBALANCE_ABS_COUNT` is removed.** The\n prefix-cache-specific load-imbalance gate it used to tune was replaced by the centralized gate\n described under [Load Balance Router](#load-balance-router-algorithmsload_balancego), which\n now applies to prefix-cache (and every other non-exclusive strategy) via\n `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` / `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` instead. The\n trigger condition also changed: the old gate fired on absolute gap alone; the new one\n additionally requires a relative `factor × (mean + 1)` check for 3+ pod clusters, so it fires\n less often on larger fleets. Setting the old variable now only logs a startup warning.\n\n## OpenTelemetry\n\nThe gateway plugins feature built-in support for distributed tracing via OpenTelemetry (OTel), empowering you to monitor and trace end-to-end requests across the entire external processing pipeline.\n\n**Tracing is opt-in by default.** Telemetry components will only initialize if you explicitly configure either ``OTEL_EXPORTER_OTLP_ENDPOINT`` or ``OTEL_EXPORTER_OTLP_TRACES_ENDPOINT``. If both are omitted, all tracing capabilities remain disabled to conserve system resources.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|\n| `OTEL_EXPORTER_OTLP_PROTOCOL` | string | `grpc` | The transport protocol for OTLP data. Valid options are `grpc`, `http`, or `http/protobuf`. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | string | `\"\"` | Base URL for all OTLP signals. **⚠️ Note:** ensure your ENDPOINT URLs include the `http://` or `https://` prefix. The SDK automatically appends signal-specific paths to this URL (e.g., appending `/v1/traces` for HTTP exports). | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | string | `\"\"` | Target URL specifically for traces. Takes precedence over the global `ENDPOINT`. **⚠️ Note:** This URL is used exactly **as-is**. The SDK will **NOT** automatically append `/v1/traces` to it. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_HEADERS` | string | `\"\"` | Key-value pairs used as headers for OTLP requests (e.g., `key1=value1,key2=value2`). Useful for passing auth tokens to backends like Datadog or Honeycomb. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_TIMEOUT` | string | `10s` | Maximum time the OTLP exporter will wait for each batch export. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_INSECURE` | bool | `false` | Set to `true` to disable TLS/HTTPS for the exporter. Force downgrade to HTTP. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_INSECURE_SKIP_VERIFY` | bool | `false` | Keeps TLS active but skips server certificate validation (useful for self-signed certs). Set to \"true\" to enable. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n\n> **Note on OpenTelemetry Configuration:**\n> Aibrix supports the standard OpenTelemetry Protocol Exporter environment variables. For advanced OTLP configurations (such as `_CERTIFICATE`, `_CLIENT_KEY`, or specific `_COMPRESSION` settings), please refer to the [OpenTelemetry Protocol Exporter](https://opentelemetry.io/docs/specs/otel/protocol/exporter/)\n","pkg/plugins/gateway/algorithms/vtc_readme.md":"# Virtual Token Counter (VTC) Routing\n\nVirtual Token Counter (VTC) is a fair scheduling algorithm for LLM serving based on the paper\n[Fairness in Serving Large Language Models (Sheng et al., 2024)](https://arxiv.org/abs/2401.00588).\nVTC tracks the weighted token service each client has received and routes new requests to the pod\nthat gives the least-served client the best chance of progressing — balancing fairness with\ncluster utilization.\n\n## vtc-basic\n\n`vtc-basic` is a Kubernetes-native simplification of VTC. Rather than managing a single global\nqueue, it selects a target pod per request by combining two scores:\n\n- **Fairness score** — how much token service this client has received relative to all clients\n within a sliding time window. Clients who have received less service get a lower (better) score.\n- **Utilization score** — the pod's current load normalized against `MAX_POD_LOAD`.\n\nThe pod with the **lowest combined score** is selected.\n\nToken counts and the adaptive window extrema exclude expired usage, including\nclients that have stopped sending requests. The in-memory tracker reclaims that\nusage on reads and updates; it does not need a request from the departed client.\n\n### Scoring diagram\n\n```\nFor each ready pod:\n\n fairness_score = client_tokens_in_window / total_tokens_in_window\n (normalized, 0.0 → 1.0; lower = client received less service)\n\n utilization_score = pod_running_requests / MAX_POD_LOAD\n (normalized, 0.0 → 1.0; lower = pod is less loaded)\n\n combined_score = FAIRNESS_WEIGHT × fairness_score\n + UTILIZATION_WEIGHT × utilization_score\n\n ──────────────────────────────────────────────────────────────\n Select pod with minimum combined_score\n```\n\n### Request lifecycle\n\n```\nIncoming Request (with user identity)\n │\n ▼\n┌──────────────────────────────────┐\n│ Look up client token count │\n│ in sliding time window │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ Compute fairness_score │\n│ per ready pod │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ Compute utilization_score │\n│ per ready pod │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ combined_score = │\n│ w_f × fairness │\n│ + w_u × utilization │\n└───────────────┬──────────────────┘\n │\n ▼\n Select min-score pod\n │\n ▼\n┌──────────────────────────────────┐\n│ After response: update client │\n│ token count in window │\n│ (input × w_i + output × w_o) │\n└──────────────────────────────────┘\n```\n\nEnd-to-end tests: [vtc_routing_test.go](../../../../test/e2e/vtc_routing_test.go)\n\n## Configuration\n\n| Environment Variable | Description | Default |\n|-----------------------------------------------|------------------------------------------------------------------------------|----------------|\n| `AIBRIX_ROUTING_ALGORITHM` | Set to `vtc-basic` to enable this routing strategy. | `prefix-aware` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | Time unit for the sliding window. Options: `seconds`, `minutes`, `hours`. | `minutes` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | Number of time units in the sliding window. | `5` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | Minimum token floor for adaptive tracking — prevents division by near-zero. | `1000` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | Maximum token cap for adaptive tracking. | `8000` |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | Weight applied to input tokens when updating a client's service counter. | `1.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | Weight applied to output tokens when updating a client's service counter. | `2.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | Normalization denominator for pod running-request count. | `100.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | Multiplier for the fairness term in the combined score (`w_f`). | `1.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | Multiplier for the utilization term in the combined score (`w_u`). | `1.0` |\n\n### Tuning guidance\n\n- **Output token weight > input token weight** (default `2.0` vs `1.0`) reflects that output\n generation is more GPU-expensive than prompt processing.\n- **Increase `FAIRNESS_WEIGHT`** relative to `UTILIZATION_WEIGHT` when strict per-client\n fairness matters more than raw throughput.\n- **Increase `UTILIZATION_WEIGHT`** to prioritize routing to less-loaded pods, trading some\n fairness for better overall cluster efficiency.\n- **Window size** controls how long past token usage influences routing. A shorter window (e.g.\n `1 minute`) makes the router more reactive; a longer window (e.g. `10 minutes`) smooths out\n bursty clients.\n\n## Other VTC variants\n\nThe original VTC paper and reference implementation\n([slora/server/router](https://github.com/Ying1123/VTC-artifact/tree/main/slora/server/router))\ndescribe additional variants:\n\n| Variant | Description |\n|--------------|-----------------------------------------------------------------------------|\n| `vtc-fair` | Pure fairness — ignores utilization entirely. |\n| `vtc-max-fair` | Maximizes minimum service across all clients. |\n| `vtc-pred` | Uses output-length prediction to improve fairness estimates. |\n\nAdapting these variants to the AIBrix pod-selection model (rather than a single shared queue)\nis a candidate for future work.\n","pkg/plugins/gateway/ratelimiter/README.md":"# ratelimiter\n\nPackage `ratelimiter` provides a distributed, Redis-backed fixed-window rate limiter used by the Aibrix gateway to enforce per-user and per-model request limits across multiple gateway instances.\n\n## Interface\n\n```go\ntype RateLimiter interface {\n Get(ctx context.Context, key string) (int64, error)\n GetLimit(ctx context.Context, key string) (int64, error)\n Incr(ctx context.Context, key string, val int64, window ...time.Duration) (int64, error)\n}\n```\n\n| Method | Description |\n|------------|-------------|\n| `Get` | Returns the current counter value for a key in the active time window. |\n| `GetLimit` | Returns the configured maximum for a key (used for limit lookups, not window counters). |\n| `Incr` | Atomically increments the counter by `val` and returns the new value. The first write sets the key TTL to the window size; later writes do not extend it. |\n\n## Implementations\n\n### `redisRateLimiter` — `NewRedisAccountRateLimiter(name, client, windowSize)`\n\nA fixed-window counter backed by Redis. Suitable for cluster-wide enforcement when multiple gateway replicas share the same Redis instance.\n\n**Counter key scheme:** `{name}:{key}:{windowMilliseconds}:counter`. The key is stable for each configured window duration and separate from the static `{name}:{key}` key used by `GetLimit`. The counter expires `windowSize` after its first write, and the next write starts a new window. This window is independent of wall-clock bucket boundaries.\n\n- Minimum `windowSize` is 1 second (smaller values are clamped up).\n- `Incr` uses a Redis Lua script to increment and set the TTL atomically on the first write.\n- A missing key (`redis.Nil`) is treated as `0`, not an error.\n\n### `noopRateLimiter` — `NewNoopRateLimiter()`\n\nA no-op implementation that always allows requests. Used when Redis is unavailable (e.g., local development without a Redis sidecar). `GetLimit` returns `math.MaxInt64`.\n\n## Usage in the gateway\n\nTwo limiter instances are created at server startup:\n\n| Instance | Constructor arg `name` | `windowSize` | Purpose |\n|--------------------|------------------------|--------------|----------------------------------|\n| `ratelimiter` | `\"aibrix\"` | 1 minute | Per-user RPM and TPM enforcement |\n| `modelRateLimiter` | `\"aibrix_model\"` | 1 second | Per-model RPS enforcement |\n\nThe separate `name` prefix ensures the two limiters' keys never collide in Redis.\n\n### Per-user limits (RPM / TPM)\n\nCounter keys follow the pattern `aibrix:{username}_{RPM|TPM}_CURRENT:60000:counter`. Limits are read from the user record resolved during request header processing.\n\n### Per-model RPS\n\nCounter keys follow the pattern `aibrix_model:{modelName}_MODEL_RPS_CURRENT:{windowMilliseconds}:counter`. The RPS limit is configured per model via the `requestsPerSecond` field in a config profile:\n\n```json\n{\n \"profiles\": {\n \"rps-limited\": {\n \"routingStrategy\": \"least-request\",\n \"requestsPerSecond\": 1\n }\n }\n}\n```\n\nThe active profile is selected by passing the `config-profile` request header. If no profile is active or `requestsPerSecond` is unset, the RPS check is skipped entirely.\n\n#### Enforcement flow\n\nPer-model RPS enforcement is handled in `HandleRequestBody` via `enforceModelRPS` and a deferred compensation step:\n\n1. **Pre-routing gate (`enforceModelRPS`)** — called before routing. It:\n - atomically increments the counter with `Incr(..., +1)` and receives the new value\n - rejects with HTTP 429 and rolls back with `Incr(..., -1)` if `newVal > limit`\n - allows the request through when `newVal <= limit`\n2. **Deferred compensation (`decrModelRPS`)** — armed immediately after a successful pre-charge. If routing later fails, the deferred call refunds quota with `Incr(..., -1)`.\n3. **Success path** — after routing succeeds and request accounting is attached, compensation is disabled, so the pre-charge remains counted.\n\nUsing incr-then-check (rather than check-then-increment) means the `INCRBY` is the sole gate. Because Redis processes `INCRBY` atomically, each concurrent caller receives a unique sequential result — eliminating the TOCTOU race that would otherwise allow over-admission during the check→increment window.\n\n### Upgrading from wall-clock bucket keys\n\nThe counter key format changed from `{name}:{key}:{timebin}` to `{name}:{key}:{windowMilliseconds}:counter`. Existing bucket keys expire on their own TTL, but the new limiter does not read their counts. A deployment therefore starts a new counter window for each key. During a rolling upgrade, old and new gateway instances count requests separately and may collectively admit more than the configured limit. Deployments that require uninterrupted strict limits should avoid serving rate-limited traffic from both versions at the same time; a cutover to the new version still starts fresh counters.\n","pkg/plugins/gateway/statesync/README.md":"# statesync\n\nGeneric Redis-backed state sync for multiple gateway (or other) replicas. Uses **per-entity keys** with `SETEX` (per-record TTL), `MGET` (batched), and `SCAN` (enumeration). Keys are namespaced via a hash tag (`aibrix:{namespace}:e:{id}`). No PUB/SUB.\n\n- **Periodic sync**: Each replica runs **Pull first** (load from Redis), then **Push** (delta or full) every sync period.\n- **Optional write-through**: After local updates you may call `Put`/`Delete` directly. The default gateway wiring uses periodic delta push only.\n- **Deletions/tombstones**: If a Syncable implements `TombstoneSupport`, deletions are written as tombstone payloads so peers can remove the local entry on the next pull; otherwise `DEL` is used.\n- **Jitter and backoff**: Random initial startup delay (up to half `syncPeriod`); per-cycle jitter is roughly ±10% of `syncPeriod` (see `syncLoop` in `redissync.go`); exponential backoff (capped at 1 min) on errors.\n- **Lifecycle**: Register all Syncables **before** `Start()`; registrations after `Start()` are silently ignored.\n\n**Per-record TTL:** Each entity is stored as its own key with `SETEX` and a default TTL of 2 minutes. A record expires if it is not rewritten within that window. The TTL of one record does not affect others.\n\nRedis key format: `aibrix:{namespace}:e:{entityId}` — `{namespace}` is the hash tag. Namespace must be non-empty.\n\n### Prefix cache (gateway default wiring)\n\nWith `AIBRIX_STATESYNC_ENABLED=true`, `cmd/plugins/main.go`:\n\n1. Calls `table.EnableDeltaSync()` to activate dirty tracking on `PrefixHashTable` (no-op cost when disabled).\n2. Registers `PrefixHashTableSyncable` with the manager.\n3. Relies on **periodic delta push + pull** — `Put` is not called on every `AddPrefix`.\n\nThis gives eventual consistency with staleness bounded by the sync interval and per-key TTL. For faster cross-replica visibility you can call `syncManager.Put(ctx, \"prefixcache\", blockIDStr, data)` on the `*statesync.RedisSync` using bytes from `EncodeBlockForSync` after each `AddPrefix` (higher Redis write load trade-off).\n\n### Delta sync and LRU eviction\n\n`EnableDeltaSync()` must be called on `PrefixHashTable` before it is registered with `*statesync.RedisSync`; without it, dirty tracking is inactive and `GetDeltaForSync` always returns an empty delta. If a block is marked dirty then **evicted locally** before the next successful push, `GetDeltaForSync` skips it silently. The corresponding Redis key remains until the per-entity TTL expires. Strong delete propagation is not guaranteed for evicted entries.\n\n## Diagrams\n\n### Technical Flow\n\n```mermaid\nsequenceDiagram\n participant SyncLoop\n participant Redis\n participant Syncable\n Note over SyncLoop: Pull first, then Push every syncPeriod\n SyncLoop->>Redis: SCAN pattern (keys for namespace)\n Redis-->>SyncLoop: keys[]\n SyncLoop->>Redis: MGET keys (batched)\n Redis-->>SyncLoop: values[]\n SyncLoop->>Syncable: ApplyRemote(id, bytes) for each\n SyncLoop->>Syncable: GetDelta or GetSnapshot\n Syncable-->>SyncLoop: updated, deleted\n SyncLoop->>Redis: SETEX key val TTL per entity, DEL deleted\n SyncLoop->>Syncable: OptionalHooks (if implemented)\n```\n\n---\n\n## Adopting statesync in a new component\n\nChanges live **only in the package that owns the state**. The `statesync` package stays generic and never imports component types.\n\n### 1. Implement `syncable.Syncable`\n\nDepend on `github.com/vllm-project/aibrix/pkg/utils/syncable` and implement:\n\n| Method | Purpose |\n|--------|---------|\n| `Namespace() string` | Stable name for this state (e.g. `\"mytracker\"`). Used in the Redis key. |\n| `GetSnapshot(ctx) (map[string][]byte, error)` | Return current local state as **entity id → serialized bytes**. |\n| `ApplyRemote(ctx, id string, data []byte) error` | Apply one entity from Redis into local state (merge or overwrite). |\n\nOptional interfaces:\n\n| Interface | Purpose |\n|-----------|---------|\n| `DeltaSyncable` | Add `GetDelta` / `ClearDirty` to push only changed entities instead of the full snapshot. |\n| `OptionalHooks` | `OnSyncStart` / `OnSyncEnd` called around each **Pull** for that Syncable (not around Push). |\n| `StalePolicy` | `IsStale` — if true, the sync layer deletes the remote key and skips applying. |\n| `TombstoneSupport` | `MakeTombstone` / `IsTombstone` / `DeleteLocal` — propagate deletes as payloads instead of DEL. |\n\n### 2. Add serialization\n\n- **Encode**: turn your struct into `[]byte` (e.g. JSON). Use a stable format all replicas can decode.\n- **Decode**: in `ApplyRemote`, unmarshal `data` and merge into or replace local state.\n\n### 3. Expose a constructor returning `syncable.Syncable`\n\nReturn an adapter that holds a pointer to your state and implements the interface. Callers pass it to `(*statesync.RedisSync).Register(...)`.\n\n### 4. Wire in the process entry point\n\n```go\nrs := statesync.New(redisClient,\n statesync.WithSyncPeriod(30*time.Second), // optional; default 10s\n statesync.WithOpTimeout(15*time.Second), // optional; default 30s\n statesync.WithKeyPrefix(\"myapp\"), // optional; default \"aibrix\"\n)\n// Register all Syncables before Start.\nrs.Register(mypkg.NewMyStateSyncable(myState))\nrs.Start()\n\n// On shutdown:\nrs.Stop() // blocks until the sync loop exits or stop-wait timeout (default 90s)\n```\n\nAvailable `statesync.New` options:\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `WithSyncPeriod(d)` | 10s | Interval between sync cycles (jitter applied). |\n| `WithOpTimeout(d)` | 30s | Per-operation context timeout for Pull/Push. |\n| `WithKeyPrefix(s)` | `\"aibrix\"` | Prefix for all Redis keys. |\n| `WithRecordTTL(d)` | 2m | Per-entity `SETEX` TTL; values ≤0 are ignored. |\n| `WithStopWaitTimeout(d)` | 90s | Max time `Stop()` waits for the background loop to exit. |\n| `WithSetexChunkSize(n)` | 25 | Pipeline chunk size for SETEX writes. |\n| `WithMGetBatchSize(n)` | 200 | Batch size for MGET during Pull. |\n\n### Redis client\n\nThe Redis client is provided by the caller (e.g. `utils.GetRedisClient()`). Configure TLS and auth in the client options for production (see `pkg/utils/redis.go`).\n\n---\n\n## Example: a new component \"TokenTracker\"\n\nA component that keeps `tokenID → lastUsedTime` and should sync across gateway replicas.\n\n### Step 1 – Implement Syncable in the component package\n\n```go\n// pkg/plugins/gateway/algorithms/vtc/token_tracker_sync.go\n\nimport (\n \"context\"\n \"encoding/json\"\n \"strconv\"\n \"time\"\n\n \"github.com/vllm-project/aibrix/pkg/utils/syncable\"\n)\n\nconst tokenTrackerNamespace = \"token_tracker\"\n\n// TokenTrackerSyncable adapts TokenTracker for statesync.\ntype TokenTrackerSyncable struct {\n Tracker *TokenTracker\n}\n\nfunc NewTokenTrackerSyncable(t *TokenTracker) syncable.Syncable {\n return &TokenTrackerSyncable{Tracker: t}\n}\n\nfunc (s *TokenTrackerSyncable) Namespace() string { return tokenTrackerNamespace }\n\nfunc (s *TokenTrackerSyncable) GetSnapshot(ctx context.Context) (map[string][]byte, error) {\n s.Tracker.mu.RLock()\n defer s.Tracker.mu.RUnlock()\n out := make(map[string][]byte, len(s.Tracker.entries))\n for id, t := range s.Tracker.entries {\n b, _ := json.Marshal(t.UnixNano())\n out[strconv.FormatInt(id, 10)] = b\n }\n return out, nil\n}\n\nfunc (s *TokenTrackerSyncable) ApplyRemote(ctx context.Context, id string, data []byte) error {\n var nano int64\n if err := json.Unmarshal(data, &nano); err != nil {\n return err\n }\n parsed, _ := strconv.ParseInt(id, 10, 64)\n remote := time.Unix(0, nano)\n s.Tracker.mu.Lock()\n defer s.Tracker.mu.Unlock()\n if existing, ok := s.Tracker.entries[parsed]; !ok || remote.After(existing) {\n s.Tracker.entries[parsed] = remote\n }\n return nil\n}\n```\n\n### Step 2 – Wire in the entry point\n\n```go\nrs := statesync.New(redisClient, statesync.WithSyncPeriod(30*time.Second))\nrs.Register(prefixcacheindexer.NewPrefixHashTableSyncable(prefixHashTable))\nrs.Register(vtc.NewTokenTrackerSyncable(tokenTracker))\nrs.Start()\ndefer rs.Stop()\n```\n\n### Step 3 – Optional write-through\n\nIf you want other replicas to see a change before the next periodic push:\n\n```go\nfunc (t *TokenTracker) RecordUse(ctx context.Context, id int64, rs *statesync.RedisSync) {\n t.mu.Lock()\n t.entries[id] = time.Now()\n t.mu.Unlock()\n if rs != nil {\n data, _ := json.Marshal(time.Now().UnixNano())\n _ = rs.Put(ctx, tokenTrackerNamespace, strconv.FormatInt(id, 10), data)\n }\n}\n```\n\n---\n\n## Reference implementation\n\nSee `pkg/utils/prefixcacheindexer/sync.go`:\n- `PrefixHashTableSyncable` and `NewPrefixHashTableSyncable(table)` — full `DeltaSyncable` implementation.\n- `GetSnapshotForSync` / `GetDeltaForSync` / `ClearDirtyForSync` / `ApplyRemoteForSync` / `EncodeBlockForSync` on `PrefixHashTable`.\n- Remember to call `table.EnableDeltaSync()` before registering if you want delta push (not required for snapshot-only sync).\n","pkg/plugins/gateway/algorithms/external/testdata/denied.json":"{\n \"apiVersion\": \"routing.aibrix.ai/v1alpha1\",\n \"kind\": \"ReplicaSelectionResponse\",\n \"metadata\": {\"requestId\": \"req-123\", \"decisionId\": \"decision-789\"},\n \"status\": {\"decision\": \"Denied\", \"reason\": \"CompliancePolicy\"}\n}\n","pkg/plugins/gateway/testdata/envoy_gateway_v1.2.8_xds.json":"{\n \"xds\": {\n \"aibrix-system/aibrix-eg\": {\n \"configs\": [\n {\n \"@type\": \"type.googleapis.com/envoy.admin.v3.ListenersConfigDump\",\n \"dynamicListeners\": [\n {\n \"activeState\": {\n \"listener\": {\n \"@type\": \"type.googleapis.com/envoy.config.listener.v3.Listener\",\n \"accessLog\": [\n {\n \"filter\": {\n \"responseFlagFilter\": {\n \"flags\": [\n \"NR\"\n ]\n }\n },\n \"name\": \"envoy.access_loggers.file\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\",\n \"logFormat\": {\n \"jsonFormat\": {\n \":authority\": \"%REQ(:AUTHORITY)%\",\n \"bytes_received\": \"%BYTES_RECEIVED%\",\n \"bytes_sent\": \"%BYTES_SENT%\",\n \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n \"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\n \"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\n \"duration\": \"%DURATION%\",\n \"method\": \"%REQ(:METHOD)%\",\n \"protocol\": \"%PROTOCOL%\",\n \"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\n \"response_code\": \"%RESPONSE_CODE%\",\n \"response_code_details\": \"%RESPONSE_CODE_DETAILS%\",\n \"response_flags\": \"%RESPONSE_FLAGS%\",\n \"route_name\": \"%ROUTE_NAME%\",\n \"start_time\": \"%START_TIME%\",\n \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n \"upstream_host\": \"%UPSTREAM_HOST%\",\n \"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\n \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\n \"user-agent\": \"%REQ(USER-AGENT)%\",\n \"x-envoy-origin-path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n \"x-envoy-upstream-service-time\": \"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\n \"x-forwarded-for\": \"%REQ(X-FORWARDED-FOR)%\",\n \"x-request-id\": \"%REQ(X-REQUEST-ID)%\"\n }\n },\n \"path\": \"/dev/stdout\"\n }\n }\n ],\n \"address\": {\n \"socketAddress\": {\n \"address\": \"0.0.0.0\",\n \"portValue\": 10080\n }\n },\n \"defaultFilterChain\": {\n \"filters\": [\n {\n \"name\": \"envoy.filters.network.http_connection_manager\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\",\n \"accessLog\": [\n {\n \"name\": \"envoy.access_loggers.file\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\",\n \"logFormat\": {\n \"jsonFormat\": {\n \":authority\": \"%REQ(:AUTHORITY)%\",\n \"bytes_received\": \"%BYTES_RECEIVED%\",\n \"bytes_sent\": \"%BYTES_SENT%\",\n \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n \"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\n \"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\n \"duration\": \"%DURATION%\",\n \"method\": \"%REQ(:METHOD)%\",\n \"protocol\": \"%PROTOCOL%\",\n \"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\n \"response_code\": \"%RESPONSE_CODE%\",\n \"response_code_details\": \"%RESPONSE_CODE_DETAILS%\",\n \"response_flags\": \"%RESPONSE_FLAGS%\",\n \"route_name\": \"%ROUTE_NAME%\",\n \"start_time\": \"%START_TIME%\",\n \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n \"upstream_host\": \"%UPSTREAM_HOST%\",\n \"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\n \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\n \"user-agent\": \"%REQ(USER-AGENT)%\",\n \"x-envoy-origin-path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n \"x-envoy-upstream-service-time\": \"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\n \"x-forwarded-for\": \"%REQ(X-FORWARDED-FOR)%\",\n \"x-request-id\": \"%REQ(X-REQUEST-ID)%\"\n }\n },\n \"path\": \"/dev/stdout\"\n }\n }\n ],\n \"commonHttpProtocolOptions\": {\n \"headersWithUnderscoresAction\": \"REJECT_REQUEST\"\n },\n \"http2ProtocolOptions\": {\n \"initialConnectionWindowSize\": 1048576,\n \"initialStreamWindowSize\": 65536,\n \"maxConcurrentStreams\": 100\n },\n \"httpFilters\": [\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"STREAMED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"STREAMED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"BUFFERED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"name\": \"envoy.filters.http.router\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\",\n \"suppressEnvoyHeaders\": true\n }\n }\n ],\n \"mergeSlashes\": true,\n \"normalizePath\": true,\n \"pathWithEscapedSlashesAction\": \"UNESCAPE_AND_REDIRECT\",\n \"rds\": {\n \"configSource\": {\n \"ads\": {},\n \"resourceApiVersion\": \"V3\"\n },\n \"routeConfigName\": \"aibrix-system/aibrix-eg/http\"\n },\n \"serverHeaderTransformation\": \"PASS_THROUGH\",\n \"statPrefix\": \"http-10080\",\n \"useRemoteAddress\": true\n }\n }\n ],\n \"name\": \"aibrix-system/aibrix-eg/http\"\n },\n \"name\": \"aibrix-system/aibrix-eg/http\",\n \"perConnectionBufferLimitBytes\": 4194304\n }\n }\n }\n ]\n },\n {\n \"@type\": \"type.googleapis.com/envoy.admin.v3.RoutesConfigDump\",\n \"dynamicRouteConfigs\": [\n {\n \"routeConfig\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.RouteConfiguration\",\n \"ignorePortInHostMatching\": true,\n \"name\": \"aibrix-system/aibrix-eg/http\",\n \"virtualHosts\": [\n {\n \"domains\": [\n \"*\"\n ],\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"Gateway\",\n \"name\": \"aibrix-eg\",\n \"namespace\": \"aibrix-system\",\n \"sectionName\": \"http\"\n }\n ]\n }\n }\n },\n \"name\": \"aibrix-system/aibrix-eg/http/*\",\n \"routes\": [\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"safeRegex\": {\n \"regex\": \"^/v1/videos/sync$|^/v1/videos/[^/]+/content$\"\n }\n },\n \"name\": \"original_route_videos_streaming\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"safeRegex\": {\n \"regex\": \"^/v1/videos(/[^/]+)?$\"\n }\n },\n \"name\": \"original_route_videos\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"prefix\": \"/\"\n },\n \"name\": \"original_route\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"path\": \"/v1/videos/sync\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos-streaming\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"safeRegex\": {\n \"regex\": \"/v1/videos/[^/]+/content\"\n }\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos-streaming\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0/match/1/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/transcriptions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/15/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/images/generations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/13/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/translations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/16/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/video/generations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/19/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/chat/completions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/voices\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/18/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/images/edits\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/14/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/speech\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/17/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/completions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/3/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/embeddings\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/4/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/systemone\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/9/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/decisions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/8/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/responses\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/1/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/classify\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/6/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/messages\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/2/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"directResponse\": {\n \"status\": 500\n },\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/batches\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-metadata-endpoint\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-metadata-endpoint/rule/0/match/2/*\"\n },\n {\n \"directResponse\": {\n \"status\": 500\n },\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/models\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-metadata-endpoint\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-metadata-endpoint/rule/0/match/0/*\"\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/videos\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/rerank\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/5/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/tokenize\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/11/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy","pkg/plugins/gateway/algorithms/external/testdata/no-decision.json":"{\n \"apiVersion\": \"routing.aibrix.ai/v1alpha1\",\n \"kind\": \"ReplicaSelectionResponse\",\n \"metadata\": {\"requestId\": \"req-123\"},\n \"status\": {\"decision\": \"NoDecision\", \"reason\": \"NoApplicablePolicy\"}\n}\n","pkg/plugins/gateway/algorithms/load_balance.md":"# Load Balance Router\n\n`load-balance` routes each request to the replica with the lowest **effective load**: the work\nalready committed to it, divided by how fast it has been observed to drain work, inflated as its\nKV cache fills. It needs no hardware labels. A mixed B40 / A100 / H20 pool teaches the gateway\nits relative speeds from observed output-token throughput, and faster replicas receive\nproportionally more traffic.\n\nSource: [load_balance.go](load_balance.go). Environment variables:\n[ENV_VARS.md](../ENV_VARS.md#load-balance-router-algorithmsload_balancego).\n\n`load-balance` is also blended in behind other strategies by default (see\n`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` and [multi_router_readme.md](multi_router_readme.md)),\nso its `ScoreAll` runs on most requests, not only when it is selected explicitly.\n\n## Score\n\n```\n running + λ · queued\nscore = ───────────────────────────────────── × (1 + α · (1 − kvFree)²)\n EWMA(output tokens completed / sec)\n\nkvFree < kvCritical ⇒ score = +Inf\n```\n\nThe lowest score wins. The three parts answer three separate questions, and each is used for one\npurpose only:\n\n| Part | Question | Source |\n|---|---|---|\n| `EWMA(output tokens/sec)` | How fast does this replica drain work? (**capacity**) | `RealtimeOutputTokenRateEWMA`, derived in the cache from completed output tokens |\n| `running + λ·queued` | How much work is already committed to it? (**load**) | `GetPodsRunningRequests` (live cross-gateway count); `num_requests_waiting` when `λ > 0` |\n| `1 + α(1 − kvFree)²` | How close is it to memory pressure? (**headroom**) | `kv_cache_usage_perc` (`KVCacheUsagePerc`) |\n\nCapacity and load are kept apart on purpose. A saturated GPU has high token throughput *because*\nit is busy; if throughput were used as \"load\", the scheduler would send it even more traffic.\nHere throughput only sets the denominator, and load comes from the committed work.\n\n### Defaults\n\n| Symbol | Meaning | Default | Variable |\n|---|---|---|---|\n| `λ` | Weight of engine-queued requests | `0` | `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` |\n| `α` | KV-pressure penalty strength | `2.0` | `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` |\n| `kvCritical` | Free-KV fraction below which a pod scores `+Inf` from `load-balance` (excluded when `load-balance` routes alone; only strongly penalized in a blend — see \"Interaction with the rest of the gateway\" below) | `0.10` | `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` |\n\n`λ` defaults to `0` because the gateway's running count already includes requests that are\nqueued inside the engine; adding `num_requests_waiting` on top would count them twice. Values\n`<= 0` for these variables fall back to the default (the loader rejects non-positive numbers).\n\n### Worked example\n\n| Replica | Running | Tokens/s | KV free | Calculation | Score |\n|---|---|---|---|---|---|\n| B40 | 30 | 6000 | 50% | 30/6000 × (1 + 2·0.50²) | **0.0075** |\n| A100 | 20 | 4000 | 30% | 20/4000 × (1 + 2·0.70²) | 0.0099 |\n| H20 | 25 | 3000 | 60% | 25/3000 × (1 + 2·0.40²) | 0.0110 |\n\nThe B40 wins despite carrying the most requests, because it drains them fastest. This example is\nasserted in `TestLoadBalanceScoreAll_HeterogeneousExample`.\n\n## Where the capacity signal comes from\n\n```\ngateway: DoneRequestTrace(outputTokens) ──► Pod.completedOutputTokens (monotonic counter)\n │ every metrics refresh (default 1s)\n ▼\n calculateRate1m: tokens/sec over ~1 minute window\n │\n ▼\n EWMA, time constant 10s ──► RealtimeOutputTokenRateEWMA (pod-scoped)\n```\n\n- The counter is **gateway-tracked**, like `completedRequests`, so it works for every engine\n (vLLM, SGLang, TRT-LLM, xLLM) without depending on engine metric names. Code:\n `donePodStats` in `pkg/cache/cache_trace.go`, `updateRealtimeOutputTokenRateEWMA` in\n `pkg/cache/cache_metrics.go`.\n- Output tokens are the request's completion-token usage. A request that finishes through the\n count-only path (no usage available) adds one completion and **zero** tokens.\n- The counter survives a brief pod delete/re-add (informer flap) the same way `completedRequests`\n does.\n- A window with no completed tokens carries no information about speed (the replica was idle,\n not slow), so the previous estimate is **kept**, not decayed toward zero. Decaying would make a\n briefly idle replica look weak and starve it of the traffic that would re-measure it.\n\n## Behavior in edge cases\n\n| Situation | Behavior |\n|---|---|\n| Pod has no token-rate estimate yet (new pod, or no usage reported) | Scored at the **mean** estimate of the pods that have one, so it competes as an average replica. A fixed fallback such as `1.0` would be off by the tokens/sec scale (thousands) and lock the pod out of the traffic it needs to be measured. |\n| No pod has an estimate | Every pod gets capacity `1.0`; the score reduces to `running × KV penalty`. |\n| Engine reports no KV usage, or `NaN` | Treated as fully free: no penalty and no guardrail. |\n| `kvFree < kvCritical` | Score `+Inf` from `load-balance`. When `load-balance` routes alone this skips the pod entirely until it recovers. When `load-balance` is blended with other strategies (the default — see below), it is only a strong penalty, not a guaranteed exclusion. |\n| **Every** pod is below `kvCritical` | The request is **not** failed. Routing falls back to the tie-break over all pods, which picks the one with the least KV usage. This fallback is specific to `load-balance` routing alone; see below for the blended case. |\n| Pod is idle (`running = 0`) | Score `0` regardless of capacity, so idle pods are preferred. |\n| Several pods share the lowest score | Ties are broken by least combined GPU+CPU KV-cache usage (`least-kv-cache` scorer), falling back to random. |\n\nKV usage is clamped to `[0, 1]` before use, so an engine that reports a value above 1 is treated\nas full rather than producing a negative free fraction.\n\n## Interaction with the rest of the gateway\n\n- **Load-imbalance gate.** The gateway applies `ApplyLoadImbalanceGate` once, ahead of whichever\n strategy routes the request, so `Route` receives an already-narrowed pod list. If that narrowed\n set is entirely KV-critical, the all-critical fallback above applies. **TODO / known\n limitation:** the gate compares raw running-request count, with no notion of a pod's capacity —\n a replica that's simply faster than its peers is expected to carry more concurrent requests\n without being more loaded (the same capacity signal `load-balance`'s own score uses above), but\n this gate can still flag it as a hotspot and exclude it before `load-balance`'s own\n capacity-aware score ever gets to run. See `getTargetPodListOnLoadImbalance` in\n [load_balance.go](load_balance.go).\n- **Multi-strategy blending: the guardrail is soft here, not hard.** `kvCritical` is a hard\n exclusion only when `load-balance` routes alone. In a blend, `ScoreAll` still returns `+Inf` for\n a critical pod, but `multiStrategyRouter.normalizeScoresArray` treats a non-finite score the\n same as \"not scored\" and assigns it `0` — the worst possible score from `load-balance`, not an\n exclusion from routing. If another blended strategy scores that same pod highly enough, it can\n still be selected.\n\n Concretely: `load-balance` is blended in behind other strategies by default (see\n `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT`), so the blended path, not the standalone one, is\n what most requests actually take. With the default `prefix-cache:5,load-balance:4` blend, a\n KV-critical pod that has the best prefix-cache score still gets `5/9` of the combined weight\n versus `4/9` for a healthy pod that only wins on `load-balance`, so the critical pod wins the\n route. `kvCritical` should therefore be read as \"how hard `load-balance` argues against routing\n to this pod,\" not as a safety limit that blocks it from being chosen — it does not, on its own,\n prevent a KV-critical pod from receiving more traffic. Something that must never route to a\n KV-critical pod needs a filter ahead of scoring (comparable to `ApplyLoadImbalanceGate`), which\n does not currently exist for KV pressure.\n- **PD disaggregation.** The PD router's decode path keeps its own request drain-rate scoring\n (`RealtimeRunningRequestsDrainRate1m`) and is unchanged.\n\n## Known limitations\n\n- **Multiple gateway replicas.** The token counter is local to each gateway, while the running\n count is the cross-gateway total. Each gateway therefore measures roughly `1/N` of a pod's\n output. Relative ranking between pods is preserved as long as the gateways spread traffic\n similarly, but absolute score values are not comparable across deployments with different\n gateway counts.\n- **Estimate lag.** The estimate follows a ~1 minute window smoothed with a 10 second time\n constant, so a sudden change in a replica's speed (for example a workload shifting from\n decode-heavy to prefill-heavy) shows up within tens of seconds, not immediately.\n- **No prefill/decode split.** Capacity is a single output-token rate. Hardware whose relative\n performance differs between prompt-heavy and decode-heavy workloads is not modeled; a natural\n extension is `capacity = α·prefill_tokens/s + β·decode_tokens/s`.\n- **Deliberately not used:** GPU utilization / SM-active. They are useful for observability but\n make a generic scheduler unstable; the score uses only drain rate, committed work, and KV\n pressure.\n\n## Debugging\n\nAt `-v=4` the gateway logs `load_balance_score` once per pod (`running_requests`,\n`capacity`, `kv_free`, and the resulting `score`) and `load_balance_selected` (with the\nwinning `score`). `+Inf` scores are logged as strings so klog's JSON formatter does not\nfail. The cache logs `Updating output token rate metric` with the completed token count,\nthe 1-minute rate, and the EWMA for each pod. The estimate is stored on the pod as\n`realtime_output_token_rate_ewma`; it is not currently exported as a Prometheus series.\n","pkg/plugins/gateway/algorithms/external/testdata/selected.json":"# Testdata Skill\n\nPath: `pkg/plugins/gateway/algorithms/external/testdata/selected.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/external/testdata/selected.json)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AIBrix\n\nAIBrix is a Kubernetes-native platform for building scalable GenAI inference\ninfrastructure. This repository contains Go controllers, gateway plugins,\ncustom resources, deployment manifests, tests, and Python runtime components.\nThis file gives coding agents the repository-specific context and safety rules\nneeded to make focused changes.\n\n## Scope and repository layout\n\n| Path | Role |\n|------|------|\n| `api/` | Kubernetes API types and CRD definitions |\n| `pkg/controller/` | Kubernetes controllers and reconciliation logic |\n| `pkg/plugins/` | Gateway plugins and request processing |\n| `pkg/` | Shared Go libraries, clients, caches, metrics, and utilities |\n| `cmd/` | Go binary entrypoints |\n| `config/` | Kustomize configuration, CRDs, RBAC, and deployment manifests |\n| `test/integration/` | Ginkgo-based integration tests |\n| `test/e2e/` | Kind and cluster-level end-to-end tests |\n| `hack/` | Code generation, verification, and CI scripts |\n| `python/aibrix/` | Python runtime, downloader, batch, metadata, and optimizer |\n| `python/aibrix_kvcache/` | Python distributed KV-cache components |\n| `apps/` | Application services and the console |\n| `docs/` | Project documentation and generated documentation assets |\n| `samples/` | Example deployments and feature samples |\n\nRules in a deeper `AGENTS.md` take precedence for files under that directory.\nThe Python runtime has additional guidance in\n[`python/aibrix/AGENTS.md`](python/aibrix/AGENTS.md).\n\n## Working rules\n\n- State the intended scope and success criteria before making a non-trivial\n change.\n- Read the closest existing implementation and its tests before introducing a\n new pattern.\n- Keep changes focused. Do not combine unrelated refactors, formatting, or\n generated output with a feature or bug fix.\n- Verify behavior from code and tests rather than inferring it from filenames.\n- Preserve unrelated user changes in the working tree.\n- Do not push branches, rewrite history, modify remote GitHub state, or send\n external messages without explicit authorization for that action.\n- Ask before upgrading dependencies or changing `.github/` workflow policy.\n\n## API and compatibility surfaces\n\nTreat the following as compatibility surfaces:\n\n- Kubernetes API types, CRD schemas, JSON/YAML tags, defaults, validation,\n list semantics, subresources, and printer columns.\n- Public HTTP endpoints, request and response fields, headers, status codes,\n and gateway plugin contracts.\n- CLI flags, configuration keys, environment variables, labels, and\n annotations used by deployment or controller logic.\n\nDo not rename, remove, or repurpose an existing field or key without an\nexplicit migration and compatibility plan. Preserve pointer and `omitempty`\nchoices when absent, zero, and explicit values have different meanings.\nKeep status fields observational; status transitions must remain consistent\nwith every controller that writes or consumes them.\n\nKeep API types declarative. Admission behavior belongs in webhooks,\nreconciliation policy belongs in controllers, and transport compatibility\nbelongs in the relevant API or gateway layer.\n\n## Build and verification\n\nRun commands from the repository root. Start with the narrowest relevant check\nand run broader checks for changes that cross package or language boundaries.\n\n| Command | Purpose |\n|---------|---------|\n| `make fmt` | Format Go code |\n| `make vet` | Run `go vet ./...` |\n| `make generate` | Regenerate Go and Kubernetes generated artifacts |\n| `make manifests` | Regenerate webhook, RBAC, and CRD manifests |\n| `make verify` | Verify generated code and CRD synchronization |\n| `make lint` | Run Go linting |\n| `make lint-all` | Run license and Go lint checks |\n| `make test` | Run Go unit tests, excluding integration and E2E packages |\n| `make test-race-condition` | Run Go unit tests with the race detector |\n| `make test-integration` | Run integration tests |\n| `make python-ci` | Run Python Ruff, mypy, and tests for `python/aibrix` |\n| `make test-e2e` | Run Kind-based end-to-end tests |\n\nFor a Go package, a focused check such as `go test ./pkg//...` is\nappropriate while iterating. For cluster-level behavior, use the existing\nhelpers and test organization under `test/e2e/`; do not replace polling with\narbitrary sleeps.\n\nWhen API types, controller-gen markers, or CRDs change, run the applicable\ngeneration and verification targets and include the resulting generated files\nin the change. Do not hand-edit generated files unless the generator explicitly\nrequires it.\n\n## Coding conventions\n\n- Follow existing Go package structure and standard Go style; use `gofmt` or\n `make fmt`.\n- Put reusable implementation in `pkg/`; keep `cmd/` entrypoints thin.\n- Use table-driven tests where they match the surrounding package style.\n- Add or update tests for behavior changes, including failure and compatibility\n cases when applicable.\n- Comments should explain non-obvious reasons or invariants, not restate code.\n- Preserve existing logging, error handling, retry, timeout, and context\n propagation patterns unless the change specifically addresses them.\n- For Python changes, follow the more specific rules in\n [`python/aibrix/AGENTS.md`](python/aibrix/AGENTS.md) and run checks from that\n subtree as documented there.\n\n## Pull requests and documentation\n\n- Keep a PR focused on one issue or coherent behavior change.\n- Use the repository [PR template](.github/PULL_REQUEST_TEMPLATE.md), link the\n relevant issue, and describe the tests actually run.\n- Update user or developer documentation when behavior, configuration, API,\n deployment, or CLI usage changes.\n- When changing any `docs/source/**/*.rst` file, update the corresponding\n `docs/source/locale/zh_CN/LC_MESSAGES/**/*.po` catalog in the same change,\n including added, modified, and removed content. Do not leave new or changed\n documentation falling back to English on the Chinese site.\n- Before submitting an RST documentation change, run `make html-all` from\n `docs/` and confirm that both the English and Simplified Chinese HTML builds\n succeed.\n- Large design changes should be discussed before implementation and recorded\n in the appropriate documentation area.\n- Review every generated or AI-assisted change yourself; do not submit code\n that you cannot explain and verify.\n\n## Further reading\n\n- [Contributing guide](CONTRIBUTING.md)\n- [Development guide](development/README.md)\n- [Test guide](test/README.md)\n- [Project documentation](docs/README.md)\n","isInternal":false,"tokens":1409,"sizeBytes":6504},{"name":"denied.json","path":"pkg/plugins/gateway/algorithms/external/testdata/denied.json","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/external/testdata/denied.json","title":"Testdata Skill","category":"anthropic-skill","format":"json","content":"{\n \"apiVersion\": \"routing.aibrix.ai/v1alpha1\",\n \"kind\": \"ReplicaSelectionResponse\",\n \"metadata\": {\"requestId\": \"req-123\", \"decisionId\": \"decision-789\"},\n \"status\": {\"decision\": \"Denied\", \"reason\": \"CompliancePolicy\"}\n}\n","frontmatter":{"apiVersion":"routing.aibrix.ai/v1alpha1","kind":"ReplicaSelectionResponse","metadata":{"requestId":"req-123","decisionId":"decision-789"},"status":{"decision":"Denied","reason":"CompliancePolicy"}},"isInternal":false,"tokens":68,"sizeBytes":223},{"name":"no-decision.json","path":"pkg/plugins/gateway/algorithms/external/testdata/no-decision.json","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/external/testdata/no-decision.json","title":"Testdata Skill","category":"anthropic-skill","format":"json","content":"{\n \"apiVersion\": \"routing.aibrix.ai/v1alpha1\",\n \"kind\": \"ReplicaSelectionResponse\",\n \"metadata\": {\"requestId\": \"req-123\"},\n \"status\": {\"decision\": \"NoDecision\", \"reason\": \"NoApplicablePolicy\"}\n}\n","frontmatter":{"apiVersion":"routing.aibrix.ai/v1alpha1","kind":"ReplicaSelectionResponse","metadata":{"requestId":"req-123"},"status":{"decision":"NoDecision","reason":"NoApplicablePolicy"}},"isInternal":false,"tokens":61,"sizeBytes":199},{"name":"selected.json","path":"pkg/plugins/gateway/algorithms/external/testdata/selected.json","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/external/testdata/selected.json","title":"Testdata Skill","category":"anthropic-skill","format":"json","content":"# Testdata Skill\n\nPath: `pkg/plugins/gateway/algorithms/external/testdata/selected.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/external/testdata/selected.json)","tokens":300,"sizeBytes":241},{"name":"load_balance.md","path":"pkg/plugins/gateway/algorithms/load_balance.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/load_balance.md","title":"Algorithms Skill","category":"anthropic-skill","format":"markdown","content":"# Load Balance Router\n\n`load-balance` routes each request to the replica with the lowest **effective load**: the work\nalready committed to it, divided by how fast it has been observed to drain work, inflated as its\nKV cache fills. It needs no hardware labels. A mixed B40 / A100 / H20 pool teaches the gateway\nits relative speeds from observed output-token throughput, and faster replicas receive\nproportionally more traffic.\n\nSource: [load_balance.go](load_balance.go). Environment variables:\n[ENV_VARS.md](../ENV_VARS.md#load-balance-router-algorithmsload_balancego).\n\n`load-balance` is also blended in behind other strategies by default (see\n`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` and [multi_router_readme.md](multi_router_readme.md)),\nso its `ScoreAll` runs on most requests, not only when it is selected explicitly.\n\n## Score\n\n```\n running + λ · queued\nscore = ───────────────────────────────────── × (1 + α · (1 − kvFree)²)\n EWMA(output tokens completed / sec)\n\nkvFree < kvCritical ⇒ score = +Inf\n```\n\nThe lowest score wins. The three parts answer three separate questions, and each is used for one\npurpose only:\n\n| Part | Question | Source |\n|---|---|---|\n| `EWMA(output tokens/sec)` | How fast does this replica drain work? (**capacity**) | `RealtimeOutputTokenRateEWMA`, derived in the cache from completed output tokens |\n| `running + λ·queued` | How much work is already committed to it? (**load**) | `GetPodsRunningRequests` (live cross-gateway count); `num_requests_waiting` when `λ > 0` |\n| `1 + α(1 − kvFree)²` | How close is it to memory pressure? (**headroom**) | `kv_cache_usage_perc` (`KVCacheUsagePerc`) |\n\nCapacity and load are kept apart on purpose. A saturated GPU has high token throughput *because*\nit is busy; if throughput were used as \"load\", the scheduler would send it even more traffic.\nHere throughput only sets the denominator, and load comes from the committed work.\n\n### Defaults\n\n| Symbol | Meaning | Default | Variable |\n|---|---|---|---|\n| `λ` | Weight of engine-queued requests | `0` | `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` |\n| `α` | KV-pressure penalty strength | `2.0` | `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` |\n| `kvCritical` | Free-KV fraction below which a pod scores `+Inf` from `load-balance` (excluded when `load-balance` routes alone; only strongly penalized in a blend — see \"Interaction with the rest of the gateway\" below) | `0.10` | `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` |\n\n`λ` defaults to `0` because the gateway's running count already includes requests that are\nqueued inside the engine; adding `num_requests_waiting` on top would count them twice. Values\n`<= 0` for these variables fall back to the default (the loader rejects non-positive numbers).\n\n### Worked example\n\n| Replica | Running | Tokens/s | KV free | Calculation | Score |\n|---|---|---|---|---|---|\n| B40 | 30 | 6000 | 50% | 30/6000 × (1 + 2·0.50²) | **0.0075** |\n| A100 | 20 | 4000 | 30% | 20/4000 × (1 + 2·0.70²) | 0.0099 |\n| H20 | 25 | 3000 | 60% | 25/3000 × (1 + 2·0.40²) | 0.0110 |\n\nThe B40 wins despite carrying the most requests, because it drains them fastest. This example is\nasserted in `TestLoadBalanceScoreAll_HeterogeneousExample`.\n\n## Where the capacity signal comes from\n\n```\ngateway: DoneRequestTrace(outputTokens) ──► Pod.completedOutputTokens (monotonic counter)\n │ every metrics refresh (default 1s)\n ▼\n calculateRate1m: tokens/sec over ~1 minute window\n │\n ▼\n EWMA, time constant 10s ──► RealtimeOutputTokenRateEWMA (pod-scoped)\n```\n\n- The counter is **gateway-tracked**, like `completedRequests`, so it works for every engine\n (vLLM, SGLang, TRT-LLM, xLLM) without depending on engine metric names. Code:\n `donePodStats` in `pkg/cache/cache_trace.go`, `updateRealtimeOutputTokenRateEWMA` in\n `pkg/cache/cache_metrics.go`.\n- Output tokens are the request's completion-token usage. A request that finishes through the\n count-only path (no usage available) adds one completion and **zero** tokens.\n- The counter survives a brief pod delete/re-add (informer flap) the same way `completedRequests`\n does.\n- A window with no completed tokens carries no information about speed (the replica was idle,\n not slow), so the previous estimate is **kept**, not decayed toward zero. Decaying would make a\n briefly idle replica look weak and starve it of the traffic that would re-measure it.\n\n## Behavior in edge cases\n\n| Situation | Behavior |\n|---|---|\n| Pod has no token-rate estimate yet (new pod, or no usage reported) | Scored at the **mean** estimate of the pods that have one, so it competes as an average replica. A fixed fallback such as `1.0` would be off by the tokens/sec scale (thousands) and lock the pod out of the traffic it needs to be measured. |\n| No pod has an estimate | Every pod gets capacity `1.0`; the score reduces to `running × KV penalty`. |\n| Engine reports no KV usage, or `NaN` | Treated as fully free: no penalty and no guardrail. |\n| `kvFree < kvCritical` | Score `+Inf` from `load-balance`. When `load-balance` routes alone this skips the pod entirely until it recovers. When `load-balance` is blended with other strategies (the default — see below), it is only a strong penalty, not a guaranteed exclusion. |\n| **Every** pod is below `kvCritical` | The request is **not** failed. Routing falls back to the tie-break over all pods, which picks the one with the least KV usage. This fallback is specific to `load-balance` routing alone; see below for the blended case. |\n| Pod is idle (`running = 0`) | Score `0` regardless of capacity, so idle pods are preferred. |\n| Several pods share the lowest score | Ties are broken by least combined GPU+CPU KV-cache usage (`least-kv-cache` scorer), falling back to random. |\n\nKV usage is clamped to `[0, 1]` before use, so an engine that reports a value above 1 is treated\nas full rather than producing a negative free fraction.\n\n## Interaction with the rest of the gateway\n\n- **Load-imbalance gate.** The gateway applies `ApplyLoadImbalanceGate` once, ahead of whichever\n strategy routes the request, so `Route` receives an already-narrowed pod list. If that narrowed\n set is entirely KV-critical, the all-critical fallback above applies. **TODO / known\n limitation:** the gate compares raw running-request count, with no notion of a pod's capacity —\n a replica that's simply faster than its peers is expected to carry more concurrent requests\n without being more loaded (the same capacity signal `load-balance`'s own score uses above), but\n this gate can still flag it as a hotspot and exclude it before `load-balance`'s own\n capacity-aware score ever gets to run. See `getTargetPodListOnLoadImbalance` in\n [load_balance.go](load_balance.go).\n- **Multi-strategy blending: the guardrail is soft here, not hard.** `kvCritical` is a hard\n exclusion only when `load-balance` routes alone. In a blend, `ScoreAll` still returns `+Inf` for\n a critical pod, but `multiStrategyRouter.normalizeScoresArray` treats a non-finite score the\n same as \"not scored\" and assigns it `0` — the worst possible score from `load-balance`, not an\n exclusion from routing. If another blended strategy scores that same pod highly enough, it can\n still be selected.\n\n Concretely: `load-balance` is blended in behind other strategies by default (see\n `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT`), so the blended path, not the standalone one, is\n what most requests actually take. With the default `prefix-cache:5,load-balance:4` blend, a\n KV-critical pod that has the best prefix-cache score still gets `5/9` of the combined weight\n versus `4/9` for a healthy pod that only wins on `load-balance`, so the critical pod wins the\n route. `kvCritical` should therefore be read as \"how hard `load-balance` argues against routing\n to this pod,\" not as a safety limit that blocks it from being chosen — it does not, on its own,\n prevent a KV-critical pod from receiving more traffic. Something that must never route to a\n KV-critical pod needs a filter ahead of scoring (comparable to `ApplyLoadImbalanceGate`), which\n does not currently exist for KV pressure.\n- **PD disaggregation.** The PD router's decode path keeps its own request drain-rate scoring\n (`RealtimeRunningRequestsDrainRate1m`) and is unchanged.\n\n## Known limitations\n\n- **Multiple gateway replicas.** The token counter is local to each gateway, while the running\n count is the cross-gateway total. Each gateway therefore measures roughly `1/N` of a pod's\n output. Relative ranking between pods is preserved as long as the gateways spread traffic\n similarly, but absolute score values are not comparable across deployments with different\n gateway counts.\n- **Estimate lag.** The estimate follows a ~1 minute window smoothed with a 10 second time\n constant, so a sudden change in a replica's speed (for example a workload shifting from\n decode-heavy to prefill-heavy) shows up within tens of seconds, not immediately.\n- **No prefill/decode split.** Capacity is a single output-token rate. Hardware whose relative\n performance differs between prompt-heavy and decode-heavy workloads is not modeled; a natural\n extension is `capacity = α·prefill_tokens/s + β·decode_tokens/s`.\n- **Deliberately not used:** GPU utilization / SM-active. They are useful for observability but\n make a generic scheduler unstable; the score uses only drain rate, committed work, and KV\n pressure.\n\n## Debugging\n\nAt `-v=4` the gateway logs `load_balance_score` once per pod (`running_requests`,\n`capacity`, `kv_free`, and the resulting `score`) and `load_balance_selected` (with the\nwinning `score`). `+Inf` scores are logged as strings so klog's JSON formatter does not\nfail. The cache logs `Updating output token rate metric` with the completed token count,\nthe 1-minute rate, and the EWMA for each pod. The estimate is stored on the pod as\n`realtime_output_token_rate_ewma`; it is not currently exported as a Prometheus series.\n","isInternal":false,"tokens":2528,"sizeBytes":10326},{"name":"multi_router_readme.md","path":"pkg/plugins/gateway/algorithms/multi_router_readme.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/multi_router_readme.md","title":"Algorithms Skill","category":"anthropic-skill","format":"markdown","content":"# Multi-Strategy Routing (Batch Soft Scoring)\n\nAIBrix Gateway supports multi-strategy routing, allowing you to combine multiple routing algorithms with different weights. Instead of making hard decisions (where one strategy completely filters out pods or dictates the single winner), the multi-strategy router uses a **Batch Soft Scoring** mechanism to evaluate pods across all specified strategies.\n\n## How it works\n\n1. **Batch Scoring:** Each configured strategy evaluates all ready pods in a single batch operation (`ScoreAll`), returning raw scores and a boolean array indicating whether each pod could be scored (e.g., handling missing metrics).\n2. **Normalization:** The raw scores from each strategy are normalized into a `[0, 1]` range:\n * **Higher is better** (e.g., `throughput`): `(score - min) / (max - min)`\n * **Lower is better** (e.g., `least-request`): `(max - score) / (max - min)`\n * If a pod lacks metrics (`scored=false`), its normalized score is `0.0` (acting as a penalty).\n3. **Weighted Aggregation:** The normalized scores are multiplied by their respective configured weights and summed up for each pod.\n4. **Top-1 Selection:** The pod with the highest total aggregated score is selected to serve the request.\n\n## Configuration Format\n\nYou can configure multi-strategy routing by setting the `AIBRIX_ROUTING_ALGORITHM` environment variable using a comma-separated list of `strategy:weight` pairs.\n\n* **Format:** `strategy1:weight1,strategy2:weight2,...`\n* If `:weight` is omitted, it defaults to `1`.\n* If a weight is `0`, the strategy is completely ignored.\n* If an invalid strategy is specified, the Gateway rejects the request with **400 Bad Request** instead of silently falling back to `random`.\n* `session-affinity` is treated as a soft-scoring strategy in multi-strategy mode: it boosts the matching pod, but the final weighted winner may still be a different pod. The response session header is updated to the final selected pod.\n With Redis-backed pinning, background sync applies its result only if the local entry has not changed since the read began. A delayed result cannot overwrite or evict a pin committed by a concurrent request.\n* A caller may send an opaque application session identifier in `x-aibrix-session-key`. When no valid gateway-issued `x-session-id` target is ready, rendezvous hashing selects a stable pod for that key. This is useful for concurrent Agent turns that start before a response cookie is available. The opaque key is never returned to the client. The `pd` router's `token_load` and `hybrid_cache_load` prefill policies read the same key to charge a multi-turn conversation only for the tokens added since its previous turn.\n\n## Examples\n\nHere are some real-world examples demonstrating how the weighted aggregation works under different scenarios.\n\nAssume we have two pods serving a model:\n* **Pod A**: High load (10 requests) but High throughput (100 tok/s).\n* **Pod B**: Low load (2 requests) but Low throughput (10 tok/s).\n\n### Example 1: `least-request` Dominates\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:10,throughput:1\"\n```\n* `least-request` (weight 10): Pod B is much better (2 vs 10). Normalized: Pod A = 0.0, Pod B = 1.0. Weighted Score: A = 0.0 * 10/11 = 0.0, B = 1.0 * 10/11 ≈ 0.91.\n* `throughput` (weight 1): Pod A is much better (100 vs 10). Normalized: Pod A = 1.0, Pod B = 0.0. Weighted Score: A = 1.0 * 1/11 ≈ 0.09, B = 0.0 * 1/11 = 0.0.\n* **Total Score**: Pod A ≈ 0.09, Pod B ≈ 0.91. **Pod B wins!**\n\n### Example 2: `throughput` Dominates\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,throughput:10\"\n```\n* `least-request` (weight 1): Weighted Score: A = 0.0 * 1/11 = 0.0, B = 1.0 * 1/11 ≈ 0.09.\n* `throughput` (weight 10): Weighted Score: A = 1.0 * 10/11 ≈ 0.91, B = 0.0 * 10/11 = 0.0.\n* **Total Score**: Pod A ≈ 0.91, Pod B ≈ 0.09. **Pod A wins!**\n\n### Example 3: Complex Scenario with Corner Cases (3 Strategies)\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request,throughput:2,least-latency:0\"\n```\n* `least-request`: No weight specified, **defaults to 1**. Weighted Score: A = 0.0 * 1/3 = 0.0, B = 1.0 * 1/3 ≈ 0.33.\n* `throughput`: Explicitly set to weight **2**. Weighted Score: A = 1.0 * 2/3 ≈ 0.67, B = 0.0 * 2/3 = 0.0.\n* `least-latency`: Weight set to **0**, so this strategy is **ignored** and does not participate in scoring.\n* **Total Score**: Pod A ≈ 0.67, Pod B ≈ 0.33. **Pod A wins!**\n\n### Example 4: Invalid Configuration Fallback\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,invalid-strategy:1\"\n```\n* Because `invalid-strategy` is not a registered algorithm, the multi-strategy parser will fail. To preserve the API contract, the Gateway will reject the request with a **400 Bad Request** error rather than silently falling back to a random router.\n\n### Example 5: Multi-Strategy Finding the Sweet Spot\nIn reality, a pod with the absolute minimum active requests might have just recovered from being severely overloaded. Multi-strategy routing prevents routing to extreme outliers by considering multiple dimensions.\n\nAssume three pods with the following metrics:\n* **Pod A**: Very low active requests (1 req), but historically massively overloaded (1500 tokens).\n* **Pod B**: Historically barely used (15 tokens), but currently slammed with active requests (50 req).\n* **Pod C**: The \"Sweet Spot\" - Medium active requests (10 req), and medium historical load (150 tokens).\n\n**Single Strategy Flaws:**\n* `AIBRIX_ROUTING_ALGORITHM=\"least-request\"`: Completely ignores historical load and picks **Pod A** (which is exhausted).\n* `AIBRIX_ROUTING_ALGORITHM=\"throughput\"`: Completely ignores current spikes and picks **Pod B** (which is currently slammed).\n\n**Multi-Strategy Solution:**\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,throughput:1\"\n```\n* `least-request` (weight 1): Normalizes active requests [1, 50]. \n * Pod A: 1.0\n * Pod B: 0.0\n * Pod C: 0.816\n* `throughput` (weight 1): Normalizes historical load [15, 1500].\n * Pod A: 0.0\n * Pod B: 1.0\n * Pod C: 0.909\n* **Total Score (Weighted)**: Pod A = 0.5, Pod B = 0.5, Pod C ≈ **0.86**. \n* **Result**: **Pod C wins!** The router correctly identifies the most balanced pod, avoiding the exhausted Pod A and the currently slammed Pod B.\n","isInternal":false,"tokens":1730,"sizeBytes":6249},{"name":"pd_readme.md","path":"pkg/plugins/gateway/algorithms/pd_readme.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/pd_readme.md","title":"Algorithms Skill","category":"anthropic-skill","format":"markdown","content":"# PD Disaggregation Router\n\n`pd/router/router.go` implements the **prefill-decode (PD) disaggregated routing algorithm** (`RouterPD = \"pd\"`). **This file (`pd_readme.md`) is the canonical documentation** for PD: request flow, scoring, engine behavior, metrics, and environment variables. PD-only settings are under **Environment Variables**; variables shared with the prefix-cache router (tokenizer, stddev factor, and related `AIBRIX_PREFIX_CACHE_*` knobs) are listed in **Inherited from Prefix Cache Router** in the same section.\n\nIn PD disaggregation, inference is split across two specialized pod roles:\n\n- **Prefill pod** — processes the prompt (context), builds the KV-cache, then transfers it.\n- **Decode pod** — generates tokens using the KV-cache transferred from the prefill pod.\n\nThe router is responsible for selecting one prefill pod and one decode pod per request, executing the prefill HTTP request synchronously (vLLM/TRT-LLM context-first) or asynchronously (SGLang/TRT-LLM generation-first), and routing the decode request through Envoy to the selected decode pod.\n\n---\n\n## High-Level Request Flow\n\n```\nClient Request\n │\n ▼\nRoute(ctx, readyPodList)\n │\n ├─► ValidateAndGetLLMEngine()\n │ Ensure all pods use the same engine (vllm / sglang / trtllm)\n │\n ├─► filterPrefillDecodePods()\n │ ┌─────────────────────────────────────────────────────┐\n │ │ collectAndBucketPods() │\n │ │ Phase 1: group by roleset (prefill / decode) │\n │ │ Phase 2: apply prompt-length bucketing filter │\n │ └─────────────────────────────────────────────────────┘\n │ │\n │ ├─► [Prompt-length bucketing ON]\n │ │ → combined pod path (see Combined Pod section)\n │ │\n │ ├─► loadImbalanceSelectPrefillPod() (fast path)\n │ ├─► loadImbalanceSelectDecodePod() (fast path)\n │ ├─► scorePrefillPods()\n │ ├─► scoreDecodePods()\n │ └─► finalPDScore() → (prefillPod, decodePod)\n │\n ├─► AddPendingDecode() + defer RemovePendingDecode() (all paths)\n │\n ├─► [prefillPod != nil]\n │ AddPrefillRequest()\n │ doPrefillRequest(ctx, prefillPod, handler)\n │ ├─ SGLang → async goroutine (bootstrap handshake; Route does not wait for completion)\n │ ├─ vLLM → sync, extract kv_transfer_params from response\n │ └─ TRT-LLM → context_first: sync, merge response params\n │ generation_first: prepare both bodies, async prefill\n │\n └─► ctx.SetTargetPod(decodePod)\n return decodePod address\n```\n\n---\n\n## Pod Classification\n\nPods are grouped using Kubernetes labels:\n\n| Label | Values | Purpose |\n|-------|--------|---------|\n| `roleset-name` | any string | Groups prefill+decode pairs into a \"roleset\" |\n| `role-name` | `prefill` / `decode` / `encode` / other | Pod's role in PD disaggregation. `encode` pods are optional and only take the encode stage (see [vLLM encode stage (EPD)](#vllm-encode-stage-epd)) |\n| `stormservice.orchestration.aibrix.ai/pod-group-index` | `\"0\"` or absent | Only pods with index `\"0\"` (or no label) host the HTTP server (multi-node TP) |\n| `stormservice.orchestration.aibrix.ai/role-replica-index` | any string | Distinct replica index within a role; used by `roleReplicaCardinality` to count unique replicas across multi-node TP groups. When absent, each routable pod counts as its own replica. |\n| `model.aibrix.ai/engine` | `vllm` / `sglang` / `trtllm` | LLM engine type |\n\nOnly rolesets with **both** at least one prefill replica and one decode replica are eligible. Incomplete rolesets are excluded. See [Roleset Eligibility Utilities](#roleset-eligibility-utilities) for diagnostic functions.\n\n---\n\n## Roleset Eligibility Utilities\n\n`pd/router/roleset.go` provides helper functions used by both the router and the gateway's pod-readiness logic.\n\n### Replica cardinality\n\nA roleset's replica count is the number of **distinct** values of `stormservice.orchestration.aibrix.ai/role-replica-index`. When that label is absent on all pods of a role, each routable pod counts as its own replica. This correctly handles multi-node tensor-parallel groups where multiple pods share one logical replica slot.\n\n```\nroleReplicaCardinality(pods):\n if any pod has role-replica-index label:\n count distinct index values\n else:\n count routable pods\n```\n\n### Eligibility checks\n\nA roleset is **eligible** when it has ≥1 prefill replica **and** ≥1 decode replica.\n\n| Function | Use |\n|----------|-----|\n| `HasEligibleRoleset(readyPods)` | Gateway check: at least one roleset is fully ready for PD routing |\n| `eligiblePDRolesets(readyPods)` | Internal: returns only complete rolesets for scoring |\n| `DescribePDRolesetEligibility(readyPods)` | Diagnostics: per-roleset replica counts and eligibility flag |\n\n`ValidateAndGetLLMEngine` (called from the gateway before `Route`) checks that all ready pods for the model share the same `model.aibrix.ai/engine` label.\n\n---\n\n## Pod Selection Flow\n\n```\ncollectAndBucketPods()\n │\n ▼\n┌───────────────────────────────┐\n│ prefillPods / decodePods │\n│ (grouped by eligible │\n│ rolesets only) │\n└───────────────────────────────┘\n │\n ▼\nloadImbalanceSelectPrefillPod()\n ┌─────────────────────────────────────────────────────────┐\n │ max(outstanding_prefill_reqs) - min > MIN_SPREAD? │\n │ → YES: narrow to single least-loaded prefill pod │\n │ align decodePods to selected pod's roleset │\n │ (continues to decode fast path below) │\n │ → NO: continue with all prefill pods │\n └─────────────────────────────────────────────────────────┘\n │\n │ NOTE: the two fast paths are independent. Both can fire on the\n │ same request if both sides are imbalanced: prefill narrows first,\n │ then decode narrows within the already-aligned decode set.\n │\n ▼\nloadImbalanceSelectDecodePod() (3 ordered checks, metrics-bearing pods only)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Effective request count = RealtimeNumRequestsRunning │\n │ + PendingDecodeTracker count │\n │ Pending counts bridge the gap between pod selection and the │\n │ first metric update, preventing thundering-herd on a newly │\n │ selected pod whose running count has not yet ticked up. │\n │ │\n │ Check 1 – Request count spread │\n │ Only pods with RealtimeNumRequestsRunning metric participate │\n │ (pods without the metric are excluded from spread; their │\n │ pending count is still stored for scoring — prevents │\n │ thundering-herd on freshly restarted pods) │\n │ max(observed running+pending) - min >= DECODE_LOAD_SPREAD? │\n │ → YES: return least-loaded observed pod; │\n │ align prefillPods to selected pod's roleset │\n │ │\n │ Check 2 – Throughput spread │\n │ Only pods with AvgGenerationThroughputToksPerS participate │\n │ max(observed throughput) - min > THROUGHPUT_SPREAD? │\n │ → YES: return lowest-throughput observed pod │\n │ │\n │ Check 3 – Drain rate score (soft path) │\n │ all pods have drain_rate > 0? │\n │ score = effective_running_reqs / drain_rate │\n │ max_score / min_score > DECODE_SCORE_RATIO? │\n │ → YES: return pod with lowest drain-rate score │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nscorePrefillPods() (per roleset, best pod wins)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Skip pods with req_count > mean + N * stddev │\n │ (N = AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR) │\n │ Skip pods with req_count > max(c, 1) * (mean + 1) │\n │ (c = AIBRIX_PREFILL_LOAD_CAP_FACTOR) │\n │ When all pods in a roleset are filtered, that roleset has no │\n │ entry in the output map and finalPDScore cannot select it. │\n │ │\n │ Policy: prefix_cache (default) │\n │ score = (100 - match_pct) * 0.1 + req_cnt / max_req_cnt │\n │ │\n │ Policy: least_request │\n │ score = req_cnt │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nscoreDecodePods() → pd.DecodeScoreRun (per roleset best pod + MaxScore + policy metadata)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Cold-start (if some pods have metrics but this pod does not): │\n │ score = 1.0 + pending_decode_count │\n │ (neutral idle-warm score; pod competes but is not favored) │\n │ │\n │ Policy: load_balancing (default) │\n │ norm_reqs = running_reqs / max_running_reqs │\n │ norm_thru = 1 - throughput / max_throughput │\n │ norm_free_gpu = free_gpu_pct / max_free_gpu_pct │\n │ score = (w_run*norm_reqs + w_thru*norm_thru) / norm_free_gpu │\n │ (w_run, w_thru from AIBRIX_DECODE_LB_WEIGHT_*, default 1) │\n │ NaN scores: skip; non-NaN Inf possible if norm_free_gpu = 0 │\n │ │\n │ Policy: least_request │\n │ score = running_reqs (with pending) │\n │ │\n │ If primary policy yields NaN, one load_balancing retry per pod │\n │ lower is better │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nfinalPDScore()\n ┌──────────────────────────────────────────────────────────────────┐\n │ For each roleset with both prefill and decode scores: │\n │ final = norm_prefill_score + norm_decode_score │\n │ Pick roleset with minimum final score │\n │ → selectedPrefillPod, selectedDecodePod │\n │ │\n │ Errors: │\n │ • prefillScores nil → Prepare failed in scorer │\n │ • prefillScores empty → all prefill pods filtered by │\n │ stddev or cap (no candidate) │\n │ • no roleset intersection → load-imbalance alignment left │\n │ prefill/decode on different sets │\n └──────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Prefill Score Policies\n\nControlled by `AIBRIX_PREFILL_SCORE_POLICY`, or overridden per request via the model **`routingConfig`** in `model.aibrix.ai/config` (see [Config profile overrides](#config-profile-overrides-for-pd-score-policies)).\n\n### `prefix_cache` (default)\n\nUses the shared `PrefixHashTable` to find how much of the request's token prefixes are already cached on each prefill pod.\n\n```\ntokens = tokenize(request.message)\nmatchedPods, hashes = prefixCacheIndexer.MatchPrefix(tokens, model, readyPodsMap)\n\nscore = (100 - match_percent) * 0.1 + req_count / max_req_count\n```\n\n- Lower score = better (more cache hits, fewer outstanding requests).\n- After routing, the prefix hashes are enqueued to `prefixUpdateCh` for asynchronous indexer update.\n- Tokenizer is selected by `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` (same as the prefix-cache router): `tiktoken` or `character` (default).\n\n### `least_request`\n\nNo prefix cache consultation. Score equals the raw running request count on the pod.\n\n```\nscore = req_count\n```\n\n---\n\n## Decode Score Policies\n\nImplementation lives in `algorithms/pd/decode_scorer.go` (`package pd`). Policies are selected by **`pd.ResolveDecodePolicy`** from `AIBRIX_DECODE_SCORE_POLICY`. Extra names can be registered at init time with **`pd.RegisterDecodePolicy`**. Per-pass results use **`pd.DecodeScoreRun`** (map of roleset → best pod/score, `MaxScore` for normalization, optional `Err`, `FallbackUsed`, `Policy`).\n\nMetrics are passed per pod via **`pd.DecodePodInput`** (running/throughput/free GPU and batch maxima). The router guarantees positive maxima from `loadImbalanceSelectDecodePod` as today.\n\nControlled by `AIBRIX_DECODE_SCORE_POLICY`, or overridden per request via **`routingConfig`** in the model config profile (see [Config profile overrides](#config-profile-overrides-for-pd-score-policies)).\n\nMetrics are pulled from the cache per pod (same maps as `loadImbalanceSelectDecodePod`):\n- `RealtimeNumRequestsRunning` + `PendingDecodeTracker` count\n- `AvgGenerationThroughputToksPerS` (per model)\n- `KVCacheUsagePerc` (free = 100 − usage%)\n\n**Cold-start gate**: if at least one pod has `RealtimeNumRequestsRunning`, pods that lack it are given a neutral **cold-start score** (`1.0 + pending_decode_count`) rather than full policy scoring. When *no* pod has that metric yet, all pods are scored normally.\n\n### `load_balancing` (default)\n\nBalances normalized queue depth, inverse throughput, and free GPU headroom:\n\n```\nnorm_running_reqs = running_reqs_with_pending / max_running_reqs\nnorm_throughput = 1 - avg_gen_throughput_tok_s / max_throughput\nnorm_free_gpu = free_gpu_percent / max_free_gpu_percent\n\ndecode_score = (w_run * norm_running_reqs + w_thru * norm_throughput) / norm_free_gpu\n```\n\n`w_run` and `w_thru` default to `1` (historical equal weighting). Lower score is better (fewer outstanding requests, higher throughput, more free GPU).\n\n### `least_request`\n\nNo throughput or GPU terms. Score equals raw running decode request count (including pending decode), same units as `RealtimeNumRequestsRunning` + pending.\n\n```\ndecode_score = running_reqs_with_pending\n```\n\n### `token_load`\n\nScores from the gateway's decode ledger in `pd.TokenLoadTracker` rather than from request counts or scraped KV usage. Each request is charged its `prompt_tokens` to the selected decode pod, under `selectMu` together with the other selection bookkeeping, and released on request completion or prefill failure (`releaseTokenLoad`, or the prefill executor for a terminal async failure). The whole prompt is charged because the decode pod receives all of its KV; there is no fixed per-request cost (`AIBRIX_TOKEN_LOAD_REQUEST_COST` is prefill-only).\n\n```\ndecode_score = decode_tokens + generated_tokens\ngenerated_tokens = rate × Σ over outstanding charges (now − routed_at)\nrate = AvgGenerationThroughputToksPerS / running requests (mean of the other pods if unknown)\n```\n\nWhen the prefill response carries `usage.prompt_tokens`, the prefill executor replaces the decode charge with it (`TokenLoadTracker.CorrectDecodeCost`), since the body-size estimate is far off for requests with media. `generated_tokens` estimates the output the pod's outstanding requests have produced so far (it sits in the decode KV too); `pd.TokenLoadTracker.DecodeGrowth` computes it in O(1) per pod from the count and the summed charge times of the outstanding charges. `AIBRIX_DECODE_TOKEN_LOAD_OUTPUT_GROWTH=false` scores prompt tokens only. The cold-start score does not apply to `token_load`: a pod without metrics is scored from the ledger like any other. The decode load-imbalance fast path is skipped under `token_load` (it still runs to fill the metric maps): it picks by request count, throughput or drain rate, and the pod holding one long prompt has the fewest requests. With Redis configured, the decode ledger is shared across gateway replicas (`AIBRIX_TOKEN_LOAD_SHARED_LEDGER`, default on): each replica publishes its per-pod charges, count and summed charge times under its instance ID (`cache.Store.PublishDecodeLedger`), and `scoreDecodePods` adds the other live replicas' values, read with the running-request counts (`GetPodsRunningRequestsAndDecodeLedger`). Without Redis the ledger is local to each replica. Each decode pod's `token_load` score is then multiplied by the candidates' mean KV capacity over its own (`decodeKVCapacities`, `AIBRIX_DECODE_TOKEN_LOAD_CAPACITY_AWARE`, default on), with capacities from the scraped `kv_cache_num_gpu_blocks` × `kv_cache_block_size` (vLLM `vllm:cache_config_info` labels) or `kv_cache_capacity_tokens` (SGLang); unknown capacities count as the mean.\n\n### Config profile overrides for PD score policies\n\nWhen the gateway resolves a model config profile (`routingCtx.ConfigProfile` from `model.aibrix.ai/config` and the `config-profile` header), the **`routingConfig`** JSON may include:\n\n| Field | Values | Effect |\n|-------|--------|--------|\n| `prefillScorePolicy` | `prefix_cache`, `least_request` | Overrides `AIBRIX_PREFILL_SCORE_POLICY` for that request |\n| `decodeScorePolicy` | `load_balancing`, `least_request`, `conductor`, `token_load` (or any name registered via `pd.RegisterDecodePolicy`) | Overrides `AIBRIX_DECODE_SCORE_POLICY` for that request |\n\nExample fragment inside a profile:\n\n```json\n\"routingConfig\": {\n \"promptLenBucketMinLength\": 0,\n \"promptLenBucketMaxLength\": 8192,\n \"prefillScorePolicy\": \"prefix_cache\",\n \"decodeScorePolicy\": \"least_request\"\n}\n```\n\nOmitted fields keep the gateway defaults from environment variables.\n\n---\n\n\n## Engine-Specific Prefill Behaviour\n\n### vLLM encode stage (EPD)\n\nWhen the selected prefill pod's roleset has `role-name: encode` pods and a vLLM `/v1/chat/completions` request carries media items (`image_url`, `audio_url`, `input_audio`, `video_url`), `Route` runs an encode stage before the prefill call:\n\n```\nGateway ──POST /v1/chat/completions {messages:[{role:user, content:[item i]}]}──► Encode Pod (one per item, in parallel)\n ◄── 200 ──────────────────────────────────────────────────────────────── Encode Pod (output stored by the EC connector)\n\nGateway ──prefill as below, request body unchanged──────────────────────────────► Prefill Pod (finds the output by the item's hash)\n```\n\nItems are spread round-robin over the roleset's encode pods. The body is not rewritten: the encoder and the prefill pod hash the same item bytes, so the prefill pod's EC connector finds the output, and encodes the item itself when it can't. A failed encode is therefore logged and counted, not returned. Each call uses the prefill request timeout. Connectors that need transfer handles passed through the request (vLLM's `*_embeds` rewrite in `disagg_epd_proxy.py`) are not supported.\n\nWhen bucketing routes the request to a combined pod instead, `Route` runs the same stage on the combined pod's roleset. For E+PD rolesets, where encode pods sit in front of pods that serve the whole request, the gateway runs it outside the PD router: with any other routing strategy, `selectTargetPod` drops encode pods from the candidates and encodes on the chosen pod's roleset before returning it.\n\n### vLLM (`AIBRIX_KV_CONNECTOR_TYPE=shfs`, default)\n\n```\nGateway ──POST /v1/... (max_tokens=1, kv_transfer_params={do_remote_decode:true})──► Prefill Pod\n ◄── response {kv_transfer_params: {remote_block_ids, ...}} ──────────────── Prefill Pod\n\nGateway updates ReqBody:\n kv_transfer_params.remote_host = prefill_pod_ip\n\nGateway ──POST /v1/... (with kv_transfer_params)──────────────────────────────────► Decode Pod\n```\n\n### vLLM (`AIBRIX_KV_CONNECTOR_TYPE=nixl`, Neuron)\n\n```\nGateway ──POST /v1/... (max_tokens=1)──────────────────────────────────────────────► Prefill Pod\n ◄── response { entire prefill response } ──────────────────────────────── Prefill Pod\n\nGateway updates ReqBody:\n disagg_prefill_resp = \n\nGateway ──POST /v1/... (with disagg_prefill_resp)──────────────────────────────────► Decode Pod\n```\n\n### SGLang (async)\n\n```\nGateway modifies ReqBody:\n bootstrap_host = prefill_pod_ip\n bootstrap_port = pod annotation / default 8998\n bootstrap_room = random int63\n rid = -<16 hex nonce> (identical on both legs)\n\nGateway ──POST (async goroutine)──────────────────────────────────────────────────► Prefill Pod\n (no wait; prefill/decode coordinate via bootstrap protocol)\n\nGateway ──POST /v1/... (updated body)─────────────────────────────────────────────► Decode Pod\n```\n\nFor SGLang, `Route()` returning a decode pod means the gateway has dispatched\nthe asynchronous prefill worker, not that the prefill HTTP request has already\ncompleted. Prefill success/failure metrics are emitted by the async worker after\nthe prefill HTTP request returns.\n\n`rid` is the engine-visible request id of the attempt. SGLang reads it from the\nrequest body (the `X-Request-Id` header is ignored) and its `/abort_request`\nendpoint matches live requests by rid alone, so injecting the same gateway-owned\nrid into both legs is what makes the decode leg cancellable. The rid always ends\nwith a fixed-width nonce so that no gateway rid can be a proper prefix of\nanother - SGLang aborts by rid prefix - and so that a client retrying with the\nsame request id (or the same traceparent) still gets a distinct rid per attempt.\nSee [PD Prefill Fail-Fast](#pd-prefill-fail-fast).\n\n### TensorRT-LLM\n\n`AIBRIX_TRT_SCHEDULE_STYLE=context_first` (default) preserves the sequential flow:\n\n```\nGateway adds disaggregated_params to prefill request:\n { request_type: \"context_only\", disagg_request_id: }\n\nGateway ──POST (sync)──────────────────────────────────────────────────────────────► Prefill Pod\n ◄── response { disaggregated_params: {...}, prompt_token_ids: [...] } ── Prefill Pod\n\nGateway updates ReqBody:\n disaggregated_params.request_type = \"generation_only\"\n prompt / prompt_token_ids from prefill response\n\nGateway ──POST /v1/... (with disaggregated_params)─────────────────────────────────► Decode Pod\n```\n\nTRT-LLM uses a Snowflake-style `disagg_request_id` (63-bit) to correlate prefill and decode:\n\n```\n[41-bit timestamp ms since 2023-01-01] [10-bit machine_id] [12-bit counter]\n```\n\nMachine ID is set via `AIBRIX_TRT_MACHINE_ID` (must be in `[0, 1024)`). Assign different machine IDs to gateway processes sharing the same TRT workers.\n\n#### Generation-first (parallel)\n\nSet `AIBRIX_TRT_SCHEDULE_STYLE=generation_first` **on the gateway plugin**, not on\nCTX/GEN workers. The setting is fixed when the PD router is constructed; an\nunknown value is logged and leaves the router on `context_first`. It does not\naffect other engines or combined pods. No automatic fallback or retry occurs after\neither leg has been dispatched.\n\nPrerequisites (TRT-LLM `1.3.0rc8`, the Python KV-cache transceiver on both roles,\nmatching model/tokenizer/chat-template) are listed in\n`docs/source/features/gateway-routing/pd-disaggregation.rst`.\n\n```\nGateway selects CTX + GEN\n ├─ GET CTX /server_info (cached)\n ├─ prepare both bodies with the same integer disagg_request_id\n ├─ goroutine: POST context_only, schedule_style=1, stream=false, max_tokens=1\n └─ return GEN address and generation_only body to Envoy immediately\n disagg_request_id = ctx_request_id = shared ID\n ctx_info_endpoint, ctx_dp_rank, optional encoded_opaque_state = CTX metadata\n schedule_style = 1 (TRT-LLM GENERATION_FIRST wire enum)\n original prompt/messages, sampling settings and stream mode preserved\n\nCTX <── engine-managed coordination / KV transfer ──> GEN\nEnvoy <──────────────────── response / SSE ────────── GEN\n```\n\nThere is no second Go-issued decode POST. GEN can initialize while CTX is still\nworking; it cannot generate without the required KV. The CTX response is not merged\nback into the already-dispatched GEN request. Both endpoints must therefore use\nmatching model/tokenizer/chat-template configuration, since GEN cannot reuse\n`prompt_token_ids` from the CTX HTTP response in this mode.\n\n**Worker discovery:** each router loads the selected CTX worker's `/server_info`\nlazily before either leg is dispatched, so newly discovered or autoscaled pods work\nwithout a gateway restart. Concurrent misses for one incarnation are coalesced;\nlookups have a 3-second timeout and a 1-minute TTL, and responses are capped at\n1 MiB. Pod UID, address and restart count identify an incarnation. A missing or\ninvalid `/server_info` fails routing before dispatch, and failed lookups are not\ncached.\n\nThe expected HTTP response follows TRT-LLM's `1.3.0rc8` OpenAI server schema:\n\n```json\n{\"disaggregated_params\":{\"ctx_info_endpoint\":\"tcp://:\",\"ctx_dp_rank\":0}}\n```\n\n`ctx_info_endpoint` must be a single string, which is what the Python transceiver\nreturns; any other JSON shape fails the lookup.\n\n`encoded_opaque_state` is also propagated when present. Only these fields are copied\nfrom `/server_info`: other keys are ignored so worker metadata can never overwrite\nthe gateway-owned `request_type`, IDs or `schedule_style`. A field the handshake\nrequires but this allowlist does not carry must be added to the handler explicitly.\nThe endpoint must be reachable from GEN. `ctx_dp_rank` must be explicitly present and nonnegative; it is **attention\nDP rank**, not TP rank, replica index, or Pod number. Rank zero is valid but is never\nsubstituted for missing data. Metadata is taken from the exact selected Pod. A\nfront-end that distributes requests across multiple DP ranks without stable\nworker/rank affinity is not supported by this Pod-level cache: expose rank-affine\nworkers or retain `context_first`. Validate nonzero-rank deployments against the\nactual engine before enabling this mode.\n\n**Failure handling:** the CTX HTTP call retains client cancellation and the existing\nprefill timeout. Failure wakes the gateway through the per-incarnation PD leg state,\nnever through a recycled RoutingContext. TRT-LLM has no SGLang `rid`/`/abort_request`\ncontract; the gateway instead fails/resets the Envoy stream, relying on TRT-LLM's\nHTTP-disconnect cancellation. Keep Envoy `failure_mode_allow=false`. TRT's SSE headers\ncan precede KV arrival, so a terminal CTX failure after response headers resets the\nstream rather than being ignored. This can also truncate an already-generating\nresponse; it never attempts to rewrite headers already sent to the client.\n\nStart validation with text-only 1P1D using the sample under\n`samples/quickstart/tensorrt/`; its README carries the smoke test and the rollback\ncommand.\n\n---\n\n## PD Prefill Fail-Fast\n\nOnly asynchronous modes (SGLang and TRT generation-first) can fail their prefill leg after `Route()`\nhas already returned a decode pod. Without fail-fast the client waits for the\ndecode leg to give up on a KV transfer that will never arrive - 300s in SGLang -\nand the decode pod holds its pre-allocated KV pages for that whole window.\n\n```\nPrefill leg fails (async worker)\n │\n ├─► record failure on the RoutingContext's PD leg state (first failure wins)\n │\n ├─► SGLang: POST /abort_request {\"rid\": ...} to decode (goroutine, twice)\n │ TRT: no abort POST; reset Envoy upstream on failure\n │\n └─► wake the ext_proc stream ──► 5xx ImmediateResponse to the client\n header x-error-pd-prefill: true\n```\n\nFailure classes, as reported in the metrics and in the client error message:\n\n| Class | Meaning | Terminal |\n|-------|---------|----------|\n| `request_setup` | The prefill HTTP request could not be built or the body could not be prepared | yes |\n| `timeout` | `AIBRIX_PREFILL_REQUEST_TIMEOUT` elapsed | yes |\n| `canceled` | The client or the gateway canceled the prefill request | yes |\n| `transport` | Connection refused, reset, DNS failure, ... | yes |\n| `http_status` | The prefill engine answered a non-2xx status | yes |\n| `bad_response` | The prefill engine answered 2xx with a body the gateway could not parse | engine-dependent |\n\n`bad_response` is terminal only for engines whose KV transfer is out of band\n(TRT generation-first): there the HTTP body is not the KV handshake, so an\nunparseable 2xx leaves the gateway unable to tell whether context processing\nsucceeded. For SGLang, whose handshake completes in the body, it is not terminal:\nthe prefill leg did run, so the decode leg is left alone and the client keeps\nwhatever the decode leg produces.\n\nThe client status code is the prefill engine's own status for `http_status`, and\n`503` for every other terminal class, since those have no upstream status. The\nresponse body is an OpenAI-shaped error carrying the class and a truncated\nupstream message, and it is sent even while the gateway is still waiting for the\ndecode leg's response headers. For SGLang, once the decode leg has started answering the client, the failure is\nonly recorded and logged. TRT generation-first instead resets the stream even\nafter headers, because TRT can send SSE headers before KV is available; it does not\nattempt to replace the already-started response.\n\n### PD Decode Watchdog\n\nFail-fast covers a prefill leg that dies. The opposite case is a prefill leg\nthat *succeeds* and a decode pod that then never answers: in SGLang the prefill\ncall only returns once the decode pod has taken the KV transfer, so from that\npoint the decode pod owes the client a response. If its scheduler dies or wedges,\nthe client would otherwise hang until Envoy's route timeout.\n\n```\nPrefill leg succeeds (async worker)\n │\n └─► mark the PD leg state (MarkPrefillSucceeded) ──► wake the ext_proc stream\n │\n arm a timer: FIRST_RESPONSE (stream) / RESPONSE (non-stream)\n │\n response headers ──► no change (sent before the first token)\n first body chunk ──► switch to STREAM_IDLE │ (streaming only)\n every later chunk ──► re-arm STREAM_IDLE │\n ▼ timer fires\n before any headers: 504 ImmediateResponse, header x-error-pd-decode: true\n + gRPC DeadlineExceeded close\n after the headers: gRPC DeadlineExceeded close only (Envoy resets\n the response the client already started reading)\n always: POST /abort_request {\"rid\": ...} to decode (goroutine, once)\n```\n\nThe watchdog runs only for requests with a gateway-owned `rid` (SGLang). Its\ntimeouts are `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` (streaming, default `60`) and\n`AIBRIX_DECODE_RESPONSE_TIMEOUT` (non-streaming, default `0`, i.e. off, since the\ndecode pod only answers a non-streaming request once the whole generation is\ndone). Both run until the first response body chunk: SGLang's streaming\nendpoint sends its headers before the first token, so the headers prove nothing.\nFrom the first chunk of a streaming response on,\n`AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` (default `120`) bounds the gap between two\nchunks.\n\n---\n\n## Request Trackers\n\n### PrefillRequestTracker\n\nTracks active **prefill** request counts per pod using `sync.Map` and `atomic.Int32`. Used by both prefill load-imbalance detection and `scorePrefillPods` (mean/stddev filter and load cap).\n\nBoth request trackers are shared by every model the router serves, so pods are identified by their pod key, `namespace/name` (`utils.GeneratePodKey(pod.Namespace, pod.Name)`), not by bare pod name.\n\n```\nAddPrefillRequest(requestID, podKey) // in Route(), after PD pod selection, before prefill HTTP\nRemovePrefillRequest(requestID) // on prefill end (executor defer) or Route error\nGetPrefillRequestCountsForPods(pods) // for scoring\n```\n\n### PendingDecodeTracker\n\nBridges the gap between **decode pod selection** and the actual decode request starting. Without this, concurrent requests could all route to the same decode pod (since `RealtimeNumRequestsRunning` hasn't updated yet).\n\n```\nAddPendingDecode(requestID, podKey) // in Route(), immediately after pod selection\nRemovePendingDecode(requestID) // deferred in Route()\nGetPendingDecodeCount(podKey) // added to running reqs in decode scoring\n```\n\nTimeline:\n```\nRoute() called\n │\n ├─ filterPrefillDecodePods() → (prefillPod, decodePod)\n │ └─ [combined path] returns prefillPod=nil\n │\n ├─ AddPendingDecode(requestID, decodePodKey)\n ├─ defer RemovePendingDecode(requestID)\n │\n ├─ [PD path] AddPrefillRequest(requestID, prefillPodKey)\n ├─ [PD path] doPrefillRequest() (may take seconds)\n │ └─ on error: RemovePrefillRequest(requestID) called immediately\n ├─ ctx.SetTargetPod(decodePod)\n └─ return address to caller\n```\n\n---\n\n## Prompt Length Bucketing\n\nEnabled by `AIBRIX_PROMPT_LENGTH_BUCKETING=true`.\n\nEach pod can declare a prompt-length range via its config profile (`promptLenBucketMinLength` / `promptLenBucketMaxLength`). A pod is eligible only when the request's prompt length falls within its range.\n\n```\ncollectAndBucketPods()\n ├─ Combined pass: collect combined-role pods (role-name is not prefill/decode)\n │ with combined=true and a prompt-length match\n ├─ Eligible rolesets: group by roleset-name (prefill + decode both present)\n └─ Bucket pass: per roleset, keep pods whose bucket covers promptLength\n only when BOTH prefill and decode sides match (half-pairs excluded)\n\nIf both bucket-filtered lists are non-empty, they replace the unfiltered prefill/decode lists atomically so the sides stay roleset-aligned.\n```\n\n### No-match guard and combined fallback\n\nWhen `AIBRIX_PROMPT_LENGTH_BUCKETING=true`, a roleset enters the bucket-filtered lists only when **both** its prefill and decode pods match the request's prompt length. A roleset with only one matching side (misconfigured bucket or a missing decode pod) is excluded from the bucket lists.\n\nIf no complete bucket-matched prefill/decode pair exists for the request — including when the prompt length is outside every declared range, or when the matching roleset is incomplete (e.g. decode pod is down) — the router falls back to a combined pod when one is available. Otherwise it returns an error rather than cross-routing to a different bucket's PD pair:\n\n```\nif AIBRIX_PROMPT_LENGTH_BUCKETING &&\n (bucketPrefills == 0 || bucketDecodes == 0):\n if combined pods available for this prompt length → route to combined (random pick)\n else → error: \"no prompt-length bucket matches prompt length N and no combined pods available\"\n```\n\nPods without a `model.aibrix.ai/config` annotation are treated as suitable for any prompt length. With bucketing enabled globally, models that do not declare explicit bucket ranges still participate using those default bounds.\n\n### Combined Pods\n\nWhen bucketing is enabled, pods with `combined=true` in their config profile can act as both prefill and decode. Combined pods use a `role-name` other than `prefill` or `decode` (for example `all`). The router falls back to a combined pod when:\n\n1. No complete bucket-matched prefill/decode pair exists for the request's prompt length (prompt outside all buckets, incomplete roleset, or decode pod unavailable), **or**\n2. `shouldPickCombined()` returns `true`:\n - At least one combined pod has queue-drain score < `0.25` (low load), **and**\n - At least one prefill pod **or** decode pod has queue-drain score > `1.0` (high load).\n\nQueue-drain score is `(waiting_reqs + prealloc_queues) / drain_rate_1m` (see `calculatePodScoreBasedOffRequestRate`).\n\n```\nshouldPickCombined():\n combinedLowLoad = any combined pod score < 0.25\n prefillHighLoad = any prefill pod score > 1.0\n decodeHighLoad = any decode pod score > 1.0 (skipped if prefillHighLoad)\n return (prefillHighLoad OR decodeHighLoad) AND combinedLowLoad\n```\n\nWhen a combined pod is selected for load imbalance, `scoreCombinedPods()` picks the lowest-score combined pod. The no-match path picks a combined pod at random.\n\nWhen a combined pod is selected, `prefillPod` is `nil` (no prefill HTTP call) and `decodePod` is the selected combined pod.\n\n### Adaptive Bucket Serving\n\nEnabled by `AIBRIX_BUCKET_SERVE=true`, which requires `AIBRIX_PROMPT_LENGTH_BUCKETING=true` as well: the plan only re-orders the rolesets that bucketing already filtered to.\n\nPlain bucketing keeps every roleset whose range covers the request as a candidate, and the scoring decides among them. Adaptive bucket serving narrows that choice. The gateway keeps a per-model picture of the prompt lengths it routes, charges every roleset the traffic only it can serve, and splits each range that several rolesets still need into one band per roleset, drawing each band against one replica-proportional share so a roleset carrying several overlaps does not draw a fresh share from each. The cut points sit at quantiles of the observed traffic. The picture is an EWMA with a 30s half-life, a model's plan is recomputed at most once every 5s, and one plan holds at most 16 affinity bands. A shared range needs a meaningful share of the model's traffic before it is split at all, and a range that cannot afford one band per needing roleset splits into as many bands as the budget left allows; an interval that is not split keeps every covering roleset a candidate.\n\n```\nbucketServeBand() (before selectMu, one call per request)\n ├─ observe promptLength for the model\n ├─ groups: one per eligible roleset, with the range its profile declares and its prefill replica count\n ├─ plan: shared ranges split at mode quantiles in replica proportion, light ones kept whole\n └─ band for promptLength → banded roleset and band upper bound\n\nfilterPrefillDecodePods()\n ├─ steps 1-5: bucket filtering and the load-imbalance fast paths\n ├─ banded roleset still has prefill and decode candidates\n │ → narrow both sides to that roleset (step 6)\n └─ otherwise keep the load fast paths' choice\n```\n\n`AIBRIX_BUCKET_SERVE_MODE` picks the unit the cut points are measured in: `throughput` (default) balances prompt-token mass, while `rps` balances request counts. Within a shared range the bands divide the observed traffic in proportion to the replica share each roleset still needs, counted in that unit. Both are per-model: a profile that selects another mode re-plans only the models it routes.\n\nThe plan is advisory. It never widens a choice, only prefers one of the rolesets that already cover the request's length. Wherever the plan holds no band, the scoring considers every covering roleset, and when the banded roleset has no candidate left after the load fast paths, the request routes exactly as it would without the plan. The steady state does narrow the choice, which is the feature, and a stale or wrong plan costs balance, not correctness. With the feature off the planner records and publishes nothing.\n\nMetrics: `pd_bucket_serve_band_total` and `pd_bucket_serve_prompt_tokens_total` count the requests and prompt tokens each banded roleset carried (label `roleset`), so a counter joins the pods that received the band, and the `pd_bucket_serve_band_max` gauge holds the highest upper prompt-length bound among the bands a roleset currently holds (a roleset can hold two bands when its range overlaps two others), deleted when no live plan holds the roleset.\n\n---\n\n## Environment Variables\n\n### Core Behaviour\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFILL_SCORE_POLICY` | `prefix_cache` | Prefill pod scoring: `prefix_cache` or `least_request`. Any other value logs a warning and falls back to `prefix_cache`. |\n| `AIBRIX_DECODE_SCORE_POLICY` | `load_balancing` | Decode pod scoring for `finalPDScore`: `load_balancing`, `least_request`, `conductor` or `token_load`. Any other value logs a warning and falls back to `load_balancing`. |\n| `AIBRIX_KV_CONNECTOR_TYPE` | `shfs` | KV transfer backend: `shfs` (GPU/SHFS), `nixl` (Neuron/NIXL), or `mooncake` (Mooncake) |\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | `30` | Prefill HTTP request timeout in seconds. Exceeding it is a terminal `timeout` prefill failure. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | `false` | Enable prompt-length-based pod bucketing |\n| `AIBRIX_BUCKET_SERVE` | `false` | Enable adaptive bucket serving on top of prompt-length bucketing |\n| `AIBRIX_BUCKET_SERVE_MODE` | `throughput` | The unit the adaptive cut points are measured in: `throughput` (prompt token mass) or `rps` (request counts) |\n\n### Prefill Load Balancing\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | `16` | Min `(max − min)` outstanding prefill request count to trigger imbalance routing |\n| `AIBRIX_PREFILL_LOAD_CAP_FACTOR` | `1.5` | Factor `c`: prefill pods with `req_count > max(c, 1) * (mean + 1)` are skipped from per-roleset selection. Unlike the stddev skip, it fires in pools of five or fewer pods. Environment-only |\n\n### Decode Load Balancing\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | `1.0` | Multiplier on normalized running-request term in `load_balancing` numerator |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | `1.0` | Multiplier on normalized inverse-throughput term in `load_balancing` numerator |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | `16.0` | Min `(max − min)` running request count spread to trigger decode imbalance routing |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | `2048.0` | Min `(max − min)` token throughput spread (tok/s) to trigger throughput-based routing |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | `1.5` | `max_drain_score / min_drain_score` ratio threshold to trigger drain-rate routing |\n\n### PD Prefill Fail-Fast\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | `3` | Per-attempt timeout in seconds of the `/abort_request` call to the decode pod. `0` disables decode aborts; the client is still failed fast |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | `2` | Delay in seconds before the second abort attempt. `0` sends a single attempt |\n\n### PD Decode Watchdog\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | `60` | Seconds a streaming SGLang request waits, after its prefill leg succeeded, for the first response body chunk (first token) from the decode pod. `0` disables |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | `0` | Same for a non-streaming request, where the first body chunk is the finished answer. `0` (default) disables |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | `120` | Longest gap allowed between two body chunks of a streaming response once the first one arrived. `0` disables |\n\n### TensorRT-LLM\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_TRT_MACHINE_ID` | `0` | 10-bit machine ID used in Snowflake disagg request ID generation (range: `[0, 1024)`); distinct per gateway process sharing TRT workers |\n| `AIBRIX_TRT_SCHEDULE_STYLE` | `context_first` | TRT-only dispatch mode: `context_first` (sequential) or `generation_first` (parallel). Read at router initialization; unknown values are logged and fall back to `context_first` |\n\n### Inherited from Prefix Cache Router\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | `character` | Tokenizer for prefix-cache prefill scoring: `tiktoken` or `character` |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `1` | Multiplier N: prefill pods with `req_count > mean + N*stddev` are skipped from per-roleset selection |\n\n---\n\n## SGLang Bootstrap Port\n\nSGLang uses a bootstrap mechanism for prefill/decode coordination. The port is resolved in order:\n\n1. Pod annotation `model.aibrix.ai/sglang-bootstrap-port`\n2. Default: `8998`\n\n---\n\n## Metrics Emitted\n\n| Metric | When |\n|--------|------|\n| `GatewayPrefillRequestFailTotal` | Engine validation fail, pod filter fail, prefill HTTP error |\n| `GatewayPrefillRequestSuccessTotal` | Prefill HTTP succeeded. For SGLang and TRT generation-first, emitted asynchronously after the background prefill request completes, not when `Route()` returns. |\n| `gateway_pd_encode_request_total{status,status_code}` | One media item sent to an encode pod. `status` is `success`, or the failure class (`http_error`, `deadline_exceeded`, ...) |\n| `PDSelectedPrefillPodTotal` | Prefill pod selected (per pod label) |\n| `PDSelectedDecodePodTotal` | Decode pod selected (per pod label) |\n| `gateway_pd_prefill_failure_total{class,stage}` | A terminal prefill failure reached the client-facing handler. `stage` is `before_response` (the client was failed fast) or `after_response` (the decode leg had already started answering) |\n| `gateway_pd_decode_abort_total{prefill_failure_class,result}` | One decode abort attempt. `result` is `ok`, `error`, `skipped_streaming`, `skipped_disabled`, `skipped_no_rid` or `skipped_no_target`. `prefill_failure_class` is `watchdog_first_response` or `watchdog_stream_idle` for an abort sent by the decode watchdog |\n| `gateway_pd_decode_watchdog_total{phase}` | The decode watchdog failed a request whose decode pod stopped responding. `phase` is `first_response` or `stream_idle` |\n\n---\n\n## Response Headers Set\n\n| Header | Value |\n|--------|-------|\n| `prefill-target-pod` | Name of the selected prefill pod |\n| `prefill-target-pod-ip` | IP of the selected prefill pod |\n| `x-error-pd-prefill` | `true`, on the error response the gateway generates when the PD prefill leg failed |\n| `x-error-pd-decode` | `true`, on the error response the gateway generates when the decode watchdog fired |\n\n---\n\n## Prefix Cache Update Pipeline\n\nAfter a prefill pod is selected, its prefix hashes are enqueued asynchronously to avoid blocking the routing path:\n\n```\nfinalPDScore()\n └─ enqueuePrefixUpdate(hashes, model, pod)\n │\n ▼ (non-blocking send; dropped if channel full)\n prefixUpdateCh (buffered, capacity 1024)\n │\n ▼ (single background goroutine)\n prefixCacheIndexer.AddPrefix(hashes, model, pod)\n```\n\n---\n\n## Initialization\n\n```go\nNewPDRouter()\n │\n ├─ cache.Get() // shared metrics cache\n ├─ prefixcacheindexer.GetSharedPrefixHashTable()\n ├─ prefillScorePolicy from AIBRIX_PREFILL_SCORE_POLICY:\n │ least_request → LeastRequestPrefillPolicy\n │ prefix_cache → PrefixCachePrefillPolicy (shared tokenizer + PrefixHashTable)\n │ (other) → log warning, same as prefix_cache\n ├─ decode score policy (package algorithms/pd, decode_scorer.go) from AIBRIX_DECODE_SCORE_POLICY:\n │ least_request → pd.LeastRequestDecodePolicy\n │ load_balancing → pd.LoadBalancingDecodePolicy\n │ (other) → log warning, same as load_balancing\n ├─ create HTTP client with connection pool\n │ MaxIdleConns=100, MaxIdleConnsPerHost=10, IdleConnTimeout=90s\n ├─ TRT handler from AIBRIX_TRT_SCHEDULE_STYLE (immutable, router-scoped)\n │ owns a lazily populated /server_info cache for generation_first\n ├─ NewPrefillRequestTracker()\n ├─ NewPendingDecodeTracker()\n └─ startPrefixUpdater() // background goroutine\n\n// pd/router/roleset.go (no per-router state; pure functions called on each request)\nHasEligibleRoleset(readyPods) // gateway: skip PD when no complete roleset\neligiblePDRolesets(readyPods) // internal: filter to complete rolesets only\nDescribePDRolesetEligibility(readyPods) // diagnostics: per-roleset replica counts\n```\n","isInternal":false,"tokens":11559,"sizeBytes":51791},{"name":"prefix_cache_readme.md","path":"pkg/plugins/gateway/algorithms/prefix_cache_readme.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/prefix_cache_readme.md","title":"Algorithms Skill","category":"anthropic-skill","format":"markdown","content":"# Prefix Cache Aware Routing\n\nPrefix Cache Aware routing improves KV cache reuse across GPU pods by steering requests whose\nprompt shares a common prefix to the pod that already has those KV blocks cached. Pod selection\namong prefix-matched candidates is still load-aware (see Step 2 below), and this router itself\nis purely about prefix caching — it does not gate or restrict candidates based on cluster-wide\nload imbalance. That concern is instead handled centrally by the gateway: `selectTargetPod` in\n`gateway.go` applies the [load-balance router](load_balance.go)'s `ApplyLoadImbalanceGate` once,\nahead of whichever strategy actually routes the request (prefix-cache included), narrowing\n`readyPods` before this router ever sees them. This replaced an older prefix-cache-specific gate\nthat lived in this file; see [ENV_VARS.md](../ENV_VARS.md#variable-dependency-notes) for the\nmigration note. The gate applies the same way when combining strategies via multi-strategy\nsoft-scoring (`AIBRIX_ROUTING_ALGORITHM=prefix-cache:W1,load-balance:W2`, see\n[multi_router_readme.md](multi_router_readme.md)): `selectTargetPod` narrows `readyPods` once,\nbefore resolving and calling into the multi-strategy router, so every sub-strategy's `ScoreAll`\n— this one included — only ever sees the already-gated pod list. It is skipped only for\nexclusive strategies (`pd`, `slo*`), which manage their own pod subsets.\n\nTwo routing modes are supported, selected automatically at startup:\n\n| Mode | When active | Prefix index |\n|------|-------------|--------------|\n| **Standard** | `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=false` (default) | Local hash table updated by the router after each request |\n| **KV Sync** | `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true` | Distributed sync indexer fed by real-time block-stored/removed events from vLLM pods |\n\nBoth modes share the same stddev-based pod selection logic described below.\n\n## Routing Flow\n\n```\n Incoming Request\n │\n ▼\n┌─────────────────────────────┐\n│ Tokenize & hash prompt │ splits prompt into fixed-size blocks,\n│ into prefix blocks │ computes hash per block\n└────────────┬────────────────┘\n │\n ▼\n┌─────────────────────────────┐\n│ Match prefix hashes │\n│ against ready pods │\n└────────────┬────────────────┘\n │\n ┌────────┴────────┐\n │ match_pods │ NO match_pods\n │ found? │──────────────────────────────────────────────┐\n └────────┬────────┘ │\n │ YES │\n┌─────────────────────────────┐ │\n│ Sort match_pods by: │ │\n│ 1. prefix_match% DESC │ │\n│ 2. running_requests ASC │ │\n└────────────┬────────────────┘ │\n │ │\n┌─────────────────────────────┐ NO pod within threshold │\n│ Select first pod where: │─────────────────────────────────────►│\n│ running_req <= │ │\n│ mean + load_factor * σ │ │\n└────────────┬────────────────┘ │\n │ │\n │ ▼\n │ Least-request fallback among all ready pods\n │ │\n ▼ ▼\n Target Pod Selected\n```\n\n## Algorithm Details\n\n### Step 1 — Tokenize & Hash\n\nThe prompt is tokenized and split into fixed-size blocks. A hash is computed for each block.\nTwo tokenizer backends are supported:\n\n| Tokenizer | How it works |\n|------------|-------------------------------------------------------|\n| `character`| Splits raw text into individual characters (default) |\n| `tiktoken` | Uses the OpenAI [tiktoken](https://github.com/openai/tiktoken) BPE tokenizer |\n\nBlock hashes are used as cache keys — a pod that has processed a request with an identical\nprefix will have those KV blocks hot in GPU memory.\n\n### Step 2 — Prefix Match & Pod Selection\n\nEach candidate pod is scored by how many of the request's prefix blocks it already holds\n(`prefix_match_percent`). Matched pods are sorted:\n\n1. **Descending** by `prefix_match_percent` — prefer higher cache hit rate.\n2. **Ascending** by `running_requests` — break ties by choosing the less loaded pod.\n\nThe first pod in this sorted list that satisfies the load threshold is selected:\n\n```\npod.running_requests <= mean_running_requests + load_factor × std_dev\npod.running_requests <= max(load_cap_factor, 1) × (mean_running_requests + 1)\n```\n\n`load_factor` (= `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR`) controls how aggressively\nthe router skips overloaded cache-holding pods. If no matched pod is within the threshold, the\nrouter falls back to the global least-loaded pod across all ready pods. `load_cap_factor`\n(= `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR`) bounds load directly: no value in n pods sits more than\n(n−1)/√n standard deviations above the mean, so at `load_factor` 2 the first check cannot skip\nany pod in a pool of five or fewer.\n\n## Configuration\n\n| Environment Variable | Description | Default |\n|---|---|---|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | Tokenizer used to split the prompt. Options: `character`, `tiktoken`. | `character` |\n| `AIBRIX_PREFIX_CACHE_BLOCK_SIZE` | Number of tokens per prefix block. Smaller blocks = finer-grained matching but more hash overhead. | `128` |\n| `AIBRIX_PREFIX_CACHE_BLOCK_NUMBER` | Maximum number of prefix cache blocks tracked per pod. | `200000` |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `load_factor` in `mean + load_factor × σ`. Higher value tolerates more load on a high-cache-hit pod. | `1` |\n| `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR` | `load_cap_factor` in `max(load_cap_factor, 1) × (mean + 1)`. Higher value tolerates more load on a high-cache-hit pod; set a large value such as `100` to effectively disable it. Applies only when `prefix-cache` routes on its own, with the auto-blend off (`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT=0`). | `1.5` |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED` | Enable KV sync routing mode (requires remote tokenizer). | `false` |\n| `AIBRIX_PREFIX_CACHE_INCLUDE_TOOLS` | Prepend the canonicalized chat request `tools` to the prompt text before hashing, so requests that share messages but carry different tools do not look like a full prefix match. Token estimates keep using the messages only. KV-sync mode tokenizing through the remote chat template forwards the raw `tools` to the tokenizer regardless of this flag, so the template places them. Set to `false` to hash the messages only. | `true` |\n\nSee [ENV_VARS.md](../ENV_VARS.md#load-balance-router-algorithmsload_balancego) for the\nload-imbalance gate's `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` /\n`AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` variables — they belong to the load-balance router, not\nthis one.\n\n### Tuning guidance\n\n- **Block size**: use `128` for `character` tokenizer and `16` for `tiktoken`. Larger blocks\n reduce hash overhead but require longer identical prefixes for a match.\n- **Standard deviation factor**: `1` (default) keeps all pods within one stddev of mean load.\n Increase to `2` to allow higher-loaded pods to still be selected for cache-hit requests.\n","isInternal":false,"tokens":1723,"sizeBytes":8650},{"name":"vtc_readme.md","path":"pkg/plugins/gateway/algorithms/vtc_readme.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/vtc_readme.md","title":"Algorithms Skill","category":"anthropic-skill","format":"markdown","content":"# Virtual Token Counter (VTC) Routing\n\nVirtual Token Counter (VTC) is a fair scheduling algorithm for LLM serving based on the paper\n[Fairness in Serving Large Language Models (Sheng et al., 2024)](https://arxiv.org/abs/2401.00588).\nVTC tracks the weighted token service each client has received and routes new requests to the pod\nthat gives the least-served client the best chance of progressing — balancing fairness with\ncluster utilization.\n\n## vtc-basic\n\n`vtc-basic` is a Kubernetes-native simplification of VTC. Rather than managing a single global\nqueue, it selects a target pod per request by combining two scores:\n\n- **Fairness score** — how much token service this client has received relative to all clients\n within a sliding time window. Clients who have received less service get a lower (better) score.\n- **Utilization score** — the pod's current load normalized against `MAX_POD_LOAD`.\n\nThe pod with the **lowest combined score** is selected.\n\nToken counts and the adaptive window extrema exclude expired usage, including\nclients that have stopped sending requests. The in-memory tracker reclaims that\nusage on reads and updates; it does not need a request from the departed client.\n\n### Scoring diagram\n\n```\nFor each ready pod:\n\n fairness_score = client_tokens_in_window / total_tokens_in_window\n (normalized, 0.0 → 1.0; lower = client received less service)\n\n utilization_score = pod_running_requests / MAX_POD_LOAD\n (normalized, 0.0 → 1.0; lower = pod is less loaded)\n\n combined_score = FAIRNESS_WEIGHT × fairness_score\n + UTILIZATION_WEIGHT × utilization_score\n\n ──────────────────────────────────────────────────────────────\n Select pod with minimum combined_score\n```\n\n### Request lifecycle\n\n```\nIncoming Request (with user identity)\n │\n ▼\n┌──────────────────────────────────┐\n│ Look up client token count │\n│ in sliding time window │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ Compute fairness_score │\n│ per ready pod │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ Compute utilization_score │\n│ per ready pod │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ combined_score = │\n│ w_f × fairness │\n│ + w_u × utilization │\n└───────────────┬──────────────────┘\n │\n ▼\n Select min-score pod\n │\n ▼\n┌──────────────────────────────────┐\n│ After response: update client │\n│ token count in window │\n│ (input × w_i + output × w_o) │\n└──────────────────────────────────┘\n```\n\nEnd-to-end tests: [vtc_routing_test.go](../../../../test/e2e/vtc_routing_test.go)\n\n## Configuration\n\n| Environment Variable | Description | Default |\n|-----------------------------------------------|------------------------------------------------------------------------------|----------------|\n| `AIBRIX_ROUTING_ALGORITHM` | Set to `vtc-basic` to enable this routing strategy. | `prefix-aware` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | Time unit for the sliding window. Options: `seconds`, `minutes`, `hours`. | `minutes` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | Number of time units in the sliding window. | `5` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | Minimum token floor for adaptive tracking — prevents division by near-zero. | `1000` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | Maximum token cap for adaptive tracking. | `8000` |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | Weight applied to input tokens when updating a client's service counter. | `1.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | Weight applied to output tokens when updating a client's service counter. | `2.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | Normalization denominator for pod running-request count. | `100.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | Multiplier for the fairness term in the combined score (`w_f`). | `1.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | Multiplier for the utilization term in the combined score (`w_u`). | `1.0` |\n\n### Tuning guidance\n\n- **Output token weight > input token weight** (default `2.0` vs `1.0`) reflects that output\n generation is more GPU-expensive than prompt processing.\n- **Increase `FAIRNESS_WEIGHT`** relative to `UTILIZATION_WEIGHT` when strict per-client\n fairness matters more than raw throughput.\n- **Increase `UTILIZATION_WEIGHT`** to prioritize routing to less-loaded pods, trading some\n fairness for better overall cluster efficiency.\n- **Window size** controls how long past token usage influences routing. A shorter window (e.g.\n `1 minute`) makes the router more reactive; a longer window (e.g. `10 minutes`) smooths out\n bursty clients.\n\n## Other VTC variants\n\nThe original VTC paper and reference implementation\n([slora/server/router](https://github.com/Ying1123/VTC-artifact/tree/main/slora/server/router))\ndescribe additional variants:\n\n| Variant | Description |\n|--------------|-----------------------------------------------------------------------------|\n| `vtc-fair` | Pure fairness — ignores utilization entirely. |\n| `vtc-max-fair` | Maximizes minimum service across all clients. |\n| `vtc-pred` | Uses output-length prediction to improve fairness estimates. |\n\nAdapting these variants to the AIBrix pod-selection model (rather than a single shared queue)\nis a candidate for future work.\n","isInternal":false,"tokens":1391,"sizeBytes":7123},{"name":"ENV_VARS.md","path":"pkg/plugins/gateway/ENV_VARS.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/ENV_VARS.md","title":"Gateway Skill","category":"anthropic-skill","format":"markdown","content":"# Gateway Plugin Environment Variables\n\nThis document covers all environment variables used in the `pkg/plugins/gateway` package and its sub-packages.\n\nVariables of type `duration` are parsed with Go's [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration), so they need a unit: `30s`, `5m`, `1h30m`. A bare number such as `30`, or a zero or negative duration such as `0s` or `-1s`, is rejected with a warning and the default is used instead.\n\n---\n\n## General Gateway\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_DISABLE_RATE_LIMITING` | bool | `false` | Disable AIBrix user RPM/TPM and model RPS quota enforcement. Redis and `requestsInflight` remain active. | [cmd/plugins/main.go](../../../cmd/plugins/main.go), [gateway.go](gateway.go) |\n| `POD_NAME` | string | `\"\"` | Kubernetes pod name. Used for logging and metric label tagging. | [gateway.go](gateway.go), [util.go](util.go) |\n| `ROUTING_ALGORITHM` | string | _(none)_ | Default routing algorithm when no per-request override is set. | [types.go](types.go), [util.go](util.go) |\n| `AIBRIX_PRIORITY_TIER_ENABLED` | bool | `false` | Forward the priority tier declared by the `x-aibrix-priority-tier` request header as the upstream vLLM request priority. | [cmd/plugins/main.go](../../../cmd/plugins/main.go), [gateway_req_priority.go](gateway_req_priority.go) |\n\nWhen `AIBRIX_PRIORITY_TIER_ENABLED=true`, the gateway maps the `x-aibrix-priority-tier`\nrequest header to the `priority` field of the backend request body: `batch` becomes `100` and\n`background` becomes `1000`. The engine serves smaller values first, so the mapping only\nde-prioritizes a request and never pulls one ahead of another; the values sit below the engine\ndefault of `0`. Requests without the header, or with a tier outside the table, are forwarded\nunchanged, and a `priority` the caller already set in the body is kept. Tier names are matched\ncase-insensitively. The deployment has to run a priority-aware vLLM scheduler\n(`--scheduling-policy=priority`). With the default `false`, the header is ignored and request\nbodies are forwarded byte for byte.\n\nWhen `AIBRIX_DISABLE_RATE_LIMITING=true`, the gateway skips AIBrix user lookup, ignores the\n`user` header as routing identity, and does not write user or model quota counters. The header\nis still forwarded unchanged. Keep Redis configured when session affinity, asynchronous jobs,\nstate synchronization, or other gateway features require it. The per-replica\n`requestsInflight` guard remains active. Because routing user identity remains empty, asynchronous\nvideo jobs use the shared ownership scope; the unauthenticated `user` header does not isolate jobs\nbetween callers in this mode.\n\n---\n\n## Response Processing\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_TTFT_THRESHOLD_S` | int (seconds) | `1` | Time-to-first-token threshold in seconds. Requests exceeding this are flagged in response processing. | [gateway_rsp_body.go](gateway_rsp_body.go) |\n\n`AIBRIX_TTFT_THRESHOLD_S` can also be set per model with the top-level `ttftThresholdS` profile field; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Redis Sync (`statesync/`)\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_STATESYNC_ENABLED` | bool | `false` | Enable cross-replica state sync via Redis. Must be `true` to activate the statesync manager. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `AIBRIX_STATESYNC_SYNC_PERIOD` | duration | `10s` | Interval at which gateway state is synced to Redis across replicas. | [statesync/redissync.go](statesync/redissync.go) |\n\n---\n\n## Prefix Cache Router (`algorithms/prefixcache/router.go`)\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | string | `\"character\"` | Tokenizer type for prefix cache hashing. Options: `character`, `tiktoken`, `remote`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | int | `1` | Factor multiplied by the standard deviation of pod loads when selecting among prefix-matched pods (`pod.req ≤ mean + factor × σ`). | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR` | float64 | `1.5` | Prefix-matched pods with more than `max(c, 1) × (mean + 1)` running requests are skipped, where `c` is this factor. Applies only when `prefix-cache` routes on its own, with the auto-blend off (`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT=0`). Unlike the `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` check, which cannot skip any pod in a pool of five or fewer at a factor of `2`, it applies to pools of any size. Values between `0` and `1` are treated as `1`; `0`, negative or unparsable values use the default `1.5`. To effectively disable the cap, set a large value such as `100`. Environment-only. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_INCLUDE_TOOLS` | bool | `true` | Prepend a canonical rendering of the chat request `tools` (keys sorted, compact) to the text used for prefix matching, since chat templates commonly render tools ahead of the messages. For templates that render tools later in the prompt (e.g. near the last user turn), requests sharing tools but not messages can see a partial match the engine does not have; such deployments can set this to `false`. Requests without tools (absent, `null`, `[]` or not an array) are unaffected. Set to `false` to match on messages only. The tools text goes into a separate prefix-match text (`RoutingContext.PrefixText()`), which feeds prefix matching in `prefix-cache`, `prefix-cache-preble` and the PD prefill policies `prefix_cache`, `conductor` (including its matched/unmatched token estimate) and `hybrid_cache_load`. The routing message (`routingCtx.Message`) stays messages-only, so token estimates (VTC, `PromptTokens`/`PromptLength`, config-profile `promptTokensGte`/`promptTokensLt`) are unchanged. In KV-sync mode, chat requests tokenized through the remote chat template always forward the raw `tools` to the tokenizer, independent of this flag, since the template itself decides where tools go. | [prefix_tools.go](prefix_tools.go) |\n| `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER` | bool | `false` | Use a remote HTTP tokenizer service instead of the local tokenizer. Requires `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED` | bool | `false` | Enable KV cache event synchronization across gateway replicas. When `true`, also requires `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_REMOTE_TOKENIZER_ENDPOINT` | string | `\"\"` | Remote tokenizer service endpoint URL. Required when `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true`. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_PUBLISH_ADDR` | string | `\"\"` | ZMQ publish address for KV cache events. Used when KV event sync is enabled. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SUBSCRIBE_ADDRS` | string | `\"\"` | ZMQ subscribe addresses for KV cache events (comma-separated). Used when KV event sync is enabled. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n\n### Remote Tokenizer Pool (`algorithms/prefixcache/router.go`)\n\nThese configure the pool of remote tokenizer connections used when `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_VLLM_TOKENIZER_ENDPOINT_TEMPLATE` | string | `\"http://%s:8000\"` | HTTP endpoint template for tokenizer pods. `%s` is replaced with the pod name. |\n| `AIBRIX_TOKENIZER_HEALTH_CHECK_PERIOD` | duration | `30s` | How often to health-check tokenizer pool members. |\n| `AIBRIX_TOKENIZER_TTL` | duration | `300s` | TTL for cached tokenizer connections in the pool. |\n| `AIBRIX_MAX_TOKENIZERS_PER_POOL` | int | `100` | Maximum number of tokenizer connections in the pool. |\n| `AIBRIX_TOKENIZER_REQUEST_TIMEOUT` | duration | `5s` | Timeout for individual remote tokenizer requests. |\n\n`AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` can also be set per request by the model config profile (`routingConfig.prefixCache.standardDeviationFactor`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Load Balance Router (`algorithms/load_balance.go`)\n\nThe load-imbalance gate below is applied centrally by the gateway (`selectTargetPod` in\n`gateway.go`), once per request, ahead of whichever strategy actually routes it — not just when\n`load-balance` is selected. It restricts candidates to the least-loaded pods before that\nstrategy runs when load is severely skewed, and it **is** applied to the Prefix Cache and\nPreble routers (this replaces the prefix-cache-specific gate that used to live in\n`prefixcache/router.go`; see the deprecation note under [Variable Dependency\nNotes](#variable-dependency-notes)). It is exempted only for exclusive strategies (`pd`,\n`slo*`), which manage their own pod subsets and would have that management disrupted by a\nblanket running-request-count filter applied ahead of time. It does not participate in\nmulti-strategy soft-scoring (`ScoreAll`) for strategies blended alongside `load-balance` — see\n`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` below for how `load-balance` itself gets\nsilently blended into other strategies to compensate.\n\n**TODO / known limitation:** the gate compares raw running-request count, with no notion of a\npod's capacity. In a heterogeneous pool this conflates \"carrying a lot of work\" with\n\"overloaded\" — a replica that's simply *faster* than its peers is expected to carry more\nconcurrent requests without being more loaded (the same capacity signal `load-balance`'s own\nscore uses), but this gate can still flag it as a hotspot and exclude it from routing before any\nstrategy's capacity-aware scoring gets a chance to run. See `getTargetPodListOnLoadImbalance` in\n[algorithms/load_balance.go](algorithms/load_balance.go).\n\nWhen `load-balance` is the (sole, non-blended) strategy that routes the request and multiple\npods tie on the lowest effective-load score, `Route()` breaks the tie using least combined\nGPU+CPU KV-cache usage (falling back to a random pick if cache metrics are unavailable for the\ntied pods) — the same secondary-signal pattern the Prefix Cache router uses to break ties in\nprefix-match percentage via request count.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` | float64 | `2.0` | Gate multiplier for 3+ pods: gate fires when `max_req > factor × (mean_req + 1)`. Ignored for 2-pod clusters (the relative check never holds there). | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` | int | `8` | Minimum absolute gap (`max_req − min_req`) required to trigger the load-imbalance gate. For 2 pods this is the sole trigger; for 3+ pods it is required alongside the factor check. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` | float64 | `0` | Weight λ of engine-queued requests (`num_requests_waiting`) added to the running count in the score: `(running + λ·queued) / capacity × …`. The default `0` scores on running requests only, since the gateway's running count already includes requests queued inside the engine; raise it only if you want queued work to count extra. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` | float64 | `2.0` | Strength α of the KV-pressure penalty: the score is multiplied by `1 + α·(1 − kv_free)²`. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` | float64 | `0.10` | Free-KV-cache fraction below which a pod scores `+Inf` from `load-balance`. When `load-balance` is the sole routing strategy this excludes the pod outright (falling back to the pod with the most KV headroom if every pod is below the threshold, instead of failing the request). When `load-balance` is blended with other strategies (the default for most requests — see \"Router Selection / Auto-Blend\" below), a `+Inf` score is only a strong penalty, not an exclusion: another strategy can still select the pod. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n\nThe five `AIBRIX_LOAD_BALANCE_*` gate and score variables can also be set per request by the model config profile (`routingConfig.loadBalance.*`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Least-Request Top-K Router (`algorithms/least/request_top_k.go`)\n\nStrategy name `least-request-top-k`. Plain `least-request` always resolves to the single\nleast-loaded pod, so when several gateway replicas read the same Redis-backed running-request\ncounter and decide at the same moment, they can all send their next request to that pod. This\nrouter instead picks uniformly at random among the ready pods that are both among the `K`\nleast-loaded and within `epsilon` requests of the least-loaded pod's count. It is also a scorer\n(`ScoreAll`), so it can be blended with other strategies; it is the load-count scorer that the\n[auto-blend](#router-selection--auto-blend-algorithmsroutergo) appends by default, so the variables below\napply to every auto-blended request, not only to requests that name this strategy.\n\nA pod that has just become eligible reads as count `0` and would otherwise be the sole eligible\ncandidate until it catches up. To prevent that, a pod's effective count is raised by a penalty\nthat starts at the lowest count among the already-warm pods and decays linearly to `0` over the\nramp window. The window is measured from when this gateway first sees the pod routable (Ready,\nwith an IP, not terminating or draining), not from when the pod was created, so time spent\nPending or loading a model does not use it up. Ramp state is local to each gateway; a gateway\nthat has just restarted sees every pod as new, so with no warm pod to compare against it applies\nno penalty until some pods age out of the window. The clock is kept, not restarted, when a pod\nflaps NotReady or is re-added with the same IP within 60 seconds; a new IP, or a longer gap, starts\nit over.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K` | int | `5` | `K`: how many of the least-loaded ready pods are eligible. Should be at least the number of gateway replicas deciding concurrently. Must be positive; any other value falls back to the default. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K_EPSILON` | int | `4` | A pod is eligible only if its count is within this many requests of the least-loaded pod's. Combined with `K` as an AND. Must be positive; any other value falls back to the default. The default is a starting point, not a validated one: tune it to the deployment's typical running-request count. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K_RAMP_WINDOW` | duration | `300s` | Window over which a newly routable pod's ramp penalty decays to `0`, counted from the first time the gateway sees the pod routable. `0` disables the ramp. Any other non-positive or unparseable value falls back to the default. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n\nThese are read once at startup and are environment-only (no `routingConfig` override).\n\n---\n\n## Router Selection / Auto-Blend (`algorithms/router.go`)\n\n`RouterManager.Select` silently blends `load-balance` and [`least-request-top-k`](#least-request-top-k-router-algorithmsleastrequest_top_kgo)\nbehind whatever strategy a caller actually asked for, so no single strategy can keep steering\ntraffic at an already-hot pod even without going through the central load-imbalance gate above.\nThe caller never sees this: `ctx.Algorithm`, response headers, and `Validate()` all continue to\nreflect exactly what was requested. The blend is skipped entirely for exclusive strategies\n(`pd`, `slo*`), for an explicit, standalone `load-balance` selection, and for a bare\n`session-affinity` selection, all of which must keep running their own dedicated routing logic.\n`session-affinity`'s own scoring is binary (1 for its resolved pod, 0 for everything else), so\nany load-balance weight below its own could never change the outcome anyway — blending it in\nwould be dead weight at best. A bare `prefix-cache` request, whose scoring is graded rather than\nbinary, still uses a 5:4 (1.25:1) ratio against `load-balance` instead of the flat weight-1\nappend, and does not receive `least-request-top-k`, so the cache-affinity signal wins an exact-tie\ndisagreement without masking real load imbalance.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` | int | `1` | Weight coefficient for the silently-appended `load-balance` scorer. Set to `0` to disable the whole auto-blend feature. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_LEAST_REQUEST_WEIGHT` | int | `1` | Weight coefficient for the silently-appended `least-request-top-k` scorer. It is added only when neither `least-request` nor `least-request-top-k` is already in the strategy (an explicit weight of `0` on either also suppresses it), and is needed so multi-port/data-parallel pod routing keeps working under the blend. Plain `least-request` is not used here because every gateway replica would score the same single least-loaded pod best. The variable keeps its `LEAST_REQUEST` name for compatibility. Set to `0` to omit it from the blend. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` | int | `5` | Primary weight used when auto-blending a bare `prefix-cache` request. With the matching load-balance weight this is a 5:4 (1.25:1) lean toward cache affinity. `0` is rejected with a warning and the default is used. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_LOAD_BALANCE_WEIGHT` | int | `4` | `load-balance` weight paired with `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` for a bare `prefix-cache` request. `0` leaves those requests with prefix-cache scoring alone. | [algorithms/router.go](algorithms/router.go) |\n\nThe four `AIBRIX_ROUTING_AUTO_BLEND_*` weights can also be set per request by the model config profile (`routingConfig.autoBlend.*`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## External Replica Router (`algorithms/external*.go`)\n\nThe optional `external` strategy sends the already-filtered candidate snapshot to one\noperator-configured HTTP endpoint. An unset endpoint disables the strategy. Invalid enabled\nconfiguration fails Gateway startup. Candidate metrics and labels are opt-in allowlists; prompts,\nrequest bodies, credentials, arbitrary client headers, and pod IPs are never sent.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_EXTERNAL_ROUTER_ENDPOINT` | URL | _(unset)_ | Complete `http` or `https` operation URL. Userinfo and fragments are rejected. |\n| `AIBRIX_EXTERNAL_ROUTER_POLICY_MODE` | enum | _(required)_ | `Advisory` or `Authoritative`. |\n| `AIBRIX_EXTERNAL_ROUTER_FAILURE_MODE` | enum | _(required)_ | `FailOpen` or `FailClosed`; Authoritative requires FailClosed. |\n| `AIBRIX_EXTERNAL_ROUTER_FALLBACK` | string | _(none)_ | Registered non-external, non-exclusive local router; required for Advisory and FailOpen. `pd` and `slo*` are rejected because they require dedicated Gateway preprocessing. |\n| `AIBRIX_EXTERNAL_ROUTER_TIMEOUT` | duration | `10ms` | Deadline for one decision exchange. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_INFLIGHT` | int | `256` | Non-blocking per-process bulkhead capacity. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_REQUEST_BYTES` | bytes | `256KiB` | Maximum encoded request size. Accepts bytes, `KiB`, or `MiB`. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_RESPONSE_BYTES` | bytes | `64KiB` | Maximum response body size. Accepts bytes, `KiB`, or `MiB`. |\n| `AIBRIX_EXTERNAL_ROUTER_FAILURE_THRESHOLD` | int | `5` | Consecutive attempted-exchange system failures before opening the circuit. |\n| `AIBRIX_EXTERNAL_ROUTER_OPEN_DURATION` | duration | `1s` | Circuit-open duration before one half-open probe. |\n| `AIBRIX_EXTERNAL_ROUTER_AUTH_TOKEN_FILE` | path | _(unset)_ | Optional non-empty bearer token file loaded at startup. |\n| `AIBRIX_EXTERNAL_ROUTER_CANDIDATE_ATTRIBUTES` | CSV | _(empty)_ | Pod-label keys allowed into candidate attributes. |\n| `AIBRIX_EXTERNAL_ROUTER_CANDIDATE_METRICS` | CSV | _(empty)_ | Any of `runningRequests`, `engineUtilization`, `kvCacheUsage`. |\n| `AIBRIX_EXTERNAL_ROUTER_POLICY_ATTRIBUTES` | CSV | _(empty)_ | Downstream extension-point allowlist for trusted RoutingContext policy attributes. Upstream AIBrix has no built-in producer; integrators must call `SetTrustedPolicyAttribute`. Raw headers are never copied automatically. |\n\nSee the [External Replica Routing guide](../../../docs/source/features/gateway-routing/external-replica-routing.rst)\nand the published OpenAPI contract for policy and protocol details.\n\n---\n\n## Preble (Prefix Cache with Histogram) Router (`algorithms/prefixcache/preble.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_PREBLE_TARGET_GPU` | string | `\"V100\"` | GPU model used for hardware-specific latency estimates in the Preble algorithm. |\n| `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` | int | `45` | Expected decode sequence length used for cache allocation decisions. |\n| `AIBRIX_ROUTER_PREBLE_SLIDING_WINDOW_PERIOD` | int (minutes) | `3` | Sliding window length in minutes for histogram metrics collection. |\n| `AIBRIX_ROUTER_PREBLE_EVICTION_LOOP_INTERVAL` | int (ms) | `1000` | Interval in milliseconds between cache eviction loop executions. |\n\n`AIBRIX_ROUTER_PREBLE_TARGET_GPU` and `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` can also be set per request by the model config profile (`routingConfig.preble.*`). The window and eviction variables stay environment-only; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## VTC (Virtual Token Counter) Router\n\n### Token Tracker (`algorithms/vtc/token_tracker.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | int | `5` | Sliding window size (in `TIME_UNIT` units) for token usage tracking. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | string | `\"minutes\"` | Time unit for the sliding window. Options: `minutes`, `seconds`, `milliseconds`. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | float64 | `1000.0` | Minimum token count threshold for adaptive load normalization. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | float64 | `8000.0` | Maximum token count threshold for adaptive load normalization. |\n\n### VTC Basic Scorer (`algorithms/vtc/vtc_basic.go`)\n\nScoring formula: `score = (fairnessWeight * normFairness + utilizationWeight * normUtilization) / normFreeGPU`\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | float64 | `100.0` | Load value at which a pod is considered fully saturated. |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | float64 | `1.0` | Weight applied to input tokens when computing pod load. |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | float64 | `2.0` | Weight applied to output tokens when computing pod load (typically higher than input). |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | float64 | `1.0` | Weight of the fairness component in the routing score. |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | float64 | `1.0` | Weight of the utilization component in the routing score. |\n\n`AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD`, `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` and `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` can also be set per request by the model config profile (`routingConfig.vtc.*`). The token tracker variables stay environment-only; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## PD (Prefill-Decode) Disaggregation Router (`algorithms/pd/router/router.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | int (seconds) | `30` | HTTP request timeout for prefill pod calls. |\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | int32 | `16` | Minimum (max − min) running-request spread across prefill pods to trigger load-imbalance routing. |\n| `AIBRIX_PREFILL_LOAD_CAP_FACTOR` | float64 | `1.5` | Prefill pods with more than `c × (mean + 1)` in-flight prefill requests are skipped, where `c` is this factor. Unlike the `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` check, it applies to pools of any size. Values between `0` and `1` are treated as `1`; `0`, negative or unparsable values use the default `1.5`. To effectively disable the cap, set a large value such as `100`. Lower values give up more prefix-cache affinity: with random (Poisson) load and no hot pod, `1.5` skips at least one pod on up to about 10% of requests with 4 prefill pods and about 20% with 6. `2` skips less often but cannot fire in a two-pod pool. |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | float64 | `16.0` | Minimum (max − min) running-request spread across decode pods to trigger load-imbalance routing. |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | float64 | `2048.0` | Minimum (max − min) token-throughput spread (tokens/s) across decode pods to trigger throughput-imbalance routing. |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | float64 | `1.5` | Max/min drain-rate score ratio above which the slowest decode pod is excluded from selection. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | bool | `false` | Route requests to prefill pods whose prompt-length bucket matches the request length. |\n| `AIBRIX_BUCKET_SERVE` | bool | `false` | Adaptive bucket serving: band the prompt-length range that several rolesets declare in common and prefer the roleset a request length is banded to. Requires `AIBRIX_PROMPT_LENGTH_BUCKETING=true`. |\n| `AIBRIX_BUCKET_SERVE_MODE` | string | `\"throughput\"` | The unit the adaptive cut points are measured in: `throughput` (prompt token mass) or `rps` (request counts). An unknown value keeps the default. |\n| `AIBRIX_KV_CONNECTOR_TYPE` | string | `\"shfs\"` | KV cache transfer backend. Options: `shfs` (GPU shared memory), `nixl` (Neuron). |\n| `AIBRIX_PREFILL_SCORE_POLICY` | string | `\"prefix_cache\"` | Strategy for selecting the prefill pod. Options: `prefix_cache`, `least_request`. |\n| `AIBRIX_DECODE_SCORE_POLICY` | string | `\"load_balancing\"` | Strategy for selecting the decode pod. Options: `load_balancing`, `least_request`, `conductor`, `token_load`. |\n| `AIBRIX_DECODE_TOKEN_LOAD_OUTPUT_GROWTH` | bool | `true` | `token_load` decode policy only: add an estimate of the output generated so far (per-request decode rate × time since routing) to the decode score. `false` scores prompt tokens only. |\n| `AIBRIX_DECODE_TOKEN_LOAD_CAPACITY_AWARE` | bool | `true` | `token_load` decode policy only: multiply each decode pod's score by the candidates' mean KV capacity divided by the pod's own (`num_gpu_blocks` × `block_size` from vLLM's `vllm:cache_config_info`, or SGLang's `sglang:max_total_num_tokens`), so pods of different sizes are compared by how full they are. Unknown capacities count as the mean; with none known, or `false`, scores are token counts. |\n| `AIBRIX_TOKEN_LOAD_SHARED_LEDGER` | bool | `true` | `token_load` decode policy only, with Redis configured: share the decode ledger across gateway replicas, so each replica adds the other live replicas' charges to its decode score. `false` keeps the ledger local to each replica. |\n\nThe prefill/decode routing thresholds (`AIBRIX_PREFILL_*`, `AIBRIX_DECODE_*`, `AIBRIX_TOKEN_LOAD_*`, `AIBRIX_HYBRID_CACHE_LOAD_FACTOR`, `AIBRIX_MIN_MATCH_PCT`, `AIBRIX_PROMPT_LENGTH_BUCKETING`, `AIBRIX_BUCKET_SERVE`,\n`AIBRIX_BUCKET_SERVE_MODE`), except `AIBRIX_PREFILL_LOAD_CAP_FACTOR`, can also be set per request by the model config profile\n(`routingConfig.pd.*`, `routingConfig.promptLengthBucketing` and `routingConfig.bucketServe*`); see [Model Config Profile Overrides](#model-config-profile-overrides). `AIBRIX_PREFILL_LOAD_CAP_FACTOR` is environment-only.\n\n### PD Prefill Fail-Fast (`algorithms/pd/abort.go`)\n\nWhen the prefill leg of a PD request fails, the decode pod is left waiting for a\nKV transfer that will never arrive until its own bootstrap timeout expires\n(300s in SGLang), holding its pre-allocated KV pages. The gateway injects its own\nrequest id (`rid`) into both legs, so on a prefill failure it can post\n`{\"rid\": ...}` to the decode pod's `/abort_request` and release those pages\nimmediately. The abort is best-effort and asynchronous: nothing waits on it, and\nit is skipped once the decode pod has started answering the client - a call\nalready on the wire when that happens is cancelled rather than left to run.\n\nThe abort is sent twice, because the first attempt can overtake the decode\nrequest itself on the pod (the engine answers `200` either way, so the gateway\ncannot tell a matched abort from a dropped one). Repeating an abort the engine\nalready applied, or never matched, is a no-op on both sides. The wait between\nthe two attempts is interruptible, so a decode leg that starts answering in the\nmeantime ends the retry immediately instead of at the end of the delay.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | int (seconds) | `3` | Per-attempt timeout of the `/abort_request` call to the decode pod. `0` disables decode aborts entirely; the prefill failure is still recorded, logged and reported to the client. |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | int (seconds) | `2` | Delay before the second abort attempt. `0` sends a single attempt. |\n\nMetrics: `gateway_pd_decode_abort_total{prefill_failure_class, result}`, where\n`result` is one of `ok`, `error`, `skipped_streaming`, `skipped_disabled`,\n`skipped_no_rid`, `skipped_no_target`.\n\n### PD Decode Watchdog (`gateway_decode_watchdog.go`)\n\nIn SGLang disaggregation the prefill leg only returns once the decode pod has\ntaken the KV transfer, so a successful prefill means the decode pod owes the\nclient a response. If the decode pod then stops answering (its scheduler died or\nwedged), nothing else notices and the client hangs until Envoy's route timeout.\nThe watchdog bounds that wait: once the prefill leg succeeded and until the\nfirst response body chunk comes back from the decode pod, it fails the request\nafter the first-response timeout and posts one best-effort `/abort_request` for\nthe request's `rid` to the decode pod. Response headers do not end this phase:\nSGLang sends the headers of a streaming response before its first token, so a\ndecode pod that wedges before the first token still delivers them. How the\nclient is failed depends on whether those headers already went out:\n\n- No headers yet: the client gets a `504` carrying the `x-error-pd-decode`\n header, and the ext_proc stream is closed with `DeadlineExceeded`.\n- Headers sent: no error response can replace a response whose status line is\n already on its way, so the ext_proc stream is only closed with\n `DeadlineExceeded` and Envoy resets the client connection. The client sees a\n `200` with a truncated (here: empty) body and no `x-error-pd-decode` header.\n\nFrom the first chunk of a streaming response on, the watchdog watches the gap\nbetween chunks instead: a stream that goes silent for\n`AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` is cut the same way, and the decode pod\ngets the same abort. It is armed only for requests that carry a gateway-owned\n`rid`, i.e. SGLang PD requests.\n\nA prefill leg that answered `200` with a body the gateway could not parse\n(`bad_response`) counts as succeeded for the watchdog: the KV transfer completed\nall the same, so fail-fast leaves the request alone and the watchdog times the\ndecode pod from that answer.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | int (seconds) | `60` | **Streaming requests only** (`\"stream\": true`). Time to first token: how long the decode pod is given to send its first response body chunk, measured from the moment the prefill leg returned successfully. Response headers alone do not count. On expiry the client gets a `504` (`did not start responding within …`), or a cut stream if the headers already went out. `0` disables the watchdog for streaming requests. |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | int (seconds) | `0` (disabled) | **Non-streaming requests only**. Same arming point, different meaning: the decode pod sends the body of a non-streaming response only once generation is finished, so this budget bounds the *whole* decode, not the time to first token. It is off by default because the safe value depends on the caller's `max_tokens` and the pod's throughput; operators who know both can set one. On expiry the client gets a `504` (`did not finish responding within …`). |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | int (seconds) | `120` | **Streaming requests only**. Longest gap allowed between two response body chunks once the first one arrived. Every chunk re-arms it. On expiry the stream is cut short. `0` disables it. |\n\nThe abort uses `AIBRIX_DECODE_ABORT_TIMEOUT` as its timeout and is counted in\n`gateway_pd_decode_abort_total` with `prefill_failure_class=\"watchdog_first_response\"`\nor `\"watchdog_stream_idle\"`.\n\nMetrics: `gateway_pd_decode_watchdog_total{phase}`, where `phase` is\n`first_response` or `stream_idle`. A request failed by the watchdog is also counted in\n`gateway_request_model_fail_total` with `status=\"pd_decode_watchdog\"` and\n`status_code=\"504\"`, including the cut streams whose client saw a `200`: watch\nfor those through this counter and the client's stream error, not through an\nerror body.\n\n### Decode Load Balancer Scorer (`algorithms/pd/decode_scorer.go`)\n\nScoring formula: `score = (wRun × normRunning + wThroughput × normInvThroughput) / normFreeGPU`\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | float64 | `1.0` | Weight for the normalized running-request term in the decode LB score. |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | float64 | `1.0` | Weight for the normalized inverse-throughput term in the decode LB score. |\n\n---\n\n## Utilities (`algorithms/pd/engine/trtllm.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_TRT_MACHINE_ID` | int64 | `0` | 10-bit machine ID (0–1023) used in Snowflake-style disaggregation request ID generation: `[timestamp:41b][machineID:10b][counter:12b]`. Panics on init if out of range. |\n\n---\n\n## Model Config Profile Overrides\n\nA model's `model.aibrix.ai/config` annotation can carry per-profile routing knobs\n(`routingConfig`) that override the matching variable above for that profile's requests only.\nAn unset knob keeps the environment default, and a value the variable would reject (a negative\nfactor, a percentage outside its range, an unknown GPU name) is ignored, so a profile can only\nnarrow or sharpen routing behavior. The values are resolved once per request and applied to that\nrequest alone; a request whose profile sets none behaves exactly as before. The user-facing\ndescription and examples live in the Config Profiles section of the gateway plugin guide\n([docs/source/features/gateway-routing/gateway-plugins.rst](../../../docs/source/features/gateway-routing/gateway-plugins.rst)).\n\n| Variable | Profile field | Notes |\n|---|---|---|\n| `AIBRIX_TTFT_THRESHOLD_S` | `ttftThresholdS` | Top-level profile field, not inside `routingConfig`. `0` counts as unset and keeps the environment default, so a profile can only change the threshold to another positive value, never to `0`. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | `routingConfig.promptLengthBucketing` | Turns bucketing on or off for the profile's requests. |\n| `AIBRIX_BUCKET_SERVE` | `routingConfig.bucketServe` | Turns the adaptive plan on or off for the profile's requests. |\n| `AIBRIX_BUCKET_SERVE_MODE` | `routingConfig.bucketServeMode` | `throughput` or `rps`; a name the planner does not know keeps the environment default. |\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | `routingConfig.pd.decodeAbortTimeout` | `0` sends the abort without waiting. |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | `routingConfig.pd.decodeAbortRetryDelay` | `0` repeats the abort immediately. |\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | `routingConfig.pd.decodeFirstResponseTimeout` | Seconds; `0` disables the decode watchdog for the profile's streaming requests. |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | `routingConfig.pd.decodeResponseTimeout` | Seconds; `0` disables the decode watchdog for the profile's non-streaming requests. |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | `routingConfig.pd.decodeStreamIdleTimeout` | Seconds; `0` disables the stream-idle phase for the profile. |\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.prefillLoadImbalanceMinSpread` | |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.decodeLoadImbalanceMinSpread` | |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.decodeThroughputImbalanceMinSpread` | |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | `routingConfig.pd.decodeScoreRatioThreshold` | |\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | `routingConfig.pd.decodeLBWeightRunning` | |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | `routingConfig.pd.decodeLBWeightThroughput` | |\n| `AIBRIX_TOKEN_LOAD_KV_WEIGHT` | `routingConfig.pd.tokenLoadKVWeight` | |\n| `AIBRIX_TOKEN_LOAD_REQUEST_COST` | `routingConfig.pd.tokenLoadRequestCost` | |\n| `AIBRIX_TOKEN_LOAD_TTL_SECONDS` | `routingConfig.pd.tokenLoadTTLSeconds` | `0` disables the sweep for the profile's requests. |\n| `AIBRIX_TOKEN_LOAD_SESSION_TTL_SECONDS` | `routingConfig.pd.tokenLoadSessionTTLSeconds` | `0` disables the session delta. |\n| `AIBRIX_HYBRID_CACHE_LOAD_FACTOR` | `routingConfig.pd.hybridCacheLoadFactor` | Accepted range 0 to 1. |\n| `AIBRIX_MIN_MATCH_PCT` | `routingConfig.pd.minMatchPct` | Accepted range 0 to 100; `0` disables the minimum-match clamp. |\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | `routingConfig.pd.prefillRequestTimeout` | Seconds. |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` | `routingConfig.loadBalance.imbalanceFactor` | |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` | `routingConfig.loadBalance.imbalanceMinGap` | |\n| `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` | `routingConfig.loadBalance.queuedWeight` | `0` is the V1 formula, running requests only. |\n| `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` | `routingConfig.loadBalance.kvPressureAlpha` | `0` drops the penalty. |\n| `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` | `routingConfig.loadBalance.kvCriticalFree` | Accepted range 0 to 1; `0` disables the guardrail. |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `routingConfig.prefixCache.standardDeviationFactor` | Also read by the PD prefill candidacy filter. |\n| `AIBRIX_ROUTER_PREBLE_TARGET_GPU` | `routingConfig.preble.targetGPU` | Known values `A6000` and `V100`; anything else is ignored. |\n| `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` | `routingConfig.preble.decodingLength` | |\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | `routingConfig.vtc.maxPodLoad` | |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | `routingConfig.vtc.fairnessWeight` | `0` drops the fairness term. |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | `routingConfig.vtc.utilizationWeight` | `0` drops the utilization term. |\n| `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` | `routingConfig.autoBlend.loadBalanceWeight` | `0` disables the auto-blend for the profile's requests. |\n| `AIBRIX_ROUTING_AUTO_BLEND_LEAST_REQUEST_WEIGHT` | `routingConfig.autoBlend.leastRequestWeight` | |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` | `routingConfig.autoBlend.prefixCacheWeight` | `0` is rejected with a warning and the environment value is kept; it would drop the caller's own strategy from the blend. |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_LOAD_BALANCE_WEIGHT` | `routingConfig.autoBlend.prefixCacheLoadBalanceWeight` | `0` leaves those requests with prefix-cache scoring alone. |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | `routingConfig.vtc.inputTokenWeight` | Also scopes the VTC token tracker: this weight is baked into the tracker, so the profile's requests get one of their own. |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | `routingConfig.vtc.outputTokenWeight` | Scopes the tracker the same way as the input weight. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | `routingConfig.vtc.tokenTrackerWindowSize` | Sliding window of the profile's tracker, in `tokenTrackerTimeUnit` units. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | `routingConfig.vtc.tokenTrackerTimeUnit` | Bucket size of that window: `minutes`, `seconds` or `milliseconds`. An unknown name is ignored instead of being normalized. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | `routingConfig.vtc.tokenTrackerMinTokens` | Floor the profile's tracker reports while its window holds little activity. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | `routingConfig.vtc.tokenTrackerMaxTokens` | Ceiling the profile's tracker reports while its window holds little activity. |\n\n### Scoped state behind the profile overrides\n\nThe VTC token tracker is constructed once from its window, time unit, token floors and the two\ntoken weights, so a profile that overrides any of the six gets a tracker of its own instead of\nretuning the shared one: trackers are keyed by the resolved values, so profiles that agree share\none, a profile that sets none keeps the shared tracker exactly as before, and the number of\ntrackers is bounded (16 per process; a profile past the bound keeps the shared tracker rather\nthan failing its requests).\n\n### Environment-only routing variables\n\nThese configure state shared by every model of one gateway process, so a profile cannot override\nthem without splitting or resizing that state for everyone:\n\n- `AIBRIX_ROUTER_PREBLE_SLIDING_WINDOW_PERIOD` and `AIBRIX_ROUTER_PREBLE_EVICTION_LOOP_INTERVAL`: the preble histogram window and its eviction loop are process-wide timers; scoping those needs a design of its own.\n- `AIBRIX_SESSION_AFFINITY_MAX_LOCAL_KEYS`: bounds the gateway-local session pin cache, whose entries are shared facts about a model and a session.\n- `AIBRIX_ROUTER_MAX_CACHED_ALGORITHM_STRINGS`: bounds the process-wide routing string caches.\n- `AIBRIX_TOKEN_LOAD_MAX_SESSIONS`: bounds the (model, session) table of the token-load tracker, which the gateway shares across models.\n\nVariables without a `routingConfig` field (tokenizer endpoints, Redis and statesync settings, the\nrate limiting switches, `AIBRIX_KV_CONNECTOR_TYPE`) stay environment-only as deployment-level\nsettings.\n\n---\n\n## Variable Dependency Notes\n\nThe following variables have interdependencies that must be satisfied together:\n\n- **KV event sync** requires all three to be set consistently:\n ```\n AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true\n AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true\n AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote\n AIBRIX_PREFIX_CACHE_REMOTE_TOKENIZER_ENDPOINT=\n ```\n\n- **Remote tokenizer pool** is initialized whenever `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`. The pool variables (`AIBRIX_VLLM_TOKENIZER_ENDPOINT_TEMPLATE`, `AIBRIX_TOKENIZER_*`, `AIBRIX_MAX_TOKENIZERS_PER_POOL`) all apply in that case.\n\n- **`AIBRIX_PREFIX_CACHE_POD_RUNNING_REQUEST_IMBALANCE_ABS_COUNT` is removed.** The\n prefix-cache-specific load-imbalance gate it used to tune was replaced by the centralized gate\n described under [Load Balance Router](#load-balance-router-algorithmsload_balancego), which\n now applies to prefix-cache (and every other non-exclusive strategy) via\n `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` / `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` instead. The\n trigger condition also changed: the old gate fired on absolute gap alone; the new one\n additionally requires a relative `factor × (mean + 1)` check for 3+ pod clusters, so it fires\n less often on larger fleets. Setting the old variable now only logs a startup warning.\n\n## OpenTelemetry\n\nThe gateway plugins feature built-in support for distributed tracing via OpenTelemetry (OTel), empowering you to monitor and trace end-to-end requests across the entire external processing pipeline.\n\n**Tracing is opt-in by default.** Telemetry components will only initialize if you explicitly configure either ``OTEL_EXPORTER_OTLP_ENDPOINT`` or ``OTEL_EXPORTER_OTLP_TRACES_ENDPOINT``. If both are omitted, all tracing capabilities remain disabled to conserve system resources.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|\n| `OTEL_EXPORTER_OTLP_PROTOCOL` | string | `grpc` | The transport protocol for OTLP data. Valid options are `grpc`, `http`, or `http/protobuf`. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | string | `\"\"` | Base URL for all OTLP signals. **⚠️ Note:** ensure your ENDPOINT URLs include the `http://` or `https://` prefix. The SDK automatically appends signal-specific paths to this URL (e.g., appending `/v1/traces` for HTTP exports). | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | string | `\"\"` | Target URL specifically for traces. Takes precedence over the global `ENDPOINT`. **⚠️ Note:** This URL is used exactly **as-is**. The SDK will **NOT** automatically append `/v1/traces` to it. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_HEADERS` | string | `\"\"` | Key-value pairs used as headers for OTLP requests (e.g., `key1=value1,key2=value2`). Useful for passing auth tokens to backends like Datadog or Honeycomb. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_TIMEOUT` | string | `10s` | Maximum time the OTLP exporter will wait for each batch export. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_INSECURE` | bool | `false` | Set to `true` to disable TLS/HTTPS for the exporter. Force downgrade to HTTP. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_INSECURE_SKIP_VERIFY` | bool | `false` | Keeps TLS active but skips server certificate validation (useful for self-signed certs). Set to \"true\" to enable. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n\n> **Note on OpenTelemetry Configuration:**\n> Aibrix supports the standard OpenTelemetry Protocol Exporter environment variables. For advanced OTLP configurations (such as `_CERTIFICATE`, `_CLIENT_KEY`, or specific `_COMPRESSION` settings), please refer to the [OpenTelemetry Protocol Exporter](https://opentelemetry.io/docs/specs/otel/protocol/exporter/)\n","isInternal":false,"tokens":11807,"sizeBytes":49090},{"name":"envoy_gateway_v1.2.8_xds.json","path":"pkg/plugins/gateway/testdata/envoy_gateway_v1.2.8_xds.json","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/testdata/envoy_gateway_v1.2.8_xds.json","title":"Testdata Skill","category":"anthropic-skill","format":"json","content":"{\n \"xds\": {\n \"aibrix-system/aibrix-eg\": {\n \"configs\": [\n {\n \"@type\": \"type.googleapis.com/envoy.admin.v3.ListenersConfigDump\",\n \"dynamicListeners\": [\n {\n \"activeState\": {\n \"listener\": {\n \"@type\": \"type.googleapis.com/envoy.config.listener.v3.Listener\",\n \"accessLog\": [\n {\n \"filter\": {\n \"responseFlagFilter\": {\n \"flags\": [\n \"NR\"\n ]\n }\n },\n \"name\": \"envoy.access_loggers.file\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\",\n \"logFormat\": {\n \"jsonFormat\": {\n \":authority\": \"%REQ(:AUTHORITY)%\",\n \"bytes_received\": \"%BYTES_RECEIVED%\",\n \"bytes_sent\": \"%BYTES_SENT%\",\n \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n \"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\n \"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\n \"duration\": \"%DURATION%\",\n \"method\": \"%REQ(:METHOD)%\",\n \"protocol\": \"%PROTOCOL%\",\n \"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\n \"response_code\": \"%RESPONSE_CODE%\",\n \"response_code_details\": \"%RESPONSE_CODE_DETAILS%\",\n \"response_flags\": \"%RESPONSE_FLAGS%\",\n \"route_name\": \"%ROUTE_NAME%\",\n \"start_time\": \"%START_TIME%\",\n \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n \"upstream_host\": \"%UPSTREAM_HOST%\",\n \"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\n \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\n \"user-agent\": \"%REQ(USER-AGENT)%\",\n \"x-envoy-origin-path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n \"x-envoy-upstream-service-time\": \"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\n \"x-forwarded-for\": \"%REQ(X-FORWARDED-FOR)%\",\n \"x-request-id\": \"%REQ(X-REQUEST-ID)%\"\n }\n },\n \"path\": \"/dev/stdout\"\n }\n }\n ],\n \"address\": {\n \"socketAddress\": {\n \"address\": \"0.0.0.0\",\n \"portValue\": 10080\n }\n },\n \"defaultFilterChain\": {\n \"filters\": [\n {\n \"name\": \"envoy.filters.network.http_connection_manager\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\",\n \"accessLog\": [\n {\n \"name\": \"envoy.access_loggers.file\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\",\n \"logFormat\": {\n \"jsonFormat\": {\n \":authority\": \"%REQ(:AUTHORITY)%\",\n \"bytes_received\": \"%BYTES_RECEIVED%\",\n \"bytes_sent\": \"%BYTES_SENT%\",\n \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n \"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\n \"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\n \"duration\": \"%DURATION%\",\n \"method\": \"%REQ(:METHOD)%\",\n \"protocol\": \"%PROTOCOL%\",\n \"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\n \"response_code\": \"%RESPONSE_CODE%\",\n \"response_code_details\": \"%RESPONSE_CODE_DETAILS%\",\n \"response_flags\": \"%RESPONSE_FLAGS%\",\n \"route_name\": \"%ROUTE_NAME%\",\n \"start_time\": \"%START_TIME%\",\n \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n \"upstream_host\": \"%UPSTREAM_HOST%\",\n \"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\n \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\n \"user-agent\": \"%REQ(USER-AGENT)%\",\n \"x-envoy-origin-path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n \"x-envoy-upstream-service-time\": \"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\n \"x-forwarded-for\": \"%REQ(X-FORWARDED-FOR)%\",\n \"x-request-id\": \"%REQ(X-REQUEST-ID)%\"\n }\n },\n \"path\": \"/dev/stdout\"\n }\n }\n ],\n \"commonHttpProtocolOptions\": {\n \"headersWithUnderscoresAction\": \"REJECT_REQUEST\"\n },\n \"http2ProtocolOptions\": {\n \"initialConnectionWindowSize\": 1048576,\n \"initialStreamWindowSize\": 65536,\n \"maxConcurrentStreams\": 100\n },\n \"httpFilters\": [\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"STREAMED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"STREAMED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"BUFFERED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"name\": \"envoy.filters.http.router\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\",\n \"suppressEnvoyHeaders\": true\n }\n }\n ],\n \"mergeSlashes\": true,\n \"normalizePath\": true,\n \"pathWithEscapedSlashesAction\": \"UNESCAPE_AND_REDIRECT\",\n \"rds\": {\n \"configSource\": {\n \"ads\": {},\n \"resourceApiVersion\": \"V3\"\n },\n \"routeConfigName\": \"aibrix-system/aibrix-eg/http\"\n },\n \"serverHeaderTransformation\": \"PASS_THROUGH\",\n \"statPrefix\": \"http-10080\",\n \"useRemoteAddress\": true\n }\n }\n ],\n \"name\": \"aibrix-system/aibrix-eg/http\"\n },\n \"name\": \"aibrix-system/aibrix-eg/http\",\n \"perConnectionBufferLimitBytes\": 4194304\n }\n }\n }\n ]\n },\n {\n \"@type\": \"type.googleapis.com/envoy.admin.v3.RoutesConfigDump\",\n \"dynamicRouteConfigs\": [\n {\n \"routeConfig\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.RouteConfiguration\",\n \"ignorePortInHostMatching\": true,\n \"name\": \"aibrix-system/aibrix-eg/http\",\n \"virtualHosts\": [\n {\n \"domains\": [\n \"*\"\n ],\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"Gateway\",\n \"name\": \"aibrix-eg\",\n \"namespace\": \"aibrix-system\",\n \"sectionName\": \"http\"\n }\n ]\n }\n }\n },\n \"name\": \"aibrix-system/aibrix-eg/http/*\",\n \"routes\": [\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"safeRegex\": {\n \"regex\": \"^/v1/videos/sync$|^/v1/videos/[^/]+/content$\"\n }\n },\n \"name\": \"original_route_videos_streaming\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"safeRegex\": {\n \"regex\": \"^/v1/videos(/[^/]+)?$\"\n }\n },\n \"name\": \"original_route_videos\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"prefix\": \"/\"\n },\n \"name\": \"original_route\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"path\": \"/v1/videos/sync\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos-streaming\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"safeRegex\": {\n \"regex\": \"/v1/videos/[^/]+/content\"\n }\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos-streaming\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0/match/1/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/transcriptions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/15/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/images/generations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/13/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/translations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/16/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/video/generations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/19/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/chat/completions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/voices\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/18/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/images/edits\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/14/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/speech\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/17/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/completions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/3/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/embeddings\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/4/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/systemone\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/9/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/decisions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/8/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/responses\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/1/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/classify\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/6/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/messages\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/2/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"directResponse\": {\n \"status\": 500\n },\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/batches\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-metadata-endpoint\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-metadata-endpoint/rule/0/match/2/*\"\n },\n {\n \"directResponse\": {\n \"status\": 500\n },\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/models\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-metadata-endpoint\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-metadata-endpoint/rule/0/match/0/*\"\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/videos\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/rerank\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/5/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/tokenize\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/11/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy","isInternal":false,"tokens":7532,"sizeBytes":50000},{"name":"README.md","path":"pkg/plugins/gateway/ratelimiter/README.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/ratelimiter/README.md","title":"ratelimiter Documentation","category":"plugin-manifest","format":"markdown","content":"# ratelimiter\n\nPackage `ratelimiter` provides a distributed, Redis-backed fixed-window rate limiter used by the Aibrix gateway to enforce per-user and per-model request limits across multiple gateway instances.\n\n## Interface\n\n```go\ntype RateLimiter interface {\n Get(ctx context.Context, key string) (int64, error)\n GetLimit(ctx context.Context, key string) (int64, error)\n Incr(ctx context.Context, key string, val int64, window ...time.Duration) (int64, error)\n}\n```\n\n| Method | Description |\n|------------|-------------|\n| `Get` | Returns the current counter value for a key in the active time window. |\n| `GetLimit` | Returns the configured maximum for a key (used for limit lookups, not window counters). |\n| `Incr` | Atomically increments the counter by `val` and returns the new value. The first write sets the key TTL to the window size; later writes do not extend it. |\n\n## Implementations\n\n### `redisRateLimiter` — `NewRedisAccountRateLimiter(name, client, windowSize)`\n\nA fixed-window counter backed by Redis. Suitable for cluster-wide enforcement when multiple gateway replicas share the same Redis instance.\n\n**Counter key scheme:** `{name}:{key}:{windowMilliseconds}:counter`. The key is stable for each configured window duration and separate from the static `{name}:{key}` key used by `GetLimit`. The counter expires `windowSize` after its first write, and the next write starts a new window. This window is independent of wall-clock bucket boundaries.\n\n- Minimum `windowSize` is 1 second (smaller values are clamped up).\n- `Incr` uses a Redis Lua script to increment and set the TTL atomically on the first write.\n- A missing key (`redis.Nil`) is treated as `0`, not an error.\n\n### `noopRateLimiter` — `NewNoopRateLimiter()`\n\nA no-op implementation that always allows requests. Used when Redis is unavailable (e.g., local development without a Redis sidecar). `GetLimit` returns `math.MaxInt64`.\n\n## Usage in the gateway\n\nTwo limiter instances are created at server startup:\n\n| Instance | Constructor arg `name` | `windowSize` | Purpose |\n|--------------------|------------------------|--------------|----------------------------------|\n| `ratelimiter` | `\"aibrix\"` | 1 minute | Per-user RPM and TPM enforcement |\n| `modelRateLimiter` | `\"aibrix_model\"` | 1 second | Per-model RPS enforcement |\n\nThe separate `name` prefix ensures the two limiters' keys never collide in Redis.\n\n### Per-user limits (RPM / TPM)\n\nCounter keys follow the pattern `aibrix:{username}_{RPM|TPM}_CURRENT:60000:counter`. Limits are read from the user record resolved during request header processing.\n\n### Per-model RPS\n\nCounter keys follow the pattern `aibrix_model:{modelName}_MODEL_RPS_CURRENT:{windowMilliseconds}:counter`. The RPS limit is configured per model via the `requestsPerSecond` field in a config profile:\n\n```json\n{\n \"profiles\": {\n \"rps-limited\": {\n \"routingStrategy\": \"least-request\",\n \"requestsPerSecond\": 1\n }\n }\n}\n```\n\nThe active profile is selected by passing the `config-profile` request header. If no profile is active or `requestsPerSecond` is unset, the RPS check is skipped entirely.\n\n#### Enforcement flow\n\nPer-model RPS enforcement is handled in `HandleRequestBody` via `enforceModelRPS` and a deferred compensation step:\n\n1. **Pre-routing gate (`enforceModelRPS`)** — called before routing. It:\n - atomically increments the counter with `Incr(..., +1)` and receives the new value\n - rejects with HTTP 429 and rolls back with `Incr(..., -1)` if `newVal > limit`\n - allows the request through when `newVal <= limit`\n2. **Deferred compensation (`decrModelRPS`)** — armed immediately after a successful pre-charge. If routing later fails, the deferred call refunds quota with `Incr(..., -1)`.\n3. **Success path** — after routing succeeds and request accounting is attached, compensation is disabled, so the pre-charge remains counted.\n\nUsing incr-then-check (rather than check-then-increment) means the `INCRBY` is the sole gate. Because Redis processes `INCRBY` atomically, each concurrent caller receives a unique sequential result — eliminating the TOCTOU race that would otherwise allow over-admission during the check→increment window.\n\n### Upgrading from wall-clock bucket keys\n\nThe counter key format changed from `{name}:{key}:{timebin}` to `{name}:{key}:{windowMilliseconds}:counter`. Existing bucket keys expire on their own TTL, but the new limiter does not read their counts. A deployment therefore starts a new counter window for each key. During a rolling upgrade, old and new gateway instances count requests separately and may collectively admit more than the configured limit. Deployments that require uninterrupted strict limits should avoid serving rate-limited traffic from both versions at the same time; a cutover to the new version still starts fresh counters.\n","isInternal":false,"tokens":1117,"sizeBytes":4934},{"name":"README.md","path":"pkg/plugins/gateway/statesync/README.md","rawUrl":"https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/statesync/README.md","title":"statesync Documentation","category":"plugin-manifest","format":"markdown","content":"# statesync\n\nGeneric Redis-backed state sync for multiple gateway (or other) replicas. Uses **per-entity keys** with `SETEX` (per-record TTL), `MGET` (batched), and `SCAN` (enumeration). Keys are namespaced via a hash tag (`aibrix:{namespace}:e:{id}`). No PUB/SUB.\n\n- **Periodic sync**: Each replica runs **Pull first** (load from Redis), then **Push** (delta or full) every sync period.\n- **Optional write-through**: After local updates you may call `Put`/`Delete` directly. The default gateway wiring uses periodic delta push only.\n- **Deletions/tombstones**: If a Syncable implements `TombstoneSupport`, deletions are written as tombstone payloads so peers can remove the local entry on the next pull; otherwise `DEL` is used.\n- **Jitter and backoff**: Random initial startup delay (up to half `syncPeriod`); per-cycle jitter is roughly ±10% of `syncPeriod` (see `syncLoop` in `redissync.go`); exponential backoff (capped at 1 min) on errors.\n- **Lifecycle**: Register all Syncables **before** `Start()`; registrations after `Start()` are silently ignored.\n\n**Per-record TTL:** Each entity is stored as its own key with `SETEX` and a default TTL of 2 minutes. A record expires if it is not rewritten within that window. The TTL of one record does not affect others.\n\nRedis key format: `aibrix:{namespace}:e:{entityId}` — `{namespace}` is the hash tag. Namespace must be non-empty.\n\n### Prefix cache (gateway default wiring)\n\nWith `AIBRIX_STATESYNC_ENABLED=true`, `cmd/plugins/main.go`:\n\n1. Calls `table.EnableDeltaSync()` to activate dirty tracking on `PrefixHashTable` (no-op cost when disabled).\n2. Registers `PrefixHashTableSyncable` with the manager.\n3. Relies on **periodic delta push + pull** — `Put` is not called on every `AddPrefix`.\n\nThis gives eventual consistency with staleness bounded by the sync interval and per-key TTL. For faster cross-replica visibility you can call `syncManager.Put(ctx, \"prefixcache\", blockIDStr, data)` on the `*statesync.RedisSync` using bytes from `EncodeBlockForSync` after each `AddPrefix` (higher Redis write load trade-off).\n\n### Delta sync and LRU eviction\n\n`EnableDeltaSync()` must be called on `PrefixHashTable` before it is registered with `*statesync.RedisSync`; without it, dirty tracking is inactive and `GetDeltaForSync` always returns an empty delta. If a block is marked dirty then **evicted locally** before the next successful push, `GetDeltaForSync` skips it silently. The corresponding Redis key remains until the per-entity TTL expires. Strong delete propagation is not guaranteed for evicted entries.\n\n## Diagrams\n\n### Technical Flow\n\n```mermaid\nsequenceDiagram\n participant SyncLoop\n participant Redis\n participant Syncable\n Note over SyncLoop: Pull first, then Push every syncPeriod\n SyncLoop->>Redis: SCAN pattern (keys for namespace)\n Redis-->>SyncLoop: keys[]\n SyncLoop->>Redis: MGET keys (batched)\n Redis-->>SyncLoop: values[]\n SyncLoop->>Syncable: ApplyRemote(id, bytes) for each\n SyncLoop->>Syncable: GetDelta or GetSnapshot\n Syncable-->>SyncLoop: updated, deleted\n SyncLoop->>Redis: SETEX key val TTL per entity, DEL deleted\n SyncLoop->>Syncable: OptionalHooks (if implemented)\n```\n\n---\n\n## Adopting statesync in a new component\n\nChanges live **only in the package that owns the state**. The `statesync` package stays generic and never imports component types.\n\n### 1. Implement `syncable.Syncable`\n\nDepend on `github.com/vllm-project/aibrix/pkg/utils/syncable` and implement:\n\n| Method | Purpose |\n|--------|---------|\n| `Namespace() string` | Stable name for this state (e.g. `\"mytracker\"`). Used in the Redis key. |\n| `GetSnapshot(ctx) (map[string][]byte, error)` | Return current local state as **entity id → serialized bytes**. |\n| `ApplyRemote(ctx, id string, data []byte) error` | Apply one entity from Redis into local state (merge or overwrite). |\n\nOptional interfaces:\n\n| Interface | Purpose |\n|-----------|---------|\n| `DeltaSyncable` | Add `GetDelta` / `ClearDirty` to push only changed entities instead of the full snapshot. |\n| `OptionalHooks` | `OnSyncStart` / `OnSyncEnd` called around each **Pull** for that Syncable (not around Push). |\n| `StalePolicy` | `IsStale` — if true, the sync layer deletes the remote key and skips applying. |\n| `TombstoneSupport` | `MakeTombstone` / `IsTombstone` / `DeleteLocal` — propagate deletes as payloads instead of DEL. |\n\n### 2. Add serialization\n\n- **Encode**: turn your struct into `[]byte` (e.g. JSON). Use a stable format all replicas can decode.\n- **Decode**: in `ApplyRemote`, unmarshal `data` and merge into or replace local state.\n\n### 3. Expose a constructor returning `syncable.Syncable`\n\nReturn an adapter that holds a pointer to your state and implements the interface. Callers pass it to `(*statesync.RedisSync).Register(...)`.\n\n### 4. Wire in the process entry point\n\n```go\nrs := statesync.New(redisClient,\n statesync.WithSyncPeriod(30*time.Second), // optional; default 10s\n statesync.WithOpTimeout(15*time.Second), // optional; default 30s\n statesync.WithKeyPrefix(\"myapp\"), // optional; default \"aibrix\"\n)\n// Register all Syncables before Start.\nrs.Register(mypkg.NewMyStateSyncable(myState))\nrs.Start()\n\n// On shutdown:\nrs.Stop() // blocks until the sync loop exits or stop-wait timeout (default 90s)\n```\n\nAvailable `statesync.New` options:\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `WithSyncPeriod(d)` | 10s | Interval between sync cycles (jitter applied). |\n| `WithOpTimeout(d)` | 30s | Per-operation context timeout for Pull/Push. |\n| `WithKeyPrefix(s)` | `\"aibrix\"` | Prefix for all Redis keys. |\n| `WithRecordTTL(d)` | 2m | Per-entity `SETEX` TTL; values ≤0 are ignored. |\n| `WithStopWaitTimeout(d)` | 90s | Max time `Stop()` waits for the background loop to exit. |\n| `WithSetexChunkSize(n)` | 25 | Pipeline chunk size for SETEX writes. |\n| `WithMGetBatchSize(n)` | 200 | Batch size for MGET during Pull. |\n\n### Redis client\n\nThe Redis client is provided by the caller (e.g. `utils.GetRedisClient()`). Configure TLS and auth in the client options for production (see `pkg/utils/redis.go`).\n\n---\n\n## Example: a new component \"TokenTracker\"\n\nA component that keeps `tokenID → lastUsedTime` and should sync across gateway replicas.\n\n### Step 1 – Implement Syncable in the component package\n\n```go\n// pkg/plugins/gateway/algorithms/vtc/token_tracker_sync.go\n\nimport (\n \"context\"\n \"encoding/json\"\n \"strconv\"\n \"time\"\n\n \"github.com/vllm-project/aibrix/pkg/utils/syncable\"\n)\n\nconst tokenTrackerNamespace = \"token_tracker\"\n\n// TokenTrackerSyncable adapts TokenTracker for statesync.\ntype TokenTrackerSyncable struct {\n Tracker *TokenTracker\n}\n\nfunc NewTokenTrackerSyncable(t *TokenTracker) syncable.Syncable {\n return &TokenTrackerSyncable{Tracker: t}\n}\n\nfunc (s *TokenTrackerSyncable) Namespace() string { return tokenTrackerNamespace }\n\nfunc (s *TokenTrackerSyncable) GetSnapshot(ctx context.Context) (map[string][]byte, error) {\n s.Tracker.mu.RLock()\n defer s.Tracker.mu.RUnlock()\n out := make(map[string][]byte, len(s.Tracker.entries))\n for id, t := range s.Tracker.entries {\n b, _ := json.Marshal(t.UnixNano())\n out[strconv.FormatInt(id, 10)] = b\n }\n return out, nil\n}\n\nfunc (s *TokenTrackerSyncable) ApplyRemote(ctx context.Context, id string, data []byte) error {\n var nano int64\n if err := json.Unmarshal(data, &nano); err != nil {\n return err\n }\n parsed, _ := strconv.ParseInt(id, 10, 64)\n remote := time.Unix(0, nano)\n s.Tracker.mu.Lock()\n defer s.Tracker.mu.Unlock()\n if existing, ok := s.Tracker.entries[parsed]; !ok || remote.After(existing) {\n s.Tracker.entries[parsed] = remote\n }\n return nil\n}\n```\n\n### Step 2 – Wire in the entry point\n\n```go\nrs := statesync.New(redisClient, statesync.WithSyncPeriod(30*time.Second))\nrs.Register(prefixcacheindexer.NewPrefixHashTableSyncable(prefixHashTable))\nrs.Register(vtc.NewTokenTrackerSyncable(tokenTracker))\nrs.Start()\ndefer rs.Stop()\n```\n\n### Step 3 – Optional write-through\n\nIf you want other replicas to see a change before the next periodic push:\n\n```go\nfunc (t *TokenTracker) RecordUse(ctx context.Context, id int64, rs *statesync.RedisSync) {\n t.mu.Lock()\n t.entries[id] = time.Now()\n t.mu.Unlock()\n if rs != nil {\n data, _ := json.Marshal(time.Now().UnixNano())\n _ = rs.Put(ctx, tokenTrackerNamespace, strconv.FormatInt(id, 10), data)\n }\n}\n```\n\n---\n\n## Reference implementation\n\nSee `pkg/utils/prefixcacheindexer/sync.go`:\n- `PrefixHashTableSyncable` and `NewPrefixHashTableSyncable(table)` — full `DeltaSyncable` implementation.\n- `GetSnapshotForSync` / `GetDeltaForSync` / `ClearDirtyForSync` / `ApplyRemoteForSync` / `EncodeBlockForSync` on `PrefixHashTable`.\n- Remember to call `table.EnableDeltaSync()` before registering if you want delta push (not required for snapshot-only sync).\n","isInternal":false,"tokens":2264,"sizeBytes":8932}],"systemPromptSnippet":"\n\n# AIBrix\n\nAIBrix is a Kubernetes-native platform for building scalable GenAI inference\ninfrastructure. This repository contains Go controllers, gateway plugins,\ncustom resources, deployment manifests, tests, and Python runtime components.\nThis file gives coding agents the repository-specific context and safety rules\nneeded to make focused changes.\n\n## Scope and repository layout\n\n| Path | Role |\n|------|------|\n| `api/` | Kubernetes API types and CRD definitions |\n| `pkg/controller/` | Kubernetes controllers and reconciliation logic |\n| `pkg/plugins/` | Gateway plugins and request processing |\n| `pkg/` | Shared Go libraries, clients, caches, metrics, and utilities |\n| `cmd/` | Go binary entrypoints |\n| `config/` | Kustomize configuration, CRDs, RBAC, and deployment manifests |\n| `test/integration/` | Ginkgo-based integration tests |\n| `test/e2e/` | Kind and cluster-level end-to-end tests |\n| `hack/` | Code generation, verification, and CI scripts |\n| `python/aibrix/` | Python runtime, downloader, batch, metadata, and optimizer |\n| `python/aibrix_kvcache/` | Python distributed KV-cache components |\n| `apps/` | Application services and the console |\n| `docs/` | Project documentation and generated documentation assets |\n| `samples/` | Example deployments and feature samples |\n\nRules in a deeper `AGENTS.md` take precedence for files under that directory.\nThe Python runtime has additional guidance in\n[`python/aibrix/AGENTS.md`](python/aibrix/AGENTS.md).\n\n## Working rules\n\n- State the intended scope and success criteria before making a non-trivial\n change.\n- Read the closest existing implementation and its tests before introducing a\n new pattern.\n- Keep changes focused. Do not combine unrelated refactors, formatting, or\n generated output with a feature or bug fix.\n- Verify behavior from code and tests rather than inferring it from filenames.\n- Preserve unrelated user changes in the working tree.\n- Do not push branches, rewrite history, modify remote GitHub state, or send\n external messages without explicit authorization for that action.\n- Ask before upgrading dependencies or changing `.github/` workflow policy.\n\n## API and compatibility surfaces\n\nTreat the following as compatibility surfaces:\n\n- Kubernetes API types, CRD schemas, JSON/YAML tags, defaults, validation,\n list semantics, subresources, and printer columns.\n- Public HTTP endpoints, request and response fields, headers, status codes,\n and gateway plugin contracts.\n- CLI flags, configuration keys, environment variables, labels, and\n annotations used by deployment or controller logic.\n\nDo not rename, remove, or repurpose an existing field or key without an\nexplicit migration and compatibility plan. Preserve pointer and `omitempty`\nchoices when absent, zero, and explicit values have different meanings.\nKeep status fields observational; status transitions must remain consistent\nwith every controller that writes or consumes them.\n\nKeep API types declarative. Admission behavior belongs in webhooks,\nreconciliation policy belongs in controllers, and transport compatibility\nbelongs in the relevant API or gateway layer.\n\n## Build and verification\n\nRun commands from the repository root. Start with the narrowest relevant check\nand run broader checks for changes that cross package or language boundaries.\n\n| Command | Purpose |\n|---------|---------|\n| `make fmt` | Format Go code |\n| `make vet` | Run `go vet ./...` |\n| `make generate` | Regenerate Go and Kubernetes generated artifacts |\n| `make manifests` | Regenerate webhook, RBAC, and CRD manifests |\n| `make verify` | Verify generated code and CRD synchronization |\n| `make lint` | Run Go linting |\n| `make lint-all` | Run license and Go lint checks |\n| `make test` | Run Go unit tests, excluding integration and E2E packages |\n| `make test-race-condition` | Run Go unit tests with the race detector |\n| `make test-integration` | Run integration tests |\n| `make python-ci` | Run Python Ruff, mypy, and tests for `python/aibrix` |\n| `make test-e2e` | Run Kind-based end-to-end tests |\n\nFor a Go package, a focused check such as `go test ./pkg//...` is\nappropriate while iterating. For cluster-level behavior, use the existing\nhelpers and test organization under `test/e2e/`; do not replace polling with\narbitrary sleeps.\n\nWhen API types, controller-gen markers, or CRDs change, run the applicable\ngeneration and verification targets and include the resulting generated files\nin the change. Do not hand-edit generated files unless the generator explicitly\nrequires it.\n\n## Coding conventions\n\n- Follow existing Go package structure and standard Go style; use `gofmt` or\n `make fmt`.\n- Put reusable implementation in `pkg/`; keep `cmd/` entrypoints thin.\n- Use table-driven tests where they match the surrounding package style.\n- Add or update tests for behavior changes, including failure and compatibility\n cases when applicable.\n- Comments should explain non-obvious reasons or invariants, not restate code.\n- Preserve existing logging, error handling, retry, timeout, and context\n propagation patterns unless the change specifically addresses them.\n- For Python changes, follow the more specific rules in\n [`python/aibrix/AGENTS.md`](python/aibrix/AGENTS.md) and run checks from that\n subtree as documented there.\n\n## Pull requests and documentation\n\n- Keep a PR focused on one issue or coherent behavior change.\n- Use the repository [PR template](.github/PULL_REQUEST_TEMPLATE.md), link the\n relevant issue, and describe the tests actually run.\n- Update user or developer documentation when behavior, configuration, API,\n deployment, or CLI usage changes.\n- When changing any `docs/source/**/*.rst` file, update the corresponding\n `docs/source/locale/zh_CN/LC_MESSAGES/**/*.po` catalog in the same change,\n including added, modified, and removed content. Do not leave new or changed\n documentation falling back to English on the Chinese site.\n- Before submitting an RST documentation change, run `make html-all` from\n `docs/` and confirm that both the English and Simplified Chinese HTML builds\n succeed.\n- Large design changes should be discussed before implementation and recorded\n in the appropriate documentation area.\n- Review every generated or AI-assisted change yourself; do not submit code\n that you cannot explain and verify.\n\n## Further reading\n\n- [Contributing guide](CONTRIBUTING.md)\n- [Development guide](development/README.md)\n- [Test guide](test/README.md)\n- [Project documentation](docs/README.md)\n\n\n{\n \"apiVersion\": \"routing.aibrix.ai/v1alpha1\",\n \"kind\": \"ReplicaSelectionResponse\",\n \"metadata\": {\"requestId\": \"req-123\", \"decisionId\": \"decision-789\"},\n \"status\": {\"decision\": \"Denied\", \"reason\": \"CompliancePolicy\"}\n}\n\n\n{\n \"apiVersion\": \"routing.aibrix.ai/v1alpha1\",\n \"kind\": \"ReplicaSelectionResponse\",\n \"metadata\": {\"requestId\": \"req-123\"},\n \"status\": {\"decision\": \"NoDecision\", \"reason\": \"NoApplicablePolicy\"}\n}\n\n\n# Testdata Skill\n\nPath: `pkg/plugins/gateway/algorithms/external/testdata/selected.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vllm-project/aibrix/HEAD/pkg/plugins/gateway/algorithms/external/testdata/selected.json)\n\n\n# Load Balance Router\n\n`load-balance` routes each request to the replica with the lowest **effective load**: the work\nalready committed to it, divided by how fast it has been observed to drain work, inflated as its\nKV cache fills. It needs no hardware labels. A mixed B40 / A100 / H20 pool teaches the gateway\nits relative speeds from observed output-token throughput, and faster replicas receive\nproportionally more traffic.\n\nSource: [load_balance.go](load_balance.go). Environment variables:\n[ENV_VARS.md](../ENV_VARS.md#load-balance-router-algorithmsload_balancego).\n\n`load-balance` is also blended in behind other strategies by default (see\n`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` and [multi_router_readme.md](multi_router_readme.md)),\nso its `ScoreAll` runs on most requests, not only when it is selected explicitly.\n\n## Score\n\n```\n running + λ · queued\nscore = ───────────────────────────────────── × (1 + α · (1 − kvFree)²)\n EWMA(output tokens completed / sec)\n\nkvFree < kvCritical ⇒ score = +Inf\n```\n\nThe lowest score wins. The three parts answer three separate questions, and each is used for one\npurpose only:\n\n| Part | Question | Source |\n|---|---|---|\n| `EWMA(output tokens/sec)` | How fast does this replica drain work? (**capacity**) | `RealtimeOutputTokenRateEWMA`, derived in the cache from completed output tokens |\n| `running + λ·queued` | How much work is already committed to it? (**load**) | `GetPodsRunningRequests` (live cross-gateway count); `num_requests_waiting` when `λ > 0` |\n| `1 + α(1 − kvFree)²` | How close is it to memory pressure? (**headroom**) | `kv_cache_usage_perc` (`KVCacheUsagePerc`) |\n\nCapacity and load are kept apart on purpose. A saturated GPU has high token throughput *because*\nit is busy; if throughput were used as \"load\", the scheduler would send it even more traffic.\nHere throughput only sets the denominator, and load comes from the committed work.\n\n### Defaults\n\n| Symbol | Meaning | Default | Variable |\n|---|---|---|---|\n| `λ` | Weight of engine-queued requests | `0` | `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` |\n| `α` | KV-pressure penalty strength | `2.0` | `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` |\n| `kvCritical` | Free-KV fraction below which a pod scores `+Inf` from `load-balance` (excluded when `load-balance` routes alone; only strongly penalized in a blend — see \"Interaction with the rest of the gateway\" below) | `0.10` | `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` |\n\n`λ` defaults to `0` because the gateway's running count already includes requests that are\nqueued inside the engine; adding `num_requests_waiting` on top would count them twice. Values\n`<= 0` for these variables fall back to the default (the loader rejects non-positive numbers).\n\n### Worked example\n\n| Replica | Running | Tokens/s | KV free | Calculation | Score |\n|---|---|---|---|---|---|\n| B40 | 30 | 6000 | 50% | 30/6000 × (1 + 2·0.50²) | **0.0075** |\n| A100 | 20 | 4000 | 30% | 20/4000 × (1 + 2·0.70²) | 0.0099 |\n| H20 | 25 | 3000 | 60% | 25/3000 × (1 + 2·0.40²) | 0.0110 |\n\nThe B40 wins despite carrying the most requests, because it drains them fastest. This example is\nasserted in `TestLoadBalanceScoreAll_HeterogeneousExample`.\n\n## Where the capacity signal comes from\n\n```\ngateway: DoneRequestTrace(outputTokens) ──► Pod.completedOutputTokens (monotonic counter)\n │ every metrics refresh (default 1s)\n ▼\n calculateRate1m: tokens/sec over ~1 minute window\n │\n ▼\n EWMA, time constant 10s ──► RealtimeOutputTokenRateEWMA (pod-scoped)\n```\n\n- The counter is **gateway-tracked**, like `completedRequests`, so it works for every engine\n (vLLM, SGLang, TRT-LLM, xLLM) without depending on engine metric names. Code:\n `donePodStats` in `pkg/cache/cache_trace.go`, `updateRealtimeOutputTokenRateEWMA` in\n `pkg/cache/cache_metrics.go`.\n- Output tokens are the request's completion-token usage. A request that finishes through the\n count-only path (no usage available) adds one completion and **zero** tokens.\n- The counter survives a brief pod delete/re-add (informer flap) the same way `completedRequests`\n does.\n- A window with no completed tokens carries no information about speed (the replica was idle,\n not slow), so the previous estimate is **kept**, not decayed toward zero. Decaying would make a\n briefly idle replica look weak and starve it of the traffic that would re-measure it.\n\n## Behavior in edge cases\n\n| Situation | Behavior |\n|---|---|\n| Pod has no token-rate estimate yet (new pod, or no usage reported) | Scored at the **mean** estimate of the pods that have one, so it competes as an average replica. A fixed fallback such as `1.0` would be off by the tokens/sec scale (thousands) and lock the pod out of the traffic it needs to be measured. |\n| No pod has an estimate | Every pod gets capacity `1.0`; the score reduces to `running × KV penalty`. |\n| Engine reports no KV usage, or `NaN` | Treated as fully free: no penalty and no guardrail. |\n| `kvFree < kvCritical` | Score `+Inf` from `load-balance`. When `load-balance` routes alone this skips the pod entirely until it recovers. When `load-balance` is blended with other strategies (the default — see below), it is only a strong penalty, not a guaranteed exclusion. |\n| **Every** pod is below `kvCritical` | The request is **not** failed. Routing falls back to the tie-break over all pods, which picks the one with the least KV usage. This fallback is specific to `load-balance` routing alone; see below for the blended case. |\n| Pod is idle (`running = 0`) | Score `0` regardless of capacity, so idle pods are preferred. |\n| Several pods share the lowest score | Ties are broken by least combined GPU+CPU KV-cache usage (`least-kv-cache` scorer), falling back to random. |\n\nKV usage is clamped to `[0, 1]` before use, so an engine that reports a value above 1 is treated\nas full rather than producing a negative free fraction.\n\n## Interaction with the rest of the gateway\n\n- **Load-imbalance gate.** The gateway applies `ApplyLoadImbalanceGate` once, ahead of whichever\n strategy routes the request, so `Route` receives an already-narrowed pod list. If that narrowed\n set is entirely KV-critical, the all-critical fallback above applies. **TODO / known\n limitation:** the gate compares raw running-request count, with no notion of a pod's capacity —\n a replica that's simply faster than its peers is expected to carry more concurrent requests\n without being more loaded (the same capacity signal `load-balance`'s own score uses above), but\n this gate can still flag it as a hotspot and exclude it before `load-balance`'s own\n capacity-aware score ever gets to run. See `getTargetPodListOnLoadImbalance` in\n [load_balance.go](load_balance.go).\n- **Multi-strategy blending: the guardrail is soft here, not hard.** `kvCritical` is a hard\n exclusion only when `load-balance` routes alone. In a blend, `ScoreAll` still returns `+Inf` for\n a critical pod, but `multiStrategyRouter.normalizeScoresArray` treats a non-finite score the\n same as \"not scored\" and assigns it `0` — the worst possible score from `load-balance`, not an\n exclusion from routing. If another blended strategy scores that same pod highly enough, it can\n still be selected.\n\n Concretely: `load-balance` is blended in behind other strategies by default (see\n `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT`), so the blended path, not the standalone one, is\n what most requests actually take. With the default `prefix-cache:5,load-balance:4` blend, a\n KV-critical pod that has the best prefix-cache score still gets `5/9` of the combined weight\n versus `4/9` for a healthy pod that only wins on `load-balance`, so the critical pod wins the\n route. `kvCritical` should therefore be read as \"how hard `load-balance` argues against routing\n to this pod,\" not as a safety limit that blocks it from being chosen — it does not, on its own,\n prevent a KV-critical pod from receiving more traffic. Something that must never route to a\n KV-critical pod needs a filter ahead of scoring (comparable to `ApplyLoadImbalanceGate`), which\n does not currently exist for KV pressure.\n- **PD disaggregation.** The PD router's decode path keeps its own request drain-rate scoring\n (`RealtimeRunningRequestsDrainRate1m`) and is unchanged.\n\n## Known limitations\n\n- **Multiple gateway replicas.** The token counter is local to each gateway, while the running\n count is the cross-gateway total. Each gateway therefore measures roughly `1/N` of a pod's\n output. Relative ranking between pods is preserved as long as the gateways spread traffic\n similarly, but absolute score values are not comparable across deployments with different\n gateway counts.\n- **Estimate lag.** The estimate follows a ~1 minute window smoothed with a 10 second time\n constant, so a sudden change in a replica's speed (for example a workload shifting from\n decode-heavy to prefill-heavy) shows up within tens of seconds, not immediately.\n- **No prefill/decode split.** Capacity is a single output-token rate. Hardware whose relative\n performance differs between prompt-heavy and decode-heavy workloads is not modeled; a natural\n extension is `capacity = α·prefill_tokens/s + β·decode_tokens/s`.\n- **Deliberately not used:** GPU utilization / SM-active. They are useful for observability but\n make a generic scheduler unstable; the score uses only drain rate, committed work, and KV\n pressure.\n\n## Debugging\n\nAt `-v=4` the gateway logs `load_balance_score` once per pod (`running_requests`,\n`capacity`, `kv_free`, and the resulting `score`) and `load_balance_selected` (with the\nwinning `score`). `+Inf` scores are logged as strings so klog's JSON formatter does not\nfail. The cache logs `Updating output token rate metric` with the completed token count,\nthe 1-minute rate, and the EWMA for each pod. The estimate is stored on the pod as\n`realtime_output_token_rate_ewma`; it is not currently exported as a Prometheus series.\n\n\n# Multi-Strategy Routing (Batch Soft Scoring)\n\nAIBrix Gateway supports multi-strategy routing, allowing you to combine multiple routing algorithms with different weights. Instead of making hard decisions (where one strategy completely filters out pods or dictates the single winner), the multi-strategy router uses a **Batch Soft Scoring** mechanism to evaluate pods across all specified strategies.\n\n## How it works\n\n1. **Batch Scoring:** Each configured strategy evaluates all ready pods in a single batch operation (`ScoreAll`), returning raw scores and a boolean array indicating whether each pod could be scored (e.g., handling missing metrics).\n2. **Normalization:** The raw scores from each strategy are normalized into a `[0, 1]` range:\n * **Higher is better** (e.g., `throughput`): `(score - min) / (max - min)`\n * **Lower is better** (e.g., `least-request`): `(max - score) / (max - min)`\n * If a pod lacks metrics (`scored=false`), its normalized score is `0.0` (acting as a penalty).\n3. **Weighted Aggregation:** The normalized scores are multiplied by their respective configured weights and summed up for each pod.\n4. **Top-1 Selection:** The pod with the highest total aggregated score is selected to serve the request.\n\n## Configuration Format\n\nYou can configure multi-strategy routing by setting the `AIBRIX_ROUTING_ALGORITHM` environment variable using a comma-separated list of `strategy:weight` pairs.\n\n* **Format:** `strategy1:weight1,strategy2:weight2,...`\n* If `:weight` is omitted, it defaults to `1`.\n* If a weight is `0`, the strategy is completely ignored.\n* If an invalid strategy is specified, the Gateway rejects the request with **400 Bad Request** instead of silently falling back to `random`.\n* `session-affinity` is treated as a soft-scoring strategy in multi-strategy mode: it boosts the matching pod, but the final weighted winner may still be a different pod. The response session header is updated to the final selected pod.\n With Redis-backed pinning, background sync applies its result only if the local entry has not changed since the read began. A delayed result cannot overwrite or evict a pin committed by a concurrent request.\n* A caller may send an opaque application session identifier in `x-aibrix-session-key`. When no valid gateway-issued `x-session-id` target is ready, rendezvous hashing selects a stable pod for that key. This is useful for concurrent Agent turns that start before a response cookie is available. The opaque key is never returned to the client. The `pd` router's `token_load` and `hybrid_cache_load` prefill policies read the same key to charge a multi-turn conversation only for the tokens added since its previous turn.\n\n## Examples\n\nHere are some real-world examples demonstrating how the weighted aggregation works under different scenarios.\n\nAssume we have two pods serving a model:\n* **Pod A**: High load (10 requests) but High throughput (100 tok/s).\n* **Pod B**: Low load (2 requests) but Low throughput (10 tok/s).\n\n### Example 1: `least-request` Dominates\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:10,throughput:1\"\n```\n* `least-request` (weight 10): Pod B is much better (2 vs 10). Normalized: Pod A = 0.0, Pod B = 1.0. Weighted Score: A = 0.0 * 10/11 = 0.0, B = 1.0 * 10/11 ≈ 0.91.\n* `throughput` (weight 1): Pod A is much better (100 vs 10). Normalized: Pod A = 1.0, Pod B = 0.0. Weighted Score: A = 1.0 * 1/11 ≈ 0.09, B = 0.0 * 1/11 = 0.0.\n* **Total Score**: Pod A ≈ 0.09, Pod B ≈ 0.91. **Pod B wins!**\n\n### Example 2: `throughput` Dominates\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,throughput:10\"\n```\n* `least-request` (weight 1): Weighted Score: A = 0.0 * 1/11 = 0.0, B = 1.0 * 1/11 ≈ 0.09.\n* `throughput` (weight 10): Weighted Score: A = 1.0 * 10/11 ≈ 0.91, B = 0.0 * 10/11 = 0.0.\n* **Total Score**: Pod A ≈ 0.91, Pod B ≈ 0.09. **Pod A wins!**\n\n### Example 3: Complex Scenario with Corner Cases (3 Strategies)\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request,throughput:2,least-latency:0\"\n```\n* `least-request`: No weight specified, **defaults to 1**. Weighted Score: A = 0.0 * 1/3 = 0.0, B = 1.0 * 1/3 ≈ 0.33.\n* `throughput`: Explicitly set to weight **2**. Weighted Score: A = 1.0 * 2/3 ≈ 0.67, B = 0.0 * 2/3 = 0.0.\n* `least-latency`: Weight set to **0**, so this strategy is **ignored** and does not participate in scoring.\n* **Total Score**: Pod A ≈ 0.67, Pod B ≈ 0.33. **Pod A wins!**\n\n### Example 4: Invalid Configuration Fallback\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,invalid-strategy:1\"\n```\n* Because `invalid-strategy` is not a registered algorithm, the multi-strategy parser will fail. To preserve the API contract, the Gateway will reject the request with a **400 Bad Request** error rather than silently falling back to a random router.\n\n### Example 5: Multi-Strategy Finding the Sweet Spot\nIn reality, a pod with the absolute minimum active requests might have just recovered from being severely overloaded. Multi-strategy routing prevents routing to extreme outliers by considering multiple dimensions.\n\nAssume three pods with the following metrics:\n* **Pod A**: Very low active requests (1 req), but historically massively overloaded (1500 tokens).\n* **Pod B**: Historically barely used (15 tokens), but currently slammed with active requests (50 req).\n* **Pod C**: The \"Sweet Spot\" - Medium active requests (10 req), and medium historical load (150 tokens).\n\n**Single Strategy Flaws:**\n* `AIBRIX_ROUTING_ALGORITHM=\"least-request\"`: Completely ignores historical load and picks **Pod A** (which is exhausted).\n* `AIBRIX_ROUTING_ALGORITHM=\"throughput\"`: Completely ignores current spikes and picks **Pod B** (which is currently slammed).\n\n**Multi-Strategy Solution:**\n```yaml\nAIBRIX_ROUTING_ALGORITHM: \"least-request:1,throughput:1\"\n```\n* `least-request` (weight 1): Normalizes active requests [1, 50]. \n * Pod A: 1.0\n * Pod B: 0.0\n * Pod C: 0.816\n* `throughput` (weight 1): Normalizes historical load [15, 1500].\n * Pod A: 0.0\n * Pod B: 1.0\n * Pod C: 0.909\n* **Total Score (Weighted)**: Pod A = 0.5, Pod B = 0.5, Pod C ≈ **0.86**. \n* **Result**: **Pod C wins!** The router correctly identifies the most balanced pod, avoiding the exhausted Pod A and the currently slammed Pod B.\n\n\n# PD Disaggregation Router\n\n`pd/router/router.go` implements the **prefill-decode (PD) disaggregated routing algorithm** (`RouterPD = \"pd\"`). **This file (`pd_readme.md`) is the canonical documentation** for PD: request flow, scoring, engine behavior, metrics, and environment variables. PD-only settings are under **Environment Variables**; variables shared with the prefix-cache router (tokenizer, stddev factor, and related `AIBRIX_PREFIX_CACHE_*` knobs) are listed in **Inherited from Prefix Cache Router** in the same section.\n\nIn PD disaggregation, inference is split across two specialized pod roles:\n\n- **Prefill pod** — processes the prompt (context), builds the KV-cache, then transfers it.\n- **Decode pod** — generates tokens using the KV-cache transferred from the prefill pod.\n\nThe router is responsible for selecting one prefill pod and one decode pod per request, executing the prefill HTTP request synchronously (vLLM/TRT-LLM context-first) or asynchronously (SGLang/TRT-LLM generation-first), and routing the decode request through Envoy to the selected decode pod.\n\n---\n\n## High-Level Request Flow\n\n```\nClient Request\n │\n ▼\nRoute(ctx, readyPodList)\n │\n ├─► ValidateAndGetLLMEngine()\n │ Ensure all pods use the same engine (vllm / sglang / trtllm)\n │\n ├─► filterPrefillDecodePods()\n │ ┌─────────────────────────────────────────────────────┐\n │ │ collectAndBucketPods() │\n │ │ Phase 1: group by roleset (prefill / decode) │\n │ │ Phase 2: apply prompt-length bucketing filter │\n │ └─────────────────────────────────────────────────────┘\n │ │\n │ ├─► [Prompt-length bucketing ON]\n │ │ → combined pod path (see Combined Pod section)\n │ │\n │ ├─► loadImbalanceSelectPrefillPod() (fast path)\n │ ├─► loadImbalanceSelectDecodePod() (fast path)\n │ ├─► scorePrefillPods()\n │ ├─► scoreDecodePods()\n │ └─► finalPDScore() → (prefillPod, decodePod)\n │\n ├─► AddPendingDecode() + defer RemovePendingDecode() (all paths)\n │\n ├─► [prefillPod != nil]\n │ AddPrefillRequest()\n │ doPrefillRequest(ctx, prefillPod, handler)\n │ ├─ SGLang → async goroutine (bootstrap handshake; Route does not wait for completion)\n │ ├─ vLLM → sync, extract kv_transfer_params from response\n │ └─ TRT-LLM → context_first: sync, merge response params\n │ generation_first: prepare both bodies, async prefill\n │\n └─► ctx.SetTargetPod(decodePod)\n return decodePod address\n```\n\n---\n\n## Pod Classification\n\nPods are grouped using Kubernetes labels:\n\n| Label | Values | Purpose |\n|-------|--------|---------|\n| `roleset-name` | any string | Groups prefill+decode pairs into a \"roleset\" |\n| `role-name` | `prefill` / `decode` / `encode` / other | Pod's role in PD disaggregation. `encode` pods are optional and only take the encode stage (see [vLLM encode stage (EPD)](#vllm-encode-stage-epd)) |\n| `stormservice.orchestration.aibrix.ai/pod-group-index` | `\"0\"` or absent | Only pods with index `\"0\"` (or no label) host the HTTP server (multi-node TP) |\n| `stormservice.orchestration.aibrix.ai/role-replica-index` | any string | Distinct replica index within a role; used by `roleReplicaCardinality` to count unique replicas across multi-node TP groups. When absent, each routable pod counts as its own replica. |\n| `model.aibrix.ai/engine` | `vllm` / `sglang` / `trtllm` | LLM engine type |\n\nOnly rolesets with **both** at least one prefill replica and one decode replica are eligible. Incomplete rolesets are excluded. See [Roleset Eligibility Utilities](#roleset-eligibility-utilities) for diagnostic functions.\n\n---\n\n## Roleset Eligibility Utilities\n\n`pd/router/roleset.go` provides helper functions used by both the router and the gateway's pod-readiness logic.\n\n### Replica cardinality\n\nA roleset's replica count is the number of **distinct** values of `stormservice.orchestration.aibrix.ai/role-replica-index`. When that label is absent on all pods of a role, each routable pod counts as its own replica. This correctly handles multi-node tensor-parallel groups where multiple pods share one logical replica slot.\n\n```\nroleReplicaCardinality(pods):\n if any pod has role-replica-index label:\n count distinct index values\n else:\n count routable pods\n```\n\n### Eligibility checks\n\nA roleset is **eligible** when it has ≥1 prefill replica **and** ≥1 decode replica.\n\n| Function | Use |\n|----------|-----|\n| `HasEligibleRoleset(readyPods)` | Gateway check: at least one roleset is fully ready for PD routing |\n| `eligiblePDRolesets(readyPods)` | Internal: returns only complete rolesets for scoring |\n| `DescribePDRolesetEligibility(readyPods)` | Diagnostics: per-roleset replica counts and eligibility flag |\n\n`ValidateAndGetLLMEngine` (called from the gateway before `Route`) checks that all ready pods for the model share the same `model.aibrix.ai/engine` label.\n\n---\n\n## Pod Selection Flow\n\n```\ncollectAndBucketPods()\n │\n ▼\n┌───────────────────────────────┐\n│ prefillPods / decodePods │\n│ (grouped by eligible │\n│ rolesets only) │\n└───────────────────────────────┘\n │\n ▼\nloadImbalanceSelectPrefillPod()\n ┌─────────────────────────────────────────────────────────┐\n │ max(outstanding_prefill_reqs) - min > MIN_SPREAD? │\n │ → YES: narrow to single least-loaded prefill pod │\n │ align decodePods to selected pod's roleset │\n │ (continues to decode fast path below) │\n │ → NO: continue with all prefill pods │\n └─────────────────────────────────────────────────────────┘\n │\n │ NOTE: the two fast paths are independent. Both can fire on the\n │ same request if both sides are imbalanced: prefill narrows first,\n │ then decode narrows within the already-aligned decode set.\n │\n ▼\nloadImbalanceSelectDecodePod() (3 ordered checks, metrics-bearing pods only)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Effective request count = RealtimeNumRequestsRunning │\n │ + PendingDecodeTracker count │\n │ Pending counts bridge the gap between pod selection and the │\n │ first metric update, preventing thundering-herd on a newly │\n │ selected pod whose running count has not yet ticked up. │\n │ │\n │ Check 1 – Request count spread │\n │ Only pods with RealtimeNumRequestsRunning metric participate │\n │ (pods without the metric are excluded from spread; their │\n │ pending count is still stored for scoring — prevents │\n │ thundering-herd on freshly restarted pods) │\n │ max(observed running+pending) - min >= DECODE_LOAD_SPREAD? │\n │ → YES: return least-loaded observed pod; │\n │ align prefillPods to selected pod's roleset │\n │ │\n │ Check 2 – Throughput spread │\n │ Only pods with AvgGenerationThroughputToksPerS participate │\n │ max(observed throughput) - min > THROUGHPUT_SPREAD? │\n │ → YES: return lowest-throughput observed pod │\n │ │\n │ Check 3 – Drain rate score (soft path) │\n │ all pods have drain_rate > 0? │\n │ score = effective_running_reqs / drain_rate │\n │ max_score / min_score > DECODE_SCORE_RATIO? │\n │ → YES: return pod with lowest drain-rate score │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nscorePrefillPods() (per roleset, best pod wins)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Skip pods with req_count > mean + N * stddev │\n │ (N = AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR) │\n │ Skip pods with req_count > max(c, 1) * (mean + 1) │\n │ (c = AIBRIX_PREFILL_LOAD_CAP_FACTOR) │\n │ When all pods in a roleset are filtered, that roleset has no │\n │ entry in the output map and finalPDScore cannot select it. │\n │ │\n │ Policy: prefix_cache (default) │\n │ score = (100 - match_pct) * 0.1 + req_cnt / max_req_cnt │\n │ │\n │ Policy: least_request │\n │ score = req_cnt │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nscoreDecodePods() → pd.DecodeScoreRun (per roleset best pod + MaxScore + policy metadata)\n ┌──────────────────────────────────────────────────────────────────┐\n │ Cold-start (if some pods have metrics but this pod does not): │\n │ score = 1.0 + pending_decode_count │\n │ (neutral idle-warm score; pod competes but is not favored) │\n │ │\n │ Policy: load_balancing (default) │\n │ norm_reqs = running_reqs / max_running_reqs │\n │ norm_thru = 1 - throughput / max_throughput │\n │ norm_free_gpu = free_gpu_pct / max_free_gpu_pct │\n │ score = (w_run*norm_reqs + w_thru*norm_thru) / norm_free_gpu │\n │ (w_run, w_thru from AIBRIX_DECODE_LB_WEIGHT_*, default 1) │\n │ NaN scores: skip; non-NaN Inf possible if norm_free_gpu = 0 │\n │ │\n │ Policy: least_request │\n │ score = running_reqs (with pending) │\n │ │\n │ If primary policy yields NaN, one load_balancing retry per pod │\n │ lower is better │\n └──────────────────────────────────────────────────────────────────┘\n │\n ▼\nfinalPDScore()\n ┌──────────────────────────────────────────────────────────────────┐\n │ For each roleset with both prefill and decode scores: │\n │ final = norm_prefill_score + norm_decode_score │\n │ Pick roleset with minimum final score │\n │ → selectedPrefillPod, selectedDecodePod │\n │ │\n │ Errors: │\n │ • prefillScores nil → Prepare failed in scorer │\n │ • prefillScores empty → all prefill pods filtered by │\n │ stddev or cap (no candidate) │\n │ • no roleset intersection → load-imbalance alignment left │\n │ prefill/decode on different sets │\n └──────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Prefill Score Policies\n\nControlled by `AIBRIX_PREFILL_SCORE_POLICY`, or overridden per request via the model **`routingConfig`** in `model.aibrix.ai/config` (see [Config profile overrides](#config-profile-overrides-for-pd-score-policies)).\n\n### `prefix_cache` (default)\n\nUses the shared `PrefixHashTable` to find how much of the request's token prefixes are already cached on each prefill pod.\n\n```\ntokens = tokenize(request.message)\nmatchedPods, hashes = prefixCacheIndexer.MatchPrefix(tokens, model, readyPodsMap)\n\nscore = (100 - match_percent) * 0.1 + req_count / max_req_count\n```\n\n- Lower score = better (more cache hits, fewer outstanding requests).\n- After routing, the prefix hashes are enqueued to `prefixUpdateCh` for asynchronous indexer update.\n- Tokenizer is selected by `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` (same as the prefix-cache router): `tiktoken` or `character` (default).\n\n### `least_request`\n\nNo prefix cache consultation. Score equals the raw running request count on the pod.\n\n```\nscore = req_count\n```\n\n---\n\n## Decode Score Policies\n\nImplementation lives in `algorithms/pd/decode_scorer.go` (`package pd`). Policies are selected by **`pd.ResolveDecodePolicy`** from `AIBRIX_DECODE_SCORE_POLICY`. Extra names can be registered at init time with **`pd.RegisterDecodePolicy`**. Per-pass results use **`pd.DecodeScoreRun`** (map of roleset → best pod/score, `MaxScore` for normalization, optional `Err`, `FallbackUsed`, `Policy`).\n\nMetrics are passed per pod via **`pd.DecodePodInput`** (running/throughput/free GPU and batch maxima). The router guarantees positive maxima from `loadImbalanceSelectDecodePod` as today.\n\nControlled by `AIBRIX_DECODE_SCORE_POLICY`, or overridden per request via **`routingConfig`** in the model config profile (see [Config profile overrides](#config-profile-overrides-for-pd-score-policies)).\n\nMetrics are pulled from the cache per pod (same maps as `loadImbalanceSelectDecodePod`):\n- `RealtimeNumRequestsRunning` + `PendingDecodeTracker` count\n- `AvgGenerationThroughputToksPerS` (per model)\n- `KVCacheUsagePerc` (free = 100 − usage%)\n\n**Cold-start gate**: if at least one pod has `RealtimeNumRequestsRunning`, pods that lack it are given a neutral **cold-start score** (`1.0 + pending_decode_count`) rather than full policy scoring. When *no* pod has that metric yet, all pods are scored normally.\n\n### `load_balancing` (default)\n\nBalances normalized queue depth, inverse throughput, and free GPU headroom:\n\n```\nnorm_running_reqs = running_reqs_with_pending / max_running_reqs\nnorm_throughput = 1 - avg_gen_throughput_tok_s / max_throughput\nnorm_free_gpu = free_gpu_percent / max_free_gpu_percent\n\ndecode_score = (w_run * norm_running_reqs + w_thru * norm_throughput) / norm_free_gpu\n```\n\n`w_run` and `w_thru` default to `1` (historical equal weighting). Lower score is better (fewer outstanding requests, higher throughput, more free GPU).\n\n### `least_request`\n\nNo throughput or GPU terms. Score equals raw running decode request count (including pending decode), same units as `RealtimeNumRequestsRunning` + pending.\n\n```\ndecode_score = running_reqs_with_pending\n```\n\n### `token_load`\n\nScores from the gateway's decode ledger in `pd.TokenLoadTracker` rather than from request counts or scraped KV usage. Each request is charged its `prompt_tokens` to the selected decode pod, under `selectMu` together with the other selection bookkeeping, and released on request completion or prefill failure (`releaseTokenLoad`, or the prefill executor for a terminal async failure). The whole prompt is charged because the decode pod receives all of its KV; there is no fixed per-request cost (`AIBRIX_TOKEN_LOAD_REQUEST_COST` is prefill-only).\n\n```\ndecode_score = decode_tokens + generated_tokens\ngenerated_tokens = rate × Σ over outstanding charges (now − routed_at)\nrate = AvgGenerationThroughputToksPerS / running requests (mean of the other pods if unknown)\n```\n\nWhen the prefill response carries `usage.prompt_tokens`, the prefill executor replaces the decode charge with it (`TokenLoadTracker.CorrectDecodeCost`), since the body-size estimate is far off for requests with media. `generated_tokens` estimates the output the pod's outstanding requests have produced so far (it sits in the decode KV too); `pd.TokenLoadTracker.DecodeGrowth` computes it in O(1) per pod from the count and the summed charge times of the outstanding charges. `AIBRIX_DECODE_TOKEN_LOAD_OUTPUT_GROWTH=false` scores prompt tokens only. The cold-start score does not apply to `token_load`: a pod without metrics is scored from the ledger like any other. The decode load-imbalance fast path is skipped under `token_load` (it still runs to fill the metric maps): it picks by request count, throughput or drain rate, and the pod holding one long prompt has the fewest requests. With Redis configured, the decode ledger is shared across gateway replicas (`AIBRIX_TOKEN_LOAD_SHARED_LEDGER`, default on): each replica publishes its per-pod charges, count and summed charge times under its instance ID (`cache.Store.PublishDecodeLedger`), and `scoreDecodePods` adds the other live replicas' values, read with the running-request counts (`GetPodsRunningRequestsAndDecodeLedger`). Without Redis the ledger is local to each replica. Each decode pod's `token_load` score is then multiplied by the candidates' mean KV capacity over its own (`decodeKVCapacities`, `AIBRIX_DECODE_TOKEN_LOAD_CAPACITY_AWARE`, default on), with capacities from the scraped `kv_cache_num_gpu_blocks` × `kv_cache_block_size` (vLLM `vllm:cache_config_info` labels) or `kv_cache_capacity_tokens` (SGLang); unknown capacities count as the mean.\n\n### Config profile overrides for PD score policies\n\nWhen the gateway resolves a model config profile (`routingCtx.ConfigProfile` from `model.aibrix.ai/config` and the `config-profile` header), the **`routingConfig`** JSON may include:\n\n| Field | Values | Effect |\n|-------|--------|--------|\n| `prefillScorePolicy` | `prefix_cache`, `least_request` | Overrides `AIBRIX_PREFILL_SCORE_POLICY` for that request |\n| `decodeScorePolicy` | `load_balancing`, `least_request`, `conductor`, `token_load` (or any name registered via `pd.RegisterDecodePolicy`) | Overrides `AIBRIX_DECODE_SCORE_POLICY` for that request |\n\nExample fragment inside a profile:\n\n```json\n\"routingConfig\": {\n \"promptLenBucketMinLength\": 0,\n \"promptLenBucketMaxLength\": 8192,\n \"prefillScorePolicy\": \"prefix_cache\",\n \"decodeScorePolicy\": \"least_request\"\n}\n```\n\nOmitted fields keep the gateway defaults from environment variables.\n\n---\n\n\n## Engine-Specific Prefill Behaviour\n\n### vLLM encode stage (EPD)\n\nWhen the selected prefill pod's roleset has `role-name: encode` pods and a vLLM `/v1/chat/completions` request carries media items (`image_url`, `audio_url`, `input_audio`, `video_url`), `Route` runs an encode stage before the prefill call:\n\n```\nGateway ──POST /v1/chat/completions {messages:[{role:user, content:[item i]}]}──► Encode Pod (one per item, in parallel)\n ◄── 200 ──────────────────────────────────────────────────────────────── Encode Pod (output stored by the EC connector)\n\nGateway ──prefill as below, request body unchanged──────────────────────────────► Prefill Pod (finds the output by the item's hash)\n```\n\nItems are spread round-robin over the roleset's encode pods. The body is not rewritten: the encoder and the prefill pod hash the same item bytes, so the prefill pod's EC connector finds the output, and encodes the item itself when it can't. A failed encode is therefore logged and counted, not returned. Each call uses the prefill request timeout. Connectors that need transfer handles passed through the request (vLLM's `*_embeds` rewrite in `disagg_epd_proxy.py`) are not supported.\n\nWhen bucketing routes the request to a combined pod instead, `Route` runs the same stage on the combined pod's roleset. For E+PD rolesets, where encode pods sit in front of pods that serve the whole request, the gateway runs it outside the PD router: with any other routing strategy, `selectTargetPod` drops encode pods from the candidates and encodes on the chosen pod's roleset before returning it.\n\n### vLLM (`AIBRIX_KV_CONNECTOR_TYPE=shfs`, default)\n\n```\nGateway ──POST /v1/... (max_tokens=1, kv_transfer_params={do_remote_decode:true})──► Prefill Pod\n ◄── response {kv_transfer_params: {remote_block_ids, ...}} ──────────────── Prefill Pod\n\nGateway updates ReqBody:\n kv_transfer_params.remote_host = prefill_pod_ip\n\nGateway ──POST /v1/... (with kv_transfer_params)──────────────────────────────────► Decode Pod\n```\n\n### vLLM (`AIBRIX_KV_CONNECTOR_TYPE=nixl`, Neuron)\n\n```\nGateway ──POST /v1/... (max_tokens=1)──────────────────────────────────────────────► Prefill Pod\n ◄── response { entire prefill response } ──────────────────────────────── Prefill Pod\n\nGateway updates ReqBody:\n disagg_prefill_resp = \n\nGateway ──POST /v1/... (with disagg_prefill_resp)──────────────────────────────────► Decode Pod\n```\n\n### SGLang (async)\n\n```\nGateway modifies ReqBody:\n bootstrap_host = prefill_pod_ip\n bootstrap_port = pod annotation / default 8998\n bootstrap_room = random int63\n rid = -<16 hex nonce> (identical on both legs)\n\nGateway ──POST (async goroutine)──────────────────────────────────────────────────► Prefill Pod\n (no wait; prefill/decode coordinate via bootstrap protocol)\n\nGateway ──POST /v1/... (updated body)─────────────────────────────────────────────► Decode Pod\n```\n\nFor SGLang, `Route()` returning a decode pod means the gateway has dispatched\nthe asynchronous prefill worker, not that the prefill HTTP request has already\ncompleted. Prefill success/failure metrics are emitted by the async worker after\nthe prefill HTTP request returns.\n\n`rid` is the engine-visible request id of the attempt. SGLang reads it from the\nrequest body (the `X-Request-Id` header is ignored) and its `/abort_request`\nendpoint matches live requests by rid alone, so injecting the same gateway-owned\nrid into both legs is what makes the decode leg cancellable. The rid always ends\nwith a fixed-width nonce so that no gateway rid can be a proper prefix of\nanother - SGLang aborts by rid prefix - and so that a client retrying with the\nsame request id (or the same traceparent) still gets a distinct rid per attempt.\nSee [PD Prefill Fail-Fast](#pd-prefill-fail-fast).\n\n### TensorRT-LLM\n\n`AIBRIX_TRT_SCHEDULE_STYLE=context_first` (default) preserves the sequential flow:\n\n```\nGateway adds disaggregated_params to prefill request:\n { request_type: \"context_only\", disagg_request_id: }\n\nGateway ──POST (sync)──────────────────────────────────────────────────────────────► Prefill Pod\n ◄── response { disaggregated_params: {...}, prompt_token_ids: [...] } ── Prefill Pod\n\nGateway updates ReqBody:\n disaggregated_params.request_type = \"generation_only\"\n prompt / prompt_token_ids from prefill response\n\nGateway ──POST /v1/... (with disaggregated_params)─────────────────────────────────► Decode Pod\n```\n\nTRT-LLM uses a Snowflake-style `disagg_request_id` (63-bit) to correlate prefill and decode:\n\n```\n[41-bit timestamp ms since 2023-01-01] [10-bit machine_id] [12-bit counter]\n```\n\nMachine ID is set via `AIBRIX_TRT_MACHINE_ID` (must be in `[0, 1024)`). Assign different machine IDs to gateway processes sharing the same TRT workers.\n\n#### Generation-first (parallel)\n\nSet `AIBRIX_TRT_SCHEDULE_STYLE=generation_first` **on the gateway plugin**, not on\nCTX/GEN workers. The setting is fixed when the PD router is constructed; an\nunknown value is logged and leaves the router on `context_first`. It does not\naffect other engines or combined pods. No automatic fallback or retry occurs after\neither leg has been dispatched.\n\nPrerequisites (TRT-LLM `1.3.0rc8`, the Python KV-cache transceiver on both roles,\nmatching model/tokenizer/chat-template) are listed in\n`docs/source/features/gateway-routing/pd-disaggregation.rst`.\n\n```\nGateway selects CTX + GEN\n ├─ GET CTX /server_info (cached)\n ├─ prepare both bodies with the same integer disagg_request_id\n ├─ goroutine: POST context_only, schedule_style=1, stream=false, max_tokens=1\n └─ return GEN address and generation_only body to Envoy immediately\n disagg_request_id = ctx_request_id = shared ID\n ctx_info_endpoint, ctx_dp_rank, optional encoded_opaque_state = CTX metadata\n schedule_style = 1 (TRT-LLM GENERATION_FIRST wire enum)\n original prompt/messages, sampling settings and stream mode preserved\n\nCTX <── engine-managed coordination / KV transfer ──> GEN\nEnvoy <──────────────────── response / SSE ────────── GEN\n```\n\nThere is no second Go-issued decode POST. GEN can initialize while CTX is still\nworking; it cannot generate without the required KV. The CTX response is not merged\nback into the already-dispatched GEN request. Both endpoints must therefore use\nmatching model/tokenizer/chat-template configuration, since GEN cannot reuse\n`prompt_token_ids` from the CTX HTTP response in this mode.\n\n**Worker discovery:** each router loads the selected CTX worker's `/server_info`\nlazily before either leg is dispatched, so newly discovered or autoscaled pods work\nwithout a gateway restart. Concurrent misses for one incarnation are coalesced;\nlookups have a 3-second timeout and a 1-minute TTL, and responses are capped at\n1 MiB. Pod UID, address and restart count identify an incarnation. A missing or\ninvalid `/server_info` fails routing before dispatch, and failed lookups are not\ncached.\n\nThe expected HTTP response follows TRT-LLM's `1.3.0rc8` OpenAI server schema:\n\n```json\n{\"disaggregated_params\":{\"ctx_info_endpoint\":\"tcp://:\",\"ctx_dp_rank\":0}}\n```\n\n`ctx_info_endpoint` must be a single string, which is what the Python transceiver\nreturns; any other JSON shape fails the lookup.\n\n`encoded_opaque_state` is also propagated when present. Only these fields are copied\nfrom `/server_info`: other keys are ignored so worker metadata can never overwrite\nthe gateway-owned `request_type`, IDs or `schedule_style`. A field the handshake\nrequires but this allowlist does not carry must be added to the handler explicitly.\nThe endpoint must be reachable from GEN. `ctx_dp_rank` must be explicitly present and nonnegative; it is **attention\nDP rank**, not TP rank, replica index, or Pod number. Rank zero is valid but is never\nsubstituted for missing data. Metadata is taken from the exact selected Pod. A\nfront-end that distributes requests across multiple DP ranks without stable\nworker/rank affinity is not supported by this Pod-level cache: expose rank-affine\nworkers or retain `context_first`. Validate nonzero-rank deployments against the\nactual engine before enabling this mode.\n\n**Failure handling:** the CTX HTTP call retains client cancellation and the existing\nprefill timeout. Failure wakes the gateway through the per-incarnation PD leg state,\nnever through a recycled RoutingContext. TRT-LLM has no SGLang `rid`/`/abort_request`\ncontract; the gateway instead fails/resets the Envoy stream, relying on TRT-LLM's\nHTTP-disconnect cancellation. Keep Envoy `failure_mode_allow=false`. TRT's SSE headers\ncan precede KV arrival, so a terminal CTX failure after response headers resets the\nstream rather than being ignored. This can also truncate an already-generating\nresponse; it never attempts to rewrite headers already sent to the client.\n\nStart validation with text-only 1P1D using the sample under\n`samples/quickstart/tensorrt/`; its README carries the smoke test and the rollback\ncommand.\n\n---\n\n## PD Prefill Fail-Fast\n\nOnly asynchronous modes (SGLang and TRT generation-first) can fail their prefill leg after `Route()`\nhas already returned a decode pod. Without fail-fast the client waits for the\ndecode leg to give up on a KV transfer that will never arrive - 300s in SGLang -\nand the decode pod holds its pre-allocated KV pages for that whole window.\n\n```\nPrefill leg fails (async worker)\n │\n ├─► record failure on the RoutingContext's PD leg state (first failure wins)\n │\n ├─► SGLang: POST /abort_request {\"rid\": ...} to decode (goroutine, twice)\n │ TRT: no abort POST; reset Envoy upstream on failure\n │\n └─► wake the ext_proc stream ──► 5xx ImmediateResponse to the client\n header x-error-pd-prefill: true\n```\n\nFailure classes, as reported in the metrics and in the client error message:\n\n| Class | Meaning | Terminal |\n|-------|---------|----------|\n| `request_setup` | The prefill HTTP request could not be built or the body could not be prepared | yes |\n| `timeout` | `AIBRIX_PREFILL_REQUEST_TIMEOUT` elapsed | yes |\n| `canceled` | The client or the gateway canceled the prefill request | yes |\n| `transport` | Connection refused, reset, DNS failure, ... | yes |\n| `http_status` | The prefill engine answered a non-2xx status | yes |\n| `bad_response` | The prefill engine answered 2xx with a body the gateway could not parse | engine-dependent |\n\n`bad_response` is terminal only for engines whose KV transfer is out of band\n(TRT generation-first): there the HTTP body is not the KV handshake, so an\nunparseable 2xx leaves the gateway unable to tell whether context processing\nsucceeded. For SGLang, whose handshake completes in the body, it is not terminal:\nthe prefill leg did run, so the decode leg is left alone and the client keeps\nwhatever the decode leg produces.\n\nThe client status code is the prefill engine's own status for `http_status`, and\n`503` for every other terminal class, since those have no upstream status. The\nresponse body is an OpenAI-shaped error carrying the class and a truncated\nupstream message, and it is sent even while the gateway is still waiting for the\ndecode leg's response headers. For SGLang, once the decode leg has started answering the client, the failure is\nonly recorded and logged. TRT generation-first instead resets the stream even\nafter headers, because TRT can send SSE headers before KV is available; it does not\nattempt to replace the already-started response.\n\n### PD Decode Watchdog\n\nFail-fast covers a prefill leg that dies. The opposite case is a prefill leg\nthat *succeeds* and a decode pod that then never answers: in SGLang the prefill\ncall only returns once the decode pod has taken the KV transfer, so from that\npoint the decode pod owes the client a response. If its scheduler dies or wedges,\nthe client would otherwise hang until Envoy's route timeout.\n\n```\nPrefill leg succeeds (async worker)\n │\n └─► mark the PD leg state (MarkPrefillSucceeded) ──► wake the ext_proc stream\n │\n arm a timer: FIRST_RESPONSE (stream) / RESPONSE (non-stream)\n │\n response headers ──► no change (sent before the first token)\n first body chunk ──► switch to STREAM_IDLE │ (streaming only)\n every later chunk ──► re-arm STREAM_IDLE │\n ▼ timer fires\n before any headers: 504 ImmediateResponse, header x-error-pd-decode: true\n + gRPC DeadlineExceeded close\n after the headers: gRPC DeadlineExceeded close only (Envoy resets\n the response the client already started reading)\n always: POST /abort_request {\"rid\": ...} to decode (goroutine, once)\n```\n\nThe watchdog runs only for requests with a gateway-owned `rid` (SGLang). Its\ntimeouts are `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` (streaming, default `60`) and\n`AIBRIX_DECODE_RESPONSE_TIMEOUT` (non-streaming, default `0`, i.e. off, since the\ndecode pod only answers a non-streaming request once the whole generation is\ndone). Both run until the first response body chunk: SGLang's streaming\nendpoint sends its headers before the first token, so the headers prove nothing.\nFrom the first chunk of a streaming response on,\n`AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` (default `120`) bounds the gap between two\nchunks.\n\n---\n\n## Request Trackers\n\n### PrefillRequestTracker\n\nTracks active **prefill** request counts per pod using `sync.Map` and `atomic.Int32`. Used by both prefill load-imbalance detection and `scorePrefillPods` (mean/stddev filter and load cap).\n\nBoth request trackers are shared by every model the router serves, so pods are identified by their pod key, `namespace/name` (`utils.GeneratePodKey(pod.Namespace, pod.Name)`), not by bare pod name.\n\n```\nAddPrefillRequest(requestID, podKey) // in Route(), after PD pod selection, before prefill HTTP\nRemovePrefillRequest(requestID) // on prefill end (executor defer) or Route error\nGetPrefillRequestCountsForPods(pods) // for scoring\n```\n\n### PendingDecodeTracker\n\nBridges the gap between **decode pod selection** and the actual decode request starting. Without this, concurrent requests could all route to the same decode pod (since `RealtimeNumRequestsRunning` hasn't updated yet).\n\n```\nAddPendingDecode(requestID, podKey) // in Route(), immediately after pod selection\nRemovePendingDecode(requestID) // deferred in Route()\nGetPendingDecodeCount(podKey) // added to running reqs in decode scoring\n```\n\nTimeline:\n```\nRoute() called\n │\n ├─ filterPrefillDecodePods() → (prefillPod, decodePod)\n │ └─ [combined path] returns prefillPod=nil\n │\n ├─ AddPendingDecode(requestID, decodePodKey)\n ├─ defer RemovePendingDecode(requestID)\n │\n ├─ [PD path] AddPrefillRequest(requestID, prefillPodKey)\n ├─ [PD path] doPrefillRequest() (may take seconds)\n │ └─ on error: RemovePrefillRequest(requestID) called immediately\n ├─ ctx.SetTargetPod(decodePod)\n └─ return address to caller\n```\n\n---\n\n## Prompt Length Bucketing\n\nEnabled by `AIBRIX_PROMPT_LENGTH_BUCKETING=true`.\n\nEach pod can declare a prompt-length range via its config profile (`promptLenBucketMinLength` / `promptLenBucketMaxLength`). A pod is eligible only when the request's prompt length falls within its range.\n\n```\ncollectAndBucketPods()\n ├─ Combined pass: collect combined-role pods (role-name is not prefill/decode)\n │ with combined=true and a prompt-length match\n ├─ Eligible rolesets: group by roleset-name (prefill + decode both present)\n └─ Bucket pass: per roleset, keep pods whose bucket covers promptLength\n only when BOTH prefill and decode sides match (half-pairs excluded)\n\nIf both bucket-filtered lists are non-empty, they replace the unfiltered prefill/decode lists atomically so the sides stay roleset-aligned.\n```\n\n### No-match guard and combined fallback\n\nWhen `AIBRIX_PROMPT_LENGTH_BUCKETING=true`, a roleset enters the bucket-filtered lists only when **both** its prefill and decode pods match the request's prompt length. A roleset with only one matching side (misconfigured bucket or a missing decode pod) is excluded from the bucket lists.\n\nIf no complete bucket-matched prefill/decode pair exists for the request — including when the prompt length is outside every declared range, or when the matching roleset is incomplete (e.g. decode pod is down) — the router falls back to a combined pod when one is available. Otherwise it returns an error rather than cross-routing to a different bucket's PD pair:\n\n```\nif AIBRIX_PROMPT_LENGTH_BUCKETING &&\n (bucketPrefills == 0 || bucketDecodes == 0):\n if combined pods available for this prompt length → route to combined (random pick)\n else → error: \"no prompt-length bucket matches prompt length N and no combined pods available\"\n```\n\nPods without a `model.aibrix.ai/config` annotation are treated as suitable for any prompt length. With bucketing enabled globally, models that do not declare explicit bucket ranges still participate using those default bounds.\n\n### Combined Pods\n\nWhen bucketing is enabled, pods with `combined=true` in their config profile can act as both prefill and decode. Combined pods use a `role-name` other than `prefill` or `decode` (for example `all`). The router falls back to a combined pod when:\n\n1. No complete bucket-matched prefill/decode pair exists for the request's prompt length (prompt outside all buckets, incomplete roleset, or decode pod unavailable), **or**\n2. `shouldPickCombined()` returns `true`:\n - At least one combined pod has queue-drain score < `0.25` (low load), **and**\n - At least one prefill pod **or** decode pod has queue-drain score > `1.0` (high load).\n\nQueue-drain score is `(waiting_reqs + prealloc_queues) / drain_rate_1m` (see `calculatePodScoreBasedOffRequestRate`).\n\n```\nshouldPickCombined():\n combinedLowLoad = any combined pod score < 0.25\n prefillHighLoad = any prefill pod score > 1.0\n decodeHighLoad = any decode pod score > 1.0 (skipped if prefillHighLoad)\n return (prefillHighLoad OR decodeHighLoad) AND combinedLowLoad\n```\n\nWhen a combined pod is selected for load imbalance, `scoreCombinedPods()` picks the lowest-score combined pod. The no-match path picks a combined pod at random.\n\nWhen a combined pod is selected, `prefillPod` is `nil` (no prefill HTTP call) and `decodePod` is the selected combined pod.\n\n### Adaptive Bucket Serving\n\nEnabled by `AIBRIX_BUCKET_SERVE=true`, which requires `AIBRIX_PROMPT_LENGTH_BUCKETING=true` as well: the plan only re-orders the rolesets that bucketing already filtered to.\n\nPlain bucketing keeps every roleset whose range covers the request as a candidate, and the scoring decides among them. Adaptive bucket serving narrows that choice. The gateway keeps a per-model picture of the prompt lengths it routes, charges every roleset the traffic only it can serve, and splits each range that several rolesets still need into one band per roleset, drawing each band against one replica-proportional share so a roleset carrying several overlaps does not draw a fresh share from each. The cut points sit at quantiles of the observed traffic. The picture is an EWMA with a 30s half-life, a model's plan is recomputed at most once every 5s, and one plan holds at most 16 affinity bands. A shared range needs a meaningful share of the model's traffic before it is split at all, and a range that cannot afford one band per needing roleset splits into as many bands as the budget left allows; an interval that is not split keeps every covering roleset a candidate.\n\n```\nbucketServeBand() (before selectMu, one call per request)\n ├─ observe promptLength for the model\n ├─ groups: one per eligible roleset, with the range its profile declares and its prefill replica count\n ├─ plan: shared ranges split at mode quantiles in replica proportion, light ones kept whole\n └─ band for promptLength → banded roleset and band upper bound\n\nfilterPrefillDecodePods()\n ├─ steps 1-5: bucket filtering and the load-imbalance fast paths\n ├─ banded roleset still has prefill and decode candidates\n │ → narrow both sides to that roleset (step 6)\n └─ otherwise keep the load fast paths' choice\n```\n\n`AIBRIX_BUCKET_SERVE_MODE` picks the unit the cut points are measured in: `throughput` (default) balances prompt-token mass, while `rps` balances request counts. Within a shared range the bands divide the observed traffic in proportion to the replica share each roleset still needs, counted in that unit. Both are per-model: a profile that selects another mode re-plans only the models it routes.\n\nThe plan is advisory. It never widens a choice, only prefers one of the rolesets that already cover the request's length. Wherever the plan holds no band, the scoring considers every covering roleset, and when the banded roleset has no candidate left after the load fast paths, the request routes exactly as it would without the plan. The steady state does narrow the choice, which is the feature, and a stale or wrong plan costs balance, not correctness. With the feature off the planner records and publishes nothing.\n\nMetrics: `pd_bucket_serve_band_total` and `pd_bucket_serve_prompt_tokens_total` count the requests and prompt tokens each banded roleset carried (label `roleset`), so a counter joins the pods that received the band, and the `pd_bucket_serve_band_max` gauge holds the highest upper prompt-length bound among the bands a roleset currently holds (a roleset can hold two bands when its range overlaps two others), deleted when no live plan holds the roleset.\n\n---\n\n## Environment Variables\n\n### Core Behaviour\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFILL_SCORE_POLICY` | `prefix_cache` | Prefill pod scoring: `prefix_cache` or `least_request`. Any other value logs a warning and falls back to `prefix_cache`. |\n| `AIBRIX_DECODE_SCORE_POLICY` | `load_balancing` | Decode pod scoring for `finalPDScore`: `load_balancing`, `least_request`, `conductor` or `token_load`. Any other value logs a warning and falls back to `load_balancing`. |\n| `AIBRIX_KV_CONNECTOR_TYPE` | `shfs` | KV transfer backend: `shfs` (GPU/SHFS), `nixl` (Neuron/NIXL), or `mooncake` (Mooncake) |\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | `30` | Prefill HTTP request timeout in seconds. Exceeding it is a terminal `timeout` prefill failure. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | `false` | Enable prompt-length-based pod bucketing |\n| `AIBRIX_BUCKET_SERVE` | `false` | Enable adaptive bucket serving on top of prompt-length bucketing |\n| `AIBRIX_BUCKET_SERVE_MODE` | `throughput` | The unit the adaptive cut points are measured in: `throughput` (prompt token mass) or `rps` (request counts) |\n\n### Prefill Load Balancing\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | `16` | Min `(max − min)` outstanding prefill request count to trigger imbalance routing |\n| `AIBRIX_PREFILL_LOAD_CAP_FACTOR` | `1.5` | Factor `c`: prefill pods with `req_count > max(c, 1) * (mean + 1)` are skipped from per-roleset selection. Unlike the stddev skip, it fires in pools of five or fewer pods. Environment-only |\n\n### Decode Load Balancing\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | `1.0` | Multiplier on normalized running-request term in `load_balancing` numerator |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | `1.0` | Multiplier on normalized inverse-throughput term in `load_balancing` numerator |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | `16.0` | Min `(max − min)` running request count spread to trigger decode imbalance routing |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | `2048.0` | Min `(max − min)` token throughput spread (tok/s) to trigger throughput-based routing |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | `1.5` | `max_drain_score / min_drain_score` ratio threshold to trigger drain-rate routing |\n\n### PD Prefill Fail-Fast\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | `3` | Per-attempt timeout in seconds of the `/abort_request` call to the decode pod. `0` disables decode aborts; the client is still failed fast |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | `2` | Delay in seconds before the second abort attempt. `0` sends a single attempt |\n\n### PD Decode Watchdog\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | `60` | Seconds a streaming SGLang request waits, after its prefill leg succeeded, for the first response body chunk (first token) from the decode pod. `0` disables |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | `0` | Same for a non-streaming request, where the first body chunk is the finished answer. `0` (default) disables |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | `120` | Longest gap allowed between two body chunks of a streaming response once the first one arrived. `0` disables |\n\n### TensorRT-LLM\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_TRT_MACHINE_ID` | `0` | 10-bit machine ID used in Snowflake disagg request ID generation (range: `[0, 1024)`); distinct per gateway process sharing TRT workers |\n| `AIBRIX_TRT_SCHEDULE_STYLE` | `context_first` | TRT-only dispatch mode: `context_first` (sequential) or `generation_first` (parallel). Read at router initialization; unknown values are logged and fall back to `context_first` |\n\n### Inherited from Prefix Cache Router\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | `character` | Tokenizer for prefix-cache prefill scoring: `tiktoken` or `character` |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `1` | Multiplier N: prefill pods with `req_count > mean + N*stddev` are skipped from per-roleset selection |\n\n---\n\n## SGLang Bootstrap Port\n\nSGLang uses a bootstrap mechanism for prefill/decode coordination. The port is resolved in order:\n\n1. Pod annotation `model.aibrix.ai/sglang-bootstrap-port`\n2. Default: `8998`\n\n---\n\n## Metrics Emitted\n\n| Metric | When |\n|--------|------|\n| `GatewayPrefillRequestFailTotal` | Engine validation fail, pod filter fail, prefill HTTP error |\n| `GatewayPrefillRequestSuccessTotal` | Prefill HTTP succeeded. For SGLang and TRT generation-first, emitted asynchronously after the background prefill request completes, not when `Route()` returns. |\n| `gateway_pd_encode_request_total{status,status_code}` | One media item sent to an encode pod. `status` is `success`, or the failure class (`http_error`, `deadline_exceeded`, ...) |\n| `PDSelectedPrefillPodTotal` | Prefill pod selected (per pod label) |\n| `PDSelectedDecodePodTotal` | Decode pod selected (per pod label) |\n| `gateway_pd_prefill_failure_total{class,stage}` | A terminal prefill failure reached the client-facing handler. `stage` is `before_response` (the client was failed fast) or `after_response` (the decode leg had already started answering) |\n| `gateway_pd_decode_abort_total{prefill_failure_class,result}` | One decode abort attempt. `result` is `ok`, `error`, `skipped_streaming`, `skipped_disabled`, `skipped_no_rid` or `skipped_no_target`. `prefill_failure_class` is `watchdog_first_response` or `watchdog_stream_idle` for an abort sent by the decode watchdog |\n| `gateway_pd_decode_watchdog_total{phase}` | The decode watchdog failed a request whose decode pod stopped responding. `phase` is `first_response` or `stream_idle` |\n\n---\n\n## Response Headers Set\n\n| Header | Value |\n|--------|-------|\n| `prefill-target-pod` | Name of the selected prefill pod |\n| `prefill-target-pod-ip` | IP of the selected prefill pod |\n| `x-error-pd-prefill` | `true`, on the error response the gateway generates when the PD prefill leg failed |\n| `x-error-pd-decode` | `true`, on the error response the gateway generates when the decode watchdog fired |\n\n---\n\n## Prefix Cache Update Pipeline\n\nAfter a prefill pod is selected, its prefix hashes are enqueued asynchronously to avoid blocking the routing path:\n\n```\nfinalPDScore()\n └─ enqueuePrefixUpdate(hashes, model, pod)\n │\n ▼ (non-blocking send; dropped if channel full)\n prefixUpdateCh (buffered, capacity 1024)\n │\n ▼ (single background goroutine)\n prefixCacheIndexer.AddPrefix(hashes, model, pod)\n```\n\n---\n\n## Initialization\n\n```go\nNewPDRouter()\n │\n ├─ cache.Get() // shared metrics cache\n ├─ prefixcacheindexer.GetSharedPrefixHashTable()\n ├─ prefillScorePolicy from AIBRIX_PREFILL_SCORE_POLICY:\n │ least_request → LeastRequestPrefillPolicy\n │ prefix_cache → PrefixCachePrefillPolicy (shared tokenizer + PrefixHashTable)\n │ (other) → log warning, same as prefix_cache\n ├─ decode score policy (package algorithms/pd, decode_scorer.go) from AIBRIX_DECODE_SCORE_POLICY:\n │ least_request → pd.LeastRequestDecodePolicy\n │ load_balancing → pd.LoadBalancingDecodePolicy\n │ (other) → log warning, same as load_balancing\n ├─ create HTTP client with connection pool\n │ MaxIdleConns=100, MaxIdleConnsPerHost=10, IdleConnTimeout=90s\n ├─ TRT handler from AIBRIX_TRT_SCHEDULE_STYLE (immutable, router-scoped)\n │ owns a lazily populated /server_info cache for generation_first\n ├─ NewPrefillRequestTracker()\n ├─ NewPendingDecodeTracker()\n └─ startPrefixUpdater() // background goroutine\n\n// pd/router/roleset.go (no per-router state; pure functions called on each request)\nHasEligibleRoleset(readyPods) // gateway: skip PD when no complete roleset\neligiblePDRolesets(readyPods) // internal: filter to complete rolesets only\nDescribePDRolesetEligibility(readyPods) // diagnostics: per-roleset replica counts\n```\n\n\n# Prefix Cache Aware Routing\n\nPrefix Cache Aware routing improves KV cache reuse across GPU pods by steering requests whose\nprompt shares a common prefix to the pod that already has those KV blocks cached. Pod selection\namong prefix-matched candidates is still load-aware (see Step 2 below), and this router itself\nis purely about prefix caching — it does not gate or restrict candidates based on cluster-wide\nload imbalance. That concern is instead handled centrally by the gateway: `selectTargetPod` in\n`gateway.go` applies the [load-balance router](load_balance.go)'s `ApplyLoadImbalanceGate` once,\nahead of whichever strategy actually routes the request (prefix-cache included), narrowing\n`readyPods` before this router ever sees them. This replaced an older prefix-cache-specific gate\nthat lived in this file; see [ENV_VARS.md](../ENV_VARS.md#variable-dependency-notes) for the\nmigration note. The gate applies the same way when combining strategies via multi-strategy\nsoft-scoring (`AIBRIX_ROUTING_ALGORITHM=prefix-cache:W1,load-balance:W2`, see\n[multi_router_readme.md](multi_router_readme.md)): `selectTargetPod` narrows `readyPods` once,\nbefore resolving and calling into the multi-strategy router, so every sub-strategy's `ScoreAll`\n— this one included — only ever sees the already-gated pod list. It is skipped only for\nexclusive strategies (`pd`, `slo*`), which manage their own pod subsets.\n\nTwo routing modes are supported, selected automatically at startup:\n\n| Mode | When active | Prefix index |\n|------|-------------|--------------|\n| **Standard** | `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=false` (default) | Local hash table updated by the router after each request |\n| **KV Sync** | `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true` | Distributed sync indexer fed by real-time block-stored/removed events from vLLM pods |\n\nBoth modes share the same stddev-based pod selection logic described below.\n\n## Routing Flow\n\n```\n Incoming Request\n │\n ▼\n┌─────────────────────────────┐\n│ Tokenize & hash prompt │ splits prompt into fixed-size blocks,\n│ into prefix blocks │ computes hash per block\n└────────────┬────────────────┘\n │\n ▼\n┌─────────────────────────────┐\n│ Match prefix hashes │\n│ against ready pods │\n└────────────┬────────────────┘\n │\n ┌────────┴────────┐\n │ match_pods │ NO match_pods\n │ found? │──────────────────────────────────────────────┐\n └────────┬────────┘ │\n │ YES │\n┌─────────────────────────────┐ │\n│ Sort match_pods by: │ │\n│ 1. prefix_match% DESC │ │\n│ 2. running_requests ASC │ │\n└────────────┬────────────────┘ │\n │ │\n┌─────────────────────────────┐ NO pod within threshold │\n│ Select first pod where: │─────────────────────────────────────►│\n│ running_req <= │ │\n│ mean + load_factor * σ │ │\n└────────────┬────────────────┘ │\n │ │\n │ ▼\n │ Least-request fallback among all ready pods\n │ │\n ▼ ▼\n Target Pod Selected\n```\n\n## Algorithm Details\n\n### Step 1 — Tokenize & Hash\n\nThe prompt is tokenized and split into fixed-size blocks. A hash is computed for each block.\nTwo tokenizer backends are supported:\n\n| Tokenizer | How it works |\n|------------|-------------------------------------------------------|\n| `character`| Splits raw text into individual characters (default) |\n| `tiktoken` | Uses the OpenAI [tiktoken](https://github.com/openai/tiktoken) BPE tokenizer |\n\nBlock hashes are used as cache keys — a pod that has processed a request with an identical\nprefix will have those KV blocks hot in GPU memory.\n\n### Step 2 — Prefix Match & Pod Selection\n\nEach candidate pod is scored by how many of the request's prefix blocks it already holds\n(`prefix_match_percent`). Matched pods are sorted:\n\n1. **Descending** by `prefix_match_percent` — prefer higher cache hit rate.\n2. **Ascending** by `running_requests` — break ties by choosing the less loaded pod.\n\nThe first pod in this sorted list that satisfies the load threshold is selected:\n\n```\npod.running_requests <= mean_running_requests + load_factor × std_dev\npod.running_requests <= max(load_cap_factor, 1) × (mean_running_requests + 1)\n```\n\n`load_factor` (= `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR`) controls how aggressively\nthe router skips overloaded cache-holding pods. If no matched pod is within the threshold, the\nrouter falls back to the global least-loaded pod across all ready pods. `load_cap_factor`\n(= `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR`) bounds load directly: no value in n pods sits more than\n(n−1)/√n standard deviations above the mean, so at `load_factor` 2 the first check cannot skip\nany pod in a pool of five or fewer.\n\n## Configuration\n\n| Environment Variable | Description | Default |\n|---|---|---|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | Tokenizer used to split the prompt. Options: `character`, `tiktoken`. | `character` |\n| `AIBRIX_PREFIX_CACHE_BLOCK_SIZE` | Number of tokens per prefix block. Smaller blocks = finer-grained matching but more hash overhead. | `128` |\n| `AIBRIX_PREFIX_CACHE_BLOCK_NUMBER` | Maximum number of prefix cache blocks tracked per pod. | `200000` |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `load_factor` in `mean + load_factor × σ`. Higher value tolerates more load on a high-cache-hit pod. | `1` |\n| `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR` | `load_cap_factor` in `max(load_cap_factor, 1) × (mean + 1)`. Higher value tolerates more load on a high-cache-hit pod; set a large value such as `100` to effectively disable it. Applies only when `prefix-cache` routes on its own, with the auto-blend off (`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT=0`). | `1.5` |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED` | Enable KV sync routing mode (requires remote tokenizer). | `false` |\n| `AIBRIX_PREFIX_CACHE_INCLUDE_TOOLS` | Prepend the canonicalized chat request `tools` to the prompt text before hashing, so requests that share messages but carry different tools do not look like a full prefix match. Token estimates keep using the messages only. KV-sync mode tokenizing through the remote chat template forwards the raw `tools` to the tokenizer regardless of this flag, so the template places them. Set to `false` to hash the messages only. | `true` |\n\nSee [ENV_VARS.md](../ENV_VARS.md#load-balance-router-algorithmsload_balancego) for the\nload-imbalance gate's `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` /\n`AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` variables — they belong to the load-balance router, not\nthis one.\n\n### Tuning guidance\n\n- **Block size**: use `128` for `character` tokenizer and `16` for `tiktoken`. Larger blocks\n reduce hash overhead but require longer identical prefixes for a match.\n- **Standard deviation factor**: `1` (default) keeps all pods within one stddev of mean load.\n Increase to `2` to allow higher-loaded pods to still be selected for cache-hit requests.\n\n\n# Virtual Token Counter (VTC) Routing\n\nVirtual Token Counter (VTC) is a fair scheduling algorithm for LLM serving based on the paper\n[Fairness in Serving Large Language Models (Sheng et al., 2024)](https://arxiv.org/abs/2401.00588).\nVTC tracks the weighted token service each client has received and routes new requests to the pod\nthat gives the least-served client the best chance of progressing — balancing fairness with\ncluster utilization.\n\n## vtc-basic\n\n`vtc-basic` is a Kubernetes-native simplification of VTC. Rather than managing a single global\nqueue, it selects a target pod per request by combining two scores:\n\n- **Fairness score** — how much token service this client has received relative to all clients\n within a sliding time window. Clients who have received less service get a lower (better) score.\n- **Utilization score** — the pod's current load normalized against `MAX_POD_LOAD`.\n\nThe pod with the **lowest combined score** is selected.\n\nToken counts and the adaptive window extrema exclude expired usage, including\nclients that have stopped sending requests. The in-memory tracker reclaims that\nusage on reads and updates; it does not need a request from the departed client.\n\n### Scoring diagram\n\n```\nFor each ready pod:\n\n fairness_score = client_tokens_in_window / total_tokens_in_window\n (normalized, 0.0 → 1.0; lower = client received less service)\n\n utilization_score = pod_running_requests / MAX_POD_LOAD\n (normalized, 0.0 → 1.0; lower = pod is less loaded)\n\n combined_score = FAIRNESS_WEIGHT × fairness_score\n + UTILIZATION_WEIGHT × utilization_score\n\n ──────────────────────────────────────────────────────────────\n Select pod with minimum combined_score\n```\n\n### Request lifecycle\n\n```\nIncoming Request (with user identity)\n │\n ▼\n┌──────────────────────────────────┐\n│ Look up client token count │\n│ in sliding time window │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ Compute fairness_score │\n│ per ready pod │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ Compute utilization_score │\n│ per ready pod │\n└───────────────┬──────────────────┘\n │\n ▼\n┌──────────────────────────────────┐\n│ combined_score = │\n│ w_f × fairness │\n│ + w_u × utilization │\n└───────────────┬──────────────────┘\n │\n ▼\n Select min-score pod\n │\n ▼\n┌──────────────────────────────────┐\n│ After response: update client │\n│ token count in window │\n│ (input × w_i + output × w_o) │\n└──────────────────────────────────┘\n```\n\nEnd-to-end tests: [vtc_routing_test.go](../../../../test/e2e/vtc_routing_test.go)\n\n## Configuration\n\n| Environment Variable | Description | Default |\n|-----------------------------------------------|------------------------------------------------------------------------------|----------------|\n| `AIBRIX_ROUTING_ALGORITHM` | Set to `vtc-basic` to enable this routing strategy. | `prefix-aware` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | Time unit for the sliding window. Options: `seconds`, `minutes`, `hours`. | `minutes` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | Number of time units in the sliding window. | `5` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | Minimum token floor for adaptive tracking — prevents division by near-zero. | `1000` |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | Maximum token cap for adaptive tracking. | `8000` |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | Weight applied to input tokens when updating a client's service counter. | `1.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | Weight applied to output tokens when updating a client's service counter. | `2.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | Normalization denominator for pod running-request count. | `100.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | Multiplier for the fairness term in the combined score (`w_f`). | `1.0` |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | Multiplier for the utilization term in the combined score (`w_u`). | `1.0` |\n\n### Tuning guidance\n\n- **Output token weight > input token weight** (default `2.0` vs `1.0`) reflects that output\n generation is more GPU-expensive than prompt processing.\n- **Increase `FAIRNESS_WEIGHT`** relative to `UTILIZATION_WEIGHT` when strict per-client\n fairness matters more than raw throughput.\n- **Increase `UTILIZATION_WEIGHT`** to prioritize routing to less-loaded pods, trading some\n fairness for better overall cluster efficiency.\n- **Window size** controls how long past token usage influences routing. A shorter window (e.g.\n `1 minute`) makes the router more reactive; a longer window (e.g. `10 minutes`) smooths out\n bursty clients.\n\n## Other VTC variants\n\nThe original VTC paper and reference implementation\n([slora/server/router](https://github.com/Ying1123/VTC-artifact/tree/main/slora/server/router))\ndescribe additional variants:\n\n| Variant | Description |\n|--------------|-----------------------------------------------------------------------------|\n| `vtc-fair` | Pure fairness — ignores utilization entirely. |\n| `vtc-max-fair` | Maximizes minimum service across all clients. |\n| `vtc-pred` | Uses output-length prediction to improve fairness estimates. |\n\nAdapting these variants to the AIBrix pod-selection model (rather than a single shared queue)\nis a candidate for future work.\n\n\n# Gateway Plugin Environment Variables\n\nThis document covers all environment variables used in the `pkg/plugins/gateway` package and its sub-packages.\n\nVariables of type `duration` are parsed with Go's [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration), so they need a unit: `30s`, `5m`, `1h30m`. A bare number such as `30`, or a zero or negative duration such as `0s` or `-1s`, is rejected with a warning and the default is used instead.\n\n---\n\n## General Gateway\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_DISABLE_RATE_LIMITING` | bool | `false` | Disable AIBrix user RPM/TPM and model RPS quota enforcement. Redis and `requestsInflight` remain active. | [cmd/plugins/main.go](../../../cmd/plugins/main.go), [gateway.go](gateway.go) |\n| `POD_NAME` | string | `\"\"` | Kubernetes pod name. Used for logging and metric label tagging. | [gateway.go](gateway.go), [util.go](util.go) |\n| `ROUTING_ALGORITHM` | string | _(none)_ | Default routing algorithm when no per-request override is set. | [types.go](types.go), [util.go](util.go) |\n| `AIBRIX_PRIORITY_TIER_ENABLED` | bool | `false` | Forward the priority tier declared by the `x-aibrix-priority-tier` request header as the upstream vLLM request priority. | [cmd/plugins/main.go](../../../cmd/plugins/main.go), [gateway_req_priority.go](gateway_req_priority.go) |\n\nWhen `AIBRIX_PRIORITY_TIER_ENABLED=true`, the gateway maps the `x-aibrix-priority-tier`\nrequest header to the `priority` field of the backend request body: `batch` becomes `100` and\n`background` becomes `1000`. The engine serves smaller values first, so the mapping only\nde-prioritizes a request and never pulls one ahead of another; the values sit below the engine\ndefault of `0`. Requests without the header, or with a tier outside the table, are forwarded\nunchanged, and a `priority` the caller already set in the body is kept. Tier names are matched\ncase-insensitively. The deployment has to run a priority-aware vLLM scheduler\n(`--scheduling-policy=priority`). With the default `false`, the header is ignored and request\nbodies are forwarded byte for byte.\n\nWhen `AIBRIX_DISABLE_RATE_LIMITING=true`, the gateway skips AIBrix user lookup, ignores the\n`user` header as routing identity, and does not write user or model quota counters. The header\nis still forwarded unchanged. Keep Redis configured when session affinity, asynchronous jobs,\nstate synchronization, or other gateway features require it. The per-replica\n`requestsInflight` guard remains active. Because routing user identity remains empty, asynchronous\nvideo jobs use the shared ownership scope; the unauthenticated `user` header does not isolate jobs\nbetween callers in this mode.\n\n---\n\n## Response Processing\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_TTFT_THRESHOLD_S` | int (seconds) | `1` | Time-to-first-token threshold in seconds. Requests exceeding this are flagged in response processing. | [gateway_rsp_body.go](gateway_rsp_body.go) |\n\n`AIBRIX_TTFT_THRESHOLD_S` can also be set per model with the top-level `ttftThresholdS` profile field; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Redis Sync (`statesync/`)\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_STATESYNC_ENABLED` | bool | `false` | Enable cross-replica state sync via Redis. Must be `true` to activate the statesync manager. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `AIBRIX_STATESYNC_SYNC_PERIOD` | duration | `10s` | Interval at which gateway state is synced to Redis across replicas. | [statesync/redissync.go](statesync/redissync.go) |\n\n---\n\n## Prefix Cache Router (`algorithms/prefixcache/router.go`)\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE` | string | `\"character\"` | Tokenizer type for prefix cache hashing. Options: `character`, `tiktoken`, `remote`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | int | `1` | Factor multiplied by the standard deviation of pod loads when selecting among prefix-matched pods (`pod.req ≤ mean + factor × σ`). | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_LOAD_CAP_FACTOR` | float64 | `1.5` | Prefix-matched pods with more than `max(c, 1) × (mean + 1)` running requests are skipped, where `c` is this factor. Applies only when `prefix-cache` routes on its own, with the auto-blend off (`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT=0`). Unlike the `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` check, which cannot skip any pod in a pool of five or fewer at a factor of `2`, it applies to pools of any size. Values between `0` and `1` are treated as `1`; `0`, negative or unparsable values use the default `1.5`. To effectively disable the cap, set a large value such as `100`. Environment-only. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_INCLUDE_TOOLS` | bool | `true` | Prepend a canonical rendering of the chat request `tools` (keys sorted, compact) to the text used for prefix matching, since chat templates commonly render tools ahead of the messages. For templates that render tools later in the prompt (e.g. near the last user turn), requests sharing tools but not messages can see a partial match the engine does not have; such deployments can set this to `false`. Requests without tools (absent, `null`, `[]` or not an array) are unaffected. Set to `false` to match on messages only. The tools text goes into a separate prefix-match text (`RoutingContext.PrefixText()`), which feeds prefix matching in `prefix-cache`, `prefix-cache-preble` and the PD prefill policies `prefix_cache`, `conductor` (including its matched/unmatched token estimate) and `hybrid_cache_load`. The routing message (`routingCtx.Message`) stays messages-only, so token estimates (VTC, `PromptTokens`/`PromptLength`, config-profile `promptTokensGte`/`promptTokensLt`) are unchanged. In KV-sync mode, chat requests tokenized through the remote chat template always forward the raw `tools` to the tokenizer, independent of this flag, since the template itself decides where tools go. | [prefix_tools.go](prefix_tools.go) |\n| `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER` | bool | `false` | Use a remote HTTP tokenizer service instead of the local tokenizer. Requires `AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED` | bool | `false` | Enable KV cache event synchronization across gateway replicas. When `true`, also requires `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`. | [algorithms/prefixcache/router.go](algorithms/prefixcache/router.go) |\n| `AIBRIX_PREFIX_CACHE_REMOTE_TOKENIZER_ENDPOINT` | string | `\"\"` | Remote tokenizer service endpoint URL. Required when `AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true`. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_PUBLISH_ADDR` | string | `\"\"` | ZMQ publish address for KV cache events. Used when KV event sync is enabled. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n| `AIBRIX_PREFIX_CACHE_KV_EVENT_SUBSCRIBE_ADDRS` | string | `\"\"` | ZMQ subscribe addresses for KV cache events (comma-separated). Used when KV event sync is enabled. | [pkg/constants/kv_event_sync.go](../../constants/kv_event_sync.go) |\n\n### Remote Tokenizer Pool (`algorithms/prefixcache/router.go`)\n\nThese configure the pool of remote tokenizer connections used when `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_VLLM_TOKENIZER_ENDPOINT_TEMPLATE` | string | `\"http://%s:8000\"` | HTTP endpoint template for tokenizer pods. `%s` is replaced with the pod name. |\n| `AIBRIX_TOKENIZER_HEALTH_CHECK_PERIOD` | duration | `30s` | How often to health-check tokenizer pool members. |\n| `AIBRIX_TOKENIZER_TTL` | duration | `300s` | TTL for cached tokenizer connections in the pool. |\n| `AIBRIX_MAX_TOKENIZERS_PER_POOL` | int | `100` | Maximum number of tokenizer connections in the pool. |\n| `AIBRIX_TOKENIZER_REQUEST_TIMEOUT` | duration | `5s` | Timeout for individual remote tokenizer requests. |\n\n`AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` can also be set per request by the model config profile (`routingConfig.prefixCache.standardDeviationFactor`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Load Balance Router (`algorithms/load_balance.go`)\n\nThe load-imbalance gate below is applied centrally by the gateway (`selectTargetPod` in\n`gateway.go`), once per request, ahead of whichever strategy actually routes it — not just when\n`load-balance` is selected. It restricts candidates to the least-loaded pods before that\nstrategy runs when load is severely skewed, and it **is** applied to the Prefix Cache and\nPreble routers (this replaces the prefix-cache-specific gate that used to live in\n`prefixcache/router.go`; see the deprecation note under [Variable Dependency\nNotes](#variable-dependency-notes)). It is exempted only for exclusive strategies (`pd`,\n`slo*`), which manage their own pod subsets and would have that management disrupted by a\nblanket running-request-count filter applied ahead of time. It does not participate in\nmulti-strategy soft-scoring (`ScoreAll`) for strategies blended alongside `load-balance` — see\n`AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` below for how `load-balance` itself gets\nsilently blended into other strategies to compensate.\n\n**TODO / known limitation:** the gate compares raw running-request count, with no notion of a\npod's capacity. In a heterogeneous pool this conflates \"carrying a lot of work\" with\n\"overloaded\" — a replica that's simply *faster* than its peers is expected to carry more\nconcurrent requests without being more loaded (the same capacity signal `load-balance`'s own\nscore uses), but this gate can still flag it as a hotspot and exclude it from routing before any\nstrategy's capacity-aware scoring gets a chance to run. See `getTargetPodListOnLoadImbalance` in\n[algorithms/load_balance.go](algorithms/load_balance.go).\n\nWhen `load-balance` is the (sole, non-blended) strategy that routes the request and multiple\npods tie on the lowest effective-load score, `Route()` breaks the tie using least combined\nGPU+CPU KV-cache usage (falling back to a random pick if cache metrics are unavailable for the\ntied pods) — the same secondary-signal pattern the Prefix Cache router uses to break ties in\nprefix-match percentage via request count.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` | float64 | `2.0` | Gate multiplier for 3+ pods: gate fires when `max_req > factor × (mean_req + 1)`. Ignored for 2-pod clusters (the relative check never holds there). | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` | int | `8` | Minimum absolute gap (`max_req − min_req`) required to trigger the load-imbalance gate. For 2 pods this is the sole trigger; for 3+ pods it is required alongside the factor check. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` | float64 | `0` | Weight λ of engine-queued requests (`num_requests_waiting`) added to the running count in the score: `(running + λ·queued) / capacity × …`. The default `0` scores on running requests only, since the gateway's running count already includes requests queued inside the engine; raise it only if you want queued work to count extra. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` | float64 | `2.0` | Strength α of the KV-pressure penalty: the score is multiplied by `1 + α·(1 − kv_free)²`. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n| `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` | float64 | `0.10` | Free-KV-cache fraction below which a pod scores `+Inf` from `load-balance`. When `load-balance` is the sole routing strategy this excludes the pod outright (falling back to the pod with the most KV headroom if every pod is below the threshold, instead of failing the request). When `load-balance` is blended with other strategies (the default for most requests — see \"Router Selection / Auto-Blend\" below), a `+Inf` score is only a strong penalty, not an exclusion: another strategy can still select the pod. Non-positive values fall back to the default. | [algorithms/load_balance.go](algorithms/load_balance.go) |\n\nThe five `AIBRIX_LOAD_BALANCE_*` gate and score variables can also be set per request by the model config profile (`routingConfig.loadBalance.*`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## Least-Request Top-K Router (`algorithms/least/request_top_k.go`)\n\nStrategy name `least-request-top-k`. Plain `least-request` always resolves to the single\nleast-loaded pod, so when several gateway replicas read the same Redis-backed running-request\ncounter and decide at the same moment, they can all send their next request to that pod. This\nrouter instead picks uniformly at random among the ready pods that are both among the `K`\nleast-loaded and within `epsilon` requests of the least-loaded pod's count. It is also a scorer\n(`ScoreAll`), so it can be blended with other strategies; it is the load-count scorer that the\n[auto-blend](#router-selection--auto-blend-algorithmsroutergo) appends by default, so the variables below\napply to every auto-blended request, not only to requests that name this strategy.\n\nA pod that has just become eligible reads as count `0` and would otherwise be the sole eligible\ncandidate until it catches up. To prevent that, a pod's effective count is raised by a penalty\nthat starts at the lowest count among the already-warm pods and decays linearly to `0` over the\nramp window. The window is measured from when this gateway first sees the pod routable (Ready,\nwith an IP, not terminating or draining), not from when the pod was created, so time spent\nPending or loading a model does not use it up. Ramp state is local to each gateway; a gateway\nthat has just restarted sees every pod as new, so with no warm pod to compare against it applies\nno penalty until some pods age out of the window. The clock is kept, not restarted, when a pod\nflaps NotReady or is re-added with the same IP within 60 seconds; a new IP, or a longer gap, starts\nit over.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K` | int | `5` | `K`: how many of the least-loaded ready pods are eligible. Should be at least the number of gateway replicas deciding concurrently. Must be positive; any other value falls back to the default. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K_EPSILON` | int | `4` | A pod is eligible only if its count is within this many requests of the least-loaded pod's. Combined with `K` as an AND. Must be positive; any other value falls back to the default. The default is a starting point, not a validated one: tune it to the deployment's typical running-request count. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n| `AIBRIX_ROUTING_LEAST_REQUEST_TOP_K_RAMP_WINDOW` | duration | `300s` | Window over which a newly routable pod's ramp penalty decays to `0`, counted from the first time the gateway sees the pod routable. `0` disables the ramp. Any other non-positive or unparseable value falls back to the default. | [algorithms/least/request_top_k.go](algorithms/least/request_top_k.go) |\n\nThese are read once at startup and are environment-only (no `routingConfig` override).\n\n---\n\n## Router Selection / Auto-Blend (`algorithms/router.go`)\n\n`RouterManager.Select` silently blends `load-balance` and [`least-request-top-k`](#least-request-top-k-router-algorithmsleastrequest_top_kgo)\nbehind whatever strategy a caller actually asked for, so no single strategy can keep steering\ntraffic at an already-hot pod even without going through the central load-imbalance gate above.\nThe caller never sees this: `ctx.Algorithm`, response headers, and `Validate()` all continue to\nreflect exactly what was requested. The blend is skipped entirely for exclusive strategies\n(`pd`, `slo*`), for an explicit, standalone `load-balance` selection, and for a bare\n`session-affinity` selection, all of which must keep running their own dedicated routing logic.\n`session-affinity`'s own scoring is binary (1 for its resolved pod, 0 for everything else), so\nany load-balance weight below its own could never change the outcome anyway — blending it in\nwould be dead weight at best. A bare `prefix-cache` request, whose scoring is graded rather than\nbinary, still uses a 5:4 (1.25:1) ratio against `load-balance` instead of the flat weight-1\nappend, and does not receive `least-request-top-k`, so the cache-affinity signal wins an exact-tie\ndisagreement without masking real load imbalance.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---|---|---|\n| `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` | int | `1` | Weight coefficient for the silently-appended `load-balance` scorer. Set to `0` to disable the whole auto-blend feature. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_LEAST_REQUEST_WEIGHT` | int | `1` | Weight coefficient for the silently-appended `least-request-top-k` scorer. It is added only when neither `least-request` nor `least-request-top-k` is already in the strategy (an explicit weight of `0` on either also suppresses it), and is needed so multi-port/data-parallel pod routing keeps working under the blend. Plain `least-request` is not used here because every gateway replica would score the same single least-loaded pod best. The variable keeps its `LEAST_REQUEST` name for compatibility. Set to `0` to omit it from the blend. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` | int | `5` | Primary weight used when auto-blending a bare `prefix-cache` request. With the matching load-balance weight this is a 5:4 (1.25:1) lean toward cache affinity. `0` is rejected with a warning and the default is used. | [algorithms/router.go](algorithms/router.go) |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_LOAD_BALANCE_WEIGHT` | int | `4` | `load-balance` weight paired with `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` for a bare `prefix-cache` request. `0` leaves those requests with prefix-cache scoring alone. | [algorithms/router.go](algorithms/router.go) |\n\nThe four `AIBRIX_ROUTING_AUTO_BLEND_*` weights can also be set per request by the model config profile (`routingConfig.autoBlend.*`); see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## External Replica Router (`algorithms/external*.go`)\n\nThe optional `external` strategy sends the already-filtered candidate snapshot to one\noperator-configured HTTP endpoint. An unset endpoint disables the strategy. Invalid enabled\nconfiguration fails Gateway startup. Candidate metrics and labels are opt-in allowlists; prompts,\nrequest bodies, credentials, arbitrary client headers, and pod IPs are never sent.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_EXTERNAL_ROUTER_ENDPOINT` | URL | _(unset)_ | Complete `http` or `https` operation URL. Userinfo and fragments are rejected. |\n| `AIBRIX_EXTERNAL_ROUTER_POLICY_MODE` | enum | _(required)_ | `Advisory` or `Authoritative`. |\n| `AIBRIX_EXTERNAL_ROUTER_FAILURE_MODE` | enum | _(required)_ | `FailOpen` or `FailClosed`; Authoritative requires FailClosed. |\n| `AIBRIX_EXTERNAL_ROUTER_FALLBACK` | string | _(none)_ | Registered non-external, non-exclusive local router; required for Advisory and FailOpen. `pd` and `slo*` are rejected because they require dedicated Gateway preprocessing. |\n| `AIBRIX_EXTERNAL_ROUTER_TIMEOUT` | duration | `10ms` | Deadline for one decision exchange. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_INFLIGHT` | int | `256` | Non-blocking per-process bulkhead capacity. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_REQUEST_BYTES` | bytes | `256KiB` | Maximum encoded request size. Accepts bytes, `KiB`, or `MiB`. |\n| `AIBRIX_EXTERNAL_ROUTER_MAX_RESPONSE_BYTES` | bytes | `64KiB` | Maximum response body size. Accepts bytes, `KiB`, or `MiB`. |\n| `AIBRIX_EXTERNAL_ROUTER_FAILURE_THRESHOLD` | int | `5` | Consecutive attempted-exchange system failures before opening the circuit. |\n| `AIBRIX_EXTERNAL_ROUTER_OPEN_DURATION` | duration | `1s` | Circuit-open duration before one half-open probe. |\n| `AIBRIX_EXTERNAL_ROUTER_AUTH_TOKEN_FILE` | path | _(unset)_ | Optional non-empty bearer token file loaded at startup. |\n| `AIBRIX_EXTERNAL_ROUTER_CANDIDATE_ATTRIBUTES` | CSV | _(empty)_ | Pod-label keys allowed into candidate attributes. |\n| `AIBRIX_EXTERNAL_ROUTER_CANDIDATE_METRICS` | CSV | _(empty)_ | Any of `runningRequests`, `engineUtilization`, `kvCacheUsage`. |\n| `AIBRIX_EXTERNAL_ROUTER_POLICY_ATTRIBUTES` | CSV | _(empty)_ | Downstream extension-point allowlist for trusted RoutingContext policy attributes. Upstream AIBrix has no built-in producer; integrators must call `SetTrustedPolicyAttribute`. Raw headers are never copied automatically. |\n\nSee the [External Replica Routing guide](../../../docs/source/features/gateway-routing/external-replica-routing.rst)\nand the published OpenAPI contract for policy and protocol details.\n\n---\n\n## Preble (Prefix Cache with Histogram) Router (`algorithms/prefixcache/preble.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_PREBLE_TARGET_GPU` | string | `\"V100\"` | GPU model used for hardware-specific latency estimates in the Preble algorithm. |\n| `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` | int | `45` | Expected decode sequence length used for cache allocation decisions. |\n| `AIBRIX_ROUTER_PREBLE_SLIDING_WINDOW_PERIOD` | int (minutes) | `3` | Sliding window length in minutes for histogram metrics collection. |\n| `AIBRIX_ROUTER_PREBLE_EVICTION_LOOP_INTERVAL` | int (ms) | `1000` | Interval in milliseconds between cache eviction loop executions. |\n\n`AIBRIX_ROUTER_PREBLE_TARGET_GPU` and `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` can also be set per request by the model config profile (`routingConfig.preble.*`). The window and eviction variables stay environment-only; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## VTC (Virtual Token Counter) Router\n\n### Token Tracker (`algorithms/vtc/token_tracker.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | int | `5` | Sliding window size (in `TIME_UNIT` units) for token usage tracking. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | string | `\"minutes\"` | Time unit for the sliding window. Options: `minutes`, `seconds`, `milliseconds`. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | float64 | `1000.0` | Minimum token count threshold for adaptive load normalization. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | float64 | `8000.0` | Maximum token count threshold for adaptive load normalization. |\n\n### VTC Basic Scorer (`algorithms/vtc/vtc_basic.go`)\n\nScoring formula: `score = (fairnessWeight * normFairness + utilizationWeight * normUtilization) / normFreeGPU`\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | float64 | `100.0` | Load value at which a pod is considered fully saturated. |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | float64 | `1.0` | Weight applied to input tokens when computing pod load. |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | float64 | `2.0` | Weight applied to output tokens when computing pod load (typically higher than input). |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | float64 | `1.0` | Weight of the fairness component in the routing score. |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | float64 | `1.0` | Weight of the utilization component in the routing score. |\n\n`AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD`, `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` and `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` can also be set per request by the model config profile (`routingConfig.vtc.*`). The token tracker variables stay environment-only; see [Model Config Profile Overrides](#model-config-profile-overrides).\n\n---\n\n## PD (Prefill-Decode) Disaggregation Router (`algorithms/pd/router/router.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | int (seconds) | `30` | HTTP request timeout for prefill pod calls. |\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | int32 | `16` | Minimum (max − min) running-request spread across prefill pods to trigger load-imbalance routing. |\n| `AIBRIX_PREFILL_LOAD_CAP_FACTOR` | float64 | `1.5` | Prefill pods with more than `c × (mean + 1)` in-flight prefill requests are skipped, where `c` is this factor. Unlike the `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` check, it applies to pools of any size. Values between `0` and `1` are treated as `1`; `0`, negative or unparsable values use the default `1.5`. To effectively disable the cap, set a large value such as `100`. Lower values give up more prefix-cache affinity: with random (Poisson) load and no hot pod, `1.5` skips at least one pod on up to about 10% of requests with 4 prefill pods and about 20% with 6. `2` skips less often but cannot fire in a two-pod pool. |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | float64 | `16.0` | Minimum (max − min) running-request spread across decode pods to trigger load-imbalance routing. |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | float64 | `2048.0` | Minimum (max − min) token-throughput spread (tokens/s) across decode pods to trigger throughput-imbalance routing. |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | float64 | `1.5` | Max/min drain-rate score ratio above which the slowest decode pod is excluded from selection. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | bool | `false` | Route requests to prefill pods whose prompt-length bucket matches the request length. |\n| `AIBRIX_BUCKET_SERVE` | bool | `false` | Adaptive bucket serving: band the prompt-length range that several rolesets declare in common and prefer the roleset a request length is banded to. Requires `AIBRIX_PROMPT_LENGTH_BUCKETING=true`. |\n| `AIBRIX_BUCKET_SERVE_MODE` | string | `\"throughput\"` | The unit the adaptive cut points are measured in: `throughput` (prompt token mass) or `rps` (request counts). An unknown value keeps the default. |\n| `AIBRIX_KV_CONNECTOR_TYPE` | string | `\"shfs\"` | KV cache transfer backend. Options: `shfs` (GPU shared memory), `nixl` (Neuron). |\n| `AIBRIX_PREFILL_SCORE_POLICY` | string | `\"prefix_cache\"` | Strategy for selecting the prefill pod. Options: `prefix_cache`, `least_request`. |\n| `AIBRIX_DECODE_SCORE_POLICY` | string | `\"load_balancing\"` | Strategy for selecting the decode pod. Options: `load_balancing`, `least_request`, `conductor`, `token_load`. |\n| `AIBRIX_DECODE_TOKEN_LOAD_OUTPUT_GROWTH` | bool | `true` | `token_load` decode policy only: add an estimate of the output generated so far (per-request decode rate × time since routing) to the decode score. `false` scores prompt tokens only. |\n| `AIBRIX_DECODE_TOKEN_LOAD_CAPACITY_AWARE` | bool | `true` | `token_load` decode policy only: multiply each decode pod's score by the candidates' mean KV capacity divided by the pod's own (`num_gpu_blocks` × `block_size` from vLLM's `vllm:cache_config_info`, or SGLang's `sglang:max_total_num_tokens`), so pods of different sizes are compared by how full they are. Unknown capacities count as the mean; with none known, or `false`, scores are token counts. |\n| `AIBRIX_TOKEN_LOAD_SHARED_LEDGER` | bool | `true` | `token_load` decode policy only, with Redis configured: share the decode ledger across gateway replicas, so each replica adds the other live replicas' charges to its decode score. `false` keeps the ledger local to each replica. |\n\nThe prefill/decode routing thresholds (`AIBRIX_PREFILL_*`, `AIBRIX_DECODE_*`, `AIBRIX_TOKEN_LOAD_*`, `AIBRIX_HYBRID_CACHE_LOAD_FACTOR`, `AIBRIX_MIN_MATCH_PCT`, `AIBRIX_PROMPT_LENGTH_BUCKETING`, `AIBRIX_BUCKET_SERVE`,\n`AIBRIX_BUCKET_SERVE_MODE`), except `AIBRIX_PREFILL_LOAD_CAP_FACTOR`, can also be set per request by the model config profile\n(`routingConfig.pd.*`, `routingConfig.promptLengthBucketing` and `routingConfig.bucketServe*`); see [Model Config Profile Overrides](#model-config-profile-overrides). `AIBRIX_PREFILL_LOAD_CAP_FACTOR` is environment-only.\n\n### PD Prefill Fail-Fast (`algorithms/pd/abort.go`)\n\nWhen the prefill leg of a PD request fails, the decode pod is left waiting for a\nKV transfer that will never arrive until its own bootstrap timeout expires\n(300s in SGLang), holding its pre-allocated KV pages. The gateway injects its own\nrequest id (`rid`) into both legs, so on a prefill failure it can post\n`{\"rid\": ...}` to the decode pod's `/abort_request` and release those pages\nimmediately. The abort is best-effort and asynchronous: nothing waits on it, and\nit is skipped once the decode pod has started answering the client - a call\nalready on the wire when that happens is cancelled rather than left to run.\n\nThe abort is sent twice, because the first attempt can overtake the decode\nrequest itself on the pod (the engine answers `200` either way, so the gateway\ncannot tell a matched abort from a dropped one). Repeating an abort the engine\nalready applied, or never matched, is a no-op on both sides. The wait between\nthe two attempts is interruptible, so a decode leg that starts answering in the\nmeantime ends the retry immediately instead of at the end of the delay.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | int (seconds) | `3` | Per-attempt timeout of the `/abort_request` call to the decode pod. `0` disables decode aborts entirely; the prefill failure is still recorded, logged and reported to the client. |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | int (seconds) | `2` | Delay before the second abort attempt. `0` sends a single attempt. |\n\nMetrics: `gateway_pd_decode_abort_total{prefill_failure_class, result}`, where\n`result` is one of `ok`, `error`, `skipped_streaming`, `skipped_disabled`,\n`skipped_no_rid`, `skipped_no_target`.\n\n### PD Decode Watchdog (`gateway_decode_watchdog.go`)\n\nIn SGLang disaggregation the prefill leg only returns once the decode pod has\ntaken the KV transfer, so a successful prefill means the decode pod owes the\nclient a response. If the decode pod then stops answering (its scheduler died or\nwedged), nothing else notices and the client hangs until Envoy's route timeout.\nThe watchdog bounds that wait: once the prefill leg succeeded and until the\nfirst response body chunk comes back from the decode pod, it fails the request\nafter the first-response timeout and posts one best-effort `/abort_request` for\nthe request's `rid` to the decode pod. Response headers do not end this phase:\nSGLang sends the headers of a streaming response before its first token, so a\ndecode pod that wedges before the first token still delivers them. How the\nclient is failed depends on whether those headers already went out:\n\n- No headers yet: the client gets a `504` carrying the `x-error-pd-decode`\n header, and the ext_proc stream is closed with `DeadlineExceeded`.\n- Headers sent: no error response can replace a response whose status line is\n already on its way, so the ext_proc stream is only closed with\n `DeadlineExceeded` and Envoy resets the client connection. The client sees a\n `200` with a truncated (here: empty) body and no `x-error-pd-decode` header.\n\nFrom the first chunk of a streaming response on, the watchdog watches the gap\nbetween chunks instead: a stream that goes silent for\n`AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` is cut the same way, and the decode pod\ngets the same abort. It is armed only for requests that carry a gateway-owned\n`rid`, i.e. SGLang PD requests.\n\nA prefill leg that answered `200` with a body the gateway could not parse\n(`bad_response`) counts as succeeded for the watchdog: the KV transfer completed\nall the same, so fail-fast leaves the request alone and the watchdog times the\ndecode pod from that answer.\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | int (seconds) | `60` | **Streaming requests only** (`\"stream\": true`). Time to first token: how long the decode pod is given to send its first response body chunk, measured from the moment the prefill leg returned successfully. Response headers alone do not count. On expiry the client gets a `504` (`did not start responding within …`), or a cut stream if the headers already went out. `0` disables the watchdog for streaming requests. |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | int (seconds) | `0` (disabled) | **Non-streaming requests only**. Same arming point, different meaning: the decode pod sends the body of a non-streaming response only once generation is finished, so this budget bounds the *whole* decode, not the time to first token. It is off by default because the safe value depends on the caller's `max_tokens` and the pod's throughput; operators who know both can set one. On expiry the client gets a `504` (`did not finish responding within …`). |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | int (seconds) | `120` | **Streaming requests only**. Longest gap allowed between two response body chunks once the first one arrived. Every chunk re-arms it. On expiry the stream is cut short. `0` disables it. |\n\nThe abort uses `AIBRIX_DECODE_ABORT_TIMEOUT` as its timeout and is counted in\n`gateway_pd_decode_abort_total` with `prefill_failure_class=\"watchdog_first_response\"`\nor `\"watchdog_stream_idle\"`.\n\nMetrics: `gateway_pd_decode_watchdog_total{phase}`, where `phase` is\n`first_response` or `stream_idle`. A request failed by the watchdog is also counted in\n`gateway_request_model_fail_total` with `status=\"pd_decode_watchdog\"` and\n`status_code=\"504\"`, including the cut streams whose client saw a `200`: watch\nfor those through this counter and the client's stream error, not through an\nerror body.\n\n### Decode Load Balancer Scorer (`algorithms/pd/decode_scorer.go`)\n\nScoring formula: `score = (wRun × normRunning + wThroughput × normInvThroughput) / normFreeGPU`\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | float64 | `1.0` | Weight for the normalized running-request term in the decode LB score. |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | float64 | `1.0` | Weight for the normalized inverse-throughput term in the decode LB score. |\n\n---\n\n## Utilities (`algorithms/pd/engine/trtllm.go`)\n\n| Variable | Type | Default | Description |\n|---|---|---|---|\n| `AIBRIX_TRT_MACHINE_ID` | int64 | `0` | 10-bit machine ID (0–1023) used in Snowflake-style disaggregation request ID generation: `[timestamp:41b][machineID:10b][counter:12b]`. Panics on init if out of range. |\n\n---\n\n## Model Config Profile Overrides\n\nA model's `model.aibrix.ai/config` annotation can carry per-profile routing knobs\n(`routingConfig`) that override the matching variable above for that profile's requests only.\nAn unset knob keeps the environment default, and a value the variable would reject (a negative\nfactor, a percentage outside its range, an unknown GPU name) is ignored, so a profile can only\nnarrow or sharpen routing behavior. The values are resolved once per request and applied to that\nrequest alone; a request whose profile sets none behaves exactly as before. The user-facing\ndescription and examples live in the Config Profiles section of the gateway plugin guide\n([docs/source/features/gateway-routing/gateway-plugins.rst](../../../docs/source/features/gateway-routing/gateway-plugins.rst)).\n\n| Variable | Profile field | Notes |\n|---|---|---|\n| `AIBRIX_TTFT_THRESHOLD_S` | `ttftThresholdS` | Top-level profile field, not inside `routingConfig`. `0` counts as unset and keeps the environment default, so a profile can only change the threshold to another positive value, never to `0`. |\n| `AIBRIX_PROMPT_LENGTH_BUCKETING` | `routingConfig.promptLengthBucketing` | Turns bucketing on or off for the profile's requests. |\n| `AIBRIX_BUCKET_SERVE` | `routingConfig.bucketServe` | Turns the adaptive plan on or off for the profile's requests. |\n| `AIBRIX_BUCKET_SERVE_MODE` | `routingConfig.bucketServeMode` | `throughput` or `rps`; a name the planner does not know keeps the environment default. |\n| `AIBRIX_DECODE_ABORT_TIMEOUT` | `routingConfig.pd.decodeAbortTimeout` | `0` sends the abort without waiting. |\n| `AIBRIX_DECODE_ABORT_RETRY_DELAY` | `routingConfig.pd.decodeAbortRetryDelay` | `0` repeats the abort immediately. |\n| `AIBRIX_DECODE_FIRST_RESPONSE_TIMEOUT` | `routingConfig.pd.decodeFirstResponseTimeout` | Seconds; `0` disables the decode watchdog for the profile's streaming requests. |\n| `AIBRIX_DECODE_RESPONSE_TIMEOUT` | `routingConfig.pd.decodeResponseTimeout` | Seconds; `0` disables the decode watchdog for the profile's non-streaming requests. |\n| `AIBRIX_DECODE_STREAM_IDLE_TIMEOUT` | `routingConfig.pd.decodeStreamIdleTimeout` | Seconds; `0` disables the stream-idle phase for the profile. |\n| `AIBRIX_PREFILL_LOAD_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.prefillLoadImbalanceMinSpread` | |\n| `AIBRIX_DECODE_LOAD_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.decodeLoadImbalanceMinSpread` | |\n| `AIBRIX_DECODE_THROUGHPUT_IMBALANCE_MIN_SPREAD` | `routingConfig.pd.decodeThroughputImbalanceMinSpread` | |\n| `AIBRIX_DECODE_SCORE_RATIO_THRESHOLD` | `routingConfig.pd.decodeScoreRatioThreshold` | |\n| `AIBRIX_DECODE_LB_WEIGHT_RUNNING` | `routingConfig.pd.decodeLBWeightRunning` | |\n| `AIBRIX_DECODE_LB_WEIGHT_THROUGHPUT` | `routingConfig.pd.decodeLBWeightThroughput` | |\n| `AIBRIX_TOKEN_LOAD_KV_WEIGHT` | `routingConfig.pd.tokenLoadKVWeight` | |\n| `AIBRIX_TOKEN_LOAD_REQUEST_COST` | `routingConfig.pd.tokenLoadRequestCost` | |\n| `AIBRIX_TOKEN_LOAD_TTL_SECONDS` | `routingConfig.pd.tokenLoadTTLSeconds` | `0` disables the sweep for the profile's requests. |\n| `AIBRIX_TOKEN_LOAD_SESSION_TTL_SECONDS` | `routingConfig.pd.tokenLoadSessionTTLSeconds` | `0` disables the session delta. |\n| `AIBRIX_HYBRID_CACHE_LOAD_FACTOR` | `routingConfig.pd.hybridCacheLoadFactor` | Accepted range 0 to 1. |\n| `AIBRIX_MIN_MATCH_PCT` | `routingConfig.pd.minMatchPct` | Accepted range 0 to 100; `0` disables the minimum-match clamp. |\n| `AIBRIX_PREFILL_REQUEST_TIMEOUT` | `routingConfig.pd.prefillRequestTimeout` | Seconds. |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` | `routingConfig.loadBalance.imbalanceFactor` | |\n| `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` | `routingConfig.loadBalance.imbalanceMinGap` | |\n| `AIBRIX_LOAD_BALANCE_QUEUED_WEIGHT` | `routingConfig.loadBalance.queuedWeight` | `0` is the V1 formula, running requests only. |\n| `AIBRIX_LOAD_BALANCE_KV_PRESSURE_ALPHA` | `routingConfig.loadBalance.kvPressureAlpha` | `0` drops the penalty. |\n| `AIBRIX_LOAD_BALANCE_KV_CRITICAL_FREE` | `routingConfig.loadBalance.kvCriticalFree` | Accepted range 0 to 1; `0` disables the guardrail. |\n| `AIBRIX_PREFIX_CACHE_STANDARD_DEVIATION_FACTOR` | `routingConfig.prefixCache.standardDeviationFactor` | Also read by the PD prefill candidacy filter. |\n| `AIBRIX_ROUTER_PREBLE_TARGET_GPU` | `routingConfig.preble.targetGPU` | Known values `A6000` and `V100`; anything else is ignored. |\n| `AIBRIX_ROUTER_PREBLE_DECODING_LENGTH` | `routingConfig.preble.decodingLength` | |\n| `AIBRIX_ROUTER_VTC_BASIC_MAX_POD_LOAD` | `routingConfig.vtc.maxPodLoad` | |\n| `AIBRIX_ROUTER_VTC_BASIC_FAIRNESS_WEIGHT` | `routingConfig.vtc.fairnessWeight` | `0` drops the fairness term. |\n| `AIBRIX_ROUTER_VTC_BASIC_UTILIZATION_WEIGHT` | `routingConfig.vtc.utilizationWeight` | `0` drops the utilization term. |\n| `AIBRIX_ROUTING_AUTO_BLEND_LOAD_BALANCE_WEIGHT` | `routingConfig.autoBlend.loadBalanceWeight` | `0` disables the auto-blend for the profile's requests. |\n| `AIBRIX_ROUTING_AUTO_BLEND_LEAST_REQUEST_WEIGHT` | `routingConfig.autoBlend.leastRequestWeight` | |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_WEIGHT` | `routingConfig.autoBlend.prefixCacheWeight` | `0` is rejected with a warning and the environment value is kept; it would drop the caller's own strategy from the blend. |\n| `AIBRIX_ROUTING_AUTO_BLEND_PREFIX_CACHE_LOAD_BALANCE_WEIGHT` | `routingConfig.autoBlend.prefixCacheLoadBalanceWeight` | `0` leaves those requests with prefix-cache scoring alone. |\n| `AIBRIX_ROUTER_VTC_BASIC_INPUT_TOKEN_WEIGHT` | `routingConfig.vtc.inputTokenWeight` | Also scopes the VTC token tracker: this weight is baked into the tracker, so the profile's requests get one of their own. |\n| `AIBRIX_ROUTER_VTC_BASIC_OUTPUT_TOKEN_WEIGHT` | `routingConfig.vtc.outputTokenWeight` | Scopes the tracker the same way as the input weight. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_WINDOW_SIZE` | `routingConfig.vtc.tokenTrackerWindowSize` | Sliding window of the profile's tracker, in `tokenTrackerTimeUnit` units. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_TIME_UNIT` | `routingConfig.vtc.tokenTrackerTimeUnit` | Bucket size of that window: `minutes`, `seconds` or `milliseconds`. An unknown name is ignored instead of being normalized. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MIN_TOKENS` | `routingConfig.vtc.tokenTrackerMinTokens` | Floor the profile's tracker reports while its window holds little activity. |\n| `AIBRIX_ROUTER_VTC_TOKEN_TRACKER_MAX_TOKENS` | `routingConfig.vtc.tokenTrackerMaxTokens` | Ceiling the profile's tracker reports while its window holds little activity. |\n\n### Scoped state behind the profile overrides\n\nThe VTC token tracker is constructed once from its window, time unit, token floors and the two\ntoken weights, so a profile that overrides any of the six gets a tracker of its own instead of\nretuning the shared one: trackers are keyed by the resolved values, so profiles that agree share\none, a profile that sets none keeps the shared tracker exactly as before, and the number of\ntrackers is bounded (16 per process; a profile past the bound keeps the shared tracker rather\nthan failing its requests).\n\n### Environment-only routing variables\n\nThese configure state shared by every model of one gateway process, so a profile cannot override\nthem without splitting or resizing that state for everyone:\n\n- `AIBRIX_ROUTER_PREBLE_SLIDING_WINDOW_PERIOD` and `AIBRIX_ROUTER_PREBLE_EVICTION_LOOP_INTERVAL`: the preble histogram window and its eviction loop are process-wide timers; scoping those needs a design of its own.\n- `AIBRIX_SESSION_AFFINITY_MAX_LOCAL_KEYS`: bounds the gateway-local session pin cache, whose entries are shared facts about a model and a session.\n- `AIBRIX_ROUTER_MAX_CACHED_ALGORITHM_STRINGS`: bounds the process-wide routing string caches.\n- `AIBRIX_TOKEN_LOAD_MAX_SESSIONS`: bounds the (model, session) table of the token-load tracker, which the gateway shares across models.\n\nVariables without a `routingConfig` field (tokenizer endpoints, Redis and statesync settings, the\nrate limiting switches, `AIBRIX_KV_CONNECTOR_TYPE`) stay environment-only as deployment-level\nsettings.\n\n---\n\n## Variable Dependency Notes\n\nThe following variables have interdependencies that must be satisfied together:\n\n- **KV event sync** requires all three to be set consistently:\n ```\n AIBRIX_PREFIX_CACHE_KV_EVENT_SYNC_ENABLED=true\n AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true\n AIBRIX_PREFIX_CACHE_TOKENIZER_TYPE=remote\n AIBRIX_PREFIX_CACHE_REMOTE_TOKENIZER_ENDPOINT=\n ```\n\n- **Remote tokenizer pool** is initialized whenever `AIBRIX_PREFIX_CACHE_USE_REMOTE_TOKENIZER=true`. The pool variables (`AIBRIX_VLLM_TOKENIZER_ENDPOINT_TEMPLATE`, `AIBRIX_TOKENIZER_*`, `AIBRIX_MAX_TOKENIZERS_PER_POOL`) all apply in that case.\n\n- **`AIBRIX_PREFIX_CACHE_POD_RUNNING_REQUEST_IMBALANCE_ABS_COUNT` is removed.** The\n prefix-cache-specific load-imbalance gate it used to tune was replaced by the centralized gate\n described under [Load Balance Router](#load-balance-router-algorithmsload_balancego), which\n now applies to prefix-cache (and every other non-exclusive strategy) via\n `AIBRIX_LOAD_BALANCE_IMBALANCE_MIN_GAP` / `AIBRIX_LOAD_BALANCE_IMBALANCE_FACTOR` instead. The\n trigger condition also changed: the old gate fired on absolute gap alone; the new one\n additionally requires a relative `factor × (mean + 1)` check for 3+ pod clusters, so it fires\n less often on larger fleets. Setting the old variable now only logs a startup warning.\n\n## OpenTelemetry\n\nThe gateway plugins feature built-in support for distributed tracing via OpenTelemetry (OTel), empowering you to monitor and trace end-to-end requests across the entire external processing pipeline.\n\n**Tracing is opt-in by default.** Telemetry components will only initialize if you explicitly configure either ``OTEL_EXPORTER_OTLP_ENDPOINT`` or ``OTEL_EXPORTER_OTLP_TRACES_ENDPOINT``. If both are omitted, all tracing capabilities remain disabled to conserve system resources.\n\n| Variable | Type | Default | Description | Source |\n|---|---|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|\n| `OTEL_EXPORTER_OTLP_PROTOCOL` | string | `grpc` | The transport protocol for OTLP data. Valid options are `grpc`, `http`, or `http/protobuf`. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | string | `\"\"` | Base URL for all OTLP signals. **⚠️ Note:** ensure your ENDPOINT URLs include the `http://` or `https://` prefix. The SDK automatically appends signal-specific paths to this URL (e.g., appending `/v1/traces` for HTTP exports). | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | string | `\"\"` | Target URL specifically for traces. Takes precedence over the global `ENDPOINT`. **⚠️ Note:** This URL is used exactly **as-is**. The SDK will **NOT** automatically append `/v1/traces` to it. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_HEADERS` | string | `\"\"` | Key-value pairs used as headers for OTLP requests (e.g., `key1=value1,key2=value2`). Useful for passing auth tokens to backends like Datadog or Honeycomb. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_TIMEOUT` | string | `10s` | Maximum time the OTLP exporter will wait for each batch export. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_INSECURE` | bool | `false` | Set to `true` to disable TLS/HTTPS for the exporter. Force downgrade to HTTP. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n| `OTEL_EXPORTER_OTLP_INSECURE_SKIP_VERIFY` | bool | `false` | Keeps TLS active but skips server certificate validation (useful for self-signed certs). Set to \"true\" to enable. | [cmd/plugins/main.go](../../../cmd/plugins/main.go) |\n\n> **Note on OpenTelemetry Configuration:**\n> Aibrix supports the standard OpenTelemetry Protocol Exporter environment variables. For advanced OTLP configurations (such as `_CERTIFICATE`, `_CLIENT_KEY`, or specific `_COMPRESSION` settings), please refer to the [OpenTelemetry Protocol Exporter](https://opentelemetry.io/docs/specs/otel/protocol/exporter/)\n\n\n{\n \"xds\": {\n \"aibrix-system/aibrix-eg\": {\n \"configs\": [\n {\n \"@type\": \"type.googleapis.com/envoy.admin.v3.ListenersConfigDump\",\n \"dynamicListeners\": [\n {\n \"activeState\": {\n \"listener\": {\n \"@type\": \"type.googleapis.com/envoy.config.listener.v3.Listener\",\n \"accessLog\": [\n {\n \"filter\": {\n \"responseFlagFilter\": {\n \"flags\": [\n \"NR\"\n ]\n }\n },\n \"name\": \"envoy.access_loggers.file\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\",\n \"logFormat\": {\n \"jsonFormat\": {\n \":authority\": \"%REQ(:AUTHORITY)%\",\n \"bytes_received\": \"%BYTES_RECEIVED%\",\n \"bytes_sent\": \"%BYTES_SENT%\",\n \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n \"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\n \"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\n \"duration\": \"%DURATION%\",\n \"method\": \"%REQ(:METHOD)%\",\n \"protocol\": \"%PROTOCOL%\",\n \"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\n \"response_code\": \"%RESPONSE_CODE%\",\n \"response_code_details\": \"%RESPONSE_CODE_DETAILS%\",\n \"response_flags\": \"%RESPONSE_FLAGS%\",\n \"route_name\": \"%ROUTE_NAME%\",\n \"start_time\": \"%START_TIME%\",\n \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n \"upstream_host\": \"%UPSTREAM_HOST%\",\n \"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\n \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\n \"user-agent\": \"%REQ(USER-AGENT)%\",\n \"x-envoy-origin-path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n \"x-envoy-upstream-service-time\": \"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\n \"x-forwarded-for\": \"%REQ(X-FORWARDED-FOR)%\",\n \"x-request-id\": \"%REQ(X-REQUEST-ID)%\"\n }\n },\n \"path\": \"/dev/stdout\"\n }\n }\n ],\n \"address\": {\n \"socketAddress\": {\n \"address\": \"0.0.0.0\",\n \"portValue\": 10080\n }\n },\n \"defaultFilterChain\": {\n \"filters\": [\n {\n \"name\": \"envoy.filters.network.http_connection_manager\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\",\n \"accessLog\": [\n {\n \"name\": \"envoy.access_loggers.file\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\",\n \"logFormat\": {\n \"jsonFormat\": {\n \":authority\": \"%REQ(:AUTHORITY)%\",\n \"bytes_received\": \"%BYTES_RECEIVED%\",\n \"bytes_sent\": \"%BYTES_SENT%\",\n \"connection_termination_details\": \"%CONNECTION_TERMINATION_DETAILS%\",\n \"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\n \"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\n \"duration\": \"%DURATION%\",\n \"method\": \"%REQ(:METHOD)%\",\n \"protocol\": \"%PROTOCOL%\",\n \"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\n \"response_code\": \"%RESPONSE_CODE%\",\n \"response_code_details\": \"%RESPONSE_CODE_DETAILS%\",\n \"response_flags\": \"%RESPONSE_FLAGS%\",\n \"route_name\": \"%ROUTE_NAME%\",\n \"start_time\": \"%START_TIME%\",\n \"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\n \"upstream_host\": \"%UPSTREAM_HOST%\",\n \"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\n \"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\n \"user-agent\": \"%REQ(USER-AGENT)%\",\n \"x-envoy-origin-path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\n \"x-envoy-upstream-service-time\": \"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\n \"x-forwarded-for\": \"%REQ(X-FORWARDED-FOR)%\",\n \"x-request-id\": \"%REQ(X-REQUEST-ID)%\"\n }\n },\n \"path\": \"/dev/stdout\"\n }\n }\n ],\n \"commonHttpProtocolOptions\": {\n \"headersWithUnderscoresAction\": \"REJECT_REQUEST\"\n },\n \"http2ProtocolOptions\": {\n \"initialConnectionWindowSize\": 1048576,\n \"initialStreamWindowSize\": 65536,\n \"maxConcurrentStreams\": 100\n },\n \"httpFilters\": [\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"STREAMED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"STREAMED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"disabled\": true,\n \"name\": \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor\",\n \"grpcService\": {\n \"envoyGrpc\": {\n \"authority\": \"aibrix-gateway-plugins.aibrix-system:50052\",\n \"clusterName\": \"envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\"\n },\n \"timeout\": \"10s\"\n },\n \"messageTimeout\": \"600s\",\n \"processingMode\": {\n \"requestBodyMode\": \"BUFFERED\",\n \"requestHeaderMode\": \"SEND\",\n \"requestTrailerMode\": \"SKIP\",\n \"responseBodyMode\": \"BUFFERED\",\n \"responseHeaderMode\": \"SEND\",\n \"responseTrailerMode\": \"SKIP\"\n }\n }\n },\n {\n \"name\": \"envoy.filters.http.router\",\n \"typedConfig\": {\n \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\",\n \"suppressEnvoyHeaders\": true\n }\n }\n ],\n \"mergeSlashes\": true,\n \"normalizePath\": true,\n \"pathWithEscapedSlashesAction\": \"UNESCAPE_AND_REDIRECT\",\n \"rds\": {\n \"configSource\": {\n \"ads\": {},\n \"resourceApiVersion\": \"V3\"\n },\n \"routeConfigName\": \"aibrix-system/aibrix-eg/http\"\n },\n \"serverHeaderTransformation\": \"PASS_THROUGH\",\n \"statPrefix\": \"http-10080\",\n \"useRemoteAddress\": true\n }\n }\n ],\n \"name\": \"aibrix-system/aibrix-eg/http\"\n },\n \"name\": \"aibrix-system/aibrix-eg/http\",\n \"perConnectionBufferLimitBytes\": 4194304\n }\n }\n }\n ]\n },\n {\n \"@type\": \"type.googleapis.com/envoy.admin.v3.RoutesConfigDump\",\n \"dynamicRouteConfigs\": [\n {\n \"routeConfig\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.RouteConfiguration\",\n \"ignorePortInHostMatching\": true,\n \"name\": \"aibrix-system/aibrix-eg/http\",\n \"virtualHosts\": [\n {\n \"domains\": [\n \"*\"\n ],\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"Gateway\",\n \"name\": \"aibrix-eg\",\n \"namespace\": \"aibrix-system\",\n \"sectionName\": \"http\"\n }\n ]\n }\n }\n },\n \"name\": \"aibrix-system/aibrix-eg/http/*\",\n \"routes\": [\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"safeRegex\": {\n \"regex\": \"^/v1/videos/sync$|^/v1/videos/[^/]+/content$\"\n }\n },\n \"name\": \"original_route_videos_streaming\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"safeRegex\": {\n \"regex\": \"^/v1/videos(/[^/]+)?$\"\n }\n },\n \"name\": \"original_route_videos\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"headers\": [\n {\n \"name\": \"routing-strategy\",\n \"stringMatch\": {\n \"safeRegex\": {\n \"regex\": \".*\"\n }\n }\n }\n ],\n \"prefix\": \"/\"\n },\n \"name\": \"original_route\",\n \"route\": {\n \"cluster\": \"original_destination_cluster\",\n \"timeout\": \"600s\"\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"path\": \"/v1/videos/sync\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos-streaming\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"safeRegex\": {\n \"regex\": \"/v1/videos/[^/]+/content\"\n }\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos-streaming\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0/match/1/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos-streaming/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-streaming-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/transcriptions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/15/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/images/generations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/13/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/translations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/16/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/video/generations\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/19/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/chat/completions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/voices\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/18/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/images/edits\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/14/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/audio/speech\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/17/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/completions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/3/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/embeddings\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/4/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/systemone\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/9/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/decisions\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/8/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/responses\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/1/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/classify\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/6/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/messages\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/2/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"directResponse\": {\n \"status\": 500\n },\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/batches\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-metadata-endpoint\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-metadata-endpoint/rule/0/match/2/*\"\n },\n {\n \"directResponse\": {\n \"status\": 500\n },\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/models\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-metadata-endpoint\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-metadata-endpoint/rule/0/match/0/*\"\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/videos\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router-videos\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router-videos/rule/0/match/0/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router-videos/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-videos-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/v1/rerank\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/5/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy.filters.http.ext_proc/envoyextensionpolicy/aibrix-system/aibrix-gateway-plugins-extension-policy/extproc/0\": {\n \"@type\": \"type.googleapis.com/envoy.config.route.v3.FilterConfig\",\n \"config\": {}\n }\n }\n },\n {\n \"match\": {\n \"pathSeparatedPrefix\": \"/tokenize\"\n },\n \"metadata\": {\n \"filterMetadata\": {\n \"envoy-gateway\": {\n \"resources\": [\n {\n \"kind\": \"HTTPRoute\",\n \"name\": \"aibrix-reserved-router\",\n \"namespace\": \"aibrix-system\"\n }\n ]\n }\n }\n },\n \"name\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/11/*\",\n \"route\": {\n \"cluster\": \"httproute/aibrix-system/aibrix-reserved-router/rule/0\",\n \"upgradeConfigs\": [\n {\n \"upgradeType\": \"websocket\"\n }\n ]\n },\n \"typedPerFilterConfig\": {\n \"envoy\n\n\n# ratelimiter\n\nPackage `ratelimiter` provides a distributed, Redis-backed fixed-window rate limiter used by the Aibrix gateway to enforce per-user and per-model request limits across multiple gateway instances.\n\n## Interface\n\n```go\ntype RateLimiter interface {\n Get(ctx context.Context, key string) (int64, error)\n GetLimit(ctx context.Context, key string) (int64, error)\n Incr(ctx context.Context, key string, val int64, window ...time.Duration) (int64, error)\n}\n```\n\n| Method | Description |\n|------------|-------------|\n| `Get` | Returns the current counter value for a key in the active time window. |\n| `GetLimit` | Returns the configured maximum for a key (used for limit lookups, not window counters). |\n| `Incr` | Atomically increments the counter by `val` and returns the new value. The first write sets the key TTL to the window size; later writes do not extend it. |\n\n## Implementations\n\n### `redisRateLimiter` — `NewRedisAccountRateLimiter(name, client, windowSize)`\n\nA fixed-window counter backed by Redis. Suitable for cluster-wide enforcement when multiple gateway replicas share the same Redis instance.\n\n**Counter key scheme:** `{name}:{key}:{windowMilliseconds}:counter`. The key is stable for each configured window duration and separate from the static `{name}:{key}` key used by `GetLimit`. The counter expires `windowSize` after its first write, and the next write starts a new window. This window is independent of wall-clock bucket boundaries.\n\n- Minimum `windowSize` is 1 second (smaller values are clamped up).\n- `Incr` uses a Redis Lua script to increment and set the TTL atomically on the first write.\n- A missing key (`redis.Nil`) is treated as `0`, not an error.\n\n### `noopRateLimiter` — `NewNoopRateLimiter()`\n\nA no-op implementation that always allows requests. Used when Redis is unavailable (e.g., local development without a Redis sidecar). `GetLimit` returns `math.MaxInt64`.\n\n## Usage in the gateway\n\nTwo limiter instances are created at server startup:\n\n| Instance | Constructor arg `name` | `windowSize` | Purpose |\n|--------------------|------------------------|--------------|----------------------------------|\n| `ratelimiter` | `\"aibrix\"` | 1 minute | Per-user RPM and TPM enforcement |\n| `modelRateLimiter` | `\"aibrix_model\"` | 1 second | Per-model RPS enforcement |\n\nThe separate `name` prefix ensures the two limiters' keys never collide in Redis.\n\n### Per-user limits (RPM / TPM)\n\nCounter keys follow the pattern `aibrix:{username}_{RPM|TPM}_CURRENT:60000:counter`. Limits are read from the user record resolved during request header processing.\n\n### Per-model RPS\n\nCounter keys follow the pattern `aibrix_model:{modelName}_MODEL_RPS_CURRENT:{windowMilliseconds}:counter`. The RPS limit is configured per model via the `requestsPerSecond` field in a config profile:\n\n```json\n{\n \"profiles\": {\n \"rps-limited\": {\n \"routingStrategy\": \"least-request\",\n \"requestsPerSecond\": 1\n }\n }\n}\n```\n\nThe active profile is selected by passing the `config-profile` request header. If no profile is active or `requestsPerSecond` is unset, the RPS check is skipped entirely.\n\n#### Enforcement flow\n\nPer-model RPS enforcement is handled in `HandleRequestBody` via `enforceModelRPS` and a deferred compensation step:\n\n1. **Pre-routing gate (`enforceModelRPS`)** — called before routing. It:\n - atomically increments the counter with `Incr(..., +1)` and receives the new value\n - rejects with HTTP 429 and rolls back with `Incr(..., -1)` if `newVal > limit`\n - allows the request through when `newVal <= limit`\n2. **Deferred compensation (`decrModelRPS`)** — armed immediately after a successful pre-charge. If routing later fails, the deferred call refunds quota with `Incr(..., -1)`.\n3. **Success path** — after routing succeeds and request accounting is attached, compensation is disabled, so the pre-charge remains counted.\n\nUsing incr-then-check (rather than check-then-increment) means the `INCRBY` is the sole gate. Because Redis processes `INCRBY` atomically, each concurrent caller receives a unique sequential result — eliminating the TOCTOU race that would otherwise allow over-admission during the check→increment window.\n\n### Upgrading from wall-clock bucket keys\n\nThe counter key format changed from `{name}:{key}:{timebin}` to `{name}:{key}:{windowMilliseconds}:counter`. Existing bucket keys expire on their own TTL, but the new limiter does not read their counts. A deployment therefore starts a new counter window for each key. During a rolling upgrade, old and new gateway instances count requests separately and may collectively admit more than the configured limit. Deployments that require uninterrupted strict limits should avoid serving rate-limited traffic from both versions at the same time; a cutover to the new version still starts fresh counters.\n\n\n# statesync\n\nGeneric Redis-backed state sync for multiple gateway (or other) replicas. Uses **per-entity keys** with `SETEX` (per-record TTL), `MGET` (batched), and `SCAN` (enumeration). Keys are namespaced via a hash tag (`aibrix:{namespace}:e:{id}`). No PUB/SUB.\n\n- **Periodic sync**: Each replica runs **Pull first** (load from Redis), then **Push** (delta or full) every sync period.\n- **Optional write-through**: After local updates you may call `Put`/`Delete` directly. The default gateway wiring uses periodic delta push only.\n- **Deletions/tombstones**: If a Syncable implements `TombstoneSupport`, deletions are written as tombstone payloads so peers can remove the local entry on the next pull; otherwise `DEL` is used.\n- **Jitter and backoff**: Random initial startup delay (up to half `syncPeriod`); per-cycle jitter is roughly ±10% of `syncPeriod` (see `syncLoop` in `redissync.go`); exponential backoff (capped at 1 min) on errors.\n- **Lifecycle**: Register all Syncables **before** `Start()`; registrations after `Start()` are silently ignored.\n\n**Per-record TTL:** Each entity is stored as its own key with `SETEX` and a default TTL of 2 minutes. A record expires if it is not rewritten within that window. The TTL of one record does not affect others.\n\nRedis key format: `aibrix:{namespace}:e:{entityId}` — `{namespace}` is the hash tag. Namespace must be non-empty.\n\n### Prefix cache (gateway default wiring)\n\nWith `AIBRIX_STATESYNC_ENABLED=true`, `cmd/plugins/main.go`:\n\n1. Calls `table.EnableDeltaSync()` to activate dirty tracking on `PrefixHashTable` (no-op cost when disabled).\n2. Registers `PrefixHashTableSyncable` with the manager.\n3. Relies on **periodic delta push + pull** — `Put` is not called on every `AddPrefix`.\n\nThis gives eventual consistency with staleness bounded by the sync interval and per-key TTL. For faster cross-replica visibility you can call `syncManager.Put(ctx, \"prefixcache\", blockIDStr, data)` on the `*statesync.RedisSync` using bytes from `EncodeBlockForSync` after each `AddPrefix` (higher Redis write load trade-off).\n\n### Delta sync and LRU eviction\n\n`EnableDeltaSync()` must be called on `PrefixHashTable` before it is registered with `*statesync.RedisSync`; without it, dirty tracking is inactive and `GetDeltaForSync` always returns an empty delta. If a block is marked dirty then **evicted locally** before the next successful push, `GetDeltaForSync` skips it silently. The corresponding Redis key remains until the per-entity TTL expires. Strong delete propagation is not guaranteed for evicted entries.\n\n## Diagrams\n\n### Technical Flow\n\n```mermaid\nsequenceDiagram\n participant SyncLoop\n participant Redis\n participant Syncable\n Note over SyncLoop: Pull first, then Push every syncPeriod\n SyncLoop->>Redis: SCAN pattern (keys for namespace)\n Redis-->>SyncLoop: keys[]\n SyncLoop->>Redis: MGET keys (batched)\n Redis-->>SyncLoop: values[]\n SyncLoop->>Syncable: ApplyRemote(id, bytes) for each\n SyncLoop->>Syncable: GetDelta or GetSnapshot\n Syncable-->>SyncLoop: updated, deleted\n SyncLoop->>Redis: SETEX key val TTL per entity, DEL deleted\n SyncLoop->>Syncable: OptionalHooks (if implemented)\n```\n\n---\n\n## Adopting statesync in a new component\n\nChanges live **only in the package that owns the state**. The `statesync` package stays generic and never imports component types.\n\n### 1. Implement `syncable.Syncable`\n\nDepend on `github.com/vllm-project/aibrix/pkg/utils/syncable` and implement:\n\n| Method | Purpose |\n|--------|---------|\n| `Namespace() string` | Stable name for this state (e.g. `\"mytracker\"`). Used in the Redis key. |\n| `GetSnapshot(ctx) (map[string][]byte, error)` | Return current local state as **entity id → serialized bytes**. |\n| `ApplyRemote(ctx, id string, data []byte) error` | Apply one entity from Redis into local state (merge or overwrite). |\n\nOptional interfaces:\n\n| Interface | Purpose |\n|-----------|---------|\n| `DeltaSyncable` | Add `GetDelta` / `ClearDirty` to push only changed entities instead of the full snapshot. |\n| `OptionalHooks` | `OnSyncStart` / `OnSyncEnd` called around each **Pull** for that Syncable (not around Push). |\n| `StalePolicy` | `IsStale` — if true, the sync layer deletes the remote key and skips applying. |\n| `TombstoneSupport` | `MakeTombstone` / `IsTombstone` / `DeleteLocal` — propagate deletes as payloads instead of DEL. |\n\n### 2. Add serialization\n\n- **Encode**: turn your struct into `[]byte` (e.g. JSON). Use a stable format all replicas can decode.\n- **Decode**: in `ApplyRemote`, unmarshal `data` and merge into or replace local state.\n\n### 3. Expose a constructor returning `syncable.Syncable`\n\nReturn an adapter that holds a pointer to your state and implements the interface. Callers pass it to `(*statesync.RedisSync).Register(...)`.\n\n### 4. Wire in the process entry point\n\n```go\nrs := statesync.New(redisClient,\n statesync.WithSyncPeriod(30*time.Second), // optional; default 10s\n statesync.WithOpTimeout(15*time.Second), // optional; default 30s\n statesync.WithKeyPrefix(\"myapp\"), // optional; default \"aibrix\"\n)\n// Register all Syncables before Start.\nrs.Register(mypkg.NewMyStateSyncable(myState))\nrs.Start()\n\n// On shutdown:\nrs.Stop() // blocks until the sync loop exits or stop-wait timeout (default 90s)\n```\n\nAvailable `statesync.New` options:\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `WithSyncPeriod(d)` | 10s | Interval between sync cycles (jitter applied). |\n| `WithOpTimeout(d)` | 30s | Per-operation context timeout for Pull/Push. |\n| `WithKeyPrefix(s)` | `\"aibrix\"` | Prefix for all Redis keys. |\n| `WithRecordTTL(d)` | 2m | Per-entity `SETEX` TTL; values ≤0 are ignored. |\n| `WithStopWaitTimeout(d)` | 90s | Max time `Stop()` waits for the background loop to exit. |\n| `WithSetexChunkSize(n)` | 25 | Pipeline chunk size for SETEX writes. |\n| `WithMGetBatchSize(n)` | 200 | Batch size for MGET during Pull. |\n\n### Redis client\n\nThe Redis client is provided by the caller (e.g. `utils.GetRedisClient()`). Configure TLS and auth in the client options for production (see `pkg/utils/redis.go`).\n\n---\n\n## Example: a new component \"TokenTracker\"\n\nA component that keeps `tokenID → lastUsedTime` and should sync across gateway replicas.\n\n### Step 1 – Implement Syncable in the component package\n\n```go\n// pkg/plugins/gateway/algorithms/vtc/token_tracker_sync.go\n\nimport (\n \"context\"\n \"encoding/json\"\n \"strconv\"\n \"time\"\n\n \"github.com/vllm-project/aibrix/pkg/utils/syncable\"\n)\n\nconst tokenTrackerNamespace = \"token_tracker\"\n\n// TokenTrackerSyncable adapts TokenTracker for statesync.\ntype TokenTrackerSyncable struct {\n Tracker *TokenTracker\n}\n\nfunc NewTokenTrackerSyncable(t *TokenTracker) syncable.Syncable {\n return &TokenTrackerSyncable{Tracker: t}\n}\n\nfunc (s *TokenTrackerSyncable) Namespace() string { return tokenTrackerNamespace }\n\nfunc (s *TokenTrackerSyncable) GetSnapshot(ctx context.Context) (map[string][]byte, error) {\n s.Tracker.mu.RLock()\n defer s.Tracker.mu.RUnlock()\n out := make(map[string][]byte, len(s.Tracker.entries))\n for id, t := range s.Tracker.entries {\n b, _ := json.Marshal(t.UnixNano())\n out[strconv.FormatInt(id, 10)] = b\n }\n return out, nil\n}\n\nfunc (s *TokenTrackerSyncable) ApplyRemote(ctx context.Context, id string, data []byte) error {\n var nano int64\n if err := json.Unmarshal(data, &nano); err != nil {\n return err\n }\n parsed, _ := strconv.ParseInt(id, 10, 64)\n remote := time.Unix(0, nano)\n s.Tracker.mu.Lock()\n defer s.Tracker.mu.Unlock()\n if existing, ok := s.Tracker.entries[parsed]; !ok || remote.After(existing) {\n s.Tracker.entries[parsed] = remote\n }\n return nil\n}\n```\n\n### Step 2 – Wire in the entry point\n\n```go\nrs := statesync.New(redisClient, statesync.WithSyncPeriod(30*time.Second))\nrs.Register(prefixcacheindexer.NewPrefixHashTableSyncable(prefixHashTable))\nrs.Register(vtc.NewTokenTrackerSyncable(tokenTracker))\nrs.Start()\ndefer rs.Stop()\n```\n\n### Step 3 – Optional write-through\n\nIf you want other replicas to see a change before the next periodic push:\n\n```go\nfunc (t *TokenTracker) RecordUse(ctx context.Context, id int64, rs *statesync.RedisSync) {\n t.mu.Lock()\n t.entries[id] = time.Now()\n t.mu.Unlock()\n if rs != nil {\n data, _ := json.Marshal(time.Now().UnixNano())\n _ = rs.Put(ctx, tokenTrackerNamespace, strconv.FormatInt(id, 10), data)\n }\n}\n```\n\n---\n\n## Reference implementation\n\nSee `pkg/utils/prefixcacheindexer/sync.go`:\n- `PrefixHashTableSyncable` and `NewPrefixHashTableSyncable(table)` — full `DeltaSyncable` implementation.\n- `GetSnapshotForSync` / `GetDeltaForSync` / `ClearDirtyForSync` / `ApplyRemoteForSync` / `EncodeBlockForSync` on `PrefixHashTable`.\n- Remember to call `table.EnableDeltaSync()` before registering if you want delta push (not required for snapshot-only sync).\n"}