## 1. Project Overview & Quickstart (vllm-project/aibrix)

## File: README.md

# AIBrix

Welcome to AIBrix, an open-source initiative designed to provide essential building blocks to construct scalable GenAI inference infrastructure. AIBrix delivers a cloud-native solution optimized for deploying, managing, and scaling large language model (LLM) inference, tailored specifically to enterprise needs.




## Latest News

### Releases
- **[2026-06-16]** AIBrix v0.7.0 is released. Check out the [release notes](https://github.com/vllm-project/aibrix/releases/tag/v0.7.0) and [Blog Post](https://aibrix.github.io/posts/2026-06-16-v0.7.0-release/) for more details.
- **[2026-03-05]** AIBrix v0.6.0 is released. Check out the [release notes](https://github.com/vllm-project/aibrix/releases/tag/v0.6.0) and [Blog Post](https://aibrix.github.io/posts/2026-03-03-v0.6.0-release/) for more details.
- **[2025-11-10]** AIBrix v0.5.0 is released. Check out the [release notes](https://github.com/vllm-project/aibrix/releases/tag/v0.5.0) and [Blog Post](https://aibrix.github.io/posts/2025-11-10-v0.5.0-release/) for more details.
- **[2025-08-05]** AIBrix v0.4.0 is released. Check out the [release notes](https://github.com/vllm-project/aibrix/releases/tag/v0.4.0) and [Blog Post](https://aibrix.github.io/posts/2025-08-04-v0.4.0-release/) for more details.
- **[2025-05-21]** AIBrix v0.3.0 is released. Check out the [release notes](https://github.com/vllm-project/aibrix/releases/tag/v0.3.0) and [Blog Post](https://aibrix.github.io/posts/2025-05-21-v0.3.0-release/) for more details.
- **[2025-03-09]** AIBrix v0.2.1 is released. DeepSeek-R1 full weights deployment is supported and gateway stability has been improved! Check [Blog Post](https://aibrix.github.io/posts/2025-03-10-deepseek-r1/) for more details.
- **[2025-02-19]** AIBrix v0.2.0 is released. Check out the [release notes](https://github.com/vllm-project/aibrix/releases/tag/v0.2.0) and [Blog Post](https://aibrix.github.io/posts/2025-02-05-v0.2.0-release/) for more details.
- **[2024-11-13]** AIBrix v0.1.0 is released. Check out the [release notes](https://github.com/vllm-project/aibrix/releases/tag/v0.1.0) and [Blog Post](https://aibrix.github.io/posts/2024-11-12-v0.1.0-release/) for more details.

### Talks and Presentations

- **[2025-11-12]** AIBrix team co-delivered a keynote at KubeCon North America 2025 [AIBrix: Kubernetes-native GenAI Inference Infrastructure](https://www.youtube.com/watch?v=7KHenRXNGAw&t=875s), providing AIBrix overview.
- **[2025-06-10]** AIBrix team delivered a talk at KubeCon China 2025 titled [AIBrix: Cost-Effective and Scalable Kubernetes Control Plane for vLLM](https://kccncchn2025.sched.com/event/1x5im/introducing-aibrix-cost-effective-and-scalable-kubernetes-control-plane-for-vllm-jiaxin-shan-liguang-xie-bytedance), discussing how the framework optimizes vLLM deployment via Kubernetes for cost efficiency and scalability.
- **[2025-04-04]** AIBrix team co-delivered a keynote at KubeCon EU 2025 with Google on [LLM-Aware Load Balancing in Kubernetes: A New Era of Efficiency](https://kccnceu2025.sched.com/event/1txC7/keynote-llm-aware-load-balancing-in-kubernetes-a-new-era-of-efficiency-clayton-coleman-distinguished-engineer-google-jiaxin-shan-software-engineer-bytedance), focusing on LLM specific routing solutions.
- **[2025-03-30]** AIBrix was featured at the [ASPLOS'25](http://asplos-conference.org/asplos2025/) workshop with the presentation [AIBrix: An Open-Source, Large-Scale LLM Inference Infrastructure for System Research](https://docs.google.com/presentation/d/1YDVsPFTIgGXnROGaJ1VKuDDAB4T5fzpE/edit), showcasing its architecture for efficient LLM inference in system research scenarios.

## Key Features

The initial release includes the following key features:

- **High-Density LoRA Management**: Streamlined support for lightweight, low-rank adaptations of models.
- **LLM Gateway and Routing**: Efficiently manage and direct traffic across multiple models and replicas.
- **LLM App-Tailored Autoscaler**: Dynamically scale inference resources based on real-time demand.
- **Unified AI Runtime**: A versatile sidecar enabling metric standardization, model downloading, and management.
- **Distributed Inference**: Scalable architecture to handle large workloads across multiple nodes.
- **Distributed KV Cache**: Enables high-capacity, cross-engine KV reuse.
- **Cost-efficient Heterogeneous Serving**: Enables mixed GPU inference to reduce costs with SLO guarantees.
- **GPU Hardware Failure Detection**: Proactive detection of GPU hardware issues.

## Architecture




## Quick Start

To get started with AIBrix, clone this repository and follow the setup instructions in the documentation. Our comprehensive guide will help you configure and deploy your first LLM infrastructure seamlessly.

```shell
# Local Testing
git clone https://github.com/vllm-project/aibrix.git
cd aibrix

# Install nightly aibrix dependencies
kubectl apply -k config/dependency --server-side

# Install nightly AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -k config/crd --server-side

# Install nightly aibrix components
kubectl apply -k config/default
```

Install stable distribution
```shell
# Install component dependencies
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-dependency-v0.7.0.yaml" --server-side

# Install AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-crds-v0.7.0.yaml" --server-side

# Install aibrix components
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-v0.7.0.yaml"
```

## Documentation

For detailed documentation on installation, configuration, and usage, please visit our [documentation page](https://aibrix.readthedocs.io/latest/).

## Contributing

We welcome contributions from the community! Check out our [contributing guidelines](./CONTRIBUTING.md) to see how you can make a difference.

Slack Channel: [#aibrix](https://vllm-dev.slack.com/archives/C08EQ883CSV)

## License

AIBrix is licensed under the [Apache 2.0 License](LICENSE).

## Support

If you have any questions or encounter any issues, please submit an issue on our [GitHub issues page](https://github.com/vllm-project/aibrix/issues).

Thank you for choosing AIBrix for your GenAI infrastructure needs!

---

## File: benchmarks/scenarios/lora/README.md

# Lora benchmark


## Prerequisite

### 1. Install dependencies

```
apt update && apt install -y vim wget zip jq
pip3 install huggingface_hub transformers openai peft vllm
```

### 1. Download Models

Public Cloud

```
import os
from huggingface_hub import snapshot_download

os.environ['HF_TOKEN']="hf_xxxxxxxx"

base_model_path = snapshot_download(repo_id="meta-llama/Llama-2-7b-hf", ignore_patterns=[ "*.bin"])
sql_lora_path = snapshot_download(repo_id="Fredh99/llama-2-7b-sft-lora", local_dir="/models/Fredh99/llama-2-7b-sft-lora")
```

Volcano Engine

```
TODO: Add Toscli tutorials
```

### 2. Merge Models

```
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig

def merge_base_and_lora(base_model_name_or_path, lora_model_name_or_path, output_path):
    # Load the base model and tokenizer
    print("Loading base model...")
    base_model = AutoModelForCausalLM.from_pretrained(base_model_name_or_path)
    tokenizer = AutoTokenizer.from_pretrained(base_model_name_or_path)
    
    # Load the LoRA weights
    print("Loading LoRA model...")
    peft_config = PeftConfig.from_pretrained(lora_model_name_or_path)
    lora_model = PeftModel.from_pretrained(base_model, lora_model_name_or_path)
    
    # Merge LoRA weights into the base model
    print("Merging LoRA weights into the base model...")
    lora_model = lora_model.merge_and_unload()
    
    # Save the merged model
    print(f"Saving the merged model to {output_path}...")
    lora_model.save_pretrained(output_path)
    tokenizer.save_pretrained(output_path)

    print(f"Merged model saved successfully at {output_path}!")

merge_base_and_lora('meta-llama/Llama-2-7b-hf', '/models/Fredh99/llama-2-7b-sft-lora', '/models/merge-model-weights')
```

### 3. Download datasets

```
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
```

### 4. Download chat template

```
wget https://raw.githubusercontent.com/chujiezheng/chat_templates/refs/heads/main/chat_templates/llama-2-chat.jinja
```

## Deployments

### Deployment 1

Merged Lora weights

```
CUDA_VISIBLE_DEVICES=0 vllm serve /models/merge-model-weights --served-model-name model-1 --host "0.0.0.0" --port "8071" --chat-template llama-2-chat.jinja
```

### Deployment 2

Unmerged Lora weights but deployed separately.

```
CUDA_VISIBLE_DEVICES=0 vllm serve meta-llama/Llama-2-7b-hf --host "0.0.0.0" --port "8071" --enable-lora --lora-modules model-1=/models/Fredh99/llama-2-7b-sft-lora --chat-template llama-2-chat.jinja --max-lora-rank=64
```

### Deployment 3

Multi-Lora deployments

> Note: we just deploy 32 models for easy management. add one more argument `--model` to control it.

```
CUDA_VISIBLE_DEVICES=0 vllm serve meta-llama/Llama-2-7b-hf --host "0.0.0.0" --port "8070" --enable-lora --lora-modules \
model-1=/models/Fredh99/llama-2-7b-sft-lora \
model-2=/models/Fredh99/llama-2-7b-sft-lora \
model-3=/models/Fredh99/llama-2-7b-sft-lora \
model-4=/models/Fredh99/llama-2-7b-sft-lora \
model-5=/models/Fredh99/llama-2-7b-sft-lora \
model-6=/models/Fredh99/llama-2-7b-sft-lora \
model-7=/models/Fredh99/llama-2-7b-sft-lora \
model-8=/models/Fredh99/llama-2-7b-sft-lora \
model-9=/models/Fredh99/llama-2-7b-sft-lora \
model-10=/models/Fredh99/llama-2-7b-sft-lora \
model-11=/models/Fredh99/llama-2-7b-sft-lora \
model-12=/models/Fredh99/llama-2-7b-sft-lora \
model-13=/models/Fredh99/llama-2-7b-sft-lora \
model-14=/models/Fredh99/llama-2-7b-sft-lora \
model-15=/models/Fredh99/llama-2-7b-sft-lora \
model-16=/models/Fredh99/llama-2-7b-sft-lora \
model-17=/models/Fredh99/llama-2-7b-sft-lora \
model-18=/models/Fredh99/llama-2-7b-sft-lora \
model-19=/models/Fredh99/llama-2-7b-sft-lora \
model-20=/models/Fredh99/llama-2-7b-sft-lora \
model-21=/models/Fredh99/llama-2-7b-sft-lora \
model-22=/models/Fredh99/llama-2-7b-sft-lora \
model-23=/models/Fredh99/llama-2-7b-sft-lora \
model-24=/models/Fredh99/llama-2-7b-sft-lora \
model-25=/models/Fredh99/llama-2-7b-sft-lora \
model-26=/models/Fredh99/llama-2-7b-sft-lora \
model-27=/models/Fredh99/llama-2-7b-sft-lora \
model-28=/models/Fredh99/llama-2-7b-sft-lora \
model-29=/models/Fredh99/llama-2-7b-sft-lora \
model-30=/models/Fredh99/llama-2-7b-sft-lora \
model-31=/models/Fredh99/llama-2-7b-sft-lora \
model-32=/models/Fredh99/llama-2-7b-sft-lora \
--chat-template llama-2-chat.jinja --max-lora-rank=64 --max-loras 32
```

> Note: by default, `--max-loras` is equal to number of loras, feel free to change it for different experiments. 

## Experiment 1 - Pin lora size 4, run difference concurrency.

deployment1
```
#!/bin/bash
for concurrency in {1..32}; do
    output_file="benchmark_merged.jsonl"
    echo "Running benchmark with concurrency ${concurrency} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency ${concurrency} \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment1"
done
```

deployment2
```
#!/bin/bash
for concurrency in {1..32}; do
    output_file="benchmark_unmerged_single_lora.jsonl"
    echo "Running benchmark with concurrency ${concurrency} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency ${concurrency} \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment2"
done
```

deployment3
```
#!/bin/bash
for concurrency in {1..32}; do
    output_file="benchmark_unmerged_multi_lora_4.jsonl"
    echo "Running benchmark with concurrency ${concurrency} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency ${concurrency} \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment3"
done
```

## Experiment 2 - Pin lora size 8, run difference concurrency.

There's no difference between deployment1 and deployment2, we can skip the tests.
> Note: we can play with server side `max-loras` and do different experiments here.

```
#!/bin/bash
for concurrency in {1..32}; do
    output_file="benchmark_unmerged_multi_lora_8_max_loras_8.jsonl"
    echo "Running benchmark with concurrency ${concurrency} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency ${concurrency} \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment3" \
        --models 8
done
```

## Experiment 3 - Pin concurrency size 1, run difference loras

```
#!/bin/bash
for model_number in {1..32}; do
    output_file="benchmark_unmerged_multi_lora_${model_number}_concurrency_1.jsonl"
    echo "Running benchmark with model number ${model_number} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency 1 \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment3" \
        --models ${model_number}
done
```

> Note: Find results in deployment1 and deployment2 concurrency=1 cases.

---

## File: benchmarks/scenarios/gateway/README.md

## Gateway Routing benchmark

## Prerequisite

### Test Dataset

```bash
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/blob/main/ShareGPT_V3_unfiltered_cleaned_split.json
```

### Client - Curl

```bash
curl -v http://localhost:8888/v1/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer sk-any-key" \
    -d '{
        "model": "deepseek-coder-7b-instruct",
        "messages": [{"role": "user", "content": "Say this is a test!"}],
        "max_tokens": 128
    }'
```

### Client - Locust

```
locust -f benchmark.py --host http://localhost:8887
```

## Experiments

experiment 1 & 2 should use exact same client setting and they are comparable.

### Experiment 1: gateway overhead (httpRoute) vs k8s service (baseline)

```bash
kubectl -n envoy-gateway-system port-forward service/envoy-aibrix-system-aibrix-eg-903790dc 8888:80
kubectl port-forward svc/deepseek-coder-7b-instruct 8887:8000 -n aibrix-system
```

> Note: we can not use port-forward in > 1 pod testing, all the traffic will go into one pod.
> Change model service and gateway service to LoadBalancer for real testing.

### Experiment 2: Three Routing Strategies

wait until cache ready, manually send some request to activate model.

```bash
> Note: this is for local testing, feel free to change to Elastic IP later.

# service port-forwarding
OUTPUT_FILE=k8s-service.jsonl locust -f benchmark.py --host http://localhost:8887 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_httproute.csv --csv-full-history --logfile benchmark_gateway_httproute.log

# gateway port-forwarding
OUTPUT_FILE=http-route.jsonl locust -f benchmark.py --host http://localhost:8888 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_httproute.csv --csv-full-history --logfile benchmark_gateway_httproute.log

OUTPUT_FILE=random.jsonl ROUTING_STRATEGY=random locust -f benchmark.py --host http://localhost:8888 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_random.csv --csv-full-history --logfile benchmark_gateway_random.log

OUTPUT_FILE=least-request.jsonl ROUTING_STRATEGY=least-request locust -f benchmark.py --host http://localhost:8888 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_least_request.csv --csv-full-history --logfile benchmark_gateway_least_request.log

OUTPUT_FILE=throughput.jsonl ROUTING_STRATEGY=throughput locust -f benchmark.py --host http://localhost:8888 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_throughput.csv --csv-full-history --logfile benchmark_gateway_throughput.log
```

## Local Testing

```bash
make docker-build-gateway-plugins
aibrix/gateway-plugins:9bd45a9915b71936ff0001a6fbfc32f10b65e480

k edit deployment aibrix-gateway-plugins

k delete pod aibrix-gateway-plugins-759b87dc65-j9qs8 # commit is exact same, we just need to update once
```

```bash
curl  http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer any_key" \
  -d '{
     "model": "llama2-70b",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'
```

> Note: We do not need model or routing strategy in the header now. this is clean and sdk compatibile.


## New Client Testing

```bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "k8s-v2.jsonl"
```

```bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "httproute-v2.jsonl"
```

update env
```bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "random-v2.jsonl"
```

```bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "least-request-v2.jsonl"
```

```bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "throughput-v2.jsonl"
```

---

## File: benchmarks/scenarios/autoscaling/README.md

# How to run autoscaling experiment


## TLDR;
You can run autoscaling benchmark experiment by simply running the command below.
`./overnight_run.sh workload/workload/25min_up_and_down/25min_up_and_down.jsonl`


What you have to check before running it
- run deployment for your application (refer to `deepseek-llm-7b-chat/deploy.yaml`)
- change the name field under scaleTargetRed in all autoscaling yaml files.
- check the deployment name in run-test.py

For example,
```yaml
...
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deepseek-llm-7b-chat (*this one)
...
```

## Workflow
`overnight_run.sh` will execute run-test.sh with given autoscaling mechanism.

`run-test.sh` will execute 
1. Preparation
    - clean up existing autoscaler resources
    - set the number of replicas to 1 of the target deployment
    - restart aibrix-controller-manager, aibrix-gateway-plugins, target application's deployment
2. Ready state check
    - Once all the deployments are in ready state, it will start experiment
3. Start logging running as background processes
    - Log application pods, aibrix-controller-manager, aibrix-gateway-plugins by streaming the output to files (it will be used for debugging, post analysis)
4. Run client with appropriate arguments (client code is in `aibrix/benchmark/generator/client.py`)
5. Stop logging processes once experiment is done
6. clean up resources
    - clean up autocaling resources
    - set the number of replicas for the application deployment back to 1.

## How to plot graphs
There are two plots that you can plot.

### Generating report
`python <aibrix_root_repo>/benchmarks/plot/plot-everything.py <experiment_home_dir>`

For example, 
`python <aibrix_root_repo>/benchmarks/plot/plot-everything.py experiment_results/25min_test`

The directories should look like
```bash
ls experiment_results/25min_test

25min_up_and_down-apa-least-request-20250209-064742
25min_up_and_down-hpa-least-request-20250209-055214
25min_up_and_down-kpa-least-request-20250209-061957
25min_up_and_down-optimizer-kpa-least-request-20250209-071531
```

### Generating per pod graph
```bash
python <aibrix_root_repo>/benchmarks/plot/plot_per_pod.py <experiment_pod_logs_dir>
```

For example,
```bash
python <aibrix_root_repo>/benchmarks/plot/plot_per_pod.py experiment_results/25min_test/25min_up_and_down-apa-least-request-20250209-064742/pod_logs
```
Move all experiment output dir under the same dir (e.g., 25min_test) to compare them in the report.

It will generate timeseries plot for the number of requests running in the pod, throughput, cache usage.


### Dependency
- Client: `aibrix/benchmark/generator/client.py` 
- Workload trace generator: `aibrix/benchmark/generator/workload_generator.py`. Please check aibrix/benchmark/generator/README.md for more information.
    - Workload generated using `synthetic` workload types with configurations specified under [this directory](workload-configs).

---

## File: benchmarks/client/README.md

## Test client locally

> **Note** All generator invocation should be done under the benchmark home (i.e., `/aibrix/benchmarks/`)

Starting vllm server:


```shell
export API_KEY=${API_KEY}
vllm serve --host 0.0.0.0 \
--port "8000" \
--model /root/models/deepseek-llm-7b-chat \
--trust-remote-code \
--max-model-len "4096" \
--api-key ${API_KEY} \
--enable-chunked-prefill
```

Using a sample workload (generated by [the workload generator](../generator/README.md)) in a client. Turn on `--streaming` to collect fine grained metrics such as `TTFT` and  `TPOT`:

```shell
export API_KEY=${API_KEY}
python -m client.client \
--workload-path "output/workload/constant/workload.jsonl" \
--endpoint "http://localhost:8888" \
--model llama-3-8b-instruct \
--api-key ${API_KEY} \
--output-token-limit 128 \
--streaming \
--output-file-path output.jsonl 
```
The output will be stored as a `.jsonl` file in `output.jsonl`

### Session affinity replay

For sessioned workloads (requests carrying `session_id`), pass `--session-key-header` to send each request's `session_id` as the `x-aibrix-session-key` header (see `HeaderSessionKey` in `pkg/plugins/gateway/types.go`). The gateway only consumes this header when the request is routed with the `session-affinity` strategy, so pair the flag with `--routing-strategy session-affinity` (or `routing_strategy: session-affinity` in the `benchmark.py` config) — with any other strategy the header is sent but ignored. Note the gateway caps session keys at 256 bytes; longer keys are dropped and the request silently falls back to load-balanced routing (the client logs a warning per over-long session). The flag is off by default and has no effect on requests without a `session_id`. When driving the client through `benchmark.py`, set `client_session_key_header: true` in the config instead.

Run analysis on metrics collected. For streaming client, we can specify a goodput target (e2e/tpot/ttft) like the following: 

```shell
python -m client.analyze --trace output.jsonl --output output --goodput-target tpot:0.5
```

By default, client treats timestamp in the workload as timestamp in milliseconds (ms). The timestamp described in the workload generator could be scaled with `--time-scale`, Say that the original workload demands one paralell disptach at time `0, 1000, 2000...`, meaning that the requests will be dispatch every second. You could use `--time-scale` to scale the intervals of timestamps described in the workload, e.g. `--time-scale 0.1` scale the intervals of timetsamps by `0.1` meaning that the requests will be dispatched at time `0, 100, 200...`.

---

## File: config/dependency/kuberay-operator/README.md

## KubeRay upstream manifests

Commands to export manifest from helm package. After you got manifest, copy to this folder.

```shell
helm template kuberay-operator kuberay/kuberay-operator --namespace aibrix-system --version 1.2.1 --include-crds --set env[0].name=ENABLE_PROBES_INJECTION --set-string env[0].value=false --set fullnameOverride=kuberay-operator --set featureGates[0].name=RayClusterStatusConditions --set featureGates[0].enabled=true --output-dir ./config/dependency
```

If you use zsh, please use `noglob helm ...` to skip the brace check.

---

## File: deployment/terraform/gcp/README.md

# AIBrix GCP Terraform Module
This module deploys an AIBrix cluster in its entirety onto a Google Container Cluster. It is the quickest way to get up and running with AIBrix. The purpose of this module is to both allow developers to quickly spin up the stack, and allow for the team to test on the GCP cloud environment.

**NOTE: This module was created to allow users to quickly spin up AIBrix on GCP. It is not currently built for production deployments. The user is is responsible for any costs incurred by running this module.**

## Quickstart

### Prerequisites
- [GCloud CLI](https://cloud.google.com/sdk/docs/install)
- A quota of at least 1 GPU within your GCP project. More information can be found on the topic [here](https://cloud.google.com/compute/resource-usage#gpu_quota).
- [Terraform CLI](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)

### Quickstart
1. Run `gcloud auth application-default login` to setup credentials to Google.
2. Install cluster auth plugin with `gcloud components install gke-gcloud-auth-plugin`.
3. Rename `terraform.tfvars.example` to `terraform.tfvars` and fill in the required variables. You can also add any optional overrides here as well.
4. Run `terraform init` to initialize the module.
5. Run `terraform plan` to see details on the resources created by this module.
6. When you are satisfied with the plan and want to create the resources, run `terraform apply`. NOTE: if you recieve `NodePool aibrix-gpu-nodes was created in the error state "ERROR"` while running the script, check your quotas for GPUs and the specific instances you're trying to deploy.
7. Wait for module to complete running. It will output a command to receive the kubernetes config file and a public IP address.
8. Run a command against the public IP:
```bash
ENDPOINT="<YOUR PUBLIC IP>"

curl http://${ENDPOINT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek-r1-distill-llama-8b",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "help me write a random generator in python"}
        ]
    }'
```

9. When you are finished testing and no longer want the resources, run `terraform destroy`. **Ensure that you complete this step once you are done trying it out, as GPUs are expensive.**

## Testing
Testing is done completely end to end, no resources need to be initialized beforehand. The testing script will spin up the entire stack, run its tests with the generic OpenAI client against the created resources, and then will destroy everything it has created. Tests are modeled after the E2E tests found in `test/e2e/e2e_test.go`.

1. Ensure you have required Prerequisites from Quickstart above.
2. Complete steps 1-3 in Quickstart steps above. Only fill in the rquired variables within `terraform.tfvars`.
3. Change directory to `/deployment/terraform` and run `go test -v -timeout 60m tests/gcp_test.go`. NOTE: this test takes a while to run as it's spinning up its own kubernetes cluster.
4. If the test times out, to ensure resource deletion, simply run the test again.
## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_aibrix_release_version"></a> [aibrix\_release\_version](#input\_aibrix\_release\_version) | The version of AIBRix to deploy. | `string` | `"v0.2.0"` | no |
| <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | Name of the GKE cluster. | `string` | `"aibrix-inference-cluster"` | no |
| <a name="input_cluster_zone"></a> [cluster\_zone](#input\_cluster\_zone) | Zone to deploy cluster within. If not provided will be deployed to default region. | `string` | `""` | no |
| <a name="input_default_region"></a> [default\_region](#input\_default\_region) | Default region to deploy resources within. | `string` | n/a | yes |
| <a name="input_deploy_example_model"></a> [deploy\_example\_model](#input\_deploy\_example\_model) | Whether to deploy the example model. | `bool` | `true` | no |
| <a name="input_node_pool_machine_count"></a> [node\_pool\_machine\_count](#input\_node\_pool\_machine\_count) | Machine count for the node pool. | `number` | `1` | no |
| <a name="input_node_pool_machine_type"></a> [node\_pool\_machine\_type](#input\_node\_pool\_machine\_type) | Machine type for the node pool. Must be in the A3, A2, or G2 series. | `string` | `"g2-standard-4"` | no |
| <a name="input_node_pool_name"></a> [node\_pool\_name](#input\_node\_pool\_name) | Name of the GPU node pool. | `string` | `"aibrix-gpu-nodes"` | no |
| <a name="input_node_pool_zone"></a> [node\_pool\_zone](#input\_node\_pool\_zone) | Zone to deploy GPU node pool within. If not provided will be deployed to zone in default region which has capacity for machine type. | `string` | `""` | no |
| <a name="input_project_id"></a> [project\_id](#input\_project\_id) | GCP project to deploy resources within. | `string` | n/a | yes |
## Outputs

| Name | Description |
|------|-------------|
| <a name="output_aibrix_service_public_ip"></a> [aibrix\_service\_public\_ip](#output\_aibrix\_service\_public\_ip) | Public IP address for AIBrix service. |
| <a name="output_configure_kubectl_command"></a> [configure\_kubectl\_command](#output\_configure\_kubectl\_command) | Command to run which will allow kubectl access. |
## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_aibrix"></a> [aibrix](#module\_aibrix) | ../kubernetes | n/a |
| <a name="module_cluster"></a> [cluster](#module\_cluster) | ./cluster | n/a |
## Providers

| Name | Version |
|------|---------|
| <a name="provider_google"></a> [google](#provider\_google) | 6.22.0 |
| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | 2.36.0 |

## 2. Official Technical Reference & Guides (vllm-project/vllm-project.github.io)

# vLLM blog source

## Local development

1. Install `jekyll` and `bundler` by running `gem install jekyll bundler`. Ask ChatGPT for help if you encounter any issues.
2. `bundle install` to install the necessary gems.
3. `rm ./Gemfile.lock` if you meet gem error.
5. `bundle exec jekyll server` to compile and start the server.

To add a new blogpost, please refer to `_posts/2023-06-20-vllm.md` as an example. Some notes:
- Google Doc can be saved as markdown format which will make your life easier.
- Note that the blogpost will only show when its date is in the past.
- Add `summary` front matter for SEO. Keep it concise, specific, and 240 characters or fewer.
- Put figures under `assets/figures/yourblogname/`.
- Make a pull request.

The blog is automatically built and deployed by GitHub Actions when `main` is pushed to.

## LaTeX Math

The blog supports LaTeX math via [MathJax](https://docs.mathjax.org/en/latest/index.html). 

It can be enabled by adding `math: true` to the document frontmatter. It has been configured to support the standard LaTeX style math notation, i.e.:

```latex
$ inline math $
```

```latex
$$
math block
$$
```

## GitHub Flavored Admonitions

The blog supports GitHub flavored admonitions via [jekyll-gfm-admonition](https://github.com/Helveg/jekyll-gfm-admonitions). It supports the following syntax:

```markdown
> [!NOTE]
> Highlights information that users should take into account, even when skimming.
> And supports multi-line text.

> [!TIP]
> Optional information to help a user be more successful.

> [!IMPORTANT]
> Crucial information necessary for users to succeed.

> [!WARNING]
> Critical content demanding immediate
> user attention due to potential risks.

> [!CAUTION]
> Negative potential consequences of an action.
> Opportunity to provide more context.
```

## Theme customization

The theme we are using is [Minima](https://github.com/jekyll/minima). If you need to customise anything from this theme, see [Overriding theme defaults](https://jekyllrb.com/docs/themes/#overriding-theme-defaults).