aibrix Documentation

Cost-efficient and pluggable Infrastructure components for GenAI inference

4,790 stars Go Go 1 file · ~8,169 tokens
RAW Doc

1. Project Overview & Quickstart (vllm-project/aibrix)

File: README.md

AIBrix

Welcome to AIBrix, an open-source initiative designed to provide essential building blocks to construct scalable GenAI inference infrastructure. AIBrix delivers a cloud-native solution optimized for deploying, managing, and scaling large language model (LLM) inference, tailored specifically to enterprise needs.

Latest News

Releases

  • [2026-06-16] AIBrix v0.7.0 is released. Check out the release notes and Blog Post for more details.
  • [2026-03-05] AIBrix v0.6.0 is released. Check out the release notes and Blog Post for more details.
  • [2025-11-10] AIBrix v0.5.0 is released. Check out the release notes and Blog Post for more details.
  • [2025-08-05] AIBrix v0.4.0 is released. Check out the release notes and Blog Post for more details.
  • [2025-05-21] AIBrix v0.3.0 is released. Check out the release notes and Blog Post for more details.
  • [2025-03-09] AIBrix v0.2.1 is released. DeepSeek-R1 full weights deployment is supported and gateway stability has been improved! Check Blog Post for more details.
  • [2025-02-19] AIBrix v0.2.0 is released. Check out the release notes and Blog Post for more details.
  • [2024-11-13] AIBrix v0.1.0 is released. Check out the release notes and Blog Post for more details.

Talks and Presentations

Key Features

The initial release includes the following key features:

  • High-Density LoRA Management: Streamlined support for lightweight, low-rank adaptations of models.
  • LLM Gateway and Routing: Efficiently manage and direct traffic across multiple models and replicas.
  • LLM App-Tailored Autoscaler: Dynamically scale inference resources based on real-time demand.
  • Unified AI Runtime: A versatile sidecar enabling metric standardization, model downloading, and management.
  • Distributed Inference: Scalable architecture to handle large workloads across multiple nodes.
  • Distributed KV Cache: Enables high-capacity, cross-engine KV reuse.
  • Cost-efficient Heterogeneous Serving: Enables mixed GPU inference to reduce costs with SLO guarantees.
  • GPU Hardware Failure Detection: Proactive detection of GPU hardware issues.

Architecture

Quick Start

To get started with AIBrix, clone this repository and follow the setup instructions in the documentation. Our comprehensive guide will help you configure and deploy your first LLM infrastructure seamlessly.

shell
# Local Testing
git clone https://github.com/vllm-project/aibrix.git
cd aibrix

# Install nightly aibrix dependencies
kubectl apply -k config/dependency --server-side

# Install nightly AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -k config/crd --server-side

# Install nightly aibrix components
kubectl apply -k config/default

Install stable distribution

shell
# Install component dependencies
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-dependency-v0.7.0.yaml" --server-side

# Install AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-crds-v0.7.0.yaml" --server-side

# Install aibrix components
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-v0.7.0.yaml"

Documentation

For detailed documentation on installation, configuration, and usage, please visit our documentation page.

Contributing

We welcome contributions from the community! Check out our contributing guidelines to see how you can make a difference.

Slack Channel: #aibrix

License

AIBrix is licensed under the Apache 2.0 License.

Support

If you have any questions or encounter any issues, please submit an issue on our GitHub issues page.

Thank you for choosing AIBrix for your GenAI infrastructure needs!

File: benchmarks/scenarios/lora/README.md

Lora benchmark

Prerequisite

1. Install dependencies

text
apt update && apt install -y vim wget zip jq
pip3 install huggingface_hub transformers openai peft vllm

1. Download Models

Public Cloud

text
import os
from huggingface_hub import snapshot_download

os.environ['HF_TOKEN']="hf_xxxxxxxx"

base_model_path = snapshot_download(repo_id="meta-llama/Llama-2-7b-hf", ignore_patterns=[ "*.bin"])
sql_lora_path = snapshot_download(repo_id="Fredh99/llama-2-7b-sft-lora", local_dir="/models/Fredh99/llama-2-7b-sft-lora")

Volcano Engine

text
TODO: Add Toscli tutorials

2. Merge Models

text
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig

def merge_base_and_lora(base_model_name_or_path, lora_model_name_or_path, output_path):
    # Load the base model and tokenizer
    print("Loading base model...")
    base_model = AutoModelForCausalLM.from_pretrained(base_model_name_or_path)
    tokenizer = AutoTokenizer.from_pretrained(base_model_name_or_path)
    
    # Load the LoRA weights
    print("Loading LoRA model...")
    peft_config = PeftConfig.from_pretrained(lora_model_name_or_path)
    lora_model = PeftModel.from_pretrained(base_model, lora_model_name_or_path)
    
    # Merge LoRA weights into the base model
    print("Merging LoRA weights into the base model...")
    lora_model = lora_model.merge_and_unload()
    
    # Save the merged model
    print(f"Saving the merged model to {output_path}...")
    lora_model.save_pretrained(output_path)
    tokenizer.save_pretrained(output_path)

    print(f"Merged model saved successfully at {output_path}!")

merge_base_and_lora('meta-llama/Llama-2-7b-hf', '/models/Fredh99/llama-2-7b-sft-lora', '/models/merge-model-weights')

3. Download datasets

text
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

4. Download chat template

text
wget https://raw.githubusercontent.com/chujiezheng/chat_templates/refs/heads/main/chat_templates/llama-2-chat.jinja

Deployments

Deployment 1

Merged Lora weights

text
CUDA_VISIBLE_DEVICES=0 vllm serve /models/merge-model-weights --served-model-name model-1 --host "0.0.0.0" --port "8071" --chat-template llama-2-chat.jinja

Deployment 2

Unmerged Lora weights but deployed separately.

text
CUDA_VISIBLE_DEVICES=0 vllm serve meta-llama/Llama-2-7b-hf --host "0.0.0.0" --port "8071" --enable-lora --lora-modules model-1=/models/Fredh99/llama-2-7b-sft-lora --chat-template llama-2-chat.jinja --max-lora-rank=64

Deployment 3

Multi-Lora deployments

Note: we just deploy 32 models for easy management. add one more argument --model to control it.

text
CUDA_VISIBLE_DEVICES=0 vllm serve meta-llama/Llama-2-7b-hf --host "0.0.0.0" --port "8070" --enable-lora --lora-modules \
model-1=/models/Fredh99/llama-2-7b-sft-lora \
model-2=/models/Fredh99/llama-2-7b-sft-lora \
model-3=/models/Fredh99/llama-2-7b-sft-lora \
model-4=/models/Fredh99/llama-2-7b-sft-lora \
model-5=/models/Fredh99/llama-2-7b-sft-lora \
model-6=/models/Fredh99/llama-2-7b-sft-lora \
model-7=/models/Fredh99/llama-2-7b-sft-lora \
model-8=/models/Fredh99/llama-2-7b-sft-lora \
model-9=/models/Fredh99/llama-2-7b-sft-lora \
model-10=/models/Fredh99/llama-2-7b-sft-lora \
model-11=/models/Fredh99/llama-2-7b-sft-lora \
model-12=/models/Fredh99/llama-2-7b-sft-lora \
model-13=/models/Fredh99/llama-2-7b-sft-lora \
model-14=/models/Fredh99/llama-2-7b-sft-lora \
model-15=/models/Fredh99/llama-2-7b-sft-lora \
model-16=/models/Fredh99/llama-2-7b-sft-lora \
model-17=/models/Fredh99/llama-2-7b-sft-lora \
model-18=/models/Fredh99/llama-2-7b-sft-lora \
model-19=/models/Fredh99/llama-2-7b-sft-lora \
model-20=/models/Fredh99/llama-2-7b-sft-lora \
model-21=/models/Fredh99/llama-2-7b-sft-lora \
model-22=/models/Fredh99/llama-2-7b-sft-lora \
model-23=/models/Fredh99/llama-2-7b-sft-lora \
model-24=/models/Fredh99/llama-2-7b-sft-lora \
model-25=/models/Fredh99/llama-2-7b-sft-lora \
model-26=/models/Fredh99/llama-2-7b-sft-lora \
model-27=/models/Fredh99/llama-2-7b-sft-lora \
model-28=/models/Fredh99/llama-2-7b-sft-lora \
model-29=/models/Fredh99/llama-2-7b-sft-lora \
model-30=/models/Fredh99/llama-2-7b-sft-lora \
model-31=/models/Fredh99/llama-2-7b-sft-lora \
model-32=/models/Fredh99/llama-2-7b-sft-lora \
--chat-template llama-2-chat.jinja --max-lora-rank=64 --max-loras 32

Note: by default, --max-loras is equal to number of loras, feel free to change it for different experiments.

Experiment 1 - Pin lora size 4, run difference concurrency.

deployment1

text
#!/bin/bash
for concurrency in {1..32}; do
    output_file="benchmark_merged.jsonl"
    echo "Running benchmark with concurrency ${concurrency} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency ${concurrency} \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment1"
done

deployment2

text
#!/bin/bash
for concurrency in {1..32}; do
    output_file="benchmark_unmerged_single_lora.jsonl"
    echo "Running benchmark with concurrency ${concurrency} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency ${concurrency} \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment2"
done

deployment3

text
#!/bin/bash
for concurrency in {1..32}; do
    output_file="benchmark_unmerged_multi_lora_4.jsonl"
    echo "Running benchmark with concurrency ${concurrency} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency ${concurrency} \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment3"
done

Experiment 2 - Pin lora size 8, run difference concurrency.

There's no difference between deployment1 and deployment2, we can skip the tests.

Note: we can play with server side max-loras and do different experiments here.

text
#!/bin/bash
for concurrency in {1..32}; do
    output_file="benchmark_unmerged_multi_lora_8_max_loras_8.jsonl"
    echo "Running benchmark with concurrency ${concurrency} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency ${concurrency} \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment3" \
        --models 8
done

Experiment 3 - Pin concurrency size 1, run difference loras

text
#!/bin/bash
for model_number in {1..32}; do
    output_file="benchmark_unmerged_multi_lora_${model_number}_concurrency_1.jsonl"
    echo "Running benchmark with model number ${model_number} and output file ${output_file}"

    python3 benchmark.py \
        --dataset-path "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json" \
        --concurrency 1 \
        --output-file-path "${output_file}" \
        --deployment-endpoints "deployment3" \
        --models ${model_number}
done

Note: Find results in deployment1 and deployment2 concurrency=1 cases.

File: benchmarks/scenarios/gateway/README.md

Gateway Routing benchmark

Prerequisite

Test Dataset

bash
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/blob/main/ShareGPT_V3_unfiltered_cleaned_split.json

Client - Curl

bash
curl -v http://localhost:8888/v1/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer sk-any-key" \
    -d '{
        "model": "deepseek-coder-7b-instruct",
        "messages": [{"role": "user", "content": "Say this is a test!"}],
        "max_tokens": 128
    }'

Client - Locust

text
locust -f benchmark.py --host http://localhost:8887

Experiments

experiment 1 & 2 should use exact same client setting and they are comparable.

Experiment 1: gateway overhead (httpRoute) vs k8s service (baseline)

bash
kubectl -n envoy-gateway-system port-forward service/envoy-aibrix-system-aibrix-eg-903790dc 8888:80
kubectl port-forward svc/deepseek-coder-7b-instruct 8887:8000 -n aibrix-system

Note: we can not use port-forward in > 1 pod testing, all the traffic will go into one pod.
Change model service and gateway service to LoadBalancer for real testing.

Experiment 2: Three Routing Strategies

wait until cache ready, manually send some request to activate model.

bash
> Note: this is for local testing, feel free to change to Elastic IP later.

# service port-forwarding
OUTPUT_FILE=k8s-service.jsonl locust -f benchmark.py --host http://localhost:8887 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_httproute.csv --csv-full-history --logfile benchmark_gateway_httproute.log

# gateway port-forwarding
OUTPUT_FILE=http-route.jsonl locust -f benchmark.py --host http://localhost:8888 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_httproute.csv --csv-full-history --logfile benchmark_gateway_httproute.log

OUTPUT_FILE=random.jsonl ROUTING_STRATEGY=random locust -f benchmark.py --host http://localhost:8888 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_random.csv --csv-full-history --logfile benchmark_gateway_random.log

OUTPUT_FILE=least-request.jsonl ROUTING_STRATEGY=least-request locust -f benchmark.py --host http://localhost:8888 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_least_request.csv --csv-full-history --logfile benchmark_gateway_least_request.log

OUTPUT_FILE=throughput.jsonl ROUTING_STRATEGY=throughput locust -f benchmark.py --host http://localhost:8888 --headless --users 30 --spawn-rate 0.08 --run-time 10m --csv benchmark_gateway_throughput.csv --csv-full-history --logfile benchmark_gateway_throughput.log

Local Testing

bash
make docker-build-gateway-plugins
aibrix/gateway-plugins:9bd45a9915b71936ff0001a6fbfc32f10b65e480

k edit deployment aibrix-gateway-plugins

k delete pod aibrix-gateway-plugins-759b87dc65-j9qs8 # commit is exact same, we just need to update once
bash
curl  http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer any_key" \
  -d '{
     "model": "llama2-70b",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

Note: We do not need model or routing strategy in the header now. this is clean and sdk compatibile.

New Client Testing

bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "k8s-v2.jsonl"
bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "httproute-v2.jsonl"

update env

bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "random-v2.jsonl"
bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "least-request-v2.jsonl"
bash
python client.py \
--dataset-path "/tmp/ShareGPT_V3_unfiltered_cleaned_split.json" \
--endpoint "http://localhost:8888" \
--num-prompts 2000 \
--interval 0.05 \
--output-file-path "throughput-v2.jsonl"

File: benchmarks/scenarios/autoscaling/README.md

How to run autoscaling experiment

TLDR;

You can run autoscaling benchmark experiment by simply running the command below.
./overnight_run.sh workload/workload/25min_up_and_down/25min_up_and_down.jsonl

What you have to check before running it

  • run deployment for your application (refer to deepseek-llm-7b-chat/deploy.yaml)
  • change the name field under scaleTargetRed in all autoscaling yaml files.
  • check the deployment name in run-test.py

For example,

yaml
...
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deepseek-llm-7b-chat (*this one)
...

Workflow

overnight_run.sh will execute run-test.sh with given autoscaling mechanism.

run-test.sh will execute

  1. Preparation
    • clean up existing autoscaler resources
    • set the number of replicas to 1 of the target deployment
    • restart aibrix-controller-manager, aibrix-gateway-plugins, target application's deployment
  2. Ready state check
    • Once all the deployments are in ready state, it will start experiment
  3. Start logging running as background processes
    • Log application pods, aibrix-controller-manager, aibrix-gateway-plugins by streaming the output to files (it will be used for debugging, post analysis)
  4. Run client with appropriate arguments (client code is in aibrix/benchmark/generator/client.py)
  5. Stop logging processes once experiment is done
  6. clean up resources
    • clean up autocaling resources
    • set the number of replicas for the application deployment back to 1.

How to plot graphs

There are two plots that you can plot.

Generating report

python <aibrix_root_repo>/benchmarks/plot/plot-everything.py <experiment_home_dir>

For example,
python <aibrix_root_repo>/benchmarks/plot/plot-everything.py experiment_results/25min_test

The directories should look like

bash
ls experiment_results/25min_test

25min_up_and_down-apa-least-request-20250209-064742
25min_up_and_down-hpa-least-request-20250209-055214
25min_up_and_down-kpa-least-request-20250209-061957
25min_up_and_down-optimizer-kpa-least-request-20250209-071531

Generating per pod graph

bash
python <aibrix_root_repo>/benchmarks/plot/plot_per_pod.py <experiment_pod_logs_dir>

For example,

bash
python <aibrix_root_repo>/benchmarks/plot/plot_per_pod.py experiment_results/25min_test/25min_up_and_down-apa-least-request-20250209-064742/pod_logs

Move all experiment output dir under the same dir (e.g., 25min_test) to compare them in the report.

It will generate timeseries plot for the number of requests running in the pod, throughput, cache usage.

Dependency

  • Client: aibrix/benchmark/generator/client.py
  • Workload trace generator: aibrix/benchmark/generator/workload_generator.py. Please check aibrix/benchmark/generator/README.md for more information.
    • Workload generated using synthetic workload types with configurations specified under this directory.

File: benchmarks/client/README.md

Test client locally

Note All generator invocation should be done under the benchmark home (i.e., /aibrix/benchmarks/)

Starting vllm server:

shell
export API_KEY=${API_KEY}
vllm serve --host 0.0.0.0 \
--port "8000" \
--model /root/models/deepseek-llm-7b-chat \
--trust-remote-code \
--max-model-len "4096" \
--api-key ${API_KEY} \
--enable-chunked-prefill

Using a sample workload (generated by the workload generator) in a client. Turn on --streaming to collect fine grained metrics such as TTFT and TPOT:

shell
export API_KEY=${API_KEY}
python -m client.client \
--workload-path "output/workload/constant/workload.jsonl" \
--endpoint "http://localhost:8888" \
--model llama-3-8b-instruct \
--api-key ${API_KEY} \
--output-token-limit 128 \
--streaming \
--output-file-path output.jsonl

The output will be stored as a .jsonl file in output.jsonl

Session affinity replay

For sessioned workloads (requests carrying session_id), pass --session-key-header to send each request's session_id as the x-aibrix-session-key header (see HeaderSessionKey in pkg/plugins/gateway/types.go). The gateway only consumes this header when the request is routed with the session-affinity strategy, so pair the flag with --routing-strategy session-affinity (or routing_strategy: session-affinity in the benchmark.py config) — with any other strategy the header is sent but ignored. Note the gateway caps session keys at 256 bytes; longer keys are dropped and the request silently falls back to load-balanced routing (the client logs a warning per over-long session). The flag is off by default and has no effect on requests without a session_id. When driving the client through benchmark.py, set client_session_key_header: true in the config instead.

Run analysis on metrics collected. For streaming client, we can specify a goodput target (e2e/tpot/ttft) like the following:

shell
python -m client.analyze --trace output.jsonl --output output --goodput-target tpot:0.5

By default, client treats timestamp in the workload as timestamp in milliseconds (ms). The timestamp described in the workload generator could be scaled with --time-scale, Say that the original workload demands one paralell disptach at time 0, 1000, 2000..., meaning that the requests will be dispatch every second. You could use --time-scale to scale the intervals of timestamps described in the workload, e.g. --time-scale 0.1 scale the intervals of timetsamps by 0.1 meaning that the requests will be dispatched at time 0, 100, 200....

File: config/dependency/kuberay-operator/README.md

KubeRay upstream manifests

Commands to export manifest from helm package. After you got manifest, copy to this folder.

shell
helm template kuberay-operator kuberay/kuberay-operator --namespace aibrix-system --version 1.2.1 --include-crds --set env[0].name=ENABLE_PROBES_INJECTION --set-string env[0].value=false --set fullnameOverride=kuberay-operator --set featureGates[0].name=RayClusterStatusConditions --set featureGates[0].enabled=true --output-dir ./config/dependency

If you use zsh, please use noglob helm ... to skip the brace check.

File: deployment/terraform/gcp/README.md

AIBrix GCP Terraform Module

This module deploys an AIBrix cluster in its entirety onto a Google Container Cluster. It is the quickest way to get up and running with AIBrix. The purpose of this module is to both allow developers to quickly spin up the stack, and allow for the team to test on the GCP cloud environment.

NOTE: This module was created to allow users to quickly spin up AIBrix on GCP. It is not currently built for production deployments. The user is is responsible for any costs incurred by running this module.

Quickstart

Prerequisites

Quickstart

  1. Run gcloud auth application-default login to setup credentials to Google.
  2. Install cluster auth plugin with gcloud components install gke-gcloud-auth-plugin.
  3. Rename terraform.tfvars.example to terraform.tfvars and fill in the required variables. You can also add any optional overrides here as well.
  4. Run terraform init to initialize the module.
  5. Run terraform plan to see details on the resources created by this module.
  6. When you are satisfied with the plan and want to create the resources, run terraform apply. NOTE: if you recieve NodePool aibrix-gpu-nodes was created in the error state "ERROR" while running the script, check your quotas for GPUs and the specific instances you're trying to deploy.
  7. Wait for module to complete running. It will output a command to receive the kubernetes config file and a public IP address.
  8. Run a command against the public IP:
bash
ENDPOINT="<YOUR PUBLIC IP>"

curl http://${ENDPOINT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek-r1-distill-llama-8b",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "help me write a random generator in python"}
        ]
    }'
  1. When you are finished testing and no longer want the resources, run terraform destroy. Ensure that you complete this step once you are done trying it out, as GPUs are expensive.

Testing

Testing is done completely end to end, no resources need to be initialized beforehand. The testing script will spin up the entire stack, run its tests with the generic OpenAI client against the created resources, and then will destroy everything it has created. Tests are modeled after the E2E tests found in test/e2e/e2e_test.go.

  1. Ensure you have required Prerequisites from Quickstart above.
  2. Complete steps 1-3 in Quickstart steps above. Only fill in the rquired variables within terraform.tfvars.
  3. Change directory to /deployment/terraform and run go test -v -timeout 60m tests/gcp_test.go. NOTE: this test takes a while to run as it's spinning up its own kubernetes cluster.
  4. If the test times out, to ensure resource deletion, simply run the test again.

Inputs

Name Description Type Default Required
aibrix_release_version The version of AIBRix to deploy. string "v0.2.0" no
cluster_name Name of the GKE cluster. string "aibrix-inference-cluster" no
cluster_zone Zone to deploy cluster within. If not provided will be deployed to default region. string "" no
default_region Default region to deploy resources within. string n/a yes
deploy_example_model Whether to deploy the example model. bool true no
node_pool_machine_count Machine count for the node pool. number 1 no
node_pool_machine_type Machine type for the node pool. Must be in the A3, A2, or G2 series. string "g2-standard-4" no
node_pool_name Name of the GPU node pool. string "aibrix-gpu-nodes" no
node_pool_zone Zone to deploy GPU node pool within. If not provided will be deployed to zone in default region which has capacity for machine type. string "" no
project_id GCP project to deploy resources within. string n/a yes

Outputs

Name Description
aibrix_service_public_ip Public IP address for AIBrix service.
configure_kubectl_command Command to run which will allow kubectl access.

Modules

Name Source Version
aibrix ../kubernetes n/a
cluster ./cluster n/a

Providers

Name Version
google 6.22.0
kubernetes 2.36.0

2. Official Technical Reference & Guides (vllm-project/vllm-project.github.io)

vLLM blog source

Local development

  1. Install jekyll and bundler by running gem install jekyll bundler. Ask ChatGPT for help if you encounter any issues.
  2. bundle install to install the necessary gems.
  3. rm ./Gemfile.lock if you meet gem error.
  4. bundle exec jekyll server to compile and start the server.

To add a new blogpost, please refer to _posts/2023-06-20-vllm.md as an example. Some notes:

  • Google Doc can be saved as markdown format which will make your life easier.
  • Note that the blogpost will only show when its date is in the past.
  • Add summary front matter for SEO. Keep it concise, specific, and 240 characters or fewer.
  • Put figures under assets/figures/yourblogname/.
  • Make a pull request.

The blog is automatically built and deployed by GitHub Actions when main is pushed to.

LaTeX Math

The blog supports LaTeX math via MathJax.

It can be enabled by adding math: true to the document frontmatter. It has been configured to support the standard LaTeX style math notation, i.e.:

latex
$ inline math $
latex
$$
math block
$$

GitHub Flavored Admonitions

The blog supports GitHub flavored admonitions via jekyll-gfm-admonition. It supports the following syntax:

markdown
> [!NOTE]
> Highlights information that users should take into account, even when skimming.
> And supports multi-line text.

> [!TIP]
> Optional information to help a user be more successful.

> [!IMPORTANT]
> Crucial information necessary for users to succeed.

> [!WARNING]
> Critical content demanding immediate
> user attention due to potential risks.

> [!CAUTION]
> Negative potential consequences of an action.
> Opportunity to provide more context.

Theme customization

The theme we are using is Minima. If you need to customise anything from this theme, see Overriding theme defaults.

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →