1. Project Overview & Quickstart (mostlygeek/llama-swap)

File: README.md

llama-swap

Run multiple generative AI models on your machine and hot-swap between them on demand. llama-swap works with any OpenAI and Anthropic API compatible server and is used by thousands of people to power their local AI workflows.

Built in Go for performance and simplicity, llama-swap has zero dependencies and is incredibly easy to set up. Get started in minutes - just one binary and one configuration file.

Features:

✅ Easy to deploy and configure: one binary, one configuration file. no external dependencies
✅ On-demand model switching for many local AI servers (llama.cpp + forks, vllm, stable-diffusion.cpp, audio.cpp, ComfyUI, etc.)
- future proof, upgrade your inference servers at any time.
✅ OpenAI API supported endpoints:
- v1/completions
- v1/chat/completions
- v1/responses
- v1/embeddings
- v1/models - list available models
- v1/audio/speech (#36)
- v1/audio/transcriptions (docs)
- v1/audio/voices
- v1/images/generations
- v1/images/edits
✅ Anthropic API supported endpoints:
- v1/messages
- v1/messages/count_tokens
✅ llama-server (llama.cpp) supported endpoints
- v1/rerank, v1/reranking, /rerank
- /infill - for code infilling
- /completion - for completion endpoint
- /models - list available models. same behavior as v1/models
- /props - requires ?model={model_id} query parameter to be provided. The autoload parameter is not supported and will be ignored.
✅ SDAPI via stable-diffusion.cpp's server
- /sdapi/v1/txt2img
- /sdapi/v1/img2img
- /sdapi/v1/loras - requires model in request body to fetch the correct loras
✅ audio.cpp supported extra endpoints
- /audioapi/v1/tasks/run
✅ /comfyui/ - ComfyUI custom endpoint (#1001) for more reliable swapping
✅ llama-swap API
- /ui - web UI
- /upstream/:model_id - direct access to upstream server (demo)
- /running - list currently running models (#61)
- POST /api/models/unload - manually unload all running models (#58)
- POST /api/models/unload/:model_id - unload a specific model
- GET /api/profiles - list configured profiles and the active selection
- PUT /api/profiles/active - activate a profile or select none
- /logs - remote log monitoring
  - GET /logs returns buffered plain text logs.
    - If Accept: text/html is sent, /logs redirects to /ui/.
  - GET /logs/stream keeps the connection open for live log streaming.
    - Stream endpoints send buffered history first by default; add ?no-history to stream only new lines.
  - GET /logs/stream/proxy streams proxy logs only.
  - GET /logs/stream/upstream streams upstream process logs only.
  - GET /logs/stream/{model_id} streams logs for one model (including IDs with slashes, like author/model).
- /health - just returns "OK"
- /metrics - system and GPU metrics for prometheus
✅ API Key support - define keys to restrict access to API endpoints
✅ Customization
- Switch model ID routing at runtime with profiles
- Run concurrent models with a custom DSL swap matrix (#643)
- Automatic unloading of models after timeout by setting a ttl
- Docker and Podman support using cmd and cmdStop together
- Preload models on startup with hooks (#235)
- Apply filters to requests to control inference with stripParams, setParams and setParamsByID

Web UI

llama-swap includes a real time web interface with a playground for testing out all sorts of local models:

View detailed token metrics:

Inspect request and responses:

Manually load and unload models:

Real time log streaming:

Installation

llama-swap can be installed in multiple ways

Docker
Homebrew (macOS and Linux)
MacPorts (macOS)
WinGet
From release binaries
From source

Docker Install ([download images](https://github.com/mostlygeek/llama-swap/pkgs/container/llama-swap))

Two types of container images are built nightly for llama-swap:

A unified container with llama-server, ik-llama-server, stable-diffusion.cpp, whisper.cpp and llama-swap built from source. This is only available for cuda and vulkan but has more capabilities. This one is recommended for use.
A legacy image that is based on llama.cpp's images and llama-swap copied into the container. Use this one if you prefer to stay close to llama.cpp's container images.

Unified container (Recommended)

shell

$ docker pull ghcr.io/mostlygeek/llama-swap:unified-cuda

# run with a custom configuration and models directory
$ docker run -it --rm --runtime nvidia -p 9292:8080 \
 -v /path/to/models:/models \
 -v /path/to/custom/config.yaml:/etc/llama-swap/config/config.yaml \
 ghcr.io/mostlygeek/llama-swap:unified-cuda

Legacy container

shell

$ docker pull ghcr.io/mostlygeek/llama-swap:cuda

# run with a custom configuration and models directory
$ docker run -it --rm --runtime nvidia -p 9292:8080 \
 -v /path/to/models:/models \
 -v /path/to/custom/config.yaml:/app/config.yaml \
 ghcr.io/mostlygeek/llama-swap:cuda

more examples

shell

# pull latest images per platform
docker pull ghcr.io/mostlygeek/llama-swap:cpu
docker pull ghcr.io/mostlygeek/llama-swap:cuda
docker pull ghcr.io/mostlygeek/llama-swap:vulkan
docker pull ghcr.io/mostlygeek/llama-swap:intel
docker pull ghcr.io/mostlygeek/llama-swap:musa

# tagged llama-swap, platform and llama-server version images
docker pull ghcr.io/mostlygeek/llama-swap:v166-cuda-b6795

# non-root cuda
docker pull ghcr.io/mostlygeek/llama-swap:cuda-non-root

Homebrew Install (macOS/Linux)

shell

brew tap mostlygeek/llama-swap
brew install llama-swap
llama-swap --config path/to/config.yaml --listen localhost:8080

MacPorts (macOS)

[!NOTE]
Maintained by MacPorts community - llama-swap port. It is not an official part of llama-swap.

shell

sudo port install llama-swap
llama-swap --config path/to/config.yaml --listen localhost:8080

WinGet Install (Windows)

[!NOTE]
WinGet is maintained by community contributor Dvd-Znf (#327). It is not an official part of llama-swap.

shell

# install
C:\> winget install llama-swap

# upgrade
C:\> winget upgrade llama-swap

Pre-built Binaries

Binaries are available on the release page for Linux, Mac, Windows and FreeBSD.

Building from source

Building requires Go and Node.js (for UI).
git clone https://github.com/mostlygeek/llama-swap.git
make clean all
look in the build/ subdirectory for the llama-swap binary

Configuration

yaml

# minimum viable config.yaml

models:
  model1:
    cmd: llama-server --port ${PORT} --model /path/to/model.gguf

That's all you need to get started:

models - holds all model configurations
model1 - the ID used in API calls
cmd - the command to run to start the server.
${PORT} - an automatically assigned port number

Almost all configuration settings are optional and can be added one step at a time:

Advanced features
- matrix to run concurrent models with a custom swap logic DSL
- hooks to run things on startup
- macros reusable snippets
Model customization
- ttl to automatically unload models
- unloadTimeout to tune graceful unloads (manual, API and ttl expiry)
- aliases to use familiar model names (e.g., "gpt-4o-mini")
- env to pass custom environment variables to inference servers
- cmdStop gracefully stop Docker/Podman containers
- useModelName to override model names sent to upstream servers
- ${PORT} automatic port variables for dynamic port assignment
- filters rewrite parts of requests before sending to the upstream server

See the configuration documentation for all options.

How does llama-swap work?

When a request is made to an OpenAI compatible endpoint, llama-swap will extract the model value and load the appropriate server configuration to serve it. If the wrong upstream server is running, it will be replaced with the correct one. This is where the "swap" part comes in. The upstream server is automatically swapped to handle the request correctly.

In the most basic configuration llama-swap handles one model at a time. For more advanced use cases, using a matrix allows multiple models to be loaded at the same time. You have complete control over how your system resources are used.

Reverse Proxy Configuration (nginx)

If you deploy llama-swap behind nginx, disable response buffering for streaming endpoints. By default, nginx buffers responses which breaks Server‑Sent Events (SSE) and streaming chat completion. (#236)

Recommended nginx configuration snippets:

nginx

# SSE for UI events/logs
location /api/events {
    proxy_pass http://your-llama-swap-backend;
    proxy_buffering off;
    proxy_cache off;
}

# Streaming chat completions (stream=true)
location /v1/chat/completions {
    proxy_pass http://your-llama-swap-backend;
    proxy_buffering off;
    proxy_cache off;
}

As a safeguard, llama-swap also sets X-Accel-Buffering: no on SSE responses. However, explicitly disabling proxy_buffering at your reverse proxy is still recommended for reliable streaming behavior.

Monitoring Logs on the CLI

# sends up to the last 10KB of logs
$ curl http://host/logs

# streams combined logs
curl -Ns http://host/logs/stream

# stream llama-swap's proxy status logs
curl -Ns http://host/logs/stream/proxy

# stream logs from upstream processes that llama-swap loads
curl -Ns http://host/logs/stream/upstream

# stream logs only from a specific model
curl -Ns http://host/logs/stream/{model_id}

# stream and filter logs with linux pipes
curl -Ns http://host/logs/stream | grep 'eval time'

# appending ?no-history will disable sending buffered history first
curl -Ns 'http://host/logs/stream?no-history'

Do I need to use llama.cpp's server (llama-server)?

Any OpenAI compatible server would work. llama-swap was originally designed for llama-server and it is the best supported.

For Python based inference servers like vllm or tabbyAPI it is recommended to run them via podman or docker. This provides clean environment isolation as well as responding correctly to SIGTERM signals for proper shutdown.

File: models/README.md

TODO improve these docs

Download a llama-server suitable for your architecture
Fetch some small models for testing / swapping between
- huggingface-cli download bartowski/Qwen2.5-1.5B-Instruct-GGUF --include "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf" --local-dir ./
- huggingface-cli download bartowski/Llama-3.2-1B-Instruct-GGUF --include "Llama-3.2-1B-Instruct-Q4_K_M.gguf" --local-dir ./
Create a new config.yaml file (see config.example.yaml) pointing to the models

2. Official Technical Reference & Guides (mostlygeek/llama-swap-docs)

File: README.md

llama-swap.wiki

Documentation website for llama-swap - a lightweight proxy for dynamic LLM model switching.

Technology Stack

VitePress 1.5+ - Static site generator with Vue 3
Vue 3 - Interactive components with Composition API
Tailwind CSS v4.0 - Styling with CSS-first configuration
TypeScript - Type-safe components
Prism.js - YAML syntax highlighting

Project Structure

text

llama-swap-wiki/
├── docs/                           # VitePress content root
│   ├── .vitepress/
│   │   ├── config.ts              # VitePress configuration
│   │   └── theme/
│   │       ├── index.ts           # Theme entry (registers components)
│   │       ├── style.css          # Tailwind v4 + Prism.js styles
│   │       ├── components/        # Vue components
│   │       └── lib/               # Config builder logic
│   ├── index.md                   # Homepage
│   ├── getting-started.md         # Installation guide
│   ├── examples.md                # Configuration examples
│   └── configuration.md           # Interactive config builder
└── package.json

Development

Prerequisites

Node.js 18+
npm

Setup

bash

npm install

Development Mode

bash

npm run dev

The dev server will start on an available port (shown in terminal).

Build

bash

npm run build

Output: docs/.vitepress/dist/

Preview Production Build

bash

npm run preview

Visit http://localhost:4173

Features

Homepage with project overview
Getting Started guide with installation instructions
Configuration examples with syntax highlighting
Interactive Configuration Builder with:
- Feature selection checkboxes
- Live YAML preview with syntax highlighting
- Line highlighting when features are toggled
- Download and copy functionality

Deployment

The site is a static build that can be deployed to:

GitHub Pages
Netlify
Vercel
Cloudflare Pages
Any static hosting service

Build Command

bash

npm run build

Output Directory

text

docs/.vitepress/dist/

Environment

Node.js 18+

License

MIT

File: docs/index.md

next: text: Configuration Explorer link: /explore

About

llama-swap is used by thousands of users for reliable for on-demand loading of AI backends like llama.cpp, vLLM, Whisper, and stable-diffusion.cpp. It follows a simple design philosophy: one binary, one configuration file, no dependencies.

It comes with no strings attached. MIT licensed with zero telemetry collection.

Installation

llama-swap is available on Linux, Mac and Windows. You can use on of the available methods or build it yourself.

::: tabs

== Docker

Images are built nightly and are available for multiple hardware platforms: CUDA, Intel, Vulkan, MUSA, and CPU inference.

bash

# Pull the container for your hardware
docker pull ghcr.io/mostlygeek/llama-swap:cuda
docker pull ghcr.io/mostlygeek/llama-swap:vulkan
docker pull ghcr.io/mostlygeek/llama-swap:intel
docker pull ghcr.io/mostlygeek/llama-swap:musa
docker pull ghcr.io/mostlygeek/llama-swap:cpu

# Pull specific versions of llama-swap and llama.cpp
docker pull ghcr.io/mostlygeek/llama-swap:v179-cuda-b7565

# Run with a custom configuration
docker run -v ./config.yaml:/app/config.yaml ghcr.io/mostlygeek/llama-swap

All available images can be found on the GitHub packages page.

== macOS (Homebrew)

bash

brew tap mostlygeek/llama-swap
brew install llama-swap

== Windows

powershell

winget install llama-swap

== Binary

Pre-built binaries are automatically built from the main branch. Download them from the releases page.

Available platforms:

Linux: x86_64, ARM64
macOS: Intel, Apple Silicon
Windows: x86_64
FreeBSD

== Source

Build a custom binary from source:

bash

git clone https://github.com/mostlygeek/llama-swap.git
cd llama-swap
make build

# Build for a custom OS and architecture
GOOS=linux GOARCH=arm64 make build

:::

1. Project Overview & Quickstart (mostlygeek/llama-swap) Copy Chapter

File: README.md Copy Chapter

llama-swap

Features: Copy Chapter

Web UI Copy Section

Installation Copy Chapter

Docker Install ([download images](https://github.com/mostlygeek/llama-swap/pkgs/container/llama-swap)) Copy Section

Unified container (Recommended)

Legacy container

Homebrew Install (macOS/Linux) Copy Section

MacPorts (macOS) Copy Section

WinGet Install (Windows) Copy Section

Pre-built Binaries Copy Section

Building from source Copy Section

Configuration Copy Chapter

How does llama-swap work? Copy Chapter

Reverse Proxy Configuration (nginx) Copy Chapter

Monitoring Logs on the CLI Copy Chapter

Do I need to use llama.cpp's server (llama-server)? Copy Chapter

File: models/README.md Copy Chapter

2. Official Technical Reference & Guides (mostlygeek/llama-swap-docs) Copy Chapter

File: README.md Copy Chapter

llama-swap.wiki

Technology Stack Copy Chapter

Project Structure Copy Chapter

Development Copy Chapter

Prerequisites Copy Section

Setup Copy Section

Development Mode Copy Section

Build Copy Section

Preview Production Build Copy Section

Features Copy Chapter

Deployment Copy Chapter

Build Command Copy Section

Output Directory Copy Section

Environment Copy Section

License Copy Chapter

File: docs/index.md Copy Chapter

next: text: Configuration Explorer link: /explore Copy Chapter

About

Installation Copy Chapter

Explore Related Technologies & Topics

1. Project Overview & Quickstart (mostlygeek/llama-swap)

File: README.md

Features:

Web UI

Installation

Docker Install ([download images](https://github.com/mostlygeek/llama-swap/pkgs/container/llama-swap))

Homebrew Install (macOS/Linux)

MacPorts (macOS)

WinGet Install (Windows)

Pre-built Binaries

Building from source

Configuration

How does llama-swap work?

Reverse Proxy Configuration (nginx)

Monitoring Logs on the CLI

Do I need to use llama.cpp's server (llama-server)?

File: models/README.md

2. Official Technical Reference & Guides (mostlygeek/llama-swap-docs)

File: README.md

Technology Stack

Project Structure

Development

Prerequisites

Setup

Development Mode

Build

Preview Production Build

Features

Deployment

Build Command

Output Directory

Environment

License

File: docs/index.md

next: text: Configuration Explorer link: /explore

Installation