## File: README.md ## 🍋 Lemonade: Refreshingly fast local AI

[Download](https://lemonade-server.ai/docs/guide/install/) | [Documentation](https://lemonade-server.ai/docs/) | [Discord](https://discord.gg/5xXzkMu8Zk)

Lemonade is the local AI server that gives you the same capabilities as cloud APIs, except 100% free and private. Use the latest models for chat, coding, speech, and image generation on your own NPU and GPU. Lemonade comes in two flavors: * **Lemonade Server** installs a service you can connect to hundreds of great apps using standard OpenAI, Anthropic, and Ollama APIs. * **Embeddable Lemonade** is a portable binary you can package into your own application to give it multi-modal local AI that auto-optimizes for your user’s PC. *This project is built by the community for every PC, with optimizations by AMD engineers to get the most from Ryzen AI, Radeon, and Strix Halo PCs.* ## Getting Started 1. **Install**: [Windows](https://github.com/lemonade-sdk/lemonade/releases/latest/download/lemonade.msi) · [Linux](#supported-platforms) · [macOS](https://github.com/lemonade-sdk/lemonade/releases) · [Docker](https://lemonade-server.ai/docs/guide/install/docker) · [Source](./docs/dev/getting-started.md) 2. **Get Models**: Browse and download with the [Model Manager](#model-library) 3. **Generate**: Try models with the built-in interfaces for chat, image gen, speech gen, and more 4. **Mobile**: Take your lemonade to go: [iOS](https://apps.apple.com/us/app/lemonade-mobile/id6757372210) · [Android](https://play.google.com/store/apps/details?id=com.lemonade.mobile.chat.ai&pli=1) · [Source](https://github.com/lemonade-sdk/lemonade-mobile) 5. **Connect**: Use Lemonade with your [favorite apps](https://lemonade-server.ai/marketplace): ## Supported Platforms | Platform | Build | |----------|-------| | [](https://lemonade-server.ai/docs/guide/install/arch/) | [](https://github.com/lemonade-sdk/lemonade/actions/workflows/linux_distro_builds.yml) | | [](https://lemonade-server.ai/docs/guide/install/debian/) | [](https://github.com/lemonade-sdk/lemonade/actions/workflows/linux_distro_builds.yml) | | [](https://lemonade-server.ai/docs/guide/install/docker/) | [](https://github.com/lemonade-sdk/lemonade/actions/workflows/build-and-push-container.yml) | | [](https://lemonade-server.ai/docs/guide/install/fedora/) | [](https://github.com/lemonade-sdk/lemonade/actions/workflows/cpp_server_build_test_release.yml) | | [](https://github.com/lemonade-sdk/lemonade/releases) | [](https://github.com/lemonade-sdk/lemonade/actions/workflows/cpp_server_build_test_release.yml) | | [](https://snapcraft.io/lemonade-server) | [](https://github.com/lemonade-sdk/lemonade-server-snap/actions/workflows/snap-build.yaml) | | [](https://lemonade-server.ai/docs/guide/install/ubuntu/) | [](https://github.com/lemonade-sdk/lemonade/actions/workflows/launchpad-ppa.yml) | | [](https://github.com/lemonade-sdk/lemonade/releases/latest/download/lemonade.msi) | [](https://github.com/lemonade-sdk/lemonade/actions/workflows/cpp_server_build_test_release.yml) | ## Using the CLI To run and chat with Gemma: ``` lemonade run Gemma-4-E2B-it-GGUF ``` To code with Lemonade models: ``` lemonade launch claude ``` Multi-modality: ``` # image gen lemonade run SDXL-Turbo # speech gen lemonade run kokoro-v1 # transcription lemonade run Whisper-Large-v3-Turbo ``` To see available models and download them: ``` lemonade list lemonade pull Gemma-4-E2B-it-GGUF ``` To manage model aliases for environment-independent naming and active-standby failover: ``` lemonade alias add production-llm Gemma-4-E2B-it-GGUF lemonade alias list # Instant active-standby failover to a different model target lemonade alias add production-llm Qwen3-0.6B-GGUF lemonade alias remove production-llm ``` To see the backends available on your PC: ``` lemonade backends ``` For hybrid setups, Lemonade can also route to any OpenAI-compatible cloud provider (Fireworks, OpenAI, OpenRouter, Together, …) alongside local models — see [Cloud Offload](./docs/guide/configuration/cloud.md). *(Experimental.)* ## Model Library Lemonade supports a wide variety of LLMs (**GGUF**, **FLM**, and **ONNX**), whisper, stable diffusion, etc. models across CPU, GPU, and NPU. Use `lemonade pull` or the built-in **Model Manager** to download models. Custom GGUF/ONNX models can be pulled from Hugging Face or ModelScope, with their source retained for future updates. **[Browse all built-in models →](https://lemonade-server.ai/models.html)** ## Supported Configurations Lemonade supports multiple inference engines for LLM, speech, TTS, and image generation, and each has its own backend and hardware requirements. | Modality | Engine | Backend | Device | OS | | --- | --- | --- | --- | --- | | **Text generation** | `llamacpp` | `system` | `x86_64`/ARM64 CPU, GPU | Linux | | `metal` | Apple Silicon GPU | macOS | | | | `cuda` | NVIDIA GPUs (Turing or newer)** | Windows, Linux | | | | `vulkan` | `x86_64` CPU, AMD iGPU, AMD dGPU; ARM64 CPU/GPU (Linux) | Windows, Linux | | | | `rocm` | AMD GPUs supported by ROCm | Windows, Linux | | | | `cpu` | `x86_64` CPU; ARM64 CPU (Linux) | Windows, Linux | | | | `flm` | `npu` | XDNA2 NPU | Windows, Linux | | | `ryzenai-llm` | `npu` | XDNA2 NPU | Windows | | | `vllm` (experimental) | `rocm` | Strix Halo iGPU (gfx1151) | Linux | | | **Speech-to-text** | `whispercpp` | `npu` | XDNA2 NPU | Windows | | `metal` | Apple Silicon GPU | macOS | | | | `vulkan` | `x86_64` CPU | Windows, Linux | | | | `rocm` | Supported AMD ROCm iGPU/dGPU families* | Windows, Linux | | | | `cpu` | `x86_64` CPU | Windows, Linux | | | | `moonshine` | `cpu` | `x86_64`/`arm64` CPU | Windows, Linux, macOS | | | **Text-to-speech** | `kokoro` | `metal` | Apple Silicon GPU | macOS | | `cpu` | `x86_64` CPU | Windows, Linux | | | | `openmoss` (experimental) | `cuda` | NVIDIA GPUs | Windows, Linux | | | `vulkan` | Vulkan-capable GPUs | Windows, Linux | | | | `rocm` | AMD GPUs (ROCm via TheRock) | Windows, Linux | | | | **Audio generation** | `thinksound` (experimental) | `cuda` | NVIDIA GPUs | Windows, Linux | | `vulkan` | Vulkan-capable GPUs | Windows, Linux | | | | `rocm` | Supported AMD ROCm iGPU/dGPU families (ROCm via TheRock) | Windows, Linux | | | | `acestep` (experimental) | `cuda` | NVIDIA GPUs | Windows, Linux | | | `vulkan` | Vulkan-capable GPUs | Windows, Linux | | | | `rocm` | Supported AMD ROCm iGPU/dGPU families (ROCm via TheRock) | Windows, Linux | | | | **Image generation** | `sd-cpp` | `metal` | Apple Silicon GPU | macOS | | `cuda` | NVIDIA GPUs (Turing or newer)** | Windows, Linux | | | | `vulkan` | Vulkan-capable GPUs | Windows, Linux | | | | `rocm` | Supported AMD ROCm iGPU/dGPU families* | Windows, Linux | | | | `cpu` | `x86_64` CPU | Windows, Linux | | | | `thenoise` (experimental) | `rocm` | Supported AMD ROCm iGPU families | Linux | | | **3D generation** | `trellis` (experimental) | `cuda` | NVIDIA GPUs | Windows, Linux | | `vulkan` | Vulkan-capable GPUs | Windows, Linux | | | | `rocm` | Supported AMD ROCm iGPU/dGPU families (ROCm via TheRock) | Windows, Linux | | | | **Text classification** | `onnxruntime` (experimental) | `cpu` | `x86_64` CPU | Windows | | `cpu` | `x86_64`/`arm64` CPU | Linux | | | | `cpu` | `arm64` CPU | macOS | | | To check exactly which recipes/backends are supported on your own machine, run: ``` lemonade backends ``` ** See supported AMD ROCm platforms* | Architecture | Platform Support | GPU Models | | --- | --- | --- | | **gfx1151** (STX Halo) | Windows, Ubuntu | Ryzen AI MAX+ Pro 395 | | **gfx120X** (RDNA4) | Windows, Ubuntu | Radeon AI PRO R9700, RX 9070 XT/GRE/9070, RX 9060 XT | | **gfx110X** (RDNA3) | Windows, Ubuntu | Radeon PRO W7900/W7800/W7700/V710, RX 7900 XTX/XT/GRE, RX 7800 XT, RX 7700 XT | *** See supported NVIDIA CUDA platforms* | Compute Capability | Architecture | GPU Models | | --- | --- | --- | | **sm_75** | Turing | RTX 20-series, GTX 16-series, T4 | | **sm_80** / **sm_86** | Ampere | RTX 30-series, A100, A40 | | **sm_89** | Ada Lovelace | RTX 40-series, L40, L4 | | **sm_90** | Hopper | H100, H200 | | **sm_100** / **sm_120** | Blackwell | RTX 50-series, B100, B200 | ## Project Roadmap Lemonade's roadmap is defined by a set of working groups. Visit the landing page [here](./docs/dev/working-groups/README.md) to learn each group's goal and roadmap. ## Integrate Embeddable Lemonade in Your Application Embeddable Lemonade is a binary version of Lemonade that you can bundle into your own app to give it a portable, auto-optimizing, multi-modal local AI stack. This lets users focus on your app, with zero Lemonade installers, branding, or telemetry. Check out the [Embeddable Lemonade guide](docs/embeddable/README.md). ## Connect Lemonade Server to Your Application You can use any OpenAI-compatible client library by configuring it to use `http://localhost:13305/v1` as the base URL. A table containing official and popular OpenAI clients on different languages is shown below. Feel free to pick and choose your preferred language. | Python | C++ | Java | C# | Node.js | Go | Ruby | Rust | PHP | |--------|-----|------|----|---------|----|-------|------|-----| | [openai-python](https://github.com/openai/openai-python) | [openai-cpp](https://github.com/olrea/openai-cpp) | [openai-java](https://github.com/openai/openai-java) | [openai-dotnet](https://github.com/openai/openai-dotnet) | [openai-node](https://github.com/openai/openai-node) | [go-openai](https://github.com/sashabaranov/go-openai) | [ruby-openai](https://github.com/alexrudall/ruby-openai) | [async-openai](https://github.com/64bit/async-openai) | [openai-php](https://github.com/openai-php/client) | ### Python Client Example ```python from openai import OpenAI # Initialize the client to use Lemonade Server client = OpenAI( base_url="http://localhost:13305/api/v1", api_key="lemonade" # required but unused ) # Create a chat completion completion = client.chat.completions.create( model="Gemma-4-E2B-it-GGUF", # or any other available model messages=[ {"role": "user", "content": "What is the capital of France?"} ] ) # Print the response print(completion.choices[0].message.content) ``` Click to learn more about the [available APIs](./docs/api/README.md) and how to [embed Lemonade](./docs/embeddable/README.md) in your own application. ## FAQ To read our frequently asked questions, see our [FAQ Guide](./docs/guide/faq.md) ## Contributing Lemonade is built by the local AI community! If you would like to contribute to this project, please check out our [contribution guide](./docs/dev/contribute.md). ## Maintainers This is a community project maintained by @amd-pworfolk @bitgamma @danielholanda @jeremyfowers @kenvandine @Geramy @ramkrishna2910 @sawansri @siavashhub @sofiageo @superm1 @vgodsoe, and sponsored by AMD. You can reach us by filing an [issue](https://github.com/lemonade-sdk/lemonade/issues), emailing [lemonade@amd.com](mailto:lemonade@amd.com), or joining our [Discord](https://discord.gg/5xXzkMu8Zk). ## Code Signing Policy Free code signing provided by [SignPath.io](https://signpath.io), certificate by [SignPath Foundation](https://signpath.org). - **Committers and reviewers**: [Maintainers](#maintainers) of this repo - **Approvers**: [Owners](https://github.com/orgs/lemonade-sdk/people?query=role%3Aowner) **Privacy policy**: This program will not transfer any information to other networked systems unless specifically requested by the user or the person installing or operating it. When the user requests a model download or registry lookup, Lemonade may contact [Hugging Face Hub](https://huggingface.co/) (see their [privacy policy](https://huggingface.co/privacy)) or [ModelScope](https://modelscope.cn/), according to the model source selected by the user or packager. ## License and Attribution This project is: - Built with C++ (server) and React (app) with ❤️ for the open source community, - Standing on the shoulders of great tools from: - [ggml/llama.cpp](https://github.com/ggml-org/llama.cpp) - [ggml/whisper.cpp](https://github.com/ggerganov/whisper.cpp) - [ggml/stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) - [kokoros](https://github.com/lucasjinreal/Kokoros) - [OnnxRuntime GenAI](https://github.com/microsoft/onnxruntime-genai) - [Hugging Face Hub](https://github.com/huggingface/huggingface_hub) - [ModelScope](https://github.com/modelscope/modelscope) - [OpenAI API](https://github.com/openai/openai-python) - [IRON/MLIR-AIE](https://github.com/Xilinx/mlir-aie) - and more... - Licensed under the [Apache 2.0 License](https://github.com/lemonade-sdk/lemonade/blob/main/LICENSE). - Portions of the project are licensed as described in [LICENSE](./LICENSE). --- ## File: contrib/launchpad-downloads/README.md # Launchpad PPA Download Statistics Tool A Python tool to retrieve download statistics for packages in Launchpad PPAs (Personal Package Archives). ## Installation 1. Install dependencies: ```bash pip install -r requirements.txt ``` Or install directly: ```bash pip install launchpadlib ``` ## Usage Basic usage: ```bash python ppa_stats.py username/ppa-name ``` ### Examples Get stats for all packages in a PPA: ```bash python ppa_stats.py developmentseed/mapbox ``` Get stats for a specific package: ```bash python ppa_stats.py developmentseed/mapbox -p tilemill ``` Show all versions including those with 0 downloads: ```bash python ppa_stats.py username/ppa-name --all ``` Show summary grouped by package name: ```bash python ppa_stats.py username/ppa-name --summary ``` ### Options - `-p, --package PACKAGE` - Query a specific package name - `--all` - Show all versions including those with 0 downloads - `--summary` - Show summary by package name instead of individual versions ## How It Works The tool uses the Launchpad API via the `launchpadlib` Python library to: 1. Connect anonymously to Launchpad's production API 2. Retrieve the specified PPA for the given user/team 3. Fetch published binary packages and their download counts 4. Display formatted statistics ## API Documentation The tool uses Launchpad's public API: - API endpoint: https://api.launchpad.net/ - No authentication required for read-only access - Uses the `getDownloadCount()` method on published binaries ## Output Format The tool displays: - Total downloads across all packages/versions - Individual package versions with download counts - Architecture information for each binary package - Sorted by download count (most popular first) ## Notes - Download statistics are provided by Launchpad and may have some delay - The tool caches API responses in `~/.launchpadlib/cache/` for performance - Anonymous access is sufficient for read-only statistics ## Sources Based on information from: - [Launchpad API Documentation](https://api.launchpad.net/) - [PPA Download Statistics Examples](https://gist.github.com/springmeyer/2778600) - [ppa-stats GitHub Project](https://github.com/hsheth2/ppa-stats) --- ## File: docs/api/anthropic.md # Anthropic-Compatible API Lemonade supports an initial Anthropic Messages compatibility endpoint for applications that call Claude-style APIs. | Endpoint | Status | Notes | |----------|--------|-------| | `POST /v1/messages` | Supported | Supports both streaming and non-streaming. Query params like `?beta=true` are accepted. | Current scope focuses on message generation parity for common fields (`model`, `messages`, `system`, `max_tokens`, `temperature`, `stream`, and basic `tools`). Unsupported or unimplemented Anthropic-specific fields are ignored and surfaced via warning logs/headers. --- ## File: docs/api/lemonade.md # Lemonade API We have designed a set of Lemonade-specific endpoints to enable client applications by extending the existing cloud-focused APIs (e.g., OpenAI). These extensions allow for a greater degree of UI/UX responsiveness in native applications by allowing applications to: - Download models at setup time. - Pre-load models at UI-loading-time, as opposed to completion-request time. - Unload models to save memory space. - Understand system resources and state to make dynamic choices. | Method | Endpoint | Description | |--------|----------|-------------| | `POST` | [`/v1/pull`](#post-v1pull) | Install a model | | `POST` | [`/v1/models/register`](#post-v1modelsregister) | Register or update a user model definition without downloading it | | `GET` | [`/v1/downloads`](#get-v1downloads) | List server-owned model download jobs | | `POST` | [`/v1/downloads/control`](#post-v1downloadscontrol) | Pause, cancel, or remove server-owned model download jobs | | `GET` | [`/v1/registry/search`](#get-v1registrysearch) | Search Hugging Face or ModelScope for model repositories | | `GET` | [`/v1/pull/variants`](#get-v1pullvariants) | Enumerate GGUF variants for a Hugging Face checkpoint | | `POST` | [`/v1/delete`](#post-v1delete) | Delete a model | | `POST` | [`/v1/load`](#post-v1load) | Load a model | | `POST` | [`/v1/unload`](#post-v1unload) | Unload a model | | `POST` | [`/v1/audio/generations`](#post-v1audiogenerations) | Generate audio (music or sound effects) from a text prompt | | `POST` | [`/v1/classify`](#post-v1classify) | Classify input text with an encoder classifier (label scores) | | `POST` | [`/v1/3d/generations`](#post-v13dgenerations) | Generate a textured 3D mesh (GLB) from an image | | `POST` | [`/v1/models/check-updates`](#post-v1modelscheck-updates) | Manually check downloaded models for upstream updates | | `GET` | [`/v1/models/{id}/files`](#get-v1modelsidfiles) | List resolved local file metadata for one model | | `GET` | [`/v1/models/{id}/options`](#get-v1modelsidoptions) | Read a model's saved, effective, and default recipe options | | `POST` | [`/v1/models/{id}/options`](#post-v1modelsidoptions) | Save recipe options for a model without loading it | | `DELETE` | [`/v1/models/{id}/options`](#delete-v1modelsidoptions) | Reset a model's recipe options to defaults | | `GET` | [`/v1/health`](#get-v1health) | Check server status, such as models loaded | | `GET` | [`/v1/stats`](#get-v1stats) | Performance statistics from the last request | | `GET` | [`/v1/system-stats`](#get-v1system-stats) | Current host resource usage | | `GET` | [`/v1/system-info`](#get-v1system-info) | System information and device enumeration | | `POST` | [`/v1/install`](#post-v1install) | Install or update a backend, or register a cloud provider | | `POST` | [`/v1/uninstall`](#post-v1uninstall) | Remove a backend or cloud provider | | `POST` | [`/v1/cloud/auth`](#post-v1cloudauth) | Set an in-memory API key for a cloud provider | | `DELETE` | [`/v1/cloud/auth/{provider}`](#delete-v1cloudauthprovider) | Clear the in-memory API key for a cloud provider | | `WS` | [`/logs/stream`](#log-streaming-api-websocket) | Log Streaming | | `GET` | [`/live`](#get-live) | Check server liveness for load balancers and orchestrators | | `GET` | [`/metrics`](#get-metrics) | Prometheus metrics scrape endpoint | | `POST` | [`/internal/telemetry/flush`](#post-internaltelemetryflush) | Force-flush all queued telemetry trace spans | | `GET` | [`/internal/aliases`](#get-internalaliases) | List all active model aliases | | `POST` | [`/internal/aliases`](#post-internalaliases) | Create or update a model alias | | `DELETE` | [`/internal/aliases/{alias}`](#delete-internalaliasesalias) | Remove a model alias | ## `POST /v1/classify` Run an encoder text-classifier (PII, prompt-safety, domain, etc.) on an input string and return per-label scores in `[0, 1]`. The target model must use the `onnxruntime` recipe. Both sequence-classification (one label set) and token-classification (aggregated span labels) models are supported. **Supported architectures:** single-sequence encoder families — BERT, DistilBERT, RoBERTa, XLM-RoBERTa, DeBERTa (v1/v2), ELECTRA, ALBERT, CamemBERT. A stock `optimum-cli export onnx` directory of one of these works as-is. A servable model directory is `model.onnx` + `tokenizer.json` + `config.json`. The `config.json` is **always required**: it declares the architecture, which is checked against the list above so an unsupported family (e.g. XLNet, which uses different segment/special-token conventions) is **rejected at load time** rather than served with wrong scores. The output contract (labels, normalization, token budget) is read from that same config; an optional `manifest.json` overrides it but does not replace the config. Without a manifest, inference assumes **single-label softmax**; a multi-label (sigmoid) model must declare `problem_type: multi_label_classification` in its config or ship a `manifest.json`. This endpoint provides the classification capability that the router's `classifier` condition type will consume; the live routing-policy wiring is tracked in [#2384](https://github.com/lemonade-sdk/lemonade/issues/2384). The endpoint is available at: - `/v1/classify` - `/api/v1/classify` - `/v0/classify` - `/api/v0/classify` ### Parameters | Field | Type | Required | Description | |-------|------|----------|-------------| | `model` | string | yes* | Classifier model id (a model with the `onnxruntime` recipe). *Optional when a classification model is already loaded; the loaded model is used and echoed in the response. | | `input` | string | yes | Text to classify. `text` is accepted as an alias. | | `top_k` | integer | no | Return only the highest-scoring `k` labels. | ### Example request ```bash curl -X POST http://localhost:13305/v1/classify -H "Content-Type: application/json" -d '{"model": "Phishing-Email-Detection-ONNX", "input": "Please verify your account at http://secure-login.example now."}' ``` ### Response format ```json { "object": "classification", "model": "Phishing-Email-Detection-ONNX", "labels": { "LABEL_1": 0.982, "LABEL_0": 0.011, "LABEL_2": 0.005, "LABEL_3": 0.002 } } ``` Label names come from the model's `id2label` — from `config.json`, or from `manifest.json` when one is present to override it; some upstream models only declare generic `LABEL_` names — see the model card for their meaning. Malformed requests (invalid JSON, missing `input`/`text`, non-string fields, non-positive `top_k`) return `400` with an `error` object before any model is loaded. ## Routing (`collection.router`) Naming a registered `collection.router` model in the `model` field of a `chat/completions` or `completions` request triggers the routing engine: the server picks a candidate by the policy's first-matching rule (fail-open to `default_model`) and forwards the request to it. No dedicated endpoint or `"auto"` model is involved. The decision is reported on the response: - Header **`x-lemonade-route`** — the matched rule id, or `default`. - Request field **`route_trace: true`** adds an **`x_lemonade_route`** object to the response body: `{ route_to, matched_rule, default_used, outputs, trace[] }` (`route_to` is the candidate that answered). For streaming responses it is attached to the first SSE event. See [Router Policies](../dev/router-policy.md) for authoring the policy. ## `POST /v1/models/check-updates` Explicitly checks downloaded Hugging Face-backed models for newer upstream commits. This is the manual counterpart to the startup update check and works even when `auto_check_model_updates=false`. Full offline mode remains authoritative: when `offline=true`, this endpoint returns HTTP 409 and does not make network requests. ### Example request ```bash curl -X POST http://localhost:13305/v1/models/check-updates ``` The same action is available from the CLI: ```bash lemonade check-updates ``` ### Response format ```json { "status": "success", "updates_available": 2, "models": [ "Qwen3-4B-GGUF", "Whisper-Tiny" ] } ``` The endpoint is available at: - `/v1/models/check-updates` - `/api/v1/models/check-updates` - `/v0/models/check-updates` - `/api/v0/models/check-updates` ## `GET /v1/models/{id}/files` List resolved local file metadata for a single model. This endpoint is intended for model-detail UIs such as the Files tab. It is per-model inventory, not system or drive storage accounting. The endpoint is available at: - `/v1/models/{id}/files` - `/api/v1/models/{id}/files` - `/v0/models/{id}/files` - `/api/v0/models/{id}/files` By default, the response does not include absolute filesystem paths. Trusted local clients that need paths for native UI actions can request them explicitly with `?include_paths=true`. Absolute paths may reveal local usernames and cache layout, so clients should only request them when that disclosure is acceptable. ### Example request ```bash curl http://localhost:13305/v1/models/Qwen3-4B/files ``` ### Response format ```json { "model_id": "Qwen3-4B", "files": [ { "name": "model.gguf", "role": "main", "size_bytes": 123456789, "exists": true }, { "name": "mmproj.gguf", "role": "mmproj", "size_bytes": 12345678, "exists": true } ] } ``` ### Optional path disclosure ```bash curl 'http://localhost:13305/v1/models/Qwen3-4B/files?include_paths=true' ``` When `include_paths=true` is supplied, each file entry also includes `path`: ```json { "name": "model.gguf", "path": "/abs/path/model.gguf", "role": "main", "size_bytes": 123456789, "exists": true } ``` ### Fields | Field | Description | |-------|-------------| | `model_id` | Public model ID for the requested model. | | `files` | Array of resolved model files known to the registry. | | `files[].name` | Base filename from the resolved path. | | `files[].path` | Absolute resolved path on the local system. Only included when `include_paths=true`; privacy-sensitive. | | `files[].role` | Checkpoint role, for example `main`, `mmproj`, or another recipe-specific role. | | `files[].size_bytes` | File size in bytes. Directories are summed recursively. Missing files report `0`. | | `files[].exists` | Whether the resolved path currently exists on disk. | ## `GET /v1/models/{id}/options` Read a model's recipe options, separated by layer, without loading it. With `POST` and `DELETE` on the same path, this manages per-model options independently of [`/v1/load`](#post-v1load). ### Example request ```bash curl http://localhost:13305/v1/models/Qwen3-0.6B-GGUF/options ``` ### Response format `effective` is the exact request body a [`POST /v1/load`](#post-v1load) for this model uses right now, with every option the recipe accepts resolved through the full priority chain. `defaults` is what a reset model would get. For `llamacpp`, with `--no-mmap` saved and the context size left automatic: ```json { "model_name": "Qwen3-0.6B-GGUF", "recipe": "llamacpp", "saved": { "llamacpp_args": "--no-mmap" }, "effective": { "auto_evict": null, "ctx_size": -1, "downsize_idle_timeout": 60, "evict_idle_timeout": 300, "evict_weight_factor": 1.0, "llamacpp_args": "--no-mmap", "llamacpp_backend": "vulkan", "llamacpp_device": "", "merge_args": true, "model_name": "Qwen3-0.6B-GGUF" }, "defaults": { "auto_evict": null, "ctx_size": -1, "downsize_idle_timeout": 60, "evict_idle_timeout": 300, "evict_weight_factor": 1.0, "llamacpp_args": "", "llamacpp_backend": "vulkan", "llamacpp_device": "", "merge_args": true, "model_name": "Qwen3-0.6B-GGUF" }, "resolved_ctx_size": 32768, "load_command": "curl -X POST $LEMONADE_BASE_URL/v1/load -H \"Content-Type: application/json\" -d '{\"ctx_size\":-1, ...}'" } ``` `load_command` is `effective` posted to [`/v1/load`](#post-v1load): the exact load the model would get right now, as a command. The base URL is left as `$LEMONADE_BASE_URL`, and the API key, when the server requires one, as `$LEMONADE_API_KEY`. The server cannot fill either in: it always listens over plain HTTP, so a client that reached it through a TLS-terminating proxy is on `https` without the server ever knowing, and the `Host` header is only the caller's own claim about where it sent the request. Export the variables to run the command as-is: ```bash export LEMONADE_BASE_URL=http://localhost:13305 ``` A client that already knows its base URL — every GUI and CLI does — can substitute the two tokens before displaying the command. The quoting is for POSIX shells: bash, zsh, and PowerShell 7.3 and newer, which is where PowerShell began passing arguments to native commands unchanged. `cmd.exe` does not treat `'` as a quote character at all. In either of those, build the request from `effective` rather than pasting the command. | Field | Description | |-------|-------------| | `model_name` | The id from the URL. It appears again inside `effective` and `defaults` so that each one is a complete `/v1/load` body. | | `recipe` | The recipe the option names belong to. | | `saved` | The model's own entry in `recipe_options.json`: only what was explicitly saved, or `{}` when nothing is. It can also hold keys this endpoint does not accept, such as `pinned` written by `/v1/load`, so replay `effective` rather than `saved`. | | `effective` | The `/v1/load` body shown above. Posting it back whole to this endpoint saves every resolved value as an override, so send only the options the user changed. | | `defaults` | What `effective` becomes if `saved` is erased, in the same shape. A `ctx_size` of `-1` means the server picks the context size automatically. | | `resolved_ctx_size` | The context size a load right now would use: the effective `ctx_size`, or the automatically computed size when that is `-1`. | | `load_command` | The curl command that reproduces the load: `effective` posted to `/v1/load`, with `$LEMONADE_BASE_URL` (and `$LEMONADE_API_KEY` on a key-protected server) left for the caller to fill in. | > Note: per-architecture defaults come from the model's GGUF metadata. For a model that has not been downloaded yet, every key is still present but carries the value it has before those defaults apply. ## `POST /v1/models/{id}/options` Save recipe options for a model without loading it. The request body is a flat object of the same recipe options [`/v1/load`](#post-v1load) accepts. The URL identifies the model; a `model_name` in the body is ignored. The request merges into the model's saved entry, so keys you don't mention are left alone. `null` removes an option, and the model falls back to the next layer of the [priority chain](#post-v1load). [`DELETE`](#delete-v1modelsidoptions) removes every saved option at once. `dry_run: true` validates and resolves the request identically but persists nothing: `effective`, `resolved_ctx_size`, and `load_command` describe the state the save would produce, while `saved` keeps reporting the entry on disk. Use it to preview a change before committing it. `ctx_size` takes a positive whole number, or `-1` to pin the model to automatic sizing even when the server-wide `ctx_size` is a specific number. A `400` reports an unrecognized option name, an option from a different recipe, a value of the wrong type, or an invalid `ctx_size`, and nothing from that request is saved. Saving never loads or reloads the model, so a model that is already running keeps its current options until it is next loaded. > Note: `pinned` is not settable here and is omitted from `effective` and `defaults`. It belongs to [`/v1/load`](#post-v1load) and `/internal/pin`. ### Example requests Save a context size without loading the model: ```bash curl -X POST http://localhost:13305/v1/models/Qwen3-0.6B-GGUF/options \ -H "Content-Type: application/json" \ -d '{"ctx_size": 8192, "llamacpp_backend": "vulkan"}' ``` Set the context size back to automatic, leaving the backend choice saved: ```bash curl -X POST http://localhost:13305/v1/models/Qwen3-0.6B-GGUF/options \ -H "Content-Type: application/json" \ -d '{"ctx_size": -1}' ``` ### Response format Same as [`GET /v1/models/{id}/options`](#get-v1modelsidoptions), reflecting the state after the write. ## `DELETE /v1/models/{id}/options` Reset a model to its defaults by erasing its `recipe_options.json` entry entirely. The model keeps the defaults that come from its registry entry and from the server's global configuration; only the user's saved overrides are removed. ### Example request ```bash curl -X DELETE http://localhost:13305/v1/models/Qwen3-0.6B-GGUF/options ``` ### Response format Same as [`GET /v1/models/{id}/options`](#get-v1modelsidoptions), with `saved` now `{}`. ## `POST /v1/models/register` Register or update a `user.*` model definition without downloading model files. Use this endpoint when registration and installation are separate actions. `POST /v1/pull` remains the install/download path and performs the same internal registration step before downloading. The endpoint is available at: - `/v1/models/register` - `/api/v1/models/register` - `/v0/models/register` - `/api/v0/models/register` ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `model_name` | Yes | Non-empty namespaced model name under `user.*`. | | `recipe` | Yes | Lemonade recipe associated with the model definition. | | `checkpoint` | No | Main checkpoint, when the recipe uses one. | | `checkpoints` | No | Named checkpoints for multi-checkpoint models. | | `source` | No | Registry or local source. Remote values are `huggingface` or `modelscope`. | | `labels` | No | Additional model labels. | | `components` | No | Already-registered component model names for collection recipes. | A checkpoint is intentionally not universally required: registration is a model metadata operation and some present or future model types may not have local weights. `/pull` remains the operation that attempts installation/download. The endpoint accepts one model definition. An embedded `models` array represents multiple definitions and remains a collection-import concern; register those component definitions first when using this endpoint. Example request: ```bash curl -X POST http://localhost:13305/v1/models/register \\ -H "Content-Type: application/json" \\ -d '{ "model_name": "user.Phi-4-Mini-GGUF", "checkpoint": "unsloth/Phi-4-mini-instruct-GGUF:Q4_K_M", "recipe": "llamacpp" }' ``` Example response: ```json { "status": "success", "model_name": "Phi-4-Mini-GGUF", "canonical_model_name": "user.Phi-4-Mini-GGUF", "model": { "id": "Phi-4-Mini-GGUF", "recipe": "llamacpp", "downloaded": false } } ``` `model_name` is the public ID exposed by `/v1/models`; `canonical_model_name` is the stable `user.*` registration ID. Registration updates `user_models.json` and invalidates the model cache, but does not start a model download. ## `POST /v1/pull` Register and install models for use with Lemonade Server. ### Parameters The Lemonade Server built-in model registry has a collection of model names that can be pulled and loaded. The `pull` endpoint can install any registered model, and it can also register-then-install any model available on Hugging Face. **Common Parameters** | Parameter | Required | Description | |-----------|----------|-------------| | `stream` | No | If `true`, returns Server-Sent Events (SSE) with download progress. Defaults to `false`. | | `subscribe` | No | Only applies when `stream=true`. If `false`, the server starts a background model download job and returns a JSON snapshot immediately instead of keeping the HTTP response subscribed to SSE progress. Defaults to `true` for backwards compatibility. | **Install a Model that is Already Registered** | Parameter | Required | Description | |-----------|----------|-------------| | `model_name` | Yes | [Lemonade Server model name](https://lemonade-server.ai/models.html) to install. | Example request: ```bash curl -X POST http://localhost:13305/v1/pull \ -H "Content-Type: application/json" \ -d '{ "model_name": "Qwen3-0.6B-GGUF" }' ``` Response format: ```json { "status":"success", "message":"Installed model: Qwen3-0.6B-GGUF" } ``` In case of an error, the status will be `error` and the message will contain the error message. **Register and Install a Model** Registration will place an entry for that model in the `user_models.json` file, which is located in the user's Lemonade cache (default: `~/.cache/lemonade`). Then, the model will be installed. Once the model is registered and installed, it will show up in the `models` endpoint alongside the built-in models and can be loaded. The `recipe` field defines which software framework and device will be used to load and run the model. > Note: the `model_name` for registering a new model must use the `user` namespace, to prevent collisions with built-in models. For example, `user.Phi-4-Mini-GGUF`. | Parameter | Required | Description | |-----------|----------|-------------| | `model_name` | Yes | Namespaced [Lemonade Server model name](https://lemonade-server.ai/models.html) to register and install. | | `recipe` | Yes | Lemonade API recipe to load the model with. | | `checkpoint` | Yes`*` | HuggingFace "main" checkpoint to install. | | `checkpoints` | No | HuggingFace checkpoints to install, for multi-checkpoint models. | | `reasoning` | No | Whether the model is a reasoning model, like DeepSeek (default: false). Adds 'reasoning' label. | | `vision` | No | Whether the model has vision capabilities for processing images (default: false). Adds 'vision' label. | | `embedding` | No | Whether the model is an embedding model (default: false). Adds 'embeddings' label. | | `reranking` | No | Whether the model is a reranking model (default: false). Adds 'reranking' label. | | `mmproj` | No | Multimodal Projector (mmproj) file to use for vision models. | A model definition requires at least a `main` checkpoint. This can be either be specified with the `checkpoint` parameter, or a `main` key in the `checkpoints` dict. Each backend serves a fixed set of [deployment modes](openai.md#model-labels), and a model deploys in exactly one of them. Naming a mode the recipe cannot serve, or naming two — whether through `labels` or through the `embedding` / `reranking` parameters — is rejected with `400` and nothing is registered: ```bash curl -X POST http://localhost:8000/api/v1/pull \ -H "Content-Type: application/json" \ -d '{"model_name": "user.Clf", "recipe": "llamacpp", "checkpoint": "example/model:Q4_K_M", "labels": ["classification"]}' ``` ```json {"error": "Model 'user.Clf': recipe 'llamacpp' cannot serve 'classification'. It serves 'chat', 'embeddings', 'reranking'. Omit the label to deploy as 'chat'."} ``` ```bash curl -X POST http://localhost:8000/api/v1/pull \ -H "Content-Type: application/json" \ -d '{"model_name": "user.Both", "recipe": "llamacpp", "checkpoint": "example/model:Q4_K_M", "labels": ["chat", "embeddings"]}' ``` ```json {"error": "Model 'user.Both': a model deploys in exactly one mode, but these labels name two: 'chat' and 'embeddings'. Register one model per mode."} ``` Omitting the deployment label entirely is always valid — the recipe's default is applied. Other checkpoint types may also be specified depending on the model type. This list is not exhaustive, and may change or grow over time as models and backends evolve: * `mmproj` - used by vision models, if not already embedded in `main` * `draft` - used by dflash, eagle, and multitoken-prediction, if not already embedded in `main` * `text_encoder` - text-to-token encoder used by image generation * `vae` - variational autoencoder used by image generation Example request: ```bash # Single checkpoint curl -X POST http://localhost:13305/v1/pull \ -H "Content-Type: application/json" \ -d '{ "model_name": "user.Phi-4-Mini-GGUF", "checkpoint": "unsloth/Phi-4-mini-instruct-GGUF:Q4_K_M", "recipe": "llamacpp" }' ``` Instead of defining a model by `checkpoint` and `mmproj`, a model can also be defined with a dict of checkpoint types and paths. These requests do the same thing, but the syntax for pulling the mmproj differs. ```bash # Multi-checkpoint curl -X POST http://localhost:13305/v1/pull \ -H "Content-Type: application/json" \ -d '{ "model_name": "user.My-Gemma3", "checkpoint": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M", "mmproj": "mmproj-model-f16.gguf", "vision": true, "recipe": "llamacpp" }' ``` ```bash # Multi-checkpoint curl -X POST http://localhost:13305/v1/pull \ -H "Content-Type: application/json" \ -d '{ "model_name": "user.My-Gemma3", "checkpoints": { "main": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M", "mmproj": "ggml-org/gemma-3-4b-it-GGUF:mmproj-model-f16.gguf" }, "vision": true, "recipe": "llamacpp" }' ``` Response format: ```json { "status":"success", "message":"Installed model: user.Phi-4-Mini-GGUF" } ``` In case of an error, the status will be `error` and the message will contain the error message. **Register an Omni-Model** An omni collection is a collection type that bundles several models into a single entry that can be loaded, pulled, or deleted as a unit. Use `recipe: "collection.omni"` with a `components` array instead of `checkpoint`. | Parameter | Required | Description | |-----------|----------|-------------| | `model_name` | Yes | Namespaced model name, e.g. `user.MyKit`. | | `recipe` | Yes | Must be `"collection.omni"`. | | `components` | Yes | Ordered, non-empty array of model names. Each component must be a regular model. | | `models` | No | Ordered array of full model definitions, one per `components` entry (the same fields as single-model registration, keyed by `model_name`). When present, component names that are not yet registered are registered from these definitions; names that already exist keep their local definition. When absent, every `components` entry must already exist in the registry (built-in or a previously registered `user.*` model). | Components do not need to be downloaded already — any not-yet-downloaded components are pulled by the same call. Deleting the collection removes only the collection entry; components stay on disk. Example request: ```bash curl -X POST http://localhost:13305/v1/pull \ -H "Content-Type: application/json" \ -d '{ "model_name": "user.MyKit", "recipe": "collection.omni", "components": ["Qwen3-0.6B-GGUF", "Whisper-Tiny", "SD-Turbo"] }' ``` ### Import an Exported Model File Files written by `lemonade export` (and the desktop app's Export button) are import-ready `/v1/pull` request bodies — POST the file contents verbatim to register and install the model. This works for regular models and collections alike; exported collection files additionally carry `components` plus a `models` array embedding each component's definition (see the `models` parameter above). For the file format and the export/import/Hugging Face workflows, see [Share a collection](../guide/configuration/custom-models.md#share-a-collection-export-import-and-hugging-face). ### Streaming Response (stream=true) When `stream=true`, the endpoint returns Server-Sent Events with real-time download progress: ``` event: progress data: {"file":"model.gguf","file_index":1,"total_files":2,"bytes_downloaded":1073741824,"bytes_total":2684354560,"percent":40} event: progress data: {"file":"config.json","file_index":2,"total_files":2,"bytes_downloaded":1024,"bytes_total":1024,"percent":100} event: complete data: {"file_index":2,"total_files":2,"percent":100} ``` **Event Types:** | Event | Description | |-------|-------------| | `progress` | Sent during download with current file and byte progress | | `complete` | Sent when all files are downloaded successfully | | `error` | Sent if download fails, with `error` field containing the message | ### Server-owned download mode (`stream=true`, `subscribe=false`) By default, `stream=true` keeps the `/v1/pull` HTTP response subscribed to Server-Sent Events until the download finishes. Clients that need download state to survive a renderer reload, tab close, or reconnect can also send `subscribe=false`. When `stream=true` and `subscribe=false`, `/v1/pull` starts a server-owned model download job and returns a JSON snapshot immediately. The job continues on the server. Clients can poll [`GET /v1/downloads`](#get-v1downloads) to restore progress and can use [`POST /v1/downloads/control`](#post-v1downloadscontrol) to pause, cancel, or remove the job. Example request: ```bash curl -X POST http://localhost:13305/v1/pull \ -H "Content-Type: application/json" \ -d '{ "model_name": "Qwen3-0.6B-GGUF", "stream": true, "subscribe": false }' ``` Example response: ```json { "id": "model:Qwen3-0.6B-GGUF", "type": "model", "model_name": "Qwen3-0.6B-GGUF", "status": "downloading", "running": true, "file": "", "file_index": 0, "total_files": 0, "bytes_downloaded": 0, "bytes_total": 0, "total_download_size": 0, "bytes_previously_downloaded": 0, "completed_files_bytes": 0, "cumulative_bytes_downloaded": 0, "overall_bytes_downloaded": 0, "percent": 0, "complete": false } ``` ## `GET /v1/downloads` List server-owned model download jobs that were started with `POST /v1/pull` using `stream=true` and `subscribe=false`. This endpoint is intended for clients that need to restore download-manager state after a reload or reconnect. Active, paused, cancelled, and errored jobs remain visible until the client removes them. Completed jobs remain visible briefly so clients can observe completion and refresh model state. ### Example request ```bash curl http://localhost:13305/v1/downloads ``` ### Response format ```json [ { "id": "model:Qwen3-0.6B-GGUF", "type": "model", "model_name": "Qwen3-0.6B-GGUF", "status": "downloading", "running": true, "file": "model.gguf", "file_index": 1, "total_files": 2, "bytes_downloaded": 1073741824, "bytes_total": 2684354560, "total_download_size": 2684355584, "bytes_previously_downloaded": 0, "completed_files_bytes": 0, "cumulative_bytes_downloaded": 1073741824, "overall_bytes_downloaded": 1073741824, "percent": 40, "complete": false } ] ``` ### Download job fields | Field | Description | |-------|-------------| | `id` | Stable download id. Model downloads use `model:`. | | `type` | Download type. Currently `model` for server-owned jobs. | | `model_name` | Lemonade model name associated with the job. | | `status` | Current state: `downloading`, `paused`, `cancelled`, `completed`, or `error`. | | `running` | Whether the download worker is still active. A terminal-looking status may still have `running=true` while the worker is releasing resources. | | `file`, `file_index`, `total_files` | Current file progress within the download. | | `bytes_downloaded`, `bytes_total`, `percent` | Current-file byte progress as reported by the downloader. | | `total_download_size` | Total expected bytes across all files when known. | | `bytes_previously_downloaded` | Bytes already present on disk for the current file when resuming or skipping existing data. | | `completed_files_bytes` | Bytes from files completed before the current file. | | `cumulative_bytes_downloaded`, `overall_bytes_downloaded` | Total bytes downloaded across the whole job. `overall_bytes_downloaded` is kept as a compatibility alias. | | `complete` | `true` when the download completed successfully. | | `error` | Error message, present only for failed jobs. | ## `POST /v1/downloads/control` Control a server-owned model download job. ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `id` | Yes | Download id returned by `POST /v1/pull` or `GET /v1/downloads`, for example `model:Qwen3-0.6B-GGUF`. | | `action` | Yes | One of `pause`, `cancel`, or `remove`. | ### Actions | Action | Description | |--------|-------------| | `pause` | Requests the worker to stop and keeps the job visible as `paused`. The worker may briefly report `running=true` while it unwinds. | | `cancel` | Requests the worker to stop and marks the job as `cancelled`. Clients should wait for `running=false` before deleting partial files. | | `remove` | Removes a stopped job from the server registry. If the worker is still running, the server keeps the job visible and treats the request as a cancel request until the worker stops. | ### Example request ```bash curl -X POST http://localhost:13305/v1/downloads/control \ -H "Content-Type: application/json" \ -d '{ "id": "model:Qwen3-0.6B-GGUF", "action": "pause" }' ``` ### Response format For `pause` and `cancel`, the endpoint returns the latest job snapshot: ```json { "id": "model:Qwen3-0.6B-GGUF", "type": "model", "model_name": "Qwen3-0.6B-GGUF", "status": "paused", "running": false, "file": "model.gguf", "file_index": 1, "total_files": 2, "bytes_downloaded": 1073741824, "bytes_total": 2684354560, "percent": 40, "complete": false } ``` For `remove`, the endpoint returns: ```json {"status":"ok"} ``` If the job is already missing and `action` is `remove`, the endpoint returns: ```json {"status":"ok","missing":true} ``` ## `GET /v1/registry/search` Search a remote model registry (Hugging Face or ModelScope) for repositories matching a text query. This endpoint returns **candidate repositories** based on registry metadata; it does not verify that a repository contains servable files. The desktop app's Model Manager follows up with [`/v1/pull/variants`](#get-v1pullvariants) on each candidate and only offers a download for repositories whose file listing passes that validation. Requires network access: returns 400 with code `lemond_offline` when the server is in offline mode. ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `query` | Yes | Search text, minimum 3 characters after trimming. `q` is accepted as an alias. | | `source` | No | Registry to search: `huggingface` (default) or `modelscope`. Aliases `hf` and `ms` are accepted; the canonical name is echoed in the response. | | `limit` | No | Maximum number of results, an integer from 1 to 50. Default 12. | | `format` | No | The only accepted value is `gguf`. Biases search and ranking toward GGUF repositories and echoes `"format": "gguf"` in the response. | Example request: ```bash curl 'http://localhost:13305/v1/registry/search?source=modelscope&query=qwen&format=gguf' ``` ### Response ```json { "source": "modelscope", "query": "qwen", "format": "gguf", "total": 128, "results": [ { "repository_id": "Qwen/Qwen2.5-3B-Instruct-GGUF", "display_name": "Qwen2.5-3B-Instruct-GGUF", "source": "modelscope", "repository_type": "model", "description": "GGUF quantizations of Qwen2.5-3B-Instruct", "tags": ["gguf", "chat"], "task": "text-generation", "downloads": 222500, "likes": 12, "has_gguf": true } ] } ``` | Field | Description | |-------|-------------| | `source`, `query` | Echoed input (`source` canonicalized to `huggingface` or `modelscope`). | | `format` | Present only when `format=gguf` was requested. | | `total` | Total match count reported by the upstream registry; may exceed the number of returned results. | | `results[]` | Up to `limit` repositories, each with `repository_id`, `display_name`, `source`, `repository_type`, `description`, `tags`, `task`, `downloads`, `likes`, and `has_gguf`. `has_gguf` is a hint derived from registry metadata, not proof of a servable model — [`/v1/pull/variants`](#get-v1pullvariants) performs the authoritative file-level validation. | ### Error responses | Status | Cause | |--------|-------| | 400 | `query` shorter than 3 characters, invalid `source`, `limit`, or `format`, or the server is in offline mode (`code: lemond_offline`). | | 429 | The upstream registry rate-limited the request. | | 502 | Other upstream transport or parsing failures; the body includes the upstream status code when available. | ## `GET /v1/pull/variants` Inspect a Hugging Face GGUF repository and enumerate the variants (quantizations and sharded folder groups) available for installation. Used by the `lemonade pull ` CLI flow and by the desktop app's model search to auto-populate the install form. The endpoint reads only public Hugging Face metadata; if the `HF_TOKEN` environment variable is set on the server, it is forwarded as a bearer token to access gated repositories. ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `checkpoint` | Yes | Hugging Face repo id, e.g. `unsloth/Qwen3-8B-GGUF`. Passed as a query string. | Example request: ```bash curl 'http://localhost:13305/v1/pull/variants?checkpoint=unsloth/Qwen3-8B-GGUF' ``` ### Response ```json { "checkpoint": "unsloth/Qwen3-8B-GGUF", "recipe": "llamacpp", "suggested_name": "Qwen3-8B-GGUF", "suggested_labels": ["vision"], "mmproj_files": ["mmproj-model-f16.gguf"], "variants": [ { "name": "Q4_K_M", "primary_file": "Qwen3-8B-Q4_K_M.gguf", "files": ["Qwen3-8B-Q4_K_M.gguf"], "sharded": false, "size_bytes": 4920000000 }, { "name": "Q8_0", "primary_file": "Q8_0/Qwen3-8B-Q8_0-00001-of-00002.gguf", "files": ["Q8_0/Qwen3-8B-Q8_0-00001-of-00002.gguf", "Q8_0/Qwen3-8B-Q8_0-00002-of-00002.gguf"], "sharded": true, "size_bytes": 8500000000 } ] } ``` | Field | Description | |-------|-------------| | `checkpoint` | Echoed input. | | `recipe` | Suggested recipe (always `llamacpp` today; future expansion may return other values). | | `suggested_name` | Repo id stripped of the `owner/` prefix; suitable for use as the `user.` model name. | | `suggested_labels` | Inferred labels — `vision` if any `mmproj-*.gguf` files exist, plus `embeddings`/`reranking` if those substrings appear in the repo id. | | `mmproj_files` | Bare filenames of `mmproj-*.gguf` files in the repo; the first one should be passed as `mmproj` to `/v1/pull` for vision models. | | `variants[]` | Top quantizations for the repo, capped at 5. Each entry has `name` (e.g. `Q4_K_M`, `UD-Q4_K_XL`), `primary_file`, `files`, `sharded`, and `size_bytes` (from the HF `?blobs=true` listing). Ranked by frequency of use in `server_models.json` (`Q4_K_M`, `UD-Q4_K_XL`, `Q8_0`, `Q4_0` first, everything else sorted lexicographically). The CLI `lemonade pull` menu adds a free-text "Other" option for quants outside the top 5. | ### Error responses | Status | Cause | |--------|-------| | 400 | `checkpoint` query parameter missing or malformed (must contain `/`). | | 404 | Hugging Face returned 404 for the checkpoint. | | 500 | Other transport or parsing failures; the response body contains an `error` message. | ## `POST /v1/delete` Delete a model by removing it from local storage. If the model is currently loaded, it will be unloaded first. > Note: deleting a collection (`recipe: "collection.omni"`) removes only the collection entry from `user_models.json`; its components stay on disk. Delete the components individually if you want to free their disk space. ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `model_name` | Yes | [Lemonade Server model name](https://lemonade-server.ai/models.html) to delete. | Example request: ```bash curl -X POST http://localhost:13305/v1/delete \ -H "Content-Type: application/json" \ -d '{ "model_name": "Qwen3-0.6B-GGUF" }' ``` Response format: ```json { "status":"success", "message":"Deleted model: Qwen3-0.6B-GGUF" } ``` In case of an error, the status will be `error` and the message will contain the error message. ## `POST /v1/load` Explicitly load a registered model into memory. This is useful to ensure that the model is loaded before you make a request. Installs the model if necessary. > Note: loading a collection (`recipe: "collection.omni"`) loads each of its components in turn. Per-model options like `ctx_size` or `llamacpp_backend` are not forwarded to components — set them on each component's own `recipe_options.json` entry instead. Recipe option fields on `/v1/load` have three-state semantics. Omitting a field keeps using its saved per-model value. Passing explicit `null` ignores only that saved key for this load and falls through to the lower default layers without changing `recipe_options.json`. Passing a concrete value overrides the saved value. `ctx_size: -1` is a concrete value meaning automatic context sizing, not a tombstone. With `save_options: true`, concrete values are persisted as usual while a `null` tombstone preserves the existing saved value for that key. ### Parameters | Parameter | Required | Applies to | Description | |-----------|----------|------------|-------------| | `model_name` | Yes | All | [Lemonade Server model name](https://lemonade-server.ai/models.html) to load. | | `pinned` | No | All | Boolean. If true, pins the loaded model to prevent LRU eviction. Defaults to `false`. | | `save_options` | No | All | Boolean. If true, saves recipe options to `recipe_options.json`. Any previously stored value for `model_name` is replaced. To save options without loading, or to change one option without resending the rest, use [`POST /v1/models/{id}/options`](#post-v1modelsidoptions) instead. | | `ctx_size` | No | llamacpp, flm, ryzenai-llm | Context size for the model. Overrides the default value. Pass `-1` to size it automatically instead of using a saved value; omit it to use the saved value. | | `llamacpp_backend` | No | llamacpp | LlamaCpp backend to use (`vulkan`, `rocm`, `metal` or `cpu`). | | `llamacpp_args` | No | llamacpp | Custom arguments to pass to llama-server. The following are NOT allowed: `-m`, `--port`, `--ctx-size`, `-ngl`, `--jinja`, `--mmproj`, `--embeddings`, `--reranking`. | | `whispercpp_backend` | No | whispercpp | WhisperCpp backend: `npu` or `cpu` on Windows; `cpu` or `vulkan` on Linux. Default is `npu` if supported. | | `whispercpp_args` | No | whispercpp | Custom arguments to pass to whisper-server. The following are NOT allowed: `-m`, `--model`, `--port`. Example: `--convert`. | | `steps` | No | sd-cpp | Number of inference steps for image generation. Default: 20. | | `cfg_scale` | No | sd-cpp | Classifier-free guidance scale for image generation. Default: 7.0. | | `width` | No | sd-cpp | Image width in pixels. Default: 512. | | `height` | No | sd-cpp | Image height in pixels. Default: 512. | | `merge_args` | No | All | Boolean. If true (default), `*_args` values from global config and per-model config are merged (per-model takes priority). If false, per-model `*_args` replace global `*_args` entirely. | **Setting Priority:** When loading a model, settings are applied in this priority order: 1. Values explicitly passed in the `load` request (highest priority) 2. Per-model values configurable in `recipe_options.json` (see below for details) 3. Values from environment variables or server startup arguments (see [Server Configuration](../guide/configuration/README.md)) 4. Default hardcoded values in `lemond` (lowest priority) ### Per-model options You can configure recipe-specific options on a per-model basis. Lemonade manages a file called `recipe_options.json` in the user's Lemonade cache (default: `~/.cache/lemonade`). The available options depend on the model's recipe: ```json { "user.Qwen2.5-Coder-1.5B-Instruct": { "ctx_size": 16384, "llamacpp_backend": "vulkan", "llamacpp_args": "-np 2 -kvu" }, "Qwen3-Coder-30B-A3B-Instruct-GGUF" : { "llamacpp_backend": "rocm" }, "whisper-large-v3-turbo-q8_0.bin": { "whispercpp_backend": "npu", "whispercpp_args": "--convert" } } ``` Note that model names include any applicable prefix, such as `user.` and `extra.`. ### Example requests Basic load: ```bash curl -X POST http://localhost:13305/v1/load \ -H "Content-Type: application/json" \ -d '{ "model_name": "Qwen3-0.6B-GGUF" }' ``` Load with custom settings: ```bash curl -X POST http://localhost:13305/v1/load \ -H "Content-Type: application/json" \ -d '{ "model_name": "Qwen3-0.6B-GGUF", "ctx_size": 8192, "llamacpp_backend": "rocm", "llamacpp_args": "--flash-attn on --no-mmap" }' ``` Load and save settings: ```bash curl -X POST http://localhost:13305/v1/load \ -H "Content-Type: application/json" \ -d '{ "model_name": "Qwen3-0.6B-GGUF", "ctx_size": 8192, "llamacpp_backend": "vulkan", "llamacpp_args": "--no-context-shift --no-mmap", "save_options": true }' ``` Load a Whisper model with NPU backend and conversion enabled: ```bash curl -X POST http://localhost:13305/v1/load \ -H "Content-Type: application/json" \ -d '{ "model_name": "whisper-large-v3-turbo-q8_0.bin", "whispercpp_backend": "npu", "whispercpp_args": "--convert" }' ``` Load an image generation model with custom settings: ```bash curl -X POST http://localhost:13305/v1/load \ -H "Content-Type: application/json" \ -d '{ "model_name": "sd-turbo", "steps": 4, "cfg_scale": 1.0, "width": 512, "height": 512 }' ``` ### Response format ```json { "status":"success", "message":"Loaded model: Qwen3-0.6B-GGUF" } ``` In case of an error, the status will be `error` and the message will contain the error message. ## `POST /v1/unload` Explicitly unload a model from memory. This is useful to free up memory while still leaving the server process running (which takes minimal resources but a few seconds to start). ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `model_name` | No | Name of the specific model to unload. If not provided, all loaded models will be unloaded. | ### Example requests Unload a specific model: ```bash curl -X POST http://localhost:13305/v1/unload \ -H "Content-Type: application/json" \ -d '{"model_name": "Qwen3-0.6B-GGUF"}' ``` Unload all models: ```bash curl -X POST http://localhost:13305/v1/unload ``` ### Response format Success response: ```json { "status": "success", "message": "Model unloaded successfully" } ``` Error response (model not found): ```json { "status": "error", "message": "Model not found: Qwen3-0.6B-GGUF" } ``` In case of an error, the status will be `error` and the message will contain the error message. ## `POST /v1/audio/generations` Audio Generation API. You provide a text prompt and receive a generated audio clip. The loaded model decides the kind of audio: music with ACE-Step models (e.g. `ACE-Step-Music`), sound effects with ThinkSound models (e.g. `ThinkSound-SFX`). This endpoint is not part of the OpenAI API (OpenAI's audio endpoints cover speech and transcription only), so it is a Lemonade-specific extension. > **Performance:** generation runs on the GPU (Vulkan, ROCm, or CUDA) and takes from seconds (short sound effects) to minutes (full-length music) depending on duration and hardware. ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `model` | Yes | The audio-generation model to use (e.g., `ThinkSound-SFX`, `ACE-Step-Music`). | | `prompt` | Yes | Text description of the music or sound effect to generate. For music, this is the style description: genre, mood, tempo, instruments, and voice. | | `lyrics` | No | Lyrics to sing (ACE-Step only). When present and not empty, the track is generated with vocals singing these lyrics. Omitting the field, an empty string, or the sentinel `[Instrumental]` (any case) produces an instrumental track. See [Lyrics](#lyrics) below for the expected format. | | `vocal_language` | No | BCP-47 language code of the lyrics, e.g. `en`, `fr`, `ja` (ACE-Step only). Default: `en`. | | `duration` | No | Length of the clip in seconds. Defaults to the backend's native default. | | `steps` | No | Number of inference steps. Lower is faster, higher can improve quality. | | `cfg` | No | Classifier-free guidance strength (ThinkSound only). | | `seed` | No | Random seed for reproducibility. | | `response_format` | No | Output encoding. Only formats the backend natively produces are accepted (currently `wav`); other values are rejected with `400 Bad Request`. Default: `wav`. | ### Lyrics ACE-Step vocals are a two-stage pipeline inside the backend: a language model first turns the style description and lyrics into audio codes, then the diffusion synthesizer renders those codes into audio. The instrumental path skips the language-model stage entirely, which also means lyrics embedded in the `prompt` field are treated as style text — they are never sung. Vocal generations take noticeably longer than instrumental ones of the same duration because of the extra language-model pass. Format the `lyrics` value the way the ACE-Step authors recommend: - Mark each song section with a structure tag on its own line: `[verse]`, `[chorus]`, `[bridge]`, `[intro]`, `[outro]`. - Write one sung phrase per line and separate sections with a blank line. - Describe the voice ("gentle female vocals", "raspy male baritone") in `prompt`, not in the lyrics. - Lyrics may be in any supported language; set `vocal_language` to match. ### Response On success the raw audio bytes are returned with the matching content type (`audio/wav`). On failure the response is JSON with an `error` object: `400` for invalid requests, `404` for unknown models, `500` when the backend reports an error, and `502` when the backend produces no output. ### Example request ```bash curl -X POST http://localhost:13305/v1/audio/generations \ -H "Content-Type: application/json" \ -d '{ "model": "ThinkSound-SFX", "prompt": "glass shattering on a stone floor", "duration": 5, "seed": 42 }' \ --output clip.wav ``` ### Example request (music with vocals) ```bash curl -X POST http://localhost:13305/v1/audio/generations \ -H "Content-Type: application/json" \ -d '{ "model": "ACE-Step-Music", "prompt": "warm acoustic folk ballad, fingerpicked guitar, gentle female vocals", "lyrics": "[verse]\nMoonlight spills across the floor\nShadows dancing by the door\n\n[chorus]\nWe sing until the morning light\nCarried on the wind tonight", "duration": 60 }' \ --output song.wav ``` ## `POST /v1/3d/generations` 3D Generation API. You provide an input image and receive a textured 3D mesh as a glTF-binary (`.glb`) file. Serves TRELLIS models (e.g. `TRELLIS-3D`). The input image must be PNG, JPEG, BMP, or GIF. This endpoint is not part of the OpenAI API, so it is a Lemonade-specific extension. > **Performance:** 3D reconstruction runs on the GPU (Vulkan, ROCm, or CUDA) and takes on the order of minutes; higher cascade resolutions take longer. ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `model` | Yes | The 3D-generation model to use (e.g., `TRELLIS-3D`). | | `image` | Yes | Base64-encoded input image (optionally a `data:` URL). | | `resolution` | No | Cascade resolution: `512`, `1024`, or `1536`. Default: `512`. | | `bg_removal` | No | Background removal mode: `threshold` or `birefnet`. Use `birefnet` for photos with real backgrounds. | | `uv` | No | UV atlas method: `xatlas` (default) or `box`. `xatlas` runs a full UV unwrap giving every face unique atlas space — best quality, but chart computation is superlinear in face count. `box` is a faster 6-plane projection with occlusion-aware bucket assignment and depth-tested rasterization; small texture artifacts remain possible in concave regions. | | `seed` | No | Random seed for reproducibility. | | `response_format` | No | Output encoding. Only formats the backend natively produces are accepted (currently `glb`); other values are rejected with `400 Bad Request`. Default: `glb`. | ### Response On success the raw mesh bytes are returned as `model/gltf-binary`. On failure the response is JSON with an `error` object: `400` for invalid requests, `404` for unknown models, `500` when the backend reports an error, and `502` when the backend produces no output. ### Example request ```bash curl -X POST http://localhost:13305/v1/3d/generations \ -H "Content-Type: application/json" \ -d "{ \"model\": \"TRELLIS-3D\", \"image\": \"$(base64 -w0 input.png)\", \"resolution\": 512, \"seed\": 42 }" \ --output model.glb ``` ## `GET /v1/health` Check the health of the server. This endpoint returns information about loaded models. ### Parameters This endpoint does not take any parameters. ### Example request ```bash curl http://localhost:13305/v1/health ``` ### Response format ```json { "status": "ok", "version":"9.3.3", "websocket_port":9000, "model_loaded": "Llama-3.2-1B-Instruct-Hybrid", "all_models_loaded": [ { "model_name": "Llama-3.2-1B-Instruct-Hybrid", "checkpoint": "amd/Llama-3.2-1B-Instruct-awq-g128-int4-asym-fp16-onnx-hybrid", "last_use": 1732123456.789, "type": "llm", "device": "gpu npu", "pinned": true, "recipe": "ryzenai-llm", "pid": 12345, "recipe_options": { "ctx_size": 4096 }, "backend_url": "http://127.0.0.1:8001/v1" }, { "model_name": "nomic-embed-text-v1-GGUF", "checkpoint": "nomic-ai/nomic-embed-text-v1-GGUF:Q4_K_S", "last_use": 1732123450.123, "type": "embedding", "device": "gpu", "pinned": false, "recipe": "llamacpp", "pid": 12346, "recipe_options": { "ctx_size": 8192, "llamacpp_args": "--no-mmap", "llamacpp_backend": "rocm" }, "backend_url": "http://127.0.0.1:8002/v1" } ], "pinned_models": { "transcription":0, "embedding":0, "image":0, "llm":1, "reranking":0, "tts":0 }, "max_models": { "transcription":1, "embedding":1, "image":1, "llm":1, "reranking":1, "tts":1 }, "telemetry": { "enabled": false }, "update_check_done": true } ``` **Field Descriptions:** - `status` - Server health status, always `"ok"` - `version` - Version number of Lemonade Server - `model_loaded` - Model name of the most recently accessed model - `update_check_done` - Whether the background HuggingFace model update check has completed at startup. Poll this field after server start to know when `update_available` fields are ready. - `all_models_loaded` - Array of all currently loaded models with details: - `model_name` - Name of the loaded model - `checkpoint` - Full checkpoint identifier - `last_use` - Unix timestamp of last access (load or inference) - `type` - Model type: `"llm"`, `"embedding"`, `"reranking"`, `"transcription"`, `"image"`, or `"tts"` - `device` - Space-separated device list: `"cpu"`, `"gpu"`, `"npu"`, or combinations like `"gpu npu"` - `pinned` - Boolean indicating if the model is currently pinned to prevent auto-eviction - `is_busy` - Boolean indicating if the model has active requests or maintenance in progress - `is_streaming` - Boolean indicating if the model is actively generating output tokens (true after first chunk arrives, false when all streaming requests complete) - `backend_url` - URL of the backend server process handling this model (useful for debugging) - `pid` - The Process ID (PID) of the backend engine handling this model - `recipe` - Backend/device recipe used to load the model (e.g., `"ryzenai-llm"`, `"llamacpp"`, `"flm"`) - `recipe_options` - Options used to load the model (e.g., `"ctx_size"`, `"llamacpp_backend"`, `"llamacpp_args"`, `"whispercpp_args"`) - `pinned_models` - Counts of pinned models currently loaded in memory per model type (e.g., `llm`, `embedding`, etc.) - `max_models` - Maximum number of models that can be loaded simultaneously per type (set via `max_loaded_models` in [Server Configuration](../guide/configuration/README.md)): - `llm` - Maximum LLM/chat models - `embedding` - Maximum embedding models - `reranking` - Maximum reranking models - `transcription` - Maximum speech-to-text models - `image` - Maximum image models - `tts` - Maximum text-to-speech models - `websocket_port` - *(optional)* Port of the WebSocket server for the [Realtime Audio Transcription API](./openai.md#ws-realtime) and [Log Streaming API](#log-streaming-api-websocket). Only present when the WebSocket server is running. The port is OS-assigned or set via `--websocket-port`. - `telemetry` - Structured telemetry state object: - `enabled` - Boolean indicating if telemetry collection is active - `captures` - *(optional)* Array of captured telemetry components (e.g., `["inputs", "outputs", "thinking"]`), only present when `enabled` is `true`. ## `GET /v1/stats` Performance statistics from the last request. ### Parameters This endpoint does not take any parameters. ### Example request ```bash curl http://localhost:13305/v1/stats ``` ### Response format ```json { "time_to_first_token": 2.14, "tokens_per_second": 33.33, "input_tokens": 128, "output_tokens": 5, "prompt_tokens": 9, "cache_tokens": 96, "request_count_total": 12, "input_tokens_total": 1536, "output_tokens_total": 60, "prompt_tokens_total": 108, "cache_tokens_total": 1152, "routing_decisions_total": 4, "routing_switches_total": 1 } ``` **Field Descriptions:** - `time_to_first_token` - Time in seconds until the first token was generated - `tokens_per_second` - Generation speed in tokens per second - `input_tokens` - Number of tokens processed - `output_tokens` - Number of tokens generated - `prompt_tokens` - Total prompt tokens including cached tokens - `cache_tokens` - Prompt tokens served from the backend's prefix cache on the last request (llama.cpp `timings.cache_n`, or `usage.prompt_tokens_details.cached_tokens` / Responses-API `input_tokens_details.cached_tokens` from OpenAI-compatible cloud providers). `null` when the last request did not report cache usage - `*_total` - Cumulative counters since server start - `routing_decisions_total` - Routing decisions made by `collection.router` dispatch - `routing_switches_total` - Routing decisions that changed a conversation's routed model (a proxy for route ping-pong; conversations are identified by a hash of the system prompt and first user message) ## `GET /v1/system-stats` Current host resource usage as measured by the Lemonade Server process. This endpoint is useful for first-party clients and dashboards that need lightweight runtime telemetry without scraping Prometheus. ### Parameters This endpoint does not take any parameters. ### Example request ```bash curl http://localhost:13305/v1/system-stats ``` ### Response format ```json { "cpu_percent": 12.3, "memory_gb": 8.4, "gpu_percent": 45.0, "vram_gb": 2.1, "npu_percent": null } ``` **Field Descriptions:** - `cpu_percent` - System CPU utilization percentage, or `null` when unavailable - `memory_gb` - System RAM currently in use, in GiB - `gpu_percent` - GPU utilization percentage, or `null` when unavailable - `vram_gb` - GPU memory currently in use, in GiB, or `null` when unavailable - `npu_percent` - NPU utilization percentage, or `null` when unavailable GPU, VRAM, and NPU telemetry availability depends on the operating system and installed drivers. Unsupported values are returned as `null`. ## `GET /metrics` Prometheus scrape endpoint for Lemonade Server. The endpoint returns Prometheus text exposition format and is intended to be scraped by Prometheus, not by Grafana directly. Unlike most Lemonade API endpoints, `/metrics` is root-level only. It is not mounted under `/api/v0/`, `/api/v1/`, `/v0/`, or `/v1/`. `HEAD /metrics` is also supported and returns `200 OK` with an empty body. ### Authentication If `LEMONADE_API_KEY` is set, `/metrics` requires bearer authentication. Either the regular API key or `LEMONADE_ADMIN_API_KEY` is accepted. If only `LEMONADE_ADMIN_API_KEY` is set and `LEMONADE_API_KEY` is unset, `/metrics` is accessible without authentication, matching regular API endpoint behavior. ### Polling and Refresh Rate The `/metrics` endpoint has no internal refresh timer. It renders the latest server state at the moment it is scraped. Polling frequency is configured in Prometheus via `scrape_interval`, for example: ```yaml global: scrape_interval: 10s ``` Grafana queries Prometheus. Grafana's dashboard refresh controls how often panels query Prometheus, but it does not control how often Prometheus scrapes Lemonade. ### Example request ```bash curl http://localhost:13305/metrics ``` With API-key auth: ```bash curl http://localhost:13305/metrics \ -H "Authorization: Bearer $LEMONADE_API_KEY" ``` ### Response format The response uses Prometheus text exposition format: ```text # HELP lemonade_server_up Whether the Lemonade server is running. # TYPE lemonade_server_up gauge lemonade_server_up 1 # HELP lemonade_server_info Lemonade server build information. # TYPE lemonade_server_info gauge lemonade_server_info{version="10.4.0"} 1 ``` Content type: ```text text/plain; version=0.0.4; charset=utf-8 ``` ### Lemonade Metric Families The authoritative metric-family list is generated by the `/metrics` implementation in [`src/cpp/server/server.cpp`](../../src/cpp/server/server.cpp). Search for `handle_metrics` and `metrics.describe(...)` to see the current names, types, labels, and descriptions. Unsupported, unavailable, null, NaN, and infinity values are omitted rather than emitted as samples. ### llama.cpp Backend Metrics When a loaded model uses the `llamacpp` recipe, Lemonade makes a best-effort scrape of the loaded backend process's private `/metrics` endpoint. Backend scrape failures do not fail the Lemonade `/metrics` response. Scraped llama.cpp metrics are normalized under the `lemonade_llamacpp_*` prefix and labeled with the same Lemonade model metadata used by `lemonade_model_info`. Lemonade starts llama.cpp backends with metrics enabled so these backend metrics are available whenever the backend supports them. ## `GET /v1/system-info` System information endpoint that provides complete hardware details and device enumeration. ### Example request ```bash curl "http://localhost:13305/v1/system-info" ``` ### Response format ``` /* Detailed source-code truncated for AI context efficiency. */ ``` **Field Descriptions:** - **System fields:** - `OS Version` - Operating system name and version - `Processor` - CPU model name - `Physical Memory` - Total RAM - `OEM System` - System/laptop model name (Windows only) - `BIOS Version` - BIOS information (Windows only) - `CPU Max Clock` - Maximum CPU clock speed (Windows only) - `Windows Power Setting` - Current power plan (Windows only) - `model_storage` - Drive-level storage information for the active configured model storage path. Values are reported in bytes for storage meters; this is not a recursive sum of Lemonade model files. - `path` - Active model storage path from server configuration - `used_bytes` - Used bytes on the model-storage drive - `total_bytes` - Total capacity of the model-storage drive - `free_bytes` - Free bytes available to the Lemonade Server process on the model-storage drive - `devices` - Hardware devices detected on the system (no software/support information) - `cpu` - CPU information (name, cores, threads) - `amd_gpu` - Array of AMD GPUs, both integrated and discrete (if present) - `nvidia_gpu` - Array of NVIDIA GPUs (if present) - `amd_npu` - AMD NPU device (if present) - `recipes` - Software recipes and their backend support status - Each recipe (e.g., `llamacpp`, `whispercpp`, `flm`) contains: - `default_backend` - Preferred backend selected by server policy for this system (present when at least one backend is not `unsupported`) - `backends` - Available backends for this recipe - Each backend contains: - `devices` - List of devices **on this system** that support this backend (empty if not supported) - `state` - Backend lifecycle state: `unsupported`, `installable`, `update_required`, or `installed` - `message` - Human-readable status text for GUI and CLI users. Required for `unsupported`, `installable`, and `update_required`; empty for `installed`. - `action` - Actionable user instruction string. For install/update cases this is typically an exact CLI command; for other states it may be empty or another actionable value (for example, a URL). - `version` - Installed or configured backend version (when available) - `cloud` - Cloud OpenAI-compatible providers configured on this server (omitted when no providers are installed). Contains: - `providers` - Array, one entry per installed provider: - `name` - Provider name used as the model-name prefix (e.g. `fireworks`). - `base_url` - Persisted base URL from `config.json`. - `env_var` - Canonical environment variable name for this provider's API key (e.g. `LEMONADE_FIREWORKS_API_KEY`). The variable's *name* is reported, never its value. - `env_var_set` - `true` if the env var is set in `lemond`'s environment. - `runtime_key_set` - `true` if an in-memory key has been supplied via `POST /v1/cloud/auth` this session. - `models_discovered` - Number of chat-capable models currently in the catalog for this provider. ## `POST /v1/install` Install or update a backend for a specific recipe/backend pair, **or** register a cloud OpenAI-compatible provider. The request body is dispatched by the `backend` field: any value other than `"cloud"` is treated as a local backend install. ### Install a local backend If the backend is already installed but outdated, this endpoint updates it to the configured version. | Parameter | Required | Description | |-----------|----------|-------------| | `recipe` | Yes | Recipe name (for example, `llamacpp`, `flm`, `whispercpp`, `sd-cpp`, `ryzenai-llm`) | | `backend` | Yes | Backend name within the recipe (for example, `vulkan`, `rocm`, `cpu`, `default`) | | `stream` | No | If `true`, returns Server-Sent Events with progress. Defaults to `false`. | | `force` | No | If `true`, bypasses hardware filtering for `unsupported` backends and attempts installation anyway. Defaults to `false`. | Example request: ```bash curl -X POST http://localhost:13305/v1/install \ -H "Content-Type: application/json" \ -d '{ "recipe": "llamacpp", "backend": "vulkan", "stream": false }' ``` Response format: ```json { "status":"success", "recipe":"llamacpp", "backend":"vulkan" } ``` In case of an error, returns an `error` field with details. ### Install a cloud provider Registers an OpenAI-compatible chat provider. The base URL is persisted to `config.json`; the optional `api_key` lives in `lemond` process memory only (cleared on restart). See the [Cloud Offload guide](../guide/configuration/cloud.md) for the full workflow. | Parameter | Required | Description | |-----------|----------|-------------| | `backend` | Yes | Must be the literal string `"cloud"`. | | `provider` | Yes | Short identifier (e.g. `fireworks`). Used as the model-name prefix. | | `base_url` | Yes | OpenAI-compatible base URL ending in `/v1` (or equivalent). | | `api_key` | No | Optional. If set, stored in process memory; honors env-wins precedence (see `/v1/cloud/auth`). | Example request: ```bash curl -X POST http://localhost:13305/v1/install \ -H "Content-Type: application/json" \ -d '{ "backend": "cloud", "provider": "fireworks", "base_url": "https://api.fireworks.ai/inference/v1" }' ``` Response format: ```json { "status": "success", "backend": "cloud", "provider": "fireworks", "base_url": "https://api.fireworks.ai/inference/v1", "models_discovered": 12, "auth_state": { "env_var_set": true, "runtime_key_set": false } } ``` `models_discovered` is `0` when no API key is resolvable. If `api_key` is supplied but the provider's env var is also set, the response includes a `warning` string explaining the env var took precedence. ## `POST /v1/uninstall` Uninstall a backend for a specific recipe/backend pair, **or** remove a cloud provider. Dispatched by the `backend` field, mirroring `/v1/install`. ### Uninstall a local backend If loaded models are using that backend, they are unloaded first. | Parameter | Required | Description | |-----------|----------|-------------| | `recipe` | Yes | Recipe name | | `backend` | Yes | Backend name | Example request: ```bash curl -X POST http://localhost:13305/v1/uninstall \ -H "Content-Type: application/json" \ -d '{ "recipe": "llamacpp", "backend": "vulkan" }' ``` Response format: ```json { "status":"success", "recipe":"llamacpp", "backend":"vulkan" } ``` In case of an error, returns an `error` field with details. ### Uninstall a cloud provider Removes the provider record from `config.json`, drops its in-memory API key (if any), and evicts every discovered model for that provider from the cache. Returns 404 if the provider was never installed. | Parameter | Required | Description | |-----------|----------|-------------| | `backend` | Yes | Must be the literal string `"cloud"`. | | `provider` | Yes | Installed provider name. | Example request: ```bash curl -X POST http://localhost:13305/v1/uninstall \ -H "Content-Type: application/json" \ -d '{ "backend": "cloud", "provider": "fireworks" }' ``` Response format: ```json { "status": "success", "backend": "cloud", "provider": "fireworks", "models_evicted": 12 } ``` ## `POST /v1/cloud/auth` Set an in-memory API key for a previously-installed cloud provider, and trigger a refresh of that provider's discovered model list. The key lives in `lemond` process memory only — it is never written to disk and is cleared on `lemond` restart. For persistence across restarts, set `LEMONADE__API_KEY` in `lemond`'s environment instead. ### Authentication precedence If `LEMONADE__API_KEY` is set in `lemond`'s environment, the env var takes precedence and this endpoint returns **409 Conflict** without storing the supplied key. This is the safety guarantee that lets an operator provision a "house" key via env without worrying about a client silently overriding it. ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `provider` | Yes | Installed provider name. | | `api_key` | Yes | API key to store in `lemond` process memory. | ### Example request ```bash curl -X POST http://localhost:13305/v1/cloud/auth \ -H "Content-Type: application/json" \ -d '{ "provider": "fireworks", "api_key": "fw-XXXXX" }' ``` ### Response format (success — 200) ```json { "provider": "fireworks", "auth_state": { "env_var_set": false, "runtime_key_set": true }, "models_discovered": 12 } ``` ### Response format (env-var conflict — 409) ```json { "error": { "type": "auth_conflict", "env_var": "LEMONADE_FIREWORKS_API_KEY", "message": "LEMONADE_FIREWORKS_API_KEY is set in the lemond process; the env var takes precedence and the supplied API key was not stored." } } ``` ### Other error responses | Status | Cause | |---|---| | `400` | Body is missing `provider` or `api_key`, or one of them is empty. | | `404` | Provider is not installed. Call `POST /v1/install` with `backend:"cloud"` first. | ## `DELETE /v1/cloud/auth/{provider}` Clear the in-memory API key for a provider. Any env-var-based key (`LEMONADE__API_KEY`) remains in effect. If no env-var key is set, the provider's discovered models are evicted from the catalog since they are no longer authenticatable. ### Example request ```bash curl -X DELETE http://localhost:13305/v1/cloud/auth/fireworks ``` ### Response format ```json { "provider": "fireworks", "cleared_runtime_key": true, "auth_state": { "env_var_set": false, "runtime_key_set": false } } ``` `cleared_runtime_key` is `false` when no in-memory key was present (e.g., the only key was from the env var). ## Log Streaming API (WebSocket) Stream server logs over WebSocket. Clients connect, send a subscribe message, and receive a snapshot of recent log history followed by live log entries as they occur. ### Connection The WebSocket server shares the same port as the [Realtime Audio Transcription API](./openai.md#ws-realtime). Discover the port via the [`/v1/health`](#get-v1health) endpoint (`websocket_port` field), then connect: ``` ws://localhost:/logs/stream ``` After connecting, send a `logs.subscribe` message to start receiving logs. ### Client → Server Messages | Message Type | Description | |--------------|-------------| | `logs.subscribe` | Subscribe to log stream. Optional `after_seq` field to resume from a specific sequence number. | ### Server → Client Messages | Message Type | Description | |--------------|-------------| | `logs.snapshot` | Initial batch of retained log entries (up to 5000). Sent once after subscribing. | | `logs.entry` | A single live log entry. Sent as new log lines are emitted. | | `error` | Error message (e.g., invalid subscribe request). | ### Example: Subscribe to Logs Subscribe from the beginning (full backlog): ```json { "type": "logs.subscribe", "after_seq": null } ``` Resume after a known sequence number (e.g., on reconnect): ```json { "type": "logs.subscribe", "after_seq": 1042 } ``` ### Example: Snapshot Response ```json { "type": "logs.snapshot", "entries": [ { "seq": 1, "timestamp": "2025-03-30 14:22:01.123", "severity": "Info", "tag": "Server", "line": "2025-03-30 14:22:01.123 [Info] (Server) Starting Lemonade Server..." } ] } ``` ### Example: Live Entry ```json { "type": "logs.entry", "entry": { "seq": 1043, "timestamp": "2025-03-30 14:22:05.456", "severity": "Info", "tag": "Router", "line": "2025-03-30 14:22:05.456 [Info] (Router) Model loaded successfully" } } ``` ### Log Entry Fields | Field | Type | Description | |-------|------|-------------| | `seq` | integer | Monotonically increasing sequence number. Use for dedup and resume. | | `timestamp` | string | Formatted timestamp from the log system. | | `severity` | string | Log level: `Trace`, `Debug`, `Info`, `Warning`, `Error`, `Fatal`. | | `tag` | string | Log source tag (e.g., `Server`, `Router`, component name). | | `line` | string | The full formatted log line. | ### Integration Notes - **Reconnection**: Track the last `seq` received and pass it as `after_seq` on reconnect to avoid duplicate entries. - **Backlog**: The server retains up to 5000 recent log entries. The snapshot may be smaller if fewer entries exist. - **Platform availability**: WebSocket log streaming is available on all platforms (Windows, Linux, and macOS). ## `GET /live` Lightweight liveness probe for load balancers and orchestrators. Unlike [`/v1/health`](#get-v1health), this endpoint does no work beyond confirming the process is up — it does not inspect loaded models or backends — so it is safe to poll at high frequency. `HEAD /live` is also supported and returns `200 OK` with an empty body. Unlike the other endpoints on this page, `/live` is not versioned and is not mounted under the `/api/v0/`, `/api/v1/`, `/v0/`, `/v1/` prefixes. ### Example request ```bash curl http://localhost:13305/live ``` ### Response format ```json {"status":"ok"} ``` ## Job Engine API Run client-posted sequences of server operations as durable, background **jobs** — steps that pass data forward, branch on results, and have a pause / interrupt / resume / delete / query lifecycle that survives client disconnect and server restart. Exclusive ops (`load`/`unload`/`chat`) hold a Router slot so normal traffic queues behind a running job. | Method | Path | Purpose | |--------|------|---------| | `POST` | `/v1/jobs` | Create a job from `{name, definition:{steps} \| steps, inputs}`; returns `202 {"id"}`, or `400` on an invalid step graph. | | `GET` | `/v1/jobs` | List job summaries. | | `GET` | `/v1/jobs/{id}` | Full job record (status, per-step state, context). | | `POST` | `/v1/jobs/{id}/pause` | Stop after the current step. | | `POST` | `/v1/jobs/{id}/interrupt` | Cancel the current step now; resumable. | | `POST` | `/v1/jobs/{id}/resume` | Continue a paused/interrupted job. | | `DELETE` | `/v1/jobs/{id}` | Remove a job. | See [`docs/dev/job-system.md`](../dev/job-system.md) for the step schema, op set, and lifecycle, and [`docs/dev/job-expression-language.md`](../dev/job-expression-language.md) for the `when`/`branch` expression grammar. ## Internal Endpoints Internal endpoints are used for server control and configuration. By default, they are secured by `LEMONADE_ADMIN_API_KEY` (if set) to separate control privileges from standard inference operations. ## `POST /internal/telemetry/flush` Forces the in-memory telemetry queue to flush all buffered trace spans immediately to the configured OTLP collector. This call blocks until all currently queued spans are serialized and sent. #### Parameters None. Example request: ```bash curl -X POST http://localhost:13305/internal/telemetry/flush ``` #### Response Format Returns a JSON object indicating successful completion of the flush operation: ```json { "status": "flushed" } ``` ## `GET /internal/aliases` Retrieves a list of all active model alias mappings. #### Parameters None. Example request: ```bash curl http://localhost:13305/internal/aliases ``` #### Response Format Returns a JSON object containing an array of active alias objects: ```json { "aliases": [ { "alias": "my-alias-1", "target": "user.custom-llama", "downloaded": true, "recipe": "llamacpp" } ] } ``` ## `POST /internal/aliases` Binds a model alias to a target model name. #### Parameters | Field | Type | Required | Description | |-------|------|----------|-------------| | `alias` | string | yes | The alias name to create or update. | | `target` | string | yes | The target model name or canonical ID (also accepted as `model`). | Example request: ```bash curl -X POST http://localhost:13305/internal/aliases \ -H "Content-Type: application/json" \ -d '{ "alias": "my-alias-1", "target": "user.custom-llama" }' ``` #### Response Format Returns a JSON object confirming the alias binding: ```json { "status": "ok", "alias": "my-alias-1", "target": "user.custom-llama" } ``` Returns HTTP `400 Bad Request` if required fields are missing or invalid. ## `DELETE /internal/aliases/{alias}` Removes an existing model alias binding by name. #### Parameters | Path Parameter | Type | Description | |----------------|------|-------------| | `alias` | string | The alias name to remove. | Example request: ```bash curl -X DELETE http://localhost:13305/internal/aliases/my-alias-1 ``` #### Response Format Returns a JSON object confirming deletion: ```json { "status": "deleted", "alias": "my-alias-1" } ``` Returns HTTP `404 Not Found` if the alias does not exist. --- ## File: docs/api/llamacpp.md # llama.cpp-Specific API This page documents Lemonade's llama.cpp-specific compatibility surface. ## Summary | Method | Endpoint | Description | Modality | |--------|----------|-------------|----------| | `POST` | [`/v1/rerank`](#post-v1rerank) | Reranking | query + documents -> relevance-scored documents | | `GET` | [`/v1/slots`](#get-v1slots) | Returns the current slots processing state | slots state | | `POST` | [`/v1/slots/{id}?action=save`](#post-v1slotsidactionsave) | Save the prompt cache of the specified slot to a file | prompt cache | | `POST` | [`/v1/slots/{id}?action=restore`](#post-v1slotsidactionrestore) | Restore the prompt cache of the specified slot from a file | prompt cache | | `POST` | [`/v1/slots/{id}?action=erase`](#post-v1slotsidactionerase) | Erase the prompt cache of the specified slot | prompt cache | | `POST` | [`/v1/tokenize`](#post-v1tokenize) | Tokenize a given text | tokenization | ## `POST /v1/rerank` Reranking API for llama.cpp-compatible reranker models. You provide a query and a list of documents, and receive relevance scores for each document. Lemonade will load the requested model automatically if it is not already loaded. > **Note:** This endpoint is part of Lemonade's llama.cpp compatibility layer. Internally, Lemonade forwards the request to llama.cpp's `/v1/rerank` endpoint. > **Note:** Lemonade also accepts `/reranking` and `/reranker` as aliases — all three route to the same handler and behave identically. > **Note:** The endpoint is available under all four path prefixes: `/api/v0/`, `/api/v1/`, `/v0/`, and `/v1/`. > **Note:** This endpoint is only available for reranker-specific models using the `llamacpp` recipe, such as `bge-reranker-v2-m3-GGUF`. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `query` | Yes | The search query text. | | | `documents` | Yes | Array of document strings to score against the query. | | | `model` | Yes | The reranking model to use. If not already loaded, Lemonade loads it before forwarding the request. | | ### Example request === "PowerShell" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/v1/rerank" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{ "model": "bge-reranker-v2-m3-GGUF", "query": "What is the capital of France?", "documents": [ "Paris is the capital of France.", "Berlin is the capital of Germany.", "Madrid is the capital of Spain." ] }' -UseBasicParsing ``` === "Bash" ```bash curl -X POST http://localhost:13305/v1/rerank \ -H "Content-Type: application/json" \ -d '{ "model": "bge-reranker-v2-m3-GGUF", "query": "What is the capital of France?", "documents": [ "Paris is the capital of France.", "Berlin is the capital of Germany.", "Madrid is the capital of Spain." ] }' ``` ### Response format ```json { "model": "bge-reranker-v2-m3-GGUF", "object": "list", "results": [ { "index": 0, "relevance_score": 8.60673713684082 }, { "index": 1, "relevance_score": -5.3886260986328125 }, { "index": 2, "relevance_score": -3.555561065673828 } ], "usage": { "prompt_tokens": 51, "total_tokens": 51 } } ``` **Field Descriptions:** - `model` - Model identifier used for reranking - `object` - Type of response object, always `"list"` - `results` - Array of all input documents with relevance scores - `index` - Original index of the document in the input array - `relevance_score` - Relevance score assigned by the model; higher means more relevant - `usage` - Token usage statistics - `prompt_tokens` - Number of tokens in the input - `total_tokens` - Total tokens processed > **Note:** Results are returned in input order. To rank documents by relevance, sort `results` by `relevance_score` in descending order on the client side. ## `GET /v1/slots` Returns the current state of all processing slots in the llama.cpp server. Slots are parallel processing contexts that can handle multiple requests concurrently. > **Note:** This endpoint is part of Lemonade's llama.cpp compatibility layer. Internally, Lemonade forwards the request to llama.cpp's `/slots` endpoint. > **Note:** This endpoint is only available when a llama.cpp model is loaded. > **Note:** This endpoint supports all four path prefixes: `/api/v0/slots`, `/api/v1/slots`, `/v0/slots`, and `/v1/slots`. ### Parameters This endpoint accepts no parameters. ### Example request === "PowerShell" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/v1/slots" ` -Method GET -UseBasicParsing ``` === "Bash" ```bash curl http://localhost:13305/v1/slots ``` ### Response format ```json [ { "id": 0, "state": "idle", "next_token": { "has_next_token": false, "n_remain": 0, "n_decoded": 0 }, "task_id": -1, "cache_tokens": 1024 }, { "id": 1, "state": "processing", "next_token": { "has_next_token": true, "n_remain": 42, "n_decoded": 15 }, "task_id": 123, "cache_tokens": 512 } ] ``` **Field Descriptions:** - `id` - Unique identifier for the slot - `state` - Current processing state ("idle", "processing", etc.) - `next_token` - Information about token generation state - `has_next_token` - Whether more tokens are expected - `n_remain` - Number of tokens remaining to generate - `n_decoded` - Number of tokens already decoded - `task_id` - Identifier of the current task being processed (-1 if idle) - `cache_tokens` - Number of cached tokens in the slot's prompt cache ## `POST /v1/slots/{id}?action=save` Save the prompt cache of a specific slot to a file. This allows you to persist the current context state for later restoration. > **Note:** This endpoint is part of Lemonade's llama.cpp compatibility layer. Internally, Lemonade forwards the request to llama.cpp's `/slots/{id}?action=save` endpoint. > **Note:** The llama.cpp server must be started with the `--slot-save-path` argument for save operations to work. See [Server Configuration](../guide/configuration/README.md) for details on configuring backend arguments. > > Example configuration: > ```bash > lemonade config set llamacpp.args="--slot-save-path /path/to/slot/saves" > ``` > **Note:** This endpoint supports all four path prefixes: `/api/v0/slots/{id}`, `/api/v1/slots/{id}`, `/v0/slots/{id}`, and `/v1/slots/{id}`. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `id` | Yes | The slot ID to save (path parameter). | | | `filename` | Yes | The filename where the slot cache should be saved (JSON body). | | ### Example request === "PowerShell" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/v1/slots/0?action=save" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{"filename": "my_conversation_cache.bin"}' -UseBasicParsing ``` === "PowerShell (/api/v1)" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/api/v1/slots/0?action=save" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{"filename": "my_conversation_cache.bin"}' -UseBasicParsing ``` === "Bash" ```bash curl -X POST "http://localhost:13305/v1/slots/0?action=save" \ -H "Content-Type: application/json" \ -d '{"filename": "my_conversation_cache.bin"}' ``` ### Response format ```json { "id_slot": 0, "filename": "my_conversation_cache.bin", "n_saved": 1024 } ``` **Field Descriptions:** - `id_slot` - The slot ID that was saved - `filename` - The filename where the cache was saved - `n_saved` - Number of tokens saved to the cache file ## `POST /v1/slots/{id}?action=restore` Restore the prompt cache of a specific slot from a previously saved file. This allows you to resume a conversation or context from where you left off. > **Note:** This endpoint is part of Lemonade's llama.cpp compatibility layer. Internally, Lemonade forwards the request to llama.cpp's `/slots/{id}?action=restore` endpoint. > **Note:** The llama.cpp server must be started with the `--slot-save-path` argument for restore operations to work. > **Note:** This endpoint supports all four path prefixes: `/api/v0/slots/{id}`, `/api/v1/slots/{id}`, `/v0/slots/{id}`, and `/v1/slots/{id}`. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `id` | Yes | The slot ID to restore to (path parameter). | | | `filename` | Yes | The filename from which to restore the slot cache (JSON body). | | ### Example request === "PowerShell" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/v1/slots/0?action=restore" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{"filename": "my_conversation_cache.bin"}' -UseBasicParsing ``` === "PowerShell (/api/v1)" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/api/v1/slots/0?action=restore" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{"filename": "my_conversation_cache.bin"}' -UseBasicParsing ``` === "Bash" ```bash curl -X POST "http://localhost:13305/v1/slots/0?action=restore" \ -H "Content-Type: application/json" \ -d '{"filename": "my_conversation_cache.bin"}' ``` ### Response format ```json { "id_slot": 0, "filename": "my_conversation_cache.bin", "n_restored": 1024 } ``` **Field Descriptions:** - `id_slot` - The slot ID that was restored - `filename` - The filename from which the cache was restored - `n_restored` - Number of tokens restored from the cache file ## `POST /v1/slots/{id}?action=erase` Erase (clear) the prompt cache of a specific slot. This removes all cached context from the slot, resetting it to an empty state. > **Note:** This endpoint is part of Lemonade's llama.cpp compatibility layer. Internally, Lemonade forwards the request to llama.cpp's `/slots/{id}?action=erase` endpoint. > **Note:** This endpoint supports all four path prefixes: `/api/v0/slots/{id}`, `/api/v1/slots/{id}`, `/v0/slots/{id}`, and `/v1/slots/{id}`. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `id` | Yes | The slot ID to erase (path parameter). | | ### Example request === "PowerShell" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/v1/slots/0?action=erase" ` -Method POST -UseBasicParsing ``` === "PowerShell (/api/v1)" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/api/v1/slots/0?action=erase" ` -Method POST -UseBasicParsing ``` === "Bash" ```bash curl -X POST "http://localhost:13305/v1/slots/0?action=erase" ``` ### Response format ```json { "id_slot": 0 } ``` **Field Descriptions:** - `id_slot` - The slot ID that was erased > **Note:** If the server returns an error, it may indicate that the slot was not found or that the operation failed. ## `POST /v1/tokenize` Tokenize a given text. Does not count towards the current model's context window. > **Note:** This endpoint is part of Lemonade's llama.cpp compatibility layer. Internally, Lemonade forwards the request to llama.cpp's `/tokenize` endpoint. > **Note:** This endpoint supports all four path prefixes: `/api/v0/tokenize`, `/api/v1/tokenize`, `/v0/tokenize`, and `/v1/tokenize`. > **Note:** Actual response values may vary for the same string across different models if the models do not share the same tokenizer. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `content` | Yes | The text to tokenize. | | | `add_special` | No | Boolean indicating if special tokens, i.e. `BOS`, should be inserted. Default: `false` | | | `parse_special` | No | Boolean indicating if special tokens should be tokenized. When `false` special tokens are treated as plaintext. Default: `true` | | | `with_pieces` | No | Boolean indicating whether to return token pieces along with IDs. Default: `false` | | ### Example request === "PowerShell" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/v1/tokenize" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{"content": "This is a string to tokenize"}' -UseBasicParsing ``` === "PowerShell (/api/v1)" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/api/v1/tokenize" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{"content": "This is a string to tokenize"}' -UseBasicParsing ``` === "Bash" ```bash curl -X POST "http://localhost:13305/v1/tokenize" \ -H "Content-Type: application/json" \ -d '{"content": "This is a string to tokenize"}' ``` ### Response format ```json { "tokens": [1919,369,264,886,310,74995] } ``` If `with_pieces` is `true`: ```json { "tokens": [ {"id": 123, "piece": "Hello"}, {"id": 456, "piece": " world"}, {"id": 789, "piece": "!"} ] } ``` **Field Descriptions:** - `tokens` - Array of token IDs --- ## File: docs/api/mcp.md # MCP Gateway Lemonade exposes its inference capabilities as a Model Context Protocol (MCP) server, so any MCP-compatible client (GitHub Copilot, Claude Desktop, MCP Inspector, Cursor, the `mcp` Python client, etc.) can call your locally running models as tools. The gateway implements the **MCP "Streamable HTTP" transport** (spec version `2025-06-18`) with the `tools` capability only. All traffic flows through a single endpoint: | Endpoint | Status | Notes | |----------|--------|-------| | `POST /mcp` | Supported | JSON-RPC 2.0 envelope. Accepts a single message or a batch array. | | `GET /mcp` | `405 Method Not Allowed` | Server-initiated SSE channel is not supported. | > **Why a single path?** The MCP specification mandates one endpoint URL per server, so `/mcp` is an intentional exception to Lemonade's quad-prefix convention. ## Authentication `/mcp` is treated as a regular API route, so it honors `LEMONADE_API_KEY` exactly like `/api/v1/chat/completions`: ```bash curl -s http://localhost:13305/mcp \ -H "Authorization: Bearer $LEMONADE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"ping"}' ``` ## Supported methods | Method | Purpose | |--------|---------| | `initialize` | Negotiate protocol version, return server identity and capabilities. | | `notifications/initialized` | Client acknowledgement; silently accepted. | | `tools/list` | Return the catalogue of callable tools (with JSON Schemas). | | `tools/call` | Invoke one of the tools below. | | `ping` | Liveness probe; returns `{}`. | ## Tools All tools auto-load (and download, if missing) the requested model on first call, exactly like `POST /v1/chat/completions`. Errors are returned as MCP results with `"isError": true` rather than JSON-RPC errors, matching the spec's guidance for tool failures. ### `lemonade_list_models` Discover what's loaded, what's downloaded, and what's recommended. Call this first if you don't already know the exact model name to pass to the other tools — passing a wrong name may trigger a multi-GB download. ```json { "name": "lemonade_list_models", "arguments": { "include_available": true, "include_suggested": true } } ``` Returns a summary text block plus a JSON-stringified text block with `{loaded, available, suggested_to_pull, recommended_chat_model}`. ### `lemonade_chat` Chat completion against any LLM in the registry. ```json { "name": "lemonade_chat", "arguments": { "model": "Qwen3-1.7B-GGUF", "messages": [ {"role": "system", "content": "You are concise."}, {"role": "user", "content": "Summarize MCP in one line."} ], "max_tokens": 64, "temperature": 0.2 } } ``` Returns one text block with the assistant content. If the model emits tool calls, a second text block containing `tool_calls: ` is appended. Reasoning models (Qwen3, DeepSeek-R1, ...) have the `` block disabled by default to keep small `max_tokens` budgets from being consumed by reasoning. Pass `"chat_template_kwargs": {"enable_thinking": true}` to opt back in. > **Picking a portable model.** The example above uses `Qwen3-1.7B-GGUF` because GGUF (llama.cpp) runs everywhere lemonade does — Windows, Linux/Docker, macOS, CPU and Vulkan/ROCm/Metal GPUs. Hybrid/NPU variants such as `*-Hybrid` (recipe `ryzenai-llm`, **Windows + AMD RyzenAI** only) or `*-FLM` (recipe `flm`, **AMD Ryzen AI NPU** only) are faster on supported hardware but unavailable on others. If your client picks one that isn't supported, the tool returns a structured error suggesting a portable alternative — prefer `lemonade_list_models` to discover what's actually available on the running server. ### `lemonade_transcribe_audio` Transcribe a local audio file with a Whisper-class model. Prefer `audio_path` (the server runs on the same machine as the caller); use `audio_base64` only when you genuinely have bytes in memory. ```json { "name": "lemonade_transcribe_audio", "arguments": { "model": "Whisper-Large-v3-Turbo", "audio_path": "C:/clips/meeting.wav", "response_format": "verbose_json" } } ``` Returns two text blocks: the bare transcript, followed by the full OpenAI-shaped response (for callers that need timestamps or segments). ### `lemonade_generate_image` Generate one or more PNGs from a prompt. **Prefer writing to disk** via `output_path` (single image) or `output_dir` (one or more) — base64 image content blocks cost tens of thousands of tokens per image and some clients surface them as opaque resource URIs. ```json { "name": "lemonade_generate_image", "arguments": { "model": "SDXL-Turbo", "prompt": "a lemon-shaped car driving across the moon", "size": "512x512", "output_path": "lemon-car.png" } } ``` When disk paths are provided, returns text block(s) with the absolute path(s). Otherwise, returns one inline image content block per image (`{"type":"image", "data":"", "mimeType":"image/png"}`). **Sandboxed disk writes.** To prevent a cross-origin or unauthenticated caller from overwriting arbitrary files, `output_path` and `output_dir` are confined to a sandbox directory: - Default: `/mcp-images`. - Override with the `LEMONADE_MCP_IMAGE_DIR` environment variable (absolute path). - Relative paths resolve against the sandbox root; absolute paths must stay within it. Paths that escape the sandbox (via `..` or symlinks) are rejected. - `output_dir` writes use auto-generated, unique filenames (`image__.png`), so concurrent callers never clobber one another's images — the returned `paths` tell you the exact names. Use `output_path` when you need an exact, caller-chosen filename (it is written as named, replacing any existing file at that path). ### `lemonade_omni` One-shot multimodal turn against a **Lemonade Omni collection** (a model bundle that pairs a planner LLM with an image model, an image-edit model, and a TTS voice under a single `collection.omni` recipe — see [the Omni docs](../dev/lemonade-omni.md)). The server runs the orchestrator's internal tool-calling loop, executes the collection's `generate_image` / `edit_image` / `text_to_speech` tools by routing to the bundled components, and returns the result as a text block plus native MCP `image` / `audio` content blocks — one per artifact, in the order they were produced. `model` is **optional** and defaults to `LMX-Omni-5.5B-Lite` (smaller and faster). Pass `model` explicitly to opt into a larger collection (e.g. `LMX-Omni-52B-Halo` on capable hardware) or any other `collection.omni` model surfaced by `lemonade_list_models`. The collection is downloaded on first use and may be multi-GB. Use `lemonade_chat` instead when you only need plain-text LLM output and don't want the planner-loop overhead. ```json { "name": "lemonade_omni", "arguments": { "messages": [ {"role": "user", "content": "Generate an image of a lemon car, then read out a one-line description."} ], "output_dir": "omni" } } ``` **Disk vs. inline output.** A single Omni turn can produce both images and audio in arbitrary order. Pass an `output_dir` to write each artifact to disk under a unique auto-generated name (`omni__.`) — the tool returns one text block per artifact with its absolute path, plus a JSON-stringified `paths` array. This is strongly preferred over inline base64 for the same reasons documented under `lemonade_generate_image` — and is the **only** way to get audio out on clients that don't render `audio` content blocks. Like `lemonade_generate_image`, `output_dir` is confined to the MCP image sandbox (see **Sandboxed disk writes** above): relative paths resolve against the sandbox root, paths escaping it are rejected, and unique filenames mean concurrent callers never clobber each other. When `output_dir` is omitted, artifacts are inlined as MCP content blocks: `{"type":"image", "data":"", "mimeType":"image/png"}` and `{"type":"audio", "data":"", "mimeType":"audio/mpeg"}`. If the planner emits app-defined tool calls (those you passed in via `tools`/`tool_choice`), an extra text block `tool_calls: ` is appended, matching `lemonade_chat`'s passthrough semantics. Passing a non-collection model (e.g. a plain LLM) returns `isError: true` with a hint to use `lemonade_chat`. ## Error model | Code | Meaning | |------|---------| | `-32700` | Body was not valid JSON. | | `-32600` | Request was not a JSON-RPC object (or batch was empty / missing `method`). | | `-32601` | Unknown JSON-RPC method (e.g. `resources/list`). | | `-32602` | Invalid `params` for a known method. | | `-32603` | Internal server error (an exception escaped a handler). | Tool-level failures (bad arguments, model load errors, backend exceptions) are returned as **successful** JSON-RPC results with `"isError": true` and a text content block describing the failure, so MCP-aware models can self-correct. ## Limitations (MVP) - No server-initiated SSE (GET /mcp returns 405). Tools return their full result in the POST response. - No session resumption (`Mcp-Session-Id` header is not issued). - `resources/*` and `prompts/*` capabilities are not implemented. - Streaming chat output is not exposed via MCP — `stream=true` is ignored. Use `POST /v1/chat/completions` directly for streamed tokens. - Embeddings and text-to-speech are not currently exposed as MCP tools; use the OpenAI-compatible endpoints (`/v1/embeddings`, `/v1/audio/speech`) for those. ## Quick test with curl ```bash # 1. Initialize curl -s http://localhost:13305/mcp -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{}}}' # 2. List tools curl -s http://localhost:13305/mcp -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' # 3. Call lemonade_chat curl -s http://localhost:13305/mcp -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"lemonade_chat","arguments":{"model":"Qwen3-1.7B-GGUF","messages":[{"role":"user","content":"hi"}],"max_tokens":16}}}' ``` --- ## File: docs/api/ollama.md # Ollama-Compatible API Lemonade supports the [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), allowing applications built for Ollama to work with Lemonade without modification. To enable auto-detection by Ollama-integrated apps, configure the server to use the Ollama default port `11434`. See [Server Configuration](../guide/configuration/README.md#settings-reference) for how to change the port. | Endpoint | Status | Notes | |----------|--------|-------| | `POST /api/chat` | Supported | Streaming and non-streaming | | `POST /api/generate` | Supported | Text completion + image generation | | `GET /api/tags` | Supported | Lists downloaded models | | `POST /api/show` | Supported | Model details | | `DELETE /api/delete` | Supported | | | `POST /api/pull` | Supported | Download with progress | | `POST /api/embed` | Supported | New embeddings format | | `POST /api/embeddings` | Supported | Legacy embeddings | | `GET /api/ps` | Supported | Running models | | `GET /api/version` | Supported | | | `POST /api/create` | Not supported | Returns 501 | | `POST /api/copy` | Not supported | Returns 501 | | `POST /api/push` | Not supported | Returns 501 | --- ## File: docs/api/openai.md # OpenAI-Compatible API This spec defines Lemonade's implementation of the [OpenAI API](https://developers.openai.com/api/docs). | Method | Endpoint | Description | Modality | |--------|----------|-------------|----------| | `POST` | [`/v1/chat/completions`](#post-v1chatcompletions) | Chat Completions | messages -> completion | | `POST` | [`/v1/completions`](#post-v1completions) | Text Completions | prompt -> completion | | `POST` | [`/v1/embeddings`](#post-v1embeddings) | Embeddings | text -> vector representations | | `POST` | [`/v1/responses`](#post-v1responses) | Responses API | prompt/messages -> event | | `POST` | [`/v1/audio/transcriptions`](#post-v1audiotranscriptions) | Audio Transcription | audio file -> text | | `POST` | [`/v1/audio/speech`](#post-v1audiospeech) | Text to speech | text -> audio | | `WS` | [`/realtime`](#ws-realtime) | Realtime Audio Transcription, OpenAI SDK compatible | streaming audio -> text | | `POST` | [`/v1/images/generations`](#post-v1imagesgenerations) | Image Generation | prompt -> image | | `POST` | [`/v1/images/edits`](#post-v1imagesedits) | Image Editing | image + prompt -> edited image | | `POST` | [`/v1/images/variations`](#post-v1imagesvariations) | Image Variations | image -> varied image | | `POST` | [`/v1/images/upscale`](#post-v1imagesupscale) | Image Upscaling | image + ESRGAN model -> upscaled image | | `GET` | [`/v1/models`](#get-v1models) | List models available locally | n/a | | `GET` | [`/v1/models/{model_id}`](#get-v1modelsmodel_id) | Retrieve a specific model by ID | n/a | ## `POST /v1/chat/completions` Chat Completions API. You provide a list of messages and receive a completion. This API will also load the model if it is not already loaded. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `messages` | Yes | Array of messages in the conversation. Each message should have a `role` ("user" or "assistant") and `content` (the message text). | | | `model` | Yes | The model to use for the completion. | | | `stream` | No | If true, tokens will be sent as they are generated. If false, the response will be sent as a single message once complete. Defaults to false. | | | `stop` | No | Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence. Can be a string or an array of strings. | | | `logprobs` | No | Include log probabilities of the output tokens. If true, returns the log probability of each output token. Defaults to false. | | | `temperature` | No | What sampling temperature to use. | | | `repeat_penalty` | No | Number between 1.0 and 2.0. 1.0 means no penalty. Higher values discourage repetition. | | | `top_k` | No | Integer that controls the number of top tokens to consider during sampling. | | | `top_p` | No | Float between 0.0 and 1.0 that controls the cumulative probability of top tokens to consider during nucleus sampling. | | | `tools` | No | A list of tools the model may call. | | | `max_tokens` | No | An upper bound for the number of tokens that can be generated for a completion. Mutually exclusive with `max_completion_tokens`. This value is now deprecated by OpenAI in favor of `max_completion_tokens` | | | `max_completion_tokens` | No | An upper bound for the number of tokens that can be generated for a completion. Mutually exclusive with `max_tokens`. | | ### Example request === "PowerShell" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/v1/chat/completions" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{ "model": "Qwen3-0.6B-GGUF", "messages": [ { "role": "user", "content": "What is the population of Paris?" } ], "stream": false }' ``` === "Bash" ```bash curl -X POST http://localhost:13305/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3-0.6B-GGUF", "messages": [ {"role": "user", "content": "What is the population of Paris?"} ], "stream": false }' ``` ### Response format === "Non-streaming responses" ```json { "id": "0", "object": "chat.completion", "created": 1742927481, "model": "Qwen3-0.6B-GGUF", "choices": [{ "index": 0, "message": { "role": "assistant", "content": "Paris has a population of approximately 2.2 million people in the city proper." }, "finish_reason": "stop" }] } ``` === "Streaming responses" For streaming responses, the API returns a stream of server-sent events (however, Open AI recommends using their streaming libraries for parsing streaming responses): ```json { "id": "0", "object": "chat.completion.chunk", "created": 1742927481, "model": "Qwen3-0.6B-GGUF", "choices": [{ "index": 0, "delta": { "role": "assistant", "content": "Paris" } }] } ``` ### Image understanding input format (OpenAI-compatible) To send images to `chat/completions`, pass a `messages[*].content` array that mixes `text` and `image_url` items. The image can be provided as a base64 data URL (for example, from `FileReader.readAsDataURL(...)` in web apps). #### Example request ```bash curl -X POST http://localhost:13305/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen2.5-VL-7B-Instruct", "messages": [ { "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD..."}} ] } ], "stream": false }' ``` #### Example response ```json { "id": "0", "object": "chat.completion", "created": 1742927481, "model": "Qwen2.5-VL-7B-Instruct", "choices": [{ "index": 0, "message": { "role": "assistant", "content": "The image shows a red apple resting on a wooden table." }, "finish_reason": "stop" }] } ``` ### Server-side tools > **Note:** Omni collection orchestration is a Lemonade-specific extension to the Chat Completions API; it is not part of the OpenAI specification. Requests that target ordinary (non-collection) models are unaffected and are fully OpenAI-compatible. When `model` names an Omni **collection** model (`recipe: "collection.omni"`, e.g., `LMX-Omni-52B-Halo`), this endpoint runs an internal tool-calling loop instead of a plain completion. The server injects the reference system prompt and tools, routes to the collection's chat component, executes the omni tools (image generation/editing, text-to-speech) against the matching components, and returns one OpenAI-compatible response. Generated media is embedded in the assistant `content`: - **images** → markdown `` - **speech** → `` Both non-streaming and `stream: true` are supported. In streaming mode the media arrives as a content delta on a `chat.completion.chunk` frame the moment its tool finishes. **Merge semantics.** A client-provided system prompt is prepended by the built-in omni system prompt. Client-provided `tools` are merged with the built-in omni tools. The server resolves omni tool calls internally; calls to client-provided tools are returned in a `finish_reason: "tool_calls"` response for the client to execute and resume. Targeting a collection name invokes the server-side loop; targeting a component LLM name bypasses it and returns a plain completion. See [Lemonade Omni Models](../dev/lemonade-omni.md) for details. #### Example request ```bash curl -X POST http://localhost:13305/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "LMX-Omni-52B-Halo", "messages": [{"role": "user", "content": "Draw a red apple on a table."}], "stream": false }' ``` #### Example response The image tool runs during the loop and its output is embedded as a markdown image in the assistant `content`: ```json { "id": "0", "object": "chat.completion", "created": 1742927481, "model": "LMX-Omni-52B-Halo", "choices": [{ "index": 0, "message": { "role": "assistant", "content": "Here is a red apple on a table.\n\n" }, "finish_reason": "stop" }] } ``` ## `POST /v1/completions` Text Completions API. You provide a prompt and receive a completion. This API will also load the model if it is not already loaded. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `prompt` | Yes | The prompt to use for the completion. | | | `model` | Yes | The model to use for the completion. | | | `stream` | No | If true, tokens will be sent as they are generated. If false, the response will be sent as a single message once complete. Defaults to false. | | | `stop` | No | Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence. Can be a string or an array of strings. | | | `echo` | No | Echo back the prompt in addition to the completion. Available on non-streaming mode. | | | `logprobs` | No | Include log probabilities of the output tokens. If true, returns the log probability of each output token. Defaults to false. Only available when `stream=False`. | | | `temperature` | No | What sampling temperature to use. | | | `repeat_penalty` | No | Number between 1.0 and 2.0. 1.0 means no penalty. Higher values discourage repetition. | | | `top_k` | No | Integer that controls the number of top tokens to consider during sampling. | | | `top_p` | No | Float between 0.0 and 1.0 that controls the cumulative probability of top tokens to consider during nucleus sampling. | | | `max_tokens` | No | An upper bound for the number of tokens that can be generated for a completion, including input tokens. | | ### Example request === "PowerShell" ```powershell Invoke-WebRequest -Uri "http://localhost:13305/v1/completions" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{ "model": "Qwen3-0.6B-GGUF", "prompt": "What is the population of Paris?", "stream": false }' ``` === "Bash" ```bash curl -X POST http://localhost:13305/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3-0.6B-GGUF", "prompt": "What is the population of Paris?", "stream": false }' ``` ### Response format The following format is used for both streaming and non-streaming responses: ```json { "id": "0", "object": "text_completion", "created": 1742927481, "model": "Qwen3-0.6B-GGUF", "choices": [{ "index": 0, "text": "Paris has a population of approximately 2.2 million people in the city proper.", "finish_reason": "stop" }], } ``` ## `POST /v1/embeddings` Embeddings API. You provide input text and receive vector representations (embeddings) that can be used for semantic search, clustering, and similarity comparisons. This API will also load the model if it is not already loaded. > **Note:** This endpoint is only available for models using the `llamacpp` or `flm` recipes. ONNX models (OGA recipes) do not support embeddings. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `input` | Yes | The input text or array of texts to embed. Can be a string or an array of strings. | | | `model` | Yes | The model to use for generating embeddings. | | | `encoding_format` | No | The format to return embeddings in. Supported values: `"float"` (default), `"base64"`. | | ### Example request === "PowerShell" ```powershell Invoke-WebRequest ` -Uri "http://localhost:13305/v1/embeddings" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{ "model": "nomic-embed-text-v1-GGUF", "input": ["Hello, world!", "How are you?"], "encoding_format": "float" }' ``` === "Bash" ```bash curl -X POST http://localhost:13305/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "model": "nomic-embed-text-v1-GGUF", "input": ["Hello, world!", "How are you?"], "encoding_format": "float" }' ``` ### Response format ```json { "object": "list", "data": [ { "object": "embedding", "index": 0, "embedding": [0.0234, -0.0567, 0.0891, ...] }, { "object": "embedding", "index": 1, "embedding": [0.0456, -0.0678, 0.1234, ...] } ], "model": "nomic-embed-text-v1-GGUF", "usage": { "prompt_tokens": 12, "total_tokens": 12 } } ``` **Field Descriptions:** - `object` - Type of response object, always `"list"` - `data` - Array of embedding objects - `object` - Type of embedding object, always `"embedding"` - `index` - Index position of the input text in the request - `embedding` - Vector representation as an array of floats - `model` - Model identifier used to generate the embeddings - `usage` - Token usage statistics - `prompt_tokens` - Number of tokens in the input - `total_tokens` - Total tokens processed ## `POST /v1/responses` Responses API. You provide an input and receive a response. This API will also load the model if it is not already loaded. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `input` | Yes | A list of dictionaries or a string input for the model to respond to. | | | `model` | Yes | The model to use for the response. | | | `max_output_tokens` | No | The maximum number of output tokens to generate. | | | `temperature` | No | What sampling temperature to use. | | | `repeat_penalty` | No | Number between 1.0 and 2.0. 1.0 means no penalty. Higher values discourage repetition. | | | `top_k` | No | Integer that controls the number of top tokens to consider during sampling. | | | `top_p` | No | Float between 0.0 and 1.0 that controls the cumulative probability of top tokens to consider during nucleus sampling. | | | `stream` | No | If true, tokens will be sent as they are generated. If false, the response will be sent as a single message once complete. Defaults to false. | | ### Streaming Events The Responses API uses semantic events for streaming. Each event is typed with a predefined schema, so you can listen for events you care about. Our initial implementation only offers support to: - `response.created` - `response.output_text.delta` - `response.completed` For a full list of event types, see the [API reference for streaming](https://platform.openai.com/docs/api-reference/responses-streaming). ### Example request === "PowerShell" ```powershell Invoke-WebRequest -Uri "http://localhost:13305/v1/responses" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{ "model": "Llama-3.2-1B-Instruct-Hybrid", "input": "What is the population of Paris?", "stream": false }' ``` === "Bash" ```bash curl -X POST http://localhost:13305/v1/responses \ -H "Content-Type: application/json" \ -d '{ "model": "Llama-3.2-1B-Instruct-Hybrid", "input": "What is the population of Paris?", "stream": false }' ``` ### Response format === "Non-streaming responses" ```json { "id": "0", "created_at": 1746225832.0, "model": "Llama-3.2-1B-Instruct-Hybrid", "object": "response", "output": [{ "id": "0", "content": [{ "annotations": [], "text": "Paris has a population of approximately 2.2 million people in the city proper." }] }] } ``` === "Streaming Responses" For streaming responses, the API returns a series of events. Refer to [OpenAI streaming guide](https://platform.openai.com/docs/guides/streaming-responses?api-mode=responses) for details. ## `POST /v1/audio/transcriptions` Audio Transcription API. You provide an audio file and receive a text transcription. This API will also load the model if it is not already loaded. > **Note:** This endpoint uses [whisper.cpp](https://github.com/ggerganov/whisper.cpp) as the backend. Whisper models are automatically downloaded when first used. > > **Limitations:** Only `wav` audio input is currently supported. The `response_format` field supports `json`, `verbose_json`, `text`, `srt`, and `vtt`. On the FastFlowLM (FLM) backend, `srt` and `vtt` are rejected with a `400` because FLM returns no segment timestamps, and `verbose_json` returns the compact shape without a `segments` field. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `file` | Yes | The audio file to transcribe. Supported formats: wav. | | | `model` | Yes | The Whisper model to use for transcription (e.g., `Whisper-Tiny`, `Whisper-Base`, `Whisper-Small`). | | | `language` | No | The language of the audio (ISO 639-1 code, e.g., `en`, `es`, `fr`). If not specified, Whisper will auto-detect the language. | | | `response_format` | No | The response format. Supported values: `json`, `verbose_json`, `text`, `srt`, `vtt`. `srt` and `vtt` require a backend that reports segment timestamps (whisper.cpp). | | ### Example request === "Windows" ```bash curl -X POST http://localhost:13305/v1/audio/transcriptions ^ -F "file=@C:\path\to\audio.wav" ^ -F "model=Whisper-Tiny" ``` === "Linux" ```bash curl -X POST http://localhost:13305/v1/audio/transcriptions \ -F "file=@/path/to/audio.wav" \ -F "model=Whisper-Tiny" ``` ### Response format ```json { "text": "Hello, this is a sample transcription of the audio file." } ``` **Field Descriptions:** - `text` - The transcribed text from the audio file ## `WS /realtime` Realtime Audio Transcription API via WebSocket (OpenAI SDK compatible). Stream audio from a microphone and receive transcriptions in real-time with Voice Activity Detection (VAD). > **Limitations:** Only 16kHz mono PCM16 audio format is supported. Uses the same Whisper models as the HTTP transcription endpoint. ### Connection WebSocket upgrades are accepted **directly on the main HTTP port** (default 13305) — the same port as all REST endpoints. Connect with the model name: ``` ws://localhost:13305/v1/realtime?model=Whisper-Tiny ``` Accepted paths are `/realtime` and `/logs/stream`, bare or under any of the standard prefixes (`/v1`, `/v0`, `/api/v1`, `/api/v0`), so OpenAI Realtime SDK clients (`/v1/realtime`) connect as-is. When `LEMONADE_API_KEY` is set, pass `?api_key=KEY` as a query parameter. A dedicated WebSocket port also remains for backward compatibility; it is OS-assigned and reported by the [`/v1/health`](./lemonade.md#get-v1health) endpoint (`websocket_port` field). New clients should prefer the main port: ``` ws://localhost:/realtime?model=Whisper-Tiny ``` Upon connection, the server sends a `session.created` message with a session ID. ### Client → Server Messages | Message Type | Description | |--------------|-------------| | `session.update` | Configure the session (set model, VAD settings, or disable turn detection) | | `input_audio_buffer.append` | Send audio data (base64-encoded PCM16) | | `input_audio_buffer.commit` | Force transcription of buffered audio | | `input_audio_buffer.clear` | Clear audio buffer without transcribing | ### Server → Client Messages | Message Type | Description | |--------------|-------------| | `session.created` | Session established, contains session ID | | `session.updated` | Session configuration updated | | `input_audio_buffer.speech_started` | VAD detected speech start | | `input_audio_buffer.speech_stopped` | VAD detected speech end, transcription triggered | | `input_audio_buffer.committed` | Audio buffer committed for transcription | | `input_audio_buffer.cleared` | Audio buffer cleared | | `conversation.item.input_audio_transcription.delta` | Interim/partial transcription (replaceable) | | `conversation.item.input_audio_transcription.completed` | Final transcription result | | `error` | Error message | ### Example: Configure Session ```json { "type": "session.update", "session": { "model": "Whisper-Tiny" } } ``` ### Example: Send Audio ```json { "type": "input_audio_buffer.append", "audio": "" } ``` Audio should be: - 16kHz sample rate - Mono (single channel) - 16-bit signed integer (PCM16) - Base64 encoded - Sent in chunks (~85ms recommended) ### Example: Transcription Result ```json { "type": "conversation.item.input_audio_transcription.completed", "transcript": "Hello, this is a test transcription." } ``` ### VAD Configuration VAD settings can be configured via `session.update`: ```json { "type": "session.update", "session": { "model": "Whisper-Tiny", "turn_detection": { "threshold": 0.01, "silence_duration_ms": 800, "prefix_padding_ms": 250 } } } ``` | Parameter | Default | Description | |-----------|---------|-------------| | `threshold` | 0.01 | RMS energy threshold for speech detection | | `silence_duration_ms` | 800 | Silence duration to trigger speech end | | `prefix_padding_ms` | 250 | Minimum speech duration before triggering | Set `turn_detection` to `null` to disable server-side VAD and use explicit commits instead: ```json { "type": "session.update", "session": { "model": "Whisper-Tiny", "turn_detection": null } } ``` ### Code Examples A complete, runnable example: - **[`realtime_transcription.py`](https://github.com/lemonade-sdk/lemonade/blob/main/examples/realtime_transcription.py)** - Python CLI for microphone streaming ```bash # Stream from microphone python examples/realtime_transcription.py --model Whisper-Tiny ``` ### Integration Notes - **Audio Format**: Server expects 16kHz mono PCM16. Higher sample rates must be downsampled client-side. - **Chunk Size**: Send audio in ~85-256ms chunks for optimal latency/efficiency. - **VAD Behavior**: Server automatically detects speech boundaries and triggers transcription on speech end. - **Manual Commit**: Set `turn_detection` to `null`, then use `input_audio_buffer.commit` to force transcription. In this mode the server buffers audio but does not emit VAD or interim transcription events. - **Clear Buffer**: Use `input_audio_buffer.clear` to discard audio without transcribing. - **Chunking**: We are still tuning the chunking to balance latency vs. accuracy. - **Migrating off the dedicated port**: Clients that discover `websocket_port` via `/v1/health` and connect there can switch to `ws://HOST:13305/v1/realtime?model=...` — the protocol (events, audio format, auth) is identical on both ports, so it is a URL change only. This also simplifies remote setups (one port to expose) and works through reverse proxies that pass `Upgrade: websocket`. Keep the `websocket_port` fallback only if you must support servers older than this release. ## `POST /v1/images/generations` Image Generation API. You provide a text prompt and receive a generated image. This API uses [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) as the backend. > **Note:** Image generation uses Stable Diffusion models. Available models include `SD-Turbo` (fast, ~4 steps), `SDXL-Turbo`, `SD-1.5`, and `SDXL-Base-1.0`. > > **Performance:** CPU inference takes ~4-5 minutes per image. GPU (Vulkan) is faster but may have compatibility issues with some hardware. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `prompt` | Yes | The text description of the image to generate. | | | `model` | Yes | The Stable Diffusion model to use (e.g., `SD-Turbo`, `SDXL-Turbo`). | | | `size` | No | The size of the generated image. Format: `WIDTHxHEIGHT` (e.g., `512x512`, `256x256`). Default: `512x512`. | | | `n` | No | Number of images to generate. Currently only `1` is supported. | | | `response_format` | No | Format of the response. Only `b64_json` (base64-encoded image) is supported. | | | `steps` | No | Number of inference steps. SD-Turbo works well with 4 steps. Default varies by model. | | | `cfg_scale` | No | Classifier-free guidance scale. SD-Turbo uses low values (~1.0). Default varies by model. | | | `seed` | No | Random seed for reproducibility. If not specified, a random seed is used. | | ### Example request === "Bash" ```bash curl -X POST http://localhost:13305/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "model": "SD-Turbo", "prompt": "A serene mountain landscape at sunset", "size": "512x512", "steps": 4, "response_format": "b64_json" }' ``` ## `POST /v1/images/edits` Image Editing API. You provide a source image and a text prompt describing the desired change, and receive an edited image. This API uses [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) as the backend. > **Note:** This endpoint accepts `multipart/form-data` requests (not JSON). Use editing-capable models such as `Flux-2-Klein-4B` or `SD-Turbo`. > > **Performance:** CPU inference takes several minutes per image. GPU (ROCm) is significantly faster. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `model` | Yes | The Stable Diffusion model to use (e.g., `Flux-2-Klein-4B`, `SD-Turbo`). | | | `image` / `image[]` | Yes | The source image file to edit (PNG). Sent as a file in multipart/form-data. | | | `prompt` | Yes | A text description of the desired edit. | | | `mask` | No | An optional mask image (PNG). White areas indicate regions to edit; black areas are preserved. | | | `size` | No | The size of the output image. Format: `WIDTHxHEIGHT` (e.g., `512x512`). Default: `512x512`. | | | `n` | No | Number of images to generate. Allowed range: `1`–`10`. Default: `1`. Values outside this range are rejected with `400 Bad Request`. | | | `response_format` | No | Format of the response. Only `b64_json` (base64-encoded image) is supported. | | | `steps` | No | Number of inference steps. Default varies by model. | | | `cfg_scale` | No | Classifier-free guidance scale. Default varies by model. | | | `seed` | No | Random seed for reproducibility. | | | `user` | No | OpenAI API compatibility field. Accepted but not forwarded to the backend. | | | `background` | No | OpenAI API compatibility field. Accepted but not forwarded to the backend. | | | `quality` | No | OpenAI API compatibility field. Accepted but not forwarded to the backend. | | | `input_fidelity` | No | OpenAI API compatibility field. Accepted but not forwarded to the backend. | | | `output_compression` | No | OpenAI API compatibility field. Accepted; silently ignored by the backend. | | ### Example request === "Bash" ```bash curl -X POST http://localhost:13305/v1/images/edits \ -F "model=Flux-2-Klein-4B" \ -F "prompt=Add a red barn and mountains in the background, photorealistic" \ -F "size=512x512" \ -F "n=1" \ -F "response_format=b64_json" \ -F "image=@/path/to/source_image.png" ``` === "Python (OpenAI client)" ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:13305/api/v1", api_key="not-needed") with open("source_image.png", "rb") as image_file: response = client.images.edit( model="Flux-2-Klein-4B", image=image_file, prompt="Add a red barn and mountains in the background, photorealistic", size="512x512", ) import base64 image_data = base64.b64decode(response.data[0].b64_json) open("edited_image.png", "wb").write(image_data) ``` --- ## `POST /v1/images/variations` Image Variations API. You provide a source image and receive a variation of it. This API uses [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) as the backend. > **Note:** This endpoint accepts `multipart/form-data` requests (not JSON). Unlike `/images/edits`, a `prompt` parameter is not supported and will be ignored — the model generates a variation based solely on the input image. > > **Performance:** CPU inference takes several minutes per image. GPU (ROCm) is significantly faster. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `model` | Yes | The Stable Diffusion model to use (e.g., `Flux-2-Klein-4B`, `SD-Turbo`). | | | `image` | Yes | The source image file (PNG). Sent as a file in multipart/form-data. | | | `size` | No | The size of the output image. Format: `WIDTHxHEIGHT` (e.g., `512x512`). Default: `512x512`. | | | `n` | No | Number of variations to generate. Integer between 1 and 10 inclusive. Default: `1`. Values outside this range result in a 400 Bad Request error. | | | `response_format` | No | Format of the response. Only `b64_json` (base64-encoded image) is supported. | | | `user` | No | OpenAI API compatibility field. Accepted but not forwarded to the backend. | | ### Example request === "Bash" ```bash curl -X POST http://localhost:13305/v1/images/variations \ -F "model=Flux-2-Klein-4B" \ -F "size=512x512" \ -F "n=1" \ -F "response_format=b64_json" \ -F "image=@/path/to/source_image.png" ``` === "Python (OpenAI client)" ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:13305/api/v1", api_key="not-needed") with open("source_image.png", "rb") as image_file: response = client.images.create_variation( model="Flux-2-Klein-4B", image=image_file, size="512x512", n=1, ) import base64 image_data = base64.b64decode(response.data[0].b64_json) open("variation.png", "wb").write(image_data) ``` --- ## `POST /v1/images/upscale` Image Upscaling API. You provide a base64-encoded image and a Real-ESRGAN model name, and receive a 4x upscaled image. This API uses the `sd-cli` binary from [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) to perform super-resolution. > **Note:** Available upscale models are `RealESRGAN-x4plus` (general-purpose, 64 MB) and `RealESRGAN-x4plus-anime` (optimized for anime-style art, 17 MB). Both produce a 4x resolution increase (e.g., 256x256 → 1024x1024). > > **Note:** Unlike `/images/edits` and `/images/variations`, this endpoint accepts a JSON body (not multipart/form-data). The image must be provided as a base64-encoded string. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `image` | Yes | Base64-encoded PNG image to upscale. | | | `model` | Yes | The ESRGAN model to use (e.g., `RealESRGAN-x4plus`, `RealESRGAN-x4plus-anime`). | | ### Example request A typical workflow is to generate an image first, then upscale it: === "Bash" ```bash # Step 1: Generate an image and save the base64 response RESPONSE=$(curl -s -X POST http://localhost:13305/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "model": "SD-Turbo", "prompt": "A serene mountain landscape at sunset", "size": "512x512", "steps": 4, "response_format": "b64_json" }') # Step 2: Build the upscale JSON payload and pipe it to curl via stdin # (base64 images are too large for command-line interpolation) echo "$RESPONSE" | python3 -c " import sys, json b64 = json.load(sys.stdin)['data'][0]['b64_json'] print(json.dumps({'image': b64, 'model': 'RealESRGAN-x4plus'})) " | curl -X POST http://localhost:13305/v1/images/upscale \ -H "Content-Type: application/json" \ -d @- ``` === "PowerShell" ```powershell # Step 1: Generate an image $genResponse = Invoke-WebRequest ` -Uri "http://localhost:13305/v1/images/generations" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body '{ "model": "SD-Turbo", "prompt": "A serene mountain landscape at sunset", "size": "512x512", "steps": 4, "response_format": "b64_json" }' # Step 2: Extract the base64 image $imageB64 = ($genResponse.Content | ConvertFrom-Json).data[0].b64_json # Step 3: Upscale the image with Real-ESRGAN $body = @{ image = $imageB64; model = "RealESRGAN-x4plus" } | ConvertTo-Json Invoke-WebRequest ` -Uri "http://localhost:13305/v1/images/upscale" ` -Method POST ` -Headers @{ "Content-Type" = "application/json" } ` -Body $body ``` === "Python (requests)" ```python import requests import base64 BASE_URL = "http://localhost:13305/api/v1" # Step 1: Generate an image gen_response = requests.post(f"{BASE_URL}/images/generations", json={ "model": "SD-Turbo", "prompt": "A serene mountain landscape at sunset", "size": "512x512", "steps": 4, "response_format": "b64_json", }) image_b64 = gen_response.json()["data"][0]["b64_json"] # Step 2: Upscale the image with Real-ESRGAN (512x512 -> 2048x2048) upscale_response = requests.post(f"{BASE_URL}/images/upscale", json={ "image": image_b64, "model": "RealESRGAN-x4plus", }) # Step 3: Save the upscaled image to a file upscaled_b64 = upscale_response.json()["data"][0]["b64_json"] with open("upscaled.png", "wb") as f: f.write(base64.b64decode(upscaled_b64)) ``` ### Response format ```json { "created": 1742927481, "data": [ { "b64_json": "" } ] } ``` **Field Descriptions:** - `created` - Unix timestamp of when the upscaled image was generated - `data` - Array containing the upscaled image - `b64_json` - Base64-encoded PNG of the upscaled image ### Error responses | Status Code | Condition | Example | |-------------|-----------|---------| | 400 | Missing `image` field | `{"error": {"message": "Missing 'image' field (base64 encoded)", "type": "invalid_request_error"}}` | | 400 | Missing `model` field | `{"error": {"message": "Missing 'model' field", "type": "invalid_request_error"}}` | | 404 | Unknown model name | `{"error": {"message": "Upscale model not found: bad-model", "type": "invalid_request_error"}}` | | 500 | Upscale failed | `{"error": {"message": "ESRGAN upscale failed", "type": "server_error"}}` | --- ## `POST /v1/audio/speech` Speech Generation API. You provide a text input and receive an audio file. Which engine serves the request depends on the model. > **Note:** Supported models are `kokoro-v1` (fixed voices, [Kokoros](https://github.com/lucasjinreal/Kokoros) backend) and the OpenMOSS family — `OpenMOSS-TTS` (voice cloning from a reference WAV) and `MOSS-VoiceGen` (voice design from a text description). > > **Limitations:** Which `response_format` values are accepted depends on the model's backend: `kokoro-v1` encodes `mp3`, `wav`, `opus`, and `pcm`, while OpenMOSS models natively produce `wav` only. A format the backend cannot encode is rejected with `400 Bad Request` listing the ones it does support. Streaming works for any TTS backend, but a backend's streamable set can be narrower than its buffered set — `kokoro-v1` streams `pcm` only, so an explicit `response_format` of `mp3` alongside `stream_format` is rejected rather than silently answered with PCM. ### Parameters | Parameter | Required | Description | Status | |-----------|----------|-------------|--------| | `input` | Yes | The text to speak. | | | `model` | Yes | The model to use (e.g., `kokoro-v1`). | | | `speed` | No | Speaking speed. Default: `1.0`. | | | `voice` | No | The voice to use. All OpenAI-defined voices can be used (`alloy`, `ash`, ...), as well as those defined by the kokoro model (`af_sky`, `am_echo`, ...). Default: `shimmer` | | | `voice` (OpenMOSS) | No | For OpenMOSS models the field is a free-text voice/style instruction instead of a fixed voice name (e.g. `a calm, deep male narrator voice`). | | | `reference_wav_b64` | No | Lemonade extension (OpenMOSS voice cloning): base64-encoded WAV sample of the voice to clone. | | | `response_format` | No | Container for the returned audio. Which values are accepted depends on the model's backend (see Limitations above). Default: `mp3` when buffered and `pcm` when streaming, falling back to the backend's first supported format when it cannot encode that default. | | | `stream_format` | No | If set, the response is streamed. Only `audio` is supported. This selects the transport only — the container still comes from `response_format`, and an explicit one is honored on both transports. Default: not set| | ### Example request === "Bash" ```bash curl -X POST http://localhost:13305/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "kokoro-v1", "input": "Lemonade can speak!", "speed": 1.0, "steps": 4, "response_format": "mp3" }' ``` ### Response format The generated audio file is returned as-is. ## `GET /v1/models` Returns a list of models available on the server in an OpenAI-compatible format. Each model object includes extended fields like `checkpoint`, `recipe`, `size`, `downloaded`, `labels`, and, when known, `max_context_window`. By default, only models available locally (downloaded) are shown, matching OpenAI API behavior. When `lemond` is configured with cloud providers, cloud-routed models appear here alongside local ones with `recipe: "cloud"` and a `cloud_provider` field. They are dot-namespaced by provider (e.g. `fireworks.kimi-k2p5`) and accept the standard chat-completions / completions requests below — see [Cloud Offload](../guide/configuration/cloud.md). ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `show_all` | No | If set to `true`, returns all models from the catalog including those not yet downloaded. Defaults to `false`. | ### Example request ```bash # Show only downloaded models (OpenAI-compatible) curl http://localhost:13305/v1/models # Show all models including not-yet-downloaded (extended usage) curl http://localhost:13305/v1/models?show_all=true ``` ### Response format ```json { "object": "list", "data": [ { "id": "Qwen3-0.6B-GGUF", "created": 1744173590, "object": "model", "owned_by": "lemonade", "checkpoint": "unsloth/Qwen3-0.6B-GGUF:Q4_0", "recipe": "llamacpp", "size": 0.38, "max_context_window": 40960, "downloaded": true, "suggested": true, "update_available": false, "labels": ["reasoning"] }, { "id": "Gemma-3-4b-it-GGUF", "created": 1744173590, "object": "model", "owned_by": "lemonade", "checkpoint": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M", "recipe": "llamacpp", "size": 3.61, "downloaded": true, "suggested": true, "labels": ["hot", "vision"] }, { "id": "SD-Turbo", "created": 1744173590, "object": "model", "owned_by": "lemonade", "checkpoint": "stabilityai/sd-turbo:sd_turbo.safetensors", "recipe": "sd-cpp", "size": 5.2, "downloaded": true, "suggested": true, "labels": ["image"], "image_defaults": { "steps": 4, "cfg_scale": 1.0, "width": 512, "height": 512 } } ] } ``` **Field Descriptions:** - `object` - Type of response object, always `"list"` - `data` - Array of model objects with the following fields: - `id` - Model identifier (used for loading and inference requests) - `created` - Unix timestamp of when the model entry was created - `object` - Type of object, always `"model"` - `owned_by` - Owner of the model, always `"lemonade"` - `checkpoint` - Full checkpoint identifier on Hugging Face - `recipe` - Backend/device recipe used to load the model (e.g., `"ryzenai-llm"`, `"llamacpp"`, `"flm"`) - `size` - Model size in GB (omitted for models without size information) - `max_context_window` - Optional integer indicating the maximum model-supported text context discovered from local static metadata. Currently populated for downloaded GGUF/llama.cpp models and installed FLM text-context models. - `downloaded` - Boolean indicating if the model is downloaded and available locally - `update_available` - Boolean indicating a newer commit exists on HuggingFace for this model. Only set for downloaded HF-backed models. `false` otherwise. - `suggested` - Boolean indicating if the model is recommended for general use - `labels` - Array of tags describing the model's capabilities and characteristics. See [Model Labels](#model-labels) for the full list. - `image_defaults` - (Image models only) Default generation parameters for the model: - `steps` - Number of inference steps (e.g., 4 for turbo models, 20 for standard models) - `cfg_scale` - Classifier-free guidance scale (e.g., 1.0 for turbo models, 7.5 for standard models) - `width` - Default image width in pixels - `height` - Default image height in pixels - `components` - (Omni collections only, `recipe: "collection.omni"`) Ordered array of the component model names that make up the collection - `models` - (Omni collections only) Ordered array embedding each component's full model object (same shape as the entries in this list), parallel to `components`. This makes a collection's `/v1/models/{model_id}` response self-contained — exporting it produces a file that can be imported elsewhere via [`/v1/pull`](./lemonade.md#post-v1pull) ### Model Labels Labels describe what a model can do. A model may carry multiple labels. **Deployment labels** — determine which backend endpoint the model is routed to. Every model names exactly one deployment mode, and a model is never given two labels that name different modes: | Label | Endpoint | Description | |-------|----------|-------------| | `chat` | `/chat/completions`, `/completions`, `/responses` | Text-generating LLM. This label is what makes a model an LLM — it is not inferred from `reasoning`/`vision`/`tool-calling`/`chat-transcription`, which are characteristics rather than deployment modes. | | `transcription` | `/audio/transcriptions` | Speech-to-text transcription model (e.g. Whisper). An omni LLM that accepts audio in a chat turn is not one of these — it carries `chat` and the `chat-transcription` capability below. | | `embeddings` | `/embeddings` | Produces text embedding vectors. Also accepted as `embedding`. | | `reranking` | `/rerank` | Scores and reranks a list of passages given a query. Also reachable at the aliases `/reranking` and `/reranker`. | | `image` | `/images/generations`, `/images/edits`, `/images/variations` | Text-to-image generation model. | | `tts` | `/audio/speech` | Text-to-speech synthesis model. | | `audio-generation` | `/audio/generations` | Text-to-audio generation model (e.g. music, sound effects). | | `classification` | `/classify` | Text classification model. Also accepted as `classifier`. | | `3d` | `/3d/generations` | Text- or image-to-3D mesh generation model. | When a model declares no deployment label at all, it inherits its recipe's default — `chat` for `llamacpp`, `flm`, `ryzenai-llm`, `vllm` and `cloud`, `transcription` for `whispercpp`, `image` for `sd-cpp`, `tts` for `kokoro`, and so on. Two label sets describe a model that cannot exist, and are refused rather than repaired: - **A mode the recipe's backend does not serve.** `/classify` is served only by `onnxruntime`, so `labels: ["classification"]` on a `llamacpp` model is an error — register it as the chat model it is. - **Two different modes.** `labels: ["chat", "embeddings"]` on a `llamacpp` model is an error even though llama.cpp serves both: the subprocess is launched for one mode, so the second would name an endpoint it was never configured to answer. Register one model per mode. The legacy `embedding` and `reranking` booleans count as mode claims here, exactly as the labels do. [`POST /v1/pull`](./lemonade.md#post-v1pull) answers `400` and registers nothing. An entry already stored in `user_models.json` — written before these rules — is skipped at startup with an error naming it, and the file is left untouched so it can be corrected by hand. **Input-modality labels** — the model accepts additional input types in `/chat/completions`: | Label | Description | |-------|-------------| | `vision` | Accepts image attachments in chat messages. | | `chat-transcription` | Accepts audio attachments in chat messages and transcribes them as part of its answer (e.g. Qwen2.5-Omni). Like `vision`, this is something a chat model can do, not a deployment mode of its own — a model carrying it also carries `chat`. It is distinct from `transcription`, which deploys a dedicated ASR model on `/audio/transcriptions`. | **Streaming labels** — capability flags for real-time features: | Label | Description | |-------|-------------| | `realtime-transcription` | Supports the WebSocket `/realtime` endpoint for live microphone transcription. | **Runtime labels** — affect backend launch defaults: | Label | Description | |-------|-------------| | `mtp` | Enables llama.cpp MTP draft decoding defaults (`--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75`); users can override these with `llamacpp_args`. | **Image capability labels** — carried alongside `image`; they refine what the model is offered for without changing its deployment mode: | Label | Description | |-------|-------------| | `edit` | Tuned for editing an input image (`/images/edits`). Also selects the model for the `edit_image` role in an omni collection. | | `upscaling` | Image upscaling model (e.g. Real-ESRGAN, `/images/upscale`). Used as a component in image pipelines rather than offered on its own. | **Characteristic labels** — informational, do not affect routing: | Label | Description | |-------|-------------| | `hot` | Featured or popular model, highlighted in the UI. | | `reasoning` | Uses extended chain-of-thought reasoning (e.g. DeepSeek, Qwen3). | | `tool-calling` | Supports function/tool calling in chat completions. | | `coding` | Tuned for code generation and software tasks. | | `experimental` | Not yet validated for production use. | ## `GET /v1/models/{model_id}` Retrieve a specific model by its ID. Returns the same model object format as the list endpoint above. ### Parameters | Parameter | Required | Description | |-----------|----------|-------------| | `model_id` | Yes | The ID of the model to retrieve. Must match one of the model IDs from the [models list](https://lemonade-server.ai/models.html). | ### Example request ```bash curl http://localhost:13305/v1/models/Qwen3-0.6B-GGUF ``` ### Response format Returns a single model object with the same fields as described in the [models list endpoint](#get-v1models) above. For Omni collections (`recipe: "collection.omni"`), the object additionally carries `components` (ordered component names) and `models` (each component's full model object) — see the [collection file documentation](../guide/configuration/custom-models.md#share-a-collection-export-import-and-hugging-face). ```json { "id": "Qwen3-0.6B-GGUF", "created": 1744173590, "object": "model", "owned_by": "lemonade", "checkpoint": "unsloth/Qwen3-0.6B-GGUF:Q4_0", "recipe": "llamacpp", "size": 0.38, "max_context_window": 40960, "downloaded": true, "suggested": true, "labels": ["reasoning"], "recipe_options": { "ctx_size": 8192, "llamacpp_args": "--no-mmap", "llamacpp_backend": "rocm" } } ``` ### Error responses If the model is not found, the endpoint returns a 404 error: ```json { "error": { "message": "Model Qwen3-0.6B-GGUF has not been found", "type": "not_found" } } ``` --- ## File: docs/api/README.md # Lemonade Endpoints Spec The Lemonade HTTP service provides a wide array of standards-compliant and custom endpoints. Our design philosophy is: 1. Ensure that Lemonade works out-of-box with all popular local AI apps. 2. Prioritize using a pre-existing standard for functionality whenever possible. 3. Add sufficient custom functionality to enable developers to build highly polished experiences. This spec details all supported endpoints. It is organized into pages that correspond to which organization (OpenAI, Ollama, Lemonade, etc.) defined the endpoints. | API | Description | |-----|-------------| | [OpenAI-Compatible API](./openai.md) | Start here for the main API surface used by most SDKs and clients. | | [Ollama-Compatible API](./ollama.md) | Use this if your client expects Ollama-style behavior and routes. | | [Anthropic-Compatible API](./anthropic.md) | Use this for clients built around Anthropic's message format. | | [MCP Gateway](./mcp.md) | Use this to expose Lemonade as a Model Context Protocol server (POST /mcp). | | [llama.cpp-Specific API](./llamacpp.md) | Reference for llama.cpp-specific compatibility and conventions. | | [Lemonade-Specific API](./lemonade.md) | Local-first API for managing lifecycle, configuration, backends, etc. | --- ## File: docs/dev/working-groups/auto-tune.md # Working Group: Auto-Tune ## Overview **Lead:** This working group is led by Michele Balistreri, whose handle is @bitgamma on GitHub and @mikkoph on Discord. **Background:** Running a model well requires choosing the right backend and tuning performance parameters such as batch size, GPU layer offload, thread count, and context size. Today, users either guess, look up Reddit threads, or ask an LLM — approaches that are error-prone and often outdated due to knowledge cutoff. Lemonade already has a `bench` command that measures TTFT, TPS, and VRAM usage across backends and parameter combinations, and a recipe system that defines per-model configuration. What is missing is a mechanism to turn benchmark data into actionable, hardware-aware defaults that apply automatically. **Why:** A user should install Lemonade and get good performance without needing to understand backend flags or run manual benchmarks. Hardware-aware defaults lower the barrier to entry and make Lemonade competitive with cloud-based alternatives where performance is abstracted away. **Goal:** Enable Lemonade instances to self-optimize models and backends by detecting the machine's hardware profile and applying community-validated performance parameters. The end state is that a user pulls a model, loads it, and Lemonade selects the best backend and tuning flags for their hardware — with the option to override or fine-tune manually. ## Contributing Please see the general [contribution guidelines](../contribute.md), then contact @mikkoph on Discord before getting started to discuss the roadmap. ## Scope This working group focuses on **performance parameters** (batch size, GPU layers, threads, context size, backend selection) that are determined by hardware characteristics. Quality parameters such as temperature, top_p, and chat template are model- or use-case-dependent and are explicitly **out of scope**. ## Roadmap > Roadmap items are high-level objectives that may span multiple issues and PRs. Details can also be re-defined. ### Phase 1: Archetype Detection & Profile Schema - [ ] Define the hardware archetype classification logic. Archetypes are identified by GPU family/architecture, VRAM capacity bucket, memory bandwidth tier, and whether memory is unified or discrete. This keeps the profile space bounded — two machines with the same archetype receive the same recommendations. - [ ] Design the profile JSON schema. A profile maps an archetype ID to recommended performance parameters per backend (e.g., `strix-halo-128gb` → `{ "llamacpp_backend": "vulkan", "llamacpp_vulkan_args": { "-b 2048 -ub 1024" }, "vllm_args": { ... } }`). The schema is backend-agnostic so it works for llama.cpp, FastFlowLM, vLLM, RyzenAI, and future backends. - [ ] Implement archetype detection in `lemond`, using data already collected by the `system-info` endpoint (GPU name, VRAM, bandwidth, unified vs. discrete memory). - [ ] Ship an initial set of hand-curated profiles for common hardware configurations (e.g., Strix Halo, Radeon 780M, Apple M3). ### Phase 2: Auto-Apply at Load Time - [ ] On startup, `lemond` detects the machine archetype and caches it. - [ ] When loading a model, the router checks for archetype-specific profile overrides and merges them into the model's recipe options. A fallback chain ensures graceful degradation: exact archetype + model overlay → archetype defaults for the backend → recipe defaults (current behavior). - [ ] Optional per-model overlays allow model-specific tuning on top of global archetype defaults (for cases where certain models benefit from different flags on the same hardware). - [ ] `lemonade status` displays the detected archetype and any active auto-tune overrides. - [ ] CLI or config option to enable/disable auto-tune (enabled by default). ### Phase 3: Profile Distribution - [ ] Profiles are fetched at runtime from a remote source, allowing updates without reinstalling Lemonade. A set of default profiles ships bundled with Lemonade as a fallback when no network is available. - [ ] Profile cache is stored locally with versioning; stale profiles are re-fetched periodically. - [ ] `lemonade bench --submit` generates a structured benchmark contribution from local benchmark runs. The submission workflow (upload endpoint, PR-based contribution, or other) will be designed based on infrastructure study. ### Phase 4: Community Data Collection & Curation - [ ] Tooling to merge community-submitted benchmarks into curated profiles. This includes selecting the best-performing parameter sets per scenario, validating consistency across submissions, and detecting outliers. - [ ] Documentation for the contribution workflow so community members can benchmark their hardware and contribute results. - [ ] Profiles cover a broad range of hardware archetypes across all supported backends. ### Phase 5: Adaptive Tuning - [ ] Runtime monitoring: if observed performance (TPS, TTFT) deviates significantly from profile expectations, `lemond` can suggest re-benchmarking or flag the profile for review. - [ ] "Learn from this run" — after a session, offer to save locally-observed good parameters into the user's config for persistence. - [ ] To be continued, based on learnings from the above.