lemonade

# AGENTS.md

This file provides guidance to agent driven code reviews when working with this repository.

## Project Overview

Lemonade is a local LLM server providing GPU and NPU acceleration for running large language models on consumer hardware. It exposes OpenAI-compatible, Ollama-compatible, and Anthropic-compatible REST APIs, plus a WebSocket Realtime API. It supports multiple backends: llama.cpp, FastFlowLM, RyzenAI, whisper.cpp, stable-diffusion.cpp, Kokoro TTS, and Moonshine.

## Architecture

### Executables

- **lemond** — Pure HTTP server. Handles REST API, routes requests to backends, manages model loading/unloading. Configured via `config.json` in the lemonade cache directory. CLI args: `[cache_dir] [--port PORT] [--host HOST]`.
- **lemonade** — CLI client (`src/cpp/cli/`). Commands: `list`, `pull`, `delete`, `run`, `status`, `logs`, `launch`, `backends`, `scan`, etc. Communicates with router via HTTP. Discovers running server via UDP beacon.
- **LemonadeServer.exe** (Windows) — SUBSYSTEM:WINDOWS GUI app that embeds `lemond` and shows a system tray icon. Auto-starts via Windows startup folder.
- **lemonade-tray** (macOS/Linux) — Lightweight tray client that connects to a running `lemond`. Platform code in `src/cpp/tray/platform/`.

### Backend Abstraction

`WrappedServer` (`src/cpp/include/lemon/wrapped_server.h`) is the abstract base class. Each backend inherits it and implements `load()`, `unload()`, `chat_completion()`, `completion()`, `responses()`, and optionally `install()` / `download_model()`. Backends run as **subprocesses** — Lemonade forwards HTTP requests to them.

| Backend | Class | Capabilities | Device | Purpose |
|---------|-------|-------------|--------|---------|
| llama.cpp | `LlamaCppServer` | Completion, Embeddings, Reranking | GPU | LLM inference — CPU/GPU (Vulkan, ROCm, Metal) |
| FastFlowLM | `FastFlowLMServer` | Completion, Embeddings, Audio | NPU | NPU inference (multi-modal: LLM, ASR, embeddings) |
| RyzenAI | `RyzenAIServer` | Completion | NPU | Hybrid NPU inference |
| vLLM | `VLLMServer` | Completion | GPU | LLM inference — ROCm on AMD iGPU/dGPU (Linux). **Experimental**, validated only on gfx1151 (Strix Halo). |
| whisper.cpp | `WhisperServer` | Audio | CPU | Audio transcription |
| stable-diffusion.cpp | `SdServer` | Image | CPU | Image generation, editing, variations |
| Kokoro | `KokoroServer` | TTS | CPU | Text-to-speech |
| Moonshine | `MoonshineServer` | Audio | CPU | Streaming speech-to-text (ONNX-based) |

Capability interfaces: `ICompletionServer`, `IEmbeddingsServer`, `IRerankingServer`, `ITranscriptionServer`, `IImageServer`, `ITextToSpeechServer` (defined in `server_capabilities.h`). Use `supports_capability<T>(server)` template for runtime checks.

### Router & Multi-Model Support

`Router` (`src/cpp/server/router.cpp`) manages a vector of `WrappedServer` instances. Routes requests based on model recipe, maintains LRU caches per model type (LLM, embedding, reranking, audio, image, TTS — see `model_types.h`), and enforces NPU exclusivity. Configurable via `--max-loaded-models`. On non-file-not-found errors, the router uses a "nuclear option" — evicts all models and retries the load.

### Model Manager & Recipe System

`ModelManager` (`src/cpp/server/model_manager.cpp`) loads the registry from `src/cpp/resources/server_models.json`. Each model has "recipes" defining which backend and config to use. Backend versions are pinned in `src/cpp/resources/backend_versions.json`. Models download from Hugging Face.

### API Routes

All core endpoints are registered under **4 path prefixes**:
- `/api/v0/` — Legacy
- `/api/v1/` — Current
- `/v0/` — Legacy short
- `/v1/` — OpenAI SDK / LiteLLM compatibility

**Core endpoints:** `chat/completions`, `completions`, `embeddings`, `reranking`, `models`, `models/{id}`, `health`, `pull`, `pull/variants`, `registry/search`, `load`, `unload`, `delete`, `params`, `install`, `uninstall`, `audio/transcriptions`, `audio/speech`, `images/generations`, `images/edits`, `images/variations`, `responses`, `stats`, `system-info`, `system-stats`, `log-level`, `logs/stream`, `jobs`, `jobs/{id}`, `jobs/{id}/pause`, `jobs/{id}/interrupt`, `jobs/{id}/resume`

**Job engine** (`POST jobs`, `GET jobs`, `GET/DELETE jobs/{id}`, `POST jobs/{id}/{pause,interrupt,resume}`): server-side sequences of ops (`system_info`, `system_stats`, `models`, `sleep`, `load`, `unload`, `chat`) with data passing, forward-only branching, and a pause/interrupt/resume lifecycle persisted across restart. Exclusive ops hold a Router slot so normal traffic queues. See `docs/dev/job-system.md` and `docs/dev/job-expression-language.md`.

**Ollama-compatible endpoints** (under `/api/` without version prefix): `chat`, `generate`, `tags`, `show`, `delete`, `pull`, `embed`, `embeddings`, `ps`, `version`

**Anthropic-compatible endpoint:** `POST /api/messages` — supports message completion, tool use, and SSE streaming.

**MCP gateway endpoint:** `POST /mcp` — Model Context Protocol (Streamable HTTP transport, spec `2025-06-18`). Single JSON-RPC 2.0 endpoint exposing 5 tools (`lemonade_list_models`, `lemonade_chat`, `lemonade_transcribe_audio`, `lemonade_generate_image`, `lemonade_omni`). GET returns 405.

**WebSocket Realtime API**: OpenAI-compatible Realtime protocol for real-time audio transcription. `/realtime` and `/logs/stream` accept WebSocket upgrades directly on the main HTTP port; a dedicated listener on an OS-assigned port (9000+, exposed via the `websocket_port` field in the `/health` response) also remains for backward compatibility.

**Internal endpoints:** `POST /internal/shutdown`, `GET /internal/aliases`, `POST /internal/aliases`, `DELETE /internal/aliases/{alias}`

Optional API key auth via `LEMONADE_API_KEY` env var (regular API endpoints) or `LEMONADE_ADMIN_API_KEY` env var (full access including internal endpoints). Clients prefer `LEMONADE_ADMIN_API_KEY` if set. CORS enabled on all routes.

### Desktop & Web App

- **Tauri app** — React 19 + TypeScript in `src/app/`, Rust host in `src/app/src-tauri/`. Uses native OS webview (WebView2 on Windows, WKWebView on macOS, webkit2gtk on Linux). Pure CSS (dark theme), context-based state. Key components: `ChatWindow.tsx`, `ModelManager.tsx`, `DownloadManager.tsx`, `BackendManager.tsx`. Feature panels: LLMChat, ImageGeneration, Transcription, TTS, Embedding, Reranking. The renderer keeps its `window.api` contract via `src/app/src/renderer/tauriShim.ts`, which maps each call to a Tauri `invoke()` or event `listen()`.
- **Web app** — Browser-only version in `src/web-app/`. Reuses the shared renderer from `src/app/src/` via webpack's `entry`/`template` paths (no OS symlinks); the `BuildWebApp.cmake` script stages both trees side-by-side under `build/web-app-staging/` for the actual webpack build. Built via CMake `BUILD_WEB_APP=ON`. Served at `/app`. A mock `window.api` is injected by the C++ server (`src/cpp/server/server.cpp`) so the shared renderer works unchanged in the browser.

### Key Dependencies

**C++ (FetchContent):** cpp-httplib, nlohmann/json, CLI11, libcurl, zstd, libwebsockets, brotli (macOS). Platform SSL: Schannel (Windows), SecureTransport (macOS), OpenSSL (Linux).

**Desktop app:** Tauri v2 (Rust), React 19, TypeScript 5.3, Webpack 5, markdown-it, highlight.js, katex. Rust crates: `tauri`, `tauri-plugin-{opener,clipboard-manager,single-instance,deep-link}`, `tokio`, `reqwest`, `serde`.

## Build Commands

CMakeLists.txt is at the repository root. Build uses CMake presets — run the setup script first, then build with `--preset`.

```bash
# 1. Setup (configures build directory and installs deps)
./setup.sh          # Linux / macOS
./setup.ps1         # Windows (PowerShell)

# 2. Build C++ server
cmake --build --preset default          # Linux / macOS (Ninja)
cmake --build --preset windows          # Windows (Visual Studio 2022)
cmake --build --preset vs18             # Windows (Visual Studio 2026)

# 3. Tauri desktop app (optional, requires Node.js 20+ and Rust via rustup)
cmake --build --preset default --target tauri-app    # Linux / macOS
cmake --build --preset windows --target tauri-app    # Windows (VS 2022)
cmake --build --preset vs18 --target tauri-app       # Windows (VS 2026)

# 4. Web app (auto-built on all platforms)
cmake --build --preset default --target web-app         # Linux / macOS
cmake --build --preset windows --target web-app         # Windows

# 5. Windows MSI installer (WiX 5.0+ required)
cmake --build --preset windows --target wix_installer_minimal  # server + web-app
cmake --build --preset windows --target wix_installer_full     # server + Tauri app + web-app

# 6. macOS signed installer
cmake --build --preset default --target package-macos

# 7. Linux .deb / .rpm
cd build && cpack            # .deb
cd build && cpack -G RPM     # .rpm
```

CMake presets: `default` (Ninja, Release), `windows` (VS 2022), `vs18` (VS 2026), `debug` (Ninja, Debug).

CMake options: `BUILD_WEB_APP` (ON by default on all platforms), `BUILD_TAURI_APP` (Linux only, include Tauri desktop app in deb), `LEMONADE_SYSTEMD_UNIT_NAME` (default: `lemond.service`).

## Testing

Integration tests in Python against a live server. Tests auto-discover the `lemonade` CLI binary from the build directory; use `--cli-binary` to override.

```bash
pip install -r test/requirements.txt

# CLI tests (no inference backend needed)
python test/server_cli2.py

# Endpoint tests (no inference backend needed)
python test/server_endpoints.py

# LLM tests (specify wrapped server and backend)
python test/server_llm.py --wrapped-server llamacpp --backend vulkan

# Audio transcription tests
python test/server_whisper.py

# Image generation tests (slow)
python test/server_sd.py
```

Test utilities in `test/utils/` with `server_base.py` as the base class. Test dependencies include `requests`, `httpx`, `openai`, `huggingface_hub`, `psutil`, `numpy`, `websockets`, and `ollama`.

### C++ unit tests

C++ unit tests live in `test/cpp/` and are wired up in the root `CMakeLists.txt`. The packaging workflow builds the `cpp-ci-tests` aggregate target and runs `ctest -L cpp-ci`, so a test only runs in CI if it is both labeled `cpp-ci` **and** a dependency of that aggregate target.

**Direct `add_test()` is disabled** (the built-in is overridden to fail with a fatal error just before the test section). Every test MUST be declared with the `add_cpp_ci_test()` helper, which forces an explicit `CI <ON|OFF>` decision at the call site so a test is never silently omitted from — or accidentally added to — CI.

**The enclosing `if()` MUST test `BUILD_TESTING`.** Distro packaging (`contrib/debian/rules`, the RPM job) configures with `BUILD_TESTING=OFF` so it does not build ~45 test binaries it then discards; calling `add_cpp_ci_test()` in that configuration is a fatal error rather than a silent return to the slow build.

```cmake
if(BUILD_TESTING AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/test/cpp/test_my_feature.cpp")
    add_executable(test_my_feature test/cpp/test_my_feature.cpp ...)
    # ...target_include_directories / target_link_libraries...

    include(CTest)
    add_cpp_ci_test(MyFeatureTest CI ON COMMAND test_my_feature)
endif()
```

```cmake
add_cpp_ci_test(<TestName>
                CI <ON|OFF>                 # required — run under `ctest -L cpp-ci`?
                COMMAND <command> [args...] # required — what CTest runs
                [DEPENDS <target>...])      # CI build deps; defaults to the
                                            # first COMMAND token (the test target)
```

- `CI ON` labels the test `cpp-ci` and makes its build target(s) a dependency of `cpp-ci-tests`.
- `CI OFF` still creates the CTest test (for local/other runs) but keeps it out of packaging CI. Use this only for tests that are intentionally excluded (e.g. tests that need a backend, are platform-gated, or are slow CMake-configuration tests).

Pass `DEPENDS` only when the CI build needs targets beyond the `COMMAND` executable. `add_cpp_ci_test` calls `register_cpp_ci_test()` internally; do not call `add_test()` or `register_cpp_ci_test()` directly.

## Code Style

### Comments & Documentation

**Default to writing no comments.** Only add a comment when the WHY is non-obvious: a hidden constraint, a subtle invariant, a workaround for a specific bug, or behavior that would surprise a reader. If removing the comment wouldn't confuse a future reader, don't write it.

**Never write comments that explain WHAT the code does** — well-named identifiers already do that. Don't reference the current task, fix, or callers ("used by X", "added for the Y flow", "handles the case from issue #123") — those belong in the PR description and rot as the codebase evolves.

**PR descriptions should be concise.** 1-3 sentences for the summary. No essays. The diff shows what changed; the description explains why and any non-obvious context. Bullet points over paragraphs.

### C++
- C++17, `lemon::` namespace
- `snake_case` for functions/variables, `CamelCase` for classes/types
- 4-space indent, `#pragma once` for headers
- Keep `#include` directives in alphabetical order within each include block
- Platform guards: `#ifdef _WIN32`, `#ifdef __APPLE__`, `#ifdef __linux__`

### Python
- **Black** formatting (v26.1.0, enforced in CI)
- Pylint with `.pylintrc`
- Pre-commit hooks: trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files

### TypeScript/React
- React 19, pure CSS (dark theme), context-based state
- UI/frontend changes are handled by core maintainers only

## Key Files

| File | Purpose |
|------|---------|
| `CMakeLists.txt` | Root build config (version, deps, targets) |
| `src/cpp/server/server.cpp` | HTTP route registration and all handlers |
| `src/cpp/server/router.cpp` | Request routing and multi-model orchestration |
| `src/cpp/server/model_manager.cpp` | Model registry, downloads, recipe resolution |
| `src/cpp/include/lemon/wrapped_server.h` | Backend abstract base class |
| `src/cpp/include/lemon/server_capabilities.h` | Backend capability interfaces |
| `src/cpp/resources/server_models.json` | Model registry |
| `src/cpp/resources/backend_versions.json` | Backend version pins |
| `docs/tools/gen_backend_boilerplate.py` | Regenerates committed artifacts from the C++ backend descriptors. Outputs: the whole of `src/cpp/resources/defaults.json` (per-recipe sections only; global keys stay hand-maintained in that file), and `<!-- BEGIN/END GENERATED -->` regions in `docs/dev/backends-reference.md`, root `README.md`, `docs/guide/cli.md`, `docs/guide/configuration/{README,multi-model,custom-models}.md`, and `docs/assets/models.js`. Don't hand-edit those regions/sections; CI runs `--check` and fails on drift. |
| `src/cpp/server/anthropic_api.cpp` | Anthropic API compatibility |
| `src/cpp/server/ollama_api.cpp` | Ollama API compatibility |
| `src/cpp/server/mcp_server.cpp` | MCP gateway (POST /mcp) |
| `src/cpp/include/lemon/websocket_server.h` | WebSocket Realtime API server |
| `src/cpp/include/lemon/model_types.h` | Model type and device type enums |
| `src/cpp/include/lemon/config_file.h` | config.json load/save/migrate |
| `src/cpp/include/lemon/recipe_options.h` | Per-recipe JSON configuration |
| `src/cpp/tray/tray_app.cpp` | Tray application UI and logic |
| `src/app/src/renderer/ModelManager.tsx` | Model management UI |
| `src/app/src/renderer/ChatWindow.tsx` | Chat interface |

## Critical Invariants

These MUST be maintained in all changes:

1. **Quad-prefix registration** — Every new endpoint MUST be registered under `/api/v0/`, `/api/v1/`, `/v0/`, AND `/v1/`. Documented exceptions: Ollama (`/api/*` without version prefix), Anthropic (`POST /v1/messages` only), and MCP (`POST /mcp`) — each of those protocols mandates a fixed URL shape that conflicts with the quad-prefix scheme.
2. **NPU exclusivity** — Exclusive-NPU recipes (`ryzenai-llm`, `whispercpp` on NPU) evict ALL other NPU models before loading. FastFlowLM (`flm`) can coexist with other FLM types (max 1 per FLM type) but not with exclusive-NPU recipes.
3. **WrappedServer contract** — New backends MUST implement all core virtual methods: `load()`, `unload()`, `chat_completion()`, `completion()`, `responses()`.
4. **Subprocess model** — Backends run as subprocesses (llama-server, whisper-server, sd-server, koko, flm, ryzenai-server, moonshine-server). They must NOT run in-process.
5. **Recipe integrity** — Changes to `server_models.json` must have valid recipes referencing backends in `backend_versions.json`. When adding or updating `vllm` models, also update `src/cpp/resources/vllm_model_config.json` if the model family needs vLLM-specific args such as tool-call parser settings.
6. **Cross-platform** — Code must compile on Windows (MSVC), Linux (GCC/Clang), macOS (AppleClang). Platform-specific code must use `#ifdef` guards.
7. **No hardcoded paths** — Use path utilities. Windows/Linux/macOS paths differ.
8. **Thread safety** — Router serves concurrent HTTP requests. Shared state must be properly guarded.
9. **Ollama compatibility** — Changes to model listing or management must not break `/api/*` Ollama endpoints.
10. **API key passthrough** — When `LEMONADE_API_KEY` is set, all API routes must enforce authentication.
11. **Many-clients-one-server topology** — A single `lemond` can be driven by multiple desktop/tray/CLI clients, potentially on different machines. Per-client UI state (layout, zoom, view selection, the client's own base URL and API key) MUST live locally in the client, never in `lemond`. Do not move `app_settings.json` behind an HTTP endpoint. **Shared infrastructure config** (cloud provider URLs, backend version pins) lives in `lemond`'s `config.json` so it's visible to every client and to the CLI. **Cloud API keys** specifically MUST NOT be written to disk: they live in `LEMONADE_<PROVIDER>_API_KEY` env vars (persistent) or in `lemond`'s process memory via `POST /v1/cloud/auth` (ephemeral, dies on restart).
12. **Web-app dependencies constrained by Debian native packaging** — `src/web-app/package.json` is kept separate from `src/app/package.json` because the native Debian package (`lemonade-server` .deb) must build using only npm modules available in Debian's `/usr/share/nodejs` (see `USE_SYSTEM_NODEJS_MODULES` in `src/web-app/webpack.config.js`). The old Electron app depended on packages Debian does not ship. Do NOT consolidate the two `package.json` files — the split is required for reproducible distro packaging.
13. **Desktop app is on-demand; `lemond` runs independently** — On Windows, `LemonadeServer.exe` (which embeds `lemond` + tray icon) is the always-on process, auto-started via the Windows startup folder. The Tauri desktop app (`lemonade-app.exe`) is opened on demand when the user wants the UI and must not be added to startup. The desktop app must not embed or manage `lemond`'s lifecycle — it discovers the already-running server (UDP beacon for local, explicit base URL for remote) and speaks to it over HTTP.

## Contributing

- Open an Issue before submitting major PRs
- UI/frontend changes are handled by core maintainers only
- Python formatting with Black is required
- PRs trigger CI for linting, formatting, and integration tests