{"owner":"exo-explore","repo":"exo","hasSkills":true,"totalSkillsCount":4,"totalTokensCount":8268,"categories":["root-instruction","cursor-rule","project-spec"],"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md",".cursorrules","docs/api.md","RULES.md"],"skills":{"AGENTS.md":"# AGENTS.md\n\nThis file provides guidance to AI coding agents when working with code in this repository.\n\n## Project Overview\n\nexo is a distributed AI inference system that connects multiple devices into a cluster. It enables running large language models across multiple machines using MLX as the inference backend and zenoh for peer-to-peer networking.\n\n## Build & Run Commands\n\n```bash\n# Build the dashboard (required before running exo)\ncd dashboard && npm install && npm run build && cd ..\n\n# Run exo (starts both master and worker with API at http://localhost:52415)\nuv run exo\n\n# Run with verbose logging\nuv run exo -v   # or -vv for more verbose\n\n# Run tests (excludes slow tests by default)\nuv run pytest\n\n# Run all tests including slow tests\nuv run pytest -m \"\"\n\n# Run a specific test file\nuv run pytest src/exo/shared/tests/test_election.py\n\n# Run a specific test function\nuv run pytest src/exo/shared/tests/test_election.py::test_function_name\n\n# Type checking (strict mode)\nuv run basedpyright\n\n# Linting\nuv run ruff check\n\n# Format code (using nix)\nnix fmt\n```\n\n## Pre-Commit Checks (REQUIRED)\n\n**IMPORTANT: Always run these checks before committing code. CI will fail if these don't pass.**\n\n```bash\n# 1. Type checking - MUST pass with 0 errors\nuv run basedpyright\n\n# 2. Linting - MUST pass\nuv run ruff check\n\n# 3. Formatting - MUST be applied\nnix fmt\n\n# 4. Tests - MUST pass\nuv run pytest\n```\n\nRun all checks in sequence:\n```bash\nuv run basedpyright && uv run ruff check && nix fmt && uv run pytest\n```\n\nIf `nix fmt` changes any files, stage them before committing. The CI runs `nix flake check` which verifies formatting, linting, and runs Rust tests.\n\n## Architecture\n\n### Node Composition\nA single exo `Node` (src/exo/main.py) runs multiple components:\n- **Router**: zenoh-based pub/sub messaging via Rust bindings (exo_rs)\n- **Worker**: Handles inference tasks, downloads models, manages runner processes\n- **Master**: Coordinates cluster state, places model instances across nodes\n- **Election**: Bully algorithm for master election\n- **API**: FastAPI server for OpenAI-compatible chat completions\n\n### Message Flow\nComponents communicate via typed pub/sub topics (src/exo/routing/topics.py):\n- `GLOBAL_EVENTS`: Master broadcasts indexed events to all workers\n- `LOCAL_EVENTS`: Workers send events to master for indexing\n- `COMMANDS`: Workers/API send commands to master\n- `ELECTION_MESSAGES`: Election protocol messages\n- `CONNECTION_MESSAGES`: zenoh connection updates\n\n### Event Sourcing\nThe system uses event sourcing for state management:\n- `State` (src/exo/shared/types/state.py): Immutable state object\n- `apply()` (src/exo/shared/apply.py): Pure function that applies events to state\n- Master indexes events and broadcasts; workers apply indexed events\n- The master only keeps recent events. A node that falls further behind (e.g. a late joiner) gets a `StateSnapshot` instead of a replay, so never assume a node has seen every event since the session started\n\n### Key Type Hierarchy\n- `src/exo/shared/types/`: Pydantic models for all shared types\n  - `events.py`: Event types (discriminated union)\n  - `commands.py`: Command types\n  - `tasks.py`: Task types for worker execution\n  - `state.py`: Cluster state model\n\n### Rust Components\nRust code in `rust/` provides:\n- `networking`: zenoh networking (gossipsub, peer discovery)\n- `exo_rs`: PyO3 bindings exposing Rust to Python\n- `system_custodian`: System-level operations\n\n### Dashboard\nSvelte 5 + TypeScript frontend in `dashboard/`. Build output goes to `dashboard/build/` and is served by the API.\n\n## Code Style Requirements\n\nFrom .cursorrules:\n- Strict, exhaustive typing - never bypass the type-checker\n- Use `Literal[...]` for enum-like sets, `typing.NewType` for primitives\n- Pydantic models with `frozen=True` and `strict=True`\n- Pure functions with injectable effect handlers for side-effects\n- Descriptive names - no abbreviations or 3-letter acronyms\n- Catch exceptions only where you can handle them meaningfully\n- Use `@final` and immutability wherever applicable\n\n## Testing\n\nTests use pytest-asyncio with `asyncio_mode = \"auto\"`. Tests are in `tests/` subdirectories alongside the code they test. The `EXO_TESTS=1` env var is set during tests.\n\n## Dashboard UI Testing & Screenshots\n\n### Building and Running the Dashboard\n```bash\n# Build the dashboard (must be done before running exo)\ncd dashboard && npm install && npm run build && cd ..\n\n# Start exo (serves the dashboard at http://localhost:52415)\nuv run exo &\nsleep 8  # Wait for server to start\n```\n\n### Taking Headless Screenshots with Playwright\nUse Playwright with headless Chromium for programmatic screenshots — no manual browser interaction needed.\n\n**Setup (one-time):**\n```bash\nnpx --yes playwright install chromium\ncd /tmp && npm init -y && npm install playwright\n```\n\n**Taking screenshots:**\n```javascript\n// Run from /tmp where playwright is installed: cd /tmp && node -e \"...\"\nconst { chromium } = require('playwright');\n(async () => {\n  const browser = await chromium.launch({ headless: true });\n  const page = await browser.newPage({ viewport: { width: 1280, height: 800 } });\n  await page.goto('http://localhost:52415', { waitUntil: 'networkidle' });\n  await page.waitForTimeout(2000);\n\n  // Inject test data into localStorage if needed (e.g., recent models)\n  await page.evaluate(() => {\n    localStorage.setItem('exo-recent-models', JSON.stringify([\n      { modelId: 'mlx-community/Qwen3-30B-A3B-4bit', launchedAt: Date.now() },\n    ]));\n  });\n  await page.reload({ waitUntil: 'networkidle' });\n  await page.waitForTimeout(2000);\n\n  // Interact with UI elements\n  await page.locator('text=SELECT MODEL').click();\n  await page.waitForTimeout(1000);\n\n  // Take screenshot\n  await page.screenshot({ path: '/tmp/screenshot.png', fullPage: false });\n  await browser.close();\n})();\n```\n\n### Uploading Images to GitHub PRs\nGitHub's API doesn't support direct image upload for PR comments. Workaround:\n\n1. **Commit images to the branch** (temporarily):\n   ```bash\n   cp /tmp/screenshot.png .\n   git add screenshot.png\n   git commit -m \"temp: add screenshots for PR\"\n   git push origin <branch>\n   COMMIT_SHA=$(git rev-parse HEAD)\n   ```\n\n2. **Post PR comment** referencing the raw image URL (uses permanent commit SHA so images survive deletion):\n   ```bash\n   gh pr comment <PR_NUMBER> --body \"![Screenshot](https://raw.githubusercontent.com/exo-explore/exo/${COMMIT_SHA}/screenshot.png)\"\n   ```\n\n3. **Remove the images** from the branch:\n   ```bash\n   git rm screenshot.png\n   git commit -m \"chore: remove temporary screenshot files\"\n   git push origin <branch>\n   ```\n   The images still render in the PR comment because they reference the permanent commit SHA.\n",".cursorrules":"# follow **every** rule exactly; report any violation instead of silently fixing it.\n\nYou must prioritize straightforward code semantics, well-named types, clear function signatures, and robust, carefully-chosen abstractions. Think about how your decisions might impact these aspects of code quality before proposing any changes.\n\nYou can use the advanced features of `typing`. You have access to all of the new features from Python 3.13, 3.12, 3.11...\n\n**When you're done making your changes, remove any redundant comments that you may have left; the comments that remain should only apply to complex segments of code, adding relevant context.**\n\n## 1. Code Discipline\n\n* Eliminate superfluous `try` / `catch` and `if` branches through strict typing and static analysis.\n* Use pure functions unless you must mutate fixed state—then wrap that state in a class.\n* Every function is **referentially transparent**: same inputs ⇒ same outputs, no hidden state, no unintended I/O.\n* Put side-effects in injectable “effect handlers”; keep core logic pure.\n\n## 2. Naming\n\n* Choose descriptive, non-abbreviated names—no 3-letter acronyms or non-standard contractions.\n* Anyone reading a function’s type signature alone should grasp its purpose without extra context.\n\n## 3. Typing\n\n* Maintain **strict, exhaustive** typing; never bypass the type-checker.\n* Default to `Literal[...]` when an enum-like set is needed.\n* Prefer built-in types; when two values share structure but differ in meaning, enforce separation:\n  * Use `typing.NewType` for primitives (zero runtime cost).\n  * For serialisable objects, add a `type: str` field that states the object’s identity.\n\n## 4. Pydantic\n\n* Read, respect, and rely on Pydantic docs.\n* Centralise a common `ConfigDict` with `frozen=True` and `strict=True` (or stricter) and reuse it everywhere.\n* For hierarchies of `BaseModel` variants, declare a discriminated union with `typing.Annotated[Base, Field(discriminator='variant')]`; publish a single `TypeAdapter[Base]` so all variants share one strict validator.\n\n## 5. IDs & UUIDs\n\n* Subclass Pydantic’s `UUID4` for custom ID types.\n* Generate fresh IDs with `uuid.uuid4()`.\n* Create idempotency keys by hashing *persisted* state plus a **function-specific salt** to avoid collisions after crashes.\n\n## 6. Error Handling\n\n* Catch an exception **only** where you can handle or transform it meaningfully.\n* State in the docstring **where** each exception is expected to be handled and **why**.\n\n## 7. Dependencies\n\n* Introduce new external dependencies only after approval.\n* Request only libraries common in production environments.\n\n## 8. Use of `@final` & Freezing\n\n* Mark classes, methods, and variables as `@final` or otherwise immutable wherever applicable.\n\n## 9. Repository Workflow\n\nIf you spot a rule violation within code that you've not been asked to work on directly, inform the user rather than patching it ad-hoc.\n\n\n---\n\n### One-Sentence Summary\n\nWrite strictly-typed, pure, self-describing Python that uses Pydantic, well-scoped side-effects, immutable state, approved dependencies, and explicit error handling\n","docs/api.md":"# EXO API – Technical Reference\n\nThis document describes the REST API exposed by the **EXO** service, as implemented in:\n\n`src/exo/api/main.py` (request and response types are in `src/exo/api/types/`)\n\nThe API is used to manage model instances in the cluster, inspect cluster state, and perform inference using multiple API-compatible interfaces.\n\nBase URL example:\n\n```\nhttp://localhost:52415\n```\n\n## 1. General / Meta Endpoints\n\n### Get Master Node ID\n\n**GET** `/node_id`\n\nReturns the identifier of the current master node.\n\n**Response (example):**\n\n```json\n{\n  \"node_id\": \"node-1234\"\n}\n```\n\n### Get Cluster State\n\n**GET** `/state`\n\nReturns the current state of the cluster, including nodes and active instances.\n\n**Response:**\nJSON object describing topology, nodes, and instances.\n\n### Get Events\n\n**GET** `/events`\n\nReturns the most recent internal events this node has applied, up to 10,000 of them (mainly for debugging and observability). Every generated token is an event, so under load this covers the last few seconds to minutes. A node that joined late, or fell far behind, catches up from a state snapshot, so it only has the events after that point.\n\n**Response:**\nArray of event objects.\n\n## 2. Model Instance Management\n\n### Create Instance\n\n**POST** `/instance`\n\nCreates a new model instance in the cluster.\n\n**Request body (example):**\n\n```json\n{\n  \"instance\": { ... }\n}\n```\n\nThe `instance` is a placement as returned by `/instance/previews` (each preview's\n`instance` field), for example the first preview without an error.\n\n**Response:**\nCommand acknowledgement. Instance creation is asynchronous; clients should wait\nfor the model to appear through `/instance/await` before sending inference\nrequests for that model.\n\n### Delete Instance\n\n**DELETE** `/instance/{instance_id}`\n\nDeletes an existing instance by ID.\n\n**Path parameters:**\n\n* `instance_id`: string, ID of the instance to delete\n\n**Response:**\nStatus / confirmation JSON.\n\n### Get Instance\n\n**GET** `/instance/{instance_id}`\n\nReturns details of a specific instance.\n\n**Path parameters:**\n\n* `instance_id`: string\n\n**Response:**\nJSON description of the instance.\n\n### Await Instance\n\n**GET** `/instance/await?model_id=...&timeout_seconds=0`\n\nWaits until API state contains an instance for the requested model. The response\nis an SSE stream so clients receive keep-alive comments while waiting.\n\n**Query parameters:**\n\n* `model_id`: string, required\n* `timeout_seconds`: float, optional, default `0`. `0` waits indefinitely;\n  positive values time out after that many seconds. Maximum positive value:\n  `300`.\n\n**Stream messages:**\n\n```text\ndata: {\"type\": \"ready\", \"instance\": {...}}\n\ndata: {\"type\": \"timeout\", \"message\": \"No instance found for model ...\"}\n```\n\nThe HTTP status is `200` for both messages because the stream starts before the\nfinal result is known. The `type` field disambiguates the terminal message.\n\n### Preview Placements\n\n**GET** `/instance/previews?model_id=...`\n\nReturns possible placement previews for a given model.\n\n**Query parameters:**\n\n* `model_id`: string, required\n\n**Response:**\nArray of placement preview objects.\n\n### Compute Placement\n\n**GET** `/instance/placement`\n\nComputes a placement for a potential instance without creating it.\n\n**Query parameters (typical):**\n\n* `model_id`: string\n* `sharding`: string or config\n* `instance_meta`: JSON-encoded metadata\n* `min_nodes`: integer\n\n**Response:**\nJSON object describing the proposed placement / instance configuration.\n\n### Place Instance\n\n**POST** `/place_instance`\n\nPlaces an instance for a model using the server's placement logic.\n\n**Request body:**\nJSON describing the instance to be placed.\n\n**Response:**\nCommand acknowledgement. The instance may not be ready immediately; wait for it\nto appear through `/instance/await` before sending inference requests.\n\n## 3. Models\n\n### List Models\n\n**GET** `/models`\n**GET** `/v1/models` (alias)\n\nReturns the list of available models and their metadata.\n\n**Query parameters:**\n\n* `status`: string (optional) - Filter by `downloaded` to show only downloaded models\n\n**Response:**\nArray of model descriptors including `is_custom` field for custom HuggingFace models.\n\n### Add Custom Model\n\n**POST** `/models/add`\n\nAdd a custom model from HuggingFace hub.\n\n**Request body (example):**\n\n```json\n{\n  \"model_id\": \"mlx-community/my-custom-model\"\n}\n```\n\n**Response:**\nModel descriptor for the added model.\n\n**Security note:**\nModels with `trust_remote_code` enabled in their configuration require explicit opt-in (default is false) for security.\n\n### Delete Custom Model\n\n**DELETE** `/models/custom/{model_id}`\n\nDelete a user-added custom model card.\n\n**Path parameters:**\n\n* `model_id`: string, ID of the custom model to delete\n\n**Response:**\nConfirmation JSON with deleted model ID.\n\n### Search Models\n\n**GET** `/models/search`\n\nSearch HuggingFace Hub for mlx-community models.\n\n**Query parameters:**\n\n* `query`: string (optional) - Search query\n* `limit`: integer (default: 20) - Maximum number of results\n\n**Response:**\nArray of HuggingFace model search results.\n\n## 4. Inference / Chat Completions\n\n### OpenAI-Compatible Chat Completions\n\n**POST** `/v1/chat/completions`\n\nExecutes a chat completion request using an OpenAI-compatible schema. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"system\", \"content\": \"You are a helpful assistant.\" },\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Request parameters:**\n\n* `model`: string, required - Model ID to use\n* `messages`: array, required - Conversation messages\n* `stream`: boolean (default: false) - Enable streaming responses\n* `max_tokens`: integer (optional) - Maximum tokens to generate\n* `temperature`: float (optional) - Sampling temperature\n* `top_p`: float (optional) - Nucleus sampling parameter\n* `top_k`: integer (optional) - Top-k sampling parameter\n* `stop`: string or array (optional) - Stop sequences\n* `seed`: integer (optional) - Random seed for reproducibility\n* `enable_thinking`: boolean (optional) - Enable thinking mode for capable models (DeepSeek V3.1, Qwen3, GLM-4.7)\n* `tools`: array (optional) - Tool definitions for function calling\n* `logprobs`: boolean (optional) - Return log probabilities\n* `top_logprobs`: integer (optional) - Number of top log probabilities to return\n\n**Response:**\nOpenAI-compatible chat completion response.\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events (SSE) with format:\n\n```\ndata: {\"id\":\"...\",\"object\":\"chat.completion\",\"created\":...,\"model\":\"...\",\"choices\":[...]}\n\ndata: [DONE]\n```\n\n**Non-streaming response includes usage statistics:**\n\n```json\n{\n  \"id\": \"...\",\n  \"object\": \"chat.completion\",\n  \"created\": 1234567890,\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"choices\": [{\n    \"index\": 0,\n    \"message\": {\n      \"role\": \"assistant\",\n      \"content\": \"Hello! How can I help you?\"\n    },\n    \"finish_reason\": \"stop\"\n  }],\n  \"usage\": {\n    \"prompt_tokens\": 15,\n    \"completion_tokens\": 8,\n    \"total_tokens\": 23\n  }\n}\n```\n\n**Cancellation:**\nYou can cancel an active generation by closing the HTTP connection. The server detects the disconnection and stops processing.\n\n### Claude Messages API\n\n**POST** `/v1/messages`\n\nExecutes a chat completion request using the Claude Messages API format. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"max_tokens\": 1024,\n  \"stream\": false\n}\n```\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events with Claude-specific event types:\n\n* `message_start` - Message generation started\n* `content_block_start` - Content block started\n* `content_block_delta` - Incremental content chunk\n* `content_block_stop` - Content block completed\n* `message_delta` - Message metadata updates\n* `message_stop` - Message generation completed\n\n**Response:**\nClaude-compatible messages response.\n\n### OpenAI Responses API\n\n**POST** `/v1/responses`\n\nExecutes a chat completion request using the OpenAI Responses API format. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"input\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Request parameters:**\n\n* `model`: string, required - Model ID to use\n* `input`: string or array, required - The prompt, or a list of input items such as messages\n* `instructions`: string (optional) - System instructions\n* `max_output_tokens`: integer (optional) - Maximum tokens to generate\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events with response-specific event types:\n\n* `response.created` - Response generation started\n* `response.in_progress` - Response is being generated\n* `response.output_item.added` - New output item added\n* `response.output_text.delta` - Incremental text chunk\n* `response.output_item.done` - Output item completed\n* `response.completed` - Response generation completed\n\n**Response:**\nOpenAI Responses API-compatible response.\n\n### Benchmarked Chat Completions\n\n**POST** `/bench/chat/completions`\n\nSame as `/v1/chat/completions`, but also returns performance and generation statistics.\n\n**Request body:**\nSame schema as `/v1/chat/completions`.\n\n**Response:**\nChat completion plus benchmarking metrics including:\n\n* `prompt_tps` - Tokens per second during prompt processing\n* `generation_tps` - Tokens per second during generation\n* `prompt_tokens` - Number of prompt tokens\n* `generation_tokens` - Number of generated tokens\n* `peak_memory_usage` - Peak memory used during generation\n\n### Cancel Command\n\n**POST** `/v1/cancel/{command_id}`\n\nCancels an active generation command (text or image). Notifies workers and closes the stream.\n\n**Path parameters:**\n\n* `command_id`: string, ID of the command to cancel\n\n**Response (example):**\n\n```json\n{\n  \"message\": \"Command cancelled.\",\n  \"command_id\": \"cmd-abc-123\"\n}\n```\n\nReturns 404 if the command is not found or already completed.\n\n## 5. Ollama API Compatibility\n\nEXO provides Ollama API compatibility for tools like OpenWebUI.\n\n### Ollama Chat\n\n**POST** `/ollama/api/chat`\n**POST** `/ollama/api/api/chat` (alias)\n**POST** `/ollama/api/v1/chat` (alias)\n\nExecute a chat request using Ollama API format.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Response:**\nOllama-compatible chat response.\n\n### Ollama Generate\n\n**POST** `/ollama/api/generate`\n\nExecute a text generation request using Ollama API format.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"prompt\": \"Hello\",\n  \"stream\": false\n}\n```\n\n**Response:**\nOllama-compatible generation response.\n\n### Ollama Tags\n\n**GET** `/ollama/api/tags`\n**GET** `/ollama/api/api/tags` (alias)\n**GET** `/ollama/api/v1/tags` (alias)\n\nReturns list of downloaded models in Ollama tags format.\n\n**Response:**\nArray of model tags with metadata.\n\n### Ollama Show\n\n**POST** `/ollama/api/show`\n\nReturns model information in Ollama show format.\n\n**Request body:**\n\n```json\n{\n  \"name\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\"\n}\n```\n\n**Response:**\nModel details including modelfile and family.\n\n### Ollama PS\n\n**GET** `/ollama/api/ps`\n\nReturns list of running models (active instances).\n\n**Response:**\nArray of active model instances.\n\n### Ollama Version\n\n**GET** `/ollama/api/version`\n**HEAD** `/ollama/` (alias)\n**HEAD** `/ollama/api/version` (alias)\n\nReturns version information for Ollama API compatibility.\n\n**Response:**\n\n```json\n{\n  \"version\": \"exo v1.0\"\n}\n```\n\n## 6. Image Generation & Editing\n\n### Image Generation\n\n**POST** `/v1/images/generations`\n\nExecutes an image generation request using an OpenAI-compatible schema with additional advanced_params. Supports both streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"prompt\": \"a robot playing chess\",\n  \"model\": \"exolabs/FLUX.1-dev\",\n  \"n\": 1,\n  \"size\": \"1024x1024\",\n  \"stream\": false,\n  \"response_format\": \"b64_json\"\n}\n```\n\n**Request parameters:**\n\n* `prompt`: string, required - Text description of the image\n* `model`: string, required - Image model ID\n* `n`: integer (default: 1) - Number of images to generate\n* `size`: string (default: \"auto\") - Image dimensions. Supported sizes:\n  - `512x512`\n  - `768x768`\n  - `1024x768`\n  - `768x1024`\n  - `1024x1024`\n  - `1024x1536`\n  - `1536x1024`\n  - `1024x1365`\n  - `1365x1024`\n* `stream`: boolean (default: false) - Enable streaming for partial images\n* `partial_images`: integer (default: 0) - Number of partial images to stream during generation\n* `response_format`: string (default: \"b64_json\") - Either `url` or `b64_json`\n* `quality`: string (default: \"medium\") - Either `high`, `medium`, or `low`\n* `output_format`: string (default: \"png\") - Either `png`, `jpeg`, or `webp`\n* `advanced_params`: object (optional) - Advanced generation parameters\n\n**Advanced Parameters (`advanced_params`):**\n\n| Parameter | Type | Constraints | Description |\n|-----------|------|-------------|-------------|\n| `seed` | int | >= 0 | Random seed for reproducible generation |\n| `num_inference_steps` | int | 1-100 | Number of denoising steps |\n| `guidance` | float | 1.0-20.0 | Classifier-free guidance scale |\n| `negative_prompt` | string | - | Text describing what to avoid in the image |\n\n**Non-streaming response:**\n\n```json\n{\n  \"created\": 1234567890,\n  \"data\": [\n    {\n      \"b64_json\": \"iVBORw0KGgoAAAANSUhEUgAA...\",\n      \"url\": null\n    }\n  ]\n}\n```\n\n**Streaming response format:**\nWhen `stream=true` and `partial_images > 0`, returns Server-Sent Events:\n\n```\ndata: {\"type\":\"partial\",\"image_index\":0,\"partial_index\":1,\"total_partials\":5,\"format\":\"png\",\"data\":{\"b64_json\":\"...\"}}\n\ndata: {\"type\":\"final\",\"image_index\":0,\"format\":\"png\",\"data\":{\"b64_json\":\"...\"}}\n\ndata: [DONE]\n```\n\n### Image Editing\n\n**POST** `/v1/images/edits`\n\nExecutes an image editing request (img2img) using FLUX.1-Kontext-dev or similar models.\n\n**Request (multipart/form-data):**\n\n* `image`: file, required - Input image to edit\n* `prompt`: string, required - Text description of desired changes\n* `model`: string, required - Image editing model ID (e.g., `exolabs/FLUX.1-Kontext-dev`)\n* `n`: integer (default: 1) - Number of edited images to generate\n* `size`: string (optional) - Output image dimensions\n* `response_format`: string (default: \"b64_json\") - Either `url` or `b64_json`\n* `input_fidelity`: string (default: \"low\") - Either `low` or `high` - Controls how closely the output follows the input image\n* `stream`: string (default: \"false\") - Enable streaming\n* `partial_images`: string (default: \"0\") - Number of partial images to stream\n* `quality`: string (default: \"medium\") - Either `high`, `medium`, or `low`\n* `output_format`: string (default: \"png\") - Either `png`, `jpeg`, or `webp`\n* `advanced_params`: string (optional) - JSON-encoded advanced parameters\n\n**Response:**\nSame format as `/v1/images/generations`.\n\n### Benchmarked Image Generation\n\n**POST** `/bench/images/generations`\n\nSame as `/v1/images/generations`, but also returns generation statistics.\n\n**Request body:**\nSame schema as `/v1/images/generations`.\n\n**Response:**\nImage generation plus benchmarking metrics including:\n\n* `seconds_per_step` - Average time per denoising step\n* `total_generation_time` - Total generation time\n* `num_inference_steps` - Number of inference steps used\n* `num_images` - Number of images generated\n* `image_width` - Output image width\n* `image_height` - Output image height\n* `peak_memory_usage` - Peak memory used during generation\n\n### Benchmarked Image Editing\n\n**POST** `/bench/images/edits`\n\nSame as `/v1/images/edits`, but also returns generation statistics.\n\n**Request:**\nSame schema as `/v1/images/edits`.\n\n**Response:**\nSame format as `/bench/images/generations`, including `generation_stats`.\n\n### List Images\n\n**GET** `/images`\n\nList all stored images.\n\n**Response:**\nArray of image metadata including URLs and expiration times.\n\n### Get Image\n\n**GET** `/images/{image_id}`\n\nRetrieve a stored image by ID.\n\n**Path parameters:**\n\n* `image_id`: string, ID of the image\n\n**Response:**\nImage file with appropriate content type.\n\n## 7. Complete Endpoint Summary\n\n```\n# General\nGET     /node_id\nGET     /state\nGET     /events\n\n# Instance Management\nPOST    /instance\nGET     /instance/await\nGET     /instance/previews\nGET     /instance/placement\nGET     /instance/{instance_id}\nDELETE  /instance/{instance_id}\nPOST    /place_instance\n\n# Models\nGET     /models\nGET     /v1/models\nPOST    /models/add\nDELETE  /models/custom/{model_id}\nGET     /models/search\n\n# Text Generation (OpenAI Chat Completions)\nPOST    /v1/chat/completions\nPOST    /bench/chat/completions\n\n# Text Generation (Claude Messages API)\nPOST    /v1/messages\n\n# Text Generation (OpenAI Responses API)\nPOST    /v1/responses\n\n# Text Generation (Ollama API)\nPOST    /ollama/api/chat\nPOST    /ollama/api/api/chat\nPOST    /ollama/api/v1/chat\nPOST    /ollama/api/generate\nGET     /ollama/api/tags\nGET     /ollama/api/api/tags\nGET     /ollama/api/v1/tags\nPOST    /ollama/api/show\nGET     /ollama/api/ps\nGET     /ollama/api/version\nHEAD    /ollama/\nHEAD    /ollama/api/version\n\n# Command Control\nPOST    /v1/cancel/{command_id}\n\n# Image Generation\nPOST    /v1/images/generations\nPOST    /bench/images/generations\nPOST    /v1/images/edits\nPOST    /bench/images/edits\nGET     /images\nGET     /images/{image_id}\n```\n\n## 8. Notes\n\n### API Compatibility\n\nEXO provides multiple API-compatible interfaces:\n\n* **OpenAI Chat Completions API** - Compatible with OpenAI clients and tools\n* **Claude Messages API** - Compatible with Anthropic's Claude API format\n* **OpenAI Responses API** - Compatible with OpenAI's Responses API format\n* **Ollama API** - Compatible with Ollama and tools like OpenWebUI\n\nExisting OpenAI, Claude, or Ollama clients can be pointed to EXO by changing the base URL.\n\n### Custom Models\n\nYou can add custom models from HuggingFace using the `/models/add` endpoint. Custom models are identified by the `is_custom` field in model list responses.\n\n**Security:** Models requiring `trust_remote_code` must be explicitly enabled (default is false) for security. Only enable this if you trust the model's remote code.\n\n### Usage Statistics\n\nChat completion responses include usage statistics with:\n\n* `prompt_tokens` - Number of tokens in the prompt\n* `completion_tokens` - Number of tokens generated\n* `total_tokens` - Sum of prompt and completion tokens\n\n### Request Cancellation\n\nYou can cancel active requests by:\n\n1. Closing the HTTP connection (for streaming requests)\n2. Calling `/v1/cancel/{command_id}` (for any request)\n\nThe server detects cancellation and stops processing immediately.\n\n### Instance Placement\n\nThe instance placement endpoints allow you to plan and preview cluster allocations before creating instances. This helps optimize resource usage across nodes.\n\n### Observability\n\nThe `/events` and `/state` endpoints are primarily intended for operational visibility and debugging.\n","RULES.md":"# Repository Rules\n\n* if you see any code that violates these rules, raise it with me directly rather than trying to fix.\n  * where applicable, file a GitHub Issue.\n* adhere to these rules strictly.\n\n## General Rules\n\n* if its possible to eliminate an extra try-catch or if-statement at runtime using type-level discipline, do it!\n* name your types, functions, and classes appropriately.\n  * no three-letter acronyms.\n  * no non-standard contractions.\n  * each data type has a meaning, pick a name which is accurate and descriptive.\n  * the average layman should be able to easily understand what your function does using the function signature alone!\n    * sometimes, there will be exceptions. eg, when you're using specific technical terms that are well understood (saga, event, etc).\n    * usually, you'll think that your code is an exception to the rules, but it won't be.\n\n## State, Functions and Classes\n\n* every function, given the same inputs, should produce the same outputs. ie, no hidden state.\n* use classes to prevent fixed state from being mutated arbitrarily (unsafely); methods provide a safe way of interfacing with state.\n* if your logic doesn't mutate fixed state, it probably belongs in a standalone function rather than a class.\n* functions shouldn't usually produce side-effects (they should be computationally pure).\n  * if, for example, you're updating a state using an event (computationally pure), and you want to trigger a saga (computational side-effect), store the logic for triggering the saga into an effect handler (a function, capable of producing side-effects, that you pass into an otherwise computationally pure function, so that it may trigger side-effects safely).\n\n## Pydantic\n\n* read the Pydantic docs.\n* respect the Pydantic docs.\n* pydantic is all you need.\n* declare and re-use a central `ConfigDict` for your use-case, you'll usually want `frozen` and `strict` to be `True`.\n\n## Unique ID (UUID) Generation\n\n* inherit from Pydantic's `UUID4` class to create your own UUID class.\n* use `uuid.uuid4()` to initialize your class with a fresh UUID where possible.\n* ensure that idempotency tags are generated by taking the salted hash of persisted state.\n  * rationale: if a node crashes and resumes from an older state, it should not accidentally re-publish the same event twice under different idempotency tags. \n  * every distinct function should feature a unique salt, so that there are no accidental collisions in idempotency tags.\n\n## Type Wrappers\n\n* reuse types that already exist in the Python standard library.\n* when two distinct data types are structurally identical (for example, different IDs which are both UUIDs but shouldn't never mixed up), make sure they can't be conflated by the type system.\n  * if you're working with a primitive data type (`str`, `int`, etc), use `NewType` (it has zero runtime overhead).\n  * if you're working with serializable data objects, consider adding a field (type `str`) that states its type.\n\n## Type Discipline\n\n* do not bypass the type-checker, preserve strict typing by any means necessary.\n* by default, use literal types (like `Literal['one', 'two']`) where an enum seems appropriate.\n\npro-tip: Python's type system is quite complex and feature-rich, so reading the documentation is often advisable; Matt discovered that Python `typing` library allows you to check that you've implemented a `match` exhaustively using `Literal` and `get_args(type)` after reading the docs.\n\n## Use of `@final`, Freezing\n\n* use wherever applicable.\n\n## Error Handling\n\n* don't try-catch for no reason.\n* make sure that you always know where and when the exceptions your code produces are meant to be handled, so that it's never a nasty surprise.\n  * always write the rationale for your error-handling down in the docstring!\n  * communicate the details to your colleagues when appropriate.\n\n## Dependencies\n\n* don't introduce any new dependencies without asking.\n* don't ask for any dependencies that aren't ubiquitous within production environments.\n\n## Commit Messages\n\n*   use the imperative mood in the subject line.\n*   prefix the subject line with a change type. our change types are:\n    *   `documentation`: documentation changes.\n    *   `feature`: a new feature.\n    *   `refactor`: a code change that neither fixes a bug nor adds a feature.\n    *   `bugfix`: a bug fix.\n    *   `chore`: routine tasks, maintenance, or tooling changes.\n    *   `test`: adding or correcting tests.\n*   restrict the subject line to fifty characters or less.\n*   capitalize the subject line.\n*   do not end the subject line with a period.\n*   separate subject from body with a blank line."},"files":{"AGENTS.md":"# AGENTS.md\n\nThis file provides guidance to AI coding agents when working with code in this repository.\n\n## Project Overview\n\nexo is a distributed AI inference system that connects multiple devices into a cluster. It enables running large language models across multiple machines using MLX as the inference backend and zenoh for peer-to-peer networking.\n\n## Build & Run Commands\n\n```bash\n# Build the dashboard (required before running exo)\ncd dashboard && npm install && npm run build && cd ..\n\n# Run exo (starts both master and worker with API at http://localhost:52415)\nuv run exo\n\n# Run with verbose logging\nuv run exo -v   # or -vv for more verbose\n\n# Run tests (excludes slow tests by default)\nuv run pytest\n\n# Run all tests including slow tests\nuv run pytest -m \"\"\n\n# Run a specific test file\nuv run pytest src/exo/shared/tests/test_election.py\n\n# Run a specific test function\nuv run pytest src/exo/shared/tests/test_election.py::test_function_name\n\n# Type checking (strict mode)\nuv run basedpyright\n\n# Linting\nuv run ruff check\n\n# Format code (using nix)\nnix fmt\n```\n\n## Pre-Commit Checks (REQUIRED)\n\n**IMPORTANT: Always run these checks before committing code. CI will fail if these don't pass.**\n\n```bash\n# 1. Type checking - MUST pass with 0 errors\nuv run basedpyright\n\n# 2. Linting - MUST pass\nuv run ruff check\n\n# 3. Formatting - MUST be applied\nnix fmt\n\n# 4. Tests - MUST pass\nuv run pytest\n```\n\nRun all checks in sequence:\n```bash\nuv run basedpyright && uv run ruff check && nix fmt && uv run pytest\n```\n\nIf `nix fmt` changes any files, stage them before committing. The CI runs `nix flake check` which verifies formatting, linting, and runs Rust tests.\n\n## Architecture\n\n### Node Composition\nA single exo `Node` (src/exo/main.py) runs multiple components:\n- **Router**: zenoh-based pub/sub messaging via Rust bindings (exo_rs)\n- **Worker**: Handles inference tasks, downloads models, manages runner processes\n- **Master**: Coordinates cluster state, places model instances across nodes\n- **Election**: Bully algorithm for master election\n- **API**: FastAPI server for OpenAI-compatible chat completions\n\n### Message Flow\nComponents communicate via typed pub/sub topics (src/exo/routing/topics.py):\n- `GLOBAL_EVENTS`: Master broadcasts indexed events to all workers\n- `LOCAL_EVENTS`: Workers send events to master for indexing\n- `COMMANDS`: Workers/API send commands to master\n- `ELECTION_MESSAGES`: Election protocol messages\n- `CONNECTION_MESSAGES`: zenoh connection updates\n\n### Event Sourcing\nThe system uses event sourcing for state management:\n- `State` (src/exo/shared/types/state.py): Immutable state object\n- `apply()` (src/exo/shared/apply.py): Pure function that applies events to state\n- Master indexes events and broadcasts; workers apply indexed events\n- The master only keeps recent events. A node that falls further behind (e.g. a late joiner) gets a `StateSnapshot` instead of a replay, so never assume a node has seen every event since the session started\n\n### Key Type Hierarchy\n- `src/exo/shared/types/`: Pydantic models for all shared types\n  - `events.py`: Event types (discriminated union)\n  - `commands.py`: Command types\n  - `tasks.py`: Task types for worker execution\n  - `state.py`: Cluster state model\n\n### Rust Components\nRust code in `rust/` provides:\n- `networking`: zenoh networking (gossipsub, peer discovery)\n- `exo_rs`: PyO3 bindings exposing Rust to Python\n- `system_custodian`: System-level operations\n\n### Dashboard\nSvelte 5 + TypeScript frontend in `dashboard/`. Build output goes to `dashboard/build/` and is served by the API.\n\n## Code Style Requirements\n\nFrom .cursorrules:\n- Strict, exhaustive typing - never bypass the type-checker\n- Use `Literal[...]` for enum-like sets, `typing.NewType` for primitives\n- Pydantic models with `frozen=True` and `strict=True`\n- Pure functions with injectable effect handlers for side-effects\n- Descriptive names - no abbreviations or 3-letter acronyms\n- Catch exceptions only where you can handle them meaningfully\n- Use `@final` and immutability wherever applicable\n\n## Testing\n\nTests use pytest-asyncio with `asyncio_mode = \"auto\"`. Tests are in `tests/` subdirectories alongside the code they test. The `EXO_TESTS=1` env var is set during tests.\n\n## Dashboard UI Testing & Screenshots\n\n### Building and Running the Dashboard\n```bash\n# Build the dashboard (must be done before running exo)\ncd dashboard && npm install && npm run build && cd ..\n\n# Start exo (serves the dashboard at http://localhost:52415)\nuv run exo &\nsleep 8  # Wait for server to start\n```\n\n### Taking Headless Screenshots with Playwright\nUse Playwright with headless Chromium for programmatic screenshots — no manual browser interaction needed.\n\n**Setup (one-time):**\n```bash\nnpx --yes playwright install chromium\ncd /tmp && npm init -y && npm install playwright\n```\n\n**Taking screenshots:**\n```javascript\n// Run from /tmp where playwright is installed: cd /tmp && node -e \"...\"\nconst { chromium } = require('playwright');\n(async () => {\n  const browser = await chromium.launch({ headless: true });\n  const page = await browser.newPage({ viewport: { width: 1280, height: 800 } });\n  await page.goto('http://localhost:52415', { waitUntil: 'networkidle' });\n  await page.waitForTimeout(2000);\n\n  // Inject test data into localStorage if needed (e.g., recent models)\n  await page.evaluate(() => {\n    localStorage.setItem('exo-recent-models', JSON.stringify([\n      { modelId: 'mlx-community/Qwen3-30B-A3B-4bit', launchedAt: Date.now() },\n    ]));\n  });\n  await page.reload({ waitUntil: 'networkidle' });\n  await page.waitForTimeout(2000);\n\n  // Interact with UI elements\n  await page.locator('text=SELECT MODEL').click();\n  await page.waitForTimeout(1000);\n\n  // Take screenshot\n  await page.screenshot({ path: '/tmp/screenshot.png', fullPage: false });\n  await browser.close();\n})();\n```\n\n### Uploading Images to GitHub PRs\nGitHub's API doesn't support direct image upload for PR comments. Workaround:\n\n1. **Commit images to the branch** (temporarily):\n   ```bash\n   cp /tmp/screenshot.png .\n   git add screenshot.png\n   git commit -m \"temp: add screenshots for PR\"\n   git push origin <branch>\n   COMMIT_SHA=$(git rev-parse HEAD)\n   ```\n\n2. **Post PR comment** referencing the raw image URL (uses permanent commit SHA so images survive deletion):\n   ```bash\n   gh pr comment <PR_NUMBER> --body \"![Screenshot](https://raw.githubusercontent.com/exo-explore/exo/${COMMIT_SHA}/screenshot.png)\"\n   ```\n\n3. **Remove the images** from the branch:\n   ```bash\n   git rm screenshot.png\n   git commit -m \"chore: remove temporary screenshot files\"\n   git push origin <branch>\n   ```\n   The images still render in the PR comment because they reference the permanent commit SHA.\n",".cursorrules":"# follow **every** rule exactly; report any violation instead of silently fixing it.\n\nYou must prioritize straightforward code semantics, well-named types, clear function signatures, and robust, carefully-chosen abstractions. Think about how your decisions might impact these aspects of code quality before proposing any changes.\n\nYou can use the advanced features of `typing`. You have access to all of the new features from Python 3.13, 3.12, 3.11...\n\n**When you're done making your changes, remove any redundant comments that you may have left; the comments that remain should only apply to complex segments of code, adding relevant context.**\n\n## 1. Code Discipline\n\n* Eliminate superfluous `try` / `catch` and `if` branches through strict typing and static analysis.\n* Use pure functions unless you must mutate fixed state—then wrap that state in a class.\n* Every function is **referentially transparent**: same inputs ⇒ same outputs, no hidden state, no unintended I/O.\n* Put side-effects in injectable “effect handlers”; keep core logic pure.\n\n## 2. Naming\n\n* Choose descriptive, non-abbreviated names—no 3-letter acronyms or non-standard contractions.\n* Anyone reading a function’s type signature alone should grasp its purpose without extra context.\n\n## 3. Typing\n\n* Maintain **strict, exhaustive** typing; never bypass the type-checker.\n* Default to `Literal[...]` when an enum-like set is needed.\n* Prefer built-in types; when two values share structure but differ in meaning, enforce separation:\n  * Use `typing.NewType` for primitives (zero runtime cost).\n  * For serialisable objects, add a `type: str` field that states the object’s identity.\n\n## 4. Pydantic\n\n* Read, respect, and rely on Pydantic docs.\n* Centralise a common `ConfigDict` with `frozen=True` and `strict=True` (or stricter) and reuse it everywhere.\n* For hierarchies of `BaseModel` variants, declare a discriminated union with `typing.Annotated[Base, Field(discriminator='variant')]`; publish a single `TypeAdapter[Base]` so all variants share one strict validator.\n\n## 5. IDs & UUIDs\n\n* Subclass Pydantic’s `UUID4` for custom ID types.\n* Generate fresh IDs with `uuid.uuid4()`.\n* Create idempotency keys by hashing *persisted* state plus a **function-specific salt** to avoid collisions after crashes.\n\n## 6. Error Handling\n\n* Catch an exception **only** where you can handle or transform it meaningfully.\n* State in the docstring **where** each exception is expected to be handled and **why**.\n\n## 7. Dependencies\n\n* Introduce new external dependencies only after approval.\n* Request only libraries common in production environments.\n\n## 8. Use of `@final` & Freezing\n\n* Mark classes, methods, and variables as `@final` or otherwise immutable wherever applicable.\n\n## 9. Repository Workflow\n\nIf you spot a rule violation within code that you've not been asked to work on directly, inform the user rather than patching it ad-hoc.\n\n\n---\n\n### One-Sentence Summary\n\nWrite strictly-typed, pure, self-describing Python that uses Pydantic, well-scoped side-effects, immutable state, approved dependencies, and explicit error handling\n","docs/api.md":"# EXO API – Technical Reference\n\nThis document describes the REST API exposed by the **EXO** service, as implemented in:\n\n`src/exo/api/main.py` (request and response types are in `src/exo/api/types/`)\n\nThe API is used to manage model instances in the cluster, inspect cluster state, and perform inference using multiple API-compatible interfaces.\n\nBase URL example:\n\n```\nhttp://localhost:52415\n```\n\n## 1. General / Meta Endpoints\n\n### Get Master Node ID\n\n**GET** `/node_id`\n\nReturns the identifier of the current master node.\n\n**Response (example):**\n\n```json\n{\n  \"node_id\": \"node-1234\"\n}\n```\n\n### Get Cluster State\n\n**GET** `/state`\n\nReturns the current state of the cluster, including nodes and active instances.\n\n**Response:**\nJSON object describing topology, nodes, and instances.\n\n### Get Events\n\n**GET** `/events`\n\nReturns the most recent internal events this node has applied, up to 10,000 of them (mainly for debugging and observability). Every generated token is an event, so under load this covers the last few seconds to minutes. A node that joined late, or fell far behind, catches up from a state snapshot, so it only has the events after that point.\n\n**Response:**\nArray of event objects.\n\n## 2. Model Instance Management\n\n### Create Instance\n\n**POST** `/instance`\n\nCreates a new model instance in the cluster.\n\n**Request body (example):**\n\n```json\n{\n  \"instance\": { ... }\n}\n```\n\nThe `instance` is a placement as returned by `/instance/previews` (each preview's\n`instance` field), for example the first preview without an error.\n\n**Response:**\nCommand acknowledgement. Instance creation is asynchronous; clients should wait\nfor the model to appear through `/instance/await` before sending inference\nrequests for that model.\n\n### Delete Instance\n\n**DELETE** `/instance/{instance_id}`\n\nDeletes an existing instance by ID.\n\n**Path parameters:**\n\n* `instance_id`: string, ID of the instance to delete\n\n**Response:**\nStatus / confirmation JSON.\n\n### Get Instance\n\n**GET** `/instance/{instance_id}`\n\nReturns details of a specific instance.\n\n**Path parameters:**\n\n* `instance_id`: string\n\n**Response:**\nJSON description of the instance.\n\n### Await Instance\n\n**GET** `/instance/await?model_id=...&timeout_seconds=0`\n\nWaits until API state contains an instance for the requested model. The response\nis an SSE stream so clients receive keep-alive comments while waiting.\n\n**Query parameters:**\n\n* `model_id`: string, required\n* `timeout_seconds`: float, optional, default `0`. `0` waits indefinitely;\n  positive values time out after that many seconds. Maximum positive value:\n  `300`.\n\n**Stream messages:**\n\n```text\ndata: {\"type\": \"ready\", \"instance\": {...}}\n\ndata: {\"type\": \"timeout\", \"message\": \"No instance found for model ...\"}\n```\n\nThe HTTP status is `200` for both messages because the stream starts before the\nfinal result is known. The `type` field disambiguates the terminal message.\n\n### Preview Placements\n\n**GET** `/instance/previews?model_id=...`\n\nReturns possible placement previews for a given model.\n\n**Query parameters:**\n\n* `model_id`: string, required\n\n**Response:**\nArray of placement preview objects.\n\n### Compute Placement\n\n**GET** `/instance/placement`\n\nComputes a placement for a potential instance without creating it.\n\n**Query parameters (typical):**\n\n* `model_id`: string\n* `sharding`: string or config\n* `instance_meta`: JSON-encoded metadata\n* `min_nodes`: integer\n\n**Response:**\nJSON object describing the proposed placement / instance configuration.\n\n### Place Instance\n\n**POST** `/place_instance`\n\nPlaces an instance for a model using the server's placement logic.\n\n**Request body:**\nJSON describing the instance to be placed.\n\n**Response:**\nCommand acknowledgement. The instance may not be ready immediately; wait for it\nto appear through `/instance/await` before sending inference requests.\n\n## 3. Models\n\n### List Models\n\n**GET** `/models`\n**GET** `/v1/models` (alias)\n\nReturns the list of available models and their metadata.\n\n**Query parameters:**\n\n* `status`: string (optional) - Filter by `downloaded` to show only downloaded models\n\n**Response:**\nArray of model descriptors including `is_custom` field for custom HuggingFace models.\n\n### Add Custom Model\n\n**POST** `/models/add`\n\nAdd a custom model from HuggingFace hub.\n\n**Request body (example):**\n\n```json\n{\n  \"model_id\": \"mlx-community/my-custom-model\"\n}\n```\n\n**Response:**\nModel descriptor for the added model.\n\n**Security note:**\nModels with `trust_remote_code` enabled in their configuration require explicit opt-in (default is false) for security.\n\n### Delete Custom Model\n\n**DELETE** `/models/custom/{model_id}`\n\nDelete a user-added custom model card.\n\n**Path parameters:**\n\n* `model_id`: string, ID of the custom model to delete\n\n**Response:**\nConfirmation JSON with deleted model ID.\n\n### Search Models\n\n**GET** `/models/search`\n\nSearch HuggingFace Hub for mlx-community models.\n\n**Query parameters:**\n\n* `query`: string (optional) - Search query\n* `limit`: integer (default: 20) - Maximum number of results\n\n**Response:**\nArray of HuggingFace model search results.\n\n## 4. Inference / Chat Completions\n\n### OpenAI-Compatible Chat Completions\n\n**POST** `/v1/chat/completions`\n\nExecutes a chat completion request using an OpenAI-compatible schema. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"system\", \"content\": \"You are a helpful assistant.\" },\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Request parameters:**\n\n* `model`: string, required - Model ID to use\n* `messages`: array, required - Conversation messages\n* `stream`: boolean (default: false) - Enable streaming responses\n* `max_tokens`: integer (optional) - Maximum tokens to generate\n* `temperature`: float (optional) - Sampling temperature\n* `top_p`: float (optional) - Nucleus sampling parameter\n* `top_k`: integer (optional) - Top-k sampling parameter\n* `stop`: string or array (optional) - Stop sequences\n* `seed`: integer (optional) - Random seed for reproducibility\n* `enable_thinking`: boolean (optional) - Enable thinking mode for capable models (DeepSeek V3.1, Qwen3, GLM-4.7)\n* `tools`: array (optional) - Tool definitions for function calling\n* `logprobs`: boolean (optional) - Return log probabilities\n* `top_logprobs`: integer (optional) - Number of top log probabilities to return\n\n**Response:**\nOpenAI-compatible chat completion response.\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events (SSE) with format:\n\n```\ndata: {\"id\":\"...\",\"object\":\"chat.completion\",\"created\":...,\"model\":\"...\",\"choices\":[...]}\n\ndata: [DONE]\n```\n\n**Non-streaming response includes usage statistics:**\n\n```json\n{\n  \"id\": \"...\",\n  \"object\": \"chat.completion\",\n  \"created\": 1234567890,\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"choices\": [{\n    \"index\": 0,\n    \"message\": {\n      \"role\": \"assistant\",\n      \"content\": \"Hello! How can I help you?\"\n    },\n    \"finish_reason\": \"stop\"\n  }],\n  \"usage\": {\n    \"prompt_tokens\": 15,\n    \"completion_tokens\": 8,\n    \"total_tokens\": 23\n  }\n}\n```\n\n**Cancellation:**\nYou can cancel an active generation by closing the HTTP connection. The server detects the disconnection and stops processing.\n\n### Claude Messages API\n\n**POST** `/v1/messages`\n\nExecutes a chat completion request using the Claude Messages API format. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"max_tokens\": 1024,\n  \"stream\": false\n}\n```\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events with Claude-specific event types:\n\n* `message_start` - Message generation started\n* `content_block_start` - Content block started\n* `content_block_delta` - Incremental content chunk\n* `content_block_stop` - Content block completed\n* `message_delta` - Message metadata updates\n* `message_stop` - Message generation completed\n\n**Response:**\nClaude-compatible messages response.\n\n### OpenAI Responses API\n\n**POST** `/v1/responses`\n\nExecutes a chat completion request using the OpenAI Responses API format. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"input\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Request parameters:**\n\n* `model`: string, required - Model ID to use\n* `input`: string or array, required - The prompt, or a list of input items such as messages\n* `instructions`: string (optional) - System instructions\n* `max_output_tokens`: integer (optional) - Maximum tokens to generate\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events with response-specific event types:\n\n* `response.created` - Response generation started\n* `response.in_progress` - Response is being generated\n* `response.output_item.added` - New output item added\n* `response.output_text.delta` - Incremental text chunk\n* `response.output_item.done` - Output item completed\n* `response.completed` - Response generation completed\n\n**Response:**\nOpenAI Responses API-compatible response.\n\n### Benchmarked Chat Completions\n\n**POST** `/bench/chat/completions`\n\nSame as `/v1/chat/completions`, but also returns performance and generation statistics.\n\n**Request body:**\nSame schema as `/v1/chat/completions`.\n\n**Response:**\nChat completion plus benchmarking metrics including:\n\n* `prompt_tps` - Tokens per second during prompt processing\n* `generation_tps` - Tokens per second during generation\n* `prompt_tokens` - Number of prompt tokens\n* `generation_tokens` - Number of generated tokens\n* `peak_memory_usage` - Peak memory used during generation\n\n### Cancel Command\n\n**POST** `/v1/cancel/{command_id}`\n\nCancels an active generation command (text or image). Notifies workers and closes the stream.\n\n**Path parameters:**\n\n* `command_id`: string, ID of the command to cancel\n\n**Response (example):**\n\n```json\n{\n  \"message\": \"Command cancelled.\",\n  \"command_id\": \"cmd-abc-123\"\n}\n```\n\nReturns 404 if the command is not found or already completed.\n\n## 5. Ollama API Compatibility\n\nEXO provides Ollama API compatibility for tools like OpenWebUI.\n\n### Ollama Chat\n\n**POST** `/ollama/api/chat`\n**POST** `/ollama/api/api/chat` (alias)\n**POST** `/ollama/api/v1/chat` (alias)\n\nExecute a chat request using Ollama API format.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Response:**\nOllama-compatible chat response.\n\n### Ollama Generate\n\n**POST** `/ollama/api/generate`\n\nExecute a text generation request using Ollama API format.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"prompt\": \"Hello\",\n  \"stream\": false\n}\n```\n\n**Response:**\nOllama-compatible generation response.\n\n### Ollama Tags\n\n**GET** `/ollama/api/tags`\n**GET** `/ollama/api/api/tags` (alias)\n**GET** `/ollama/api/v1/tags` (alias)\n\nReturns list of downloaded models in Ollama tags format.\n\n**Response:**\nArray of model tags with metadata.\n\n### Ollama Show\n\n**POST** `/ollama/api/show`\n\nReturns model information in Ollama show format.\n\n**Request body:**\n\n```json\n{\n  \"name\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\"\n}\n```\n\n**Response:**\nModel details including modelfile and family.\n\n### Ollama PS\n\n**GET** `/ollama/api/ps`\n\nReturns list of running models (active instances).\n\n**Response:**\nArray of active model instances.\n\n### Ollama Version\n\n**GET** `/ollama/api/version`\n**HEAD** `/ollama/` (alias)\n**HEAD** `/ollama/api/version` (alias)\n\nReturns version information for Ollama API compatibility.\n\n**Response:**\n\n```json\n{\n  \"version\": \"exo v1.0\"\n}\n```\n\n## 6. Image Generation & Editing\n\n### Image Generation\n\n**POST** `/v1/images/generations`\n\nExecutes an image generation request using an OpenAI-compatible schema with additional advanced_params. Supports both streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"prompt\": \"a robot playing chess\",\n  \"model\": \"exolabs/FLUX.1-dev\",\n  \"n\": 1,\n  \"size\": \"1024x1024\",\n  \"stream\": false,\n  \"response_format\": \"b64_json\"\n}\n```\n\n**Request parameters:**\n\n* `prompt`: string, required - Text description of the image\n* `model`: string, required - Image model ID\n* `n`: integer (default: 1) - Number of images to generate\n* `size`: string (default: \"auto\") - Image dimensions. Supported sizes:\n  - `512x512`\n  - `768x768`\n  - `1024x768`\n  - `768x1024`\n  - `1024x1024`\n  - `1024x1536`\n  - `1536x1024`\n  - `1024x1365`\n  - `1365x1024`\n* `stream`: boolean (default: false) - Enable streaming for partial images\n* `partial_images`: integer (default: 0) - Number of partial images to stream during generation\n* `response_format`: string (default: \"b64_json\") - Either `url` or `b64_json`\n* `quality`: string (default: \"medium\") - Either `high`, `medium`, or `low`\n* `output_format`: string (default: \"png\") - Either `png`, `jpeg`, or `webp`\n* `advanced_params`: object (optional) - Advanced generation parameters\n\n**Advanced Parameters (`advanced_params`):**\n\n| Parameter | Type | Constraints | Description |\n|-----------|------|-------------|-------------|\n| `seed` | int | >= 0 | Random seed for reproducible generation |\n| `num_inference_steps` | int | 1-100 | Number of denoising steps |\n| `guidance` | float | 1.0-20.0 | Classifier-free guidance scale |\n| `negative_prompt` | string | - | Text describing what to avoid in the image |\n\n**Non-streaming response:**\n\n```json\n{\n  \"created\": 1234567890,\n  \"data\": [\n    {\n      \"b64_json\": \"iVBORw0KGgoAAAANSUhEUgAA...\",\n      \"url\": null\n    }\n  ]\n}\n```\n\n**Streaming response format:**\nWhen `stream=true` and `partial_images > 0`, returns Server-Sent Events:\n\n```\ndata: {\"type\":\"partial\",\"image_index\":0,\"partial_index\":1,\"total_partials\":5,\"format\":\"png\",\"data\":{\"b64_json\":\"...\"}}\n\ndata: {\"type\":\"final\",\"image_index\":0,\"format\":\"png\",\"data\":{\"b64_json\":\"...\"}}\n\ndata: [DONE]\n```\n\n### Image Editing\n\n**POST** `/v1/images/edits`\n\nExecutes an image editing request (img2img) using FLUX.1-Kontext-dev or similar models.\n\n**Request (multipart/form-data):**\n\n* `image`: file, required - Input image to edit\n* `prompt`: string, required - Text description of desired changes\n* `model`: string, required - Image editing model ID (e.g., `exolabs/FLUX.1-Kontext-dev`)\n* `n`: integer (default: 1) - Number of edited images to generate\n* `size`: string (optional) - Output image dimensions\n* `response_format`: string (default: \"b64_json\") - Either `url` or `b64_json`\n* `input_fidelity`: string (default: \"low\") - Either `low` or `high` - Controls how closely the output follows the input image\n* `stream`: string (default: \"false\") - Enable streaming\n* `partial_images`: string (default: \"0\") - Number of partial images to stream\n* `quality`: string (default: \"medium\") - Either `high`, `medium`, or `low`\n* `output_format`: string (default: \"png\") - Either `png`, `jpeg`, or `webp`\n* `advanced_params`: string (optional) - JSON-encoded advanced parameters\n\n**Response:**\nSame format as `/v1/images/generations`.\n\n### Benchmarked Image Generation\n\n**POST** `/bench/images/generations`\n\nSame as `/v1/images/generations`, but also returns generation statistics.\n\n**Request body:**\nSame schema as `/v1/images/generations`.\n\n**Response:**\nImage generation plus benchmarking metrics including:\n\n* `seconds_per_step` - Average time per denoising step\n* `total_generation_time` - Total generation time\n* `num_inference_steps` - Number of inference steps used\n* `num_images` - Number of images generated\n* `image_width` - Output image width\n* `image_height` - Output image height\n* `peak_memory_usage` - Peak memory used during generation\n\n### Benchmarked Image Editing\n\n**POST** `/bench/images/edits`\n\nSame as `/v1/images/edits`, but also returns generation statistics.\n\n**Request:**\nSame schema as `/v1/images/edits`.\n\n**Response:**\nSame format as `/bench/images/generations`, including `generation_stats`.\n\n### List Images\n\n**GET** `/images`\n\nList all stored images.\n\n**Response:**\nArray of image metadata including URLs and expiration times.\n\n### Get Image\n\n**GET** `/images/{image_id}`\n\nRetrieve a stored image by ID.\n\n**Path parameters:**\n\n* `image_id`: string, ID of the image\n\n**Response:**\nImage file with appropriate content type.\n\n## 7. Complete Endpoint Summary\n\n```\n# General\nGET     /node_id\nGET     /state\nGET     /events\n\n# Instance Management\nPOST    /instance\nGET     /instance/await\nGET     /instance/previews\nGET     /instance/placement\nGET     /instance/{instance_id}\nDELETE  /instance/{instance_id}\nPOST    /place_instance\n\n# Models\nGET     /models\nGET     /v1/models\nPOST    /models/add\nDELETE  /models/custom/{model_id}\nGET     /models/search\n\n# Text Generation (OpenAI Chat Completions)\nPOST    /v1/chat/completions\nPOST    /bench/chat/completions\n\n# Text Generation (Claude Messages API)\nPOST    /v1/messages\n\n# Text Generation (OpenAI Responses API)\nPOST    /v1/responses\n\n# Text Generation (Ollama API)\nPOST    /ollama/api/chat\nPOST    /ollama/api/api/chat\nPOST    /ollama/api/v1/chat\nPOST    /ollama/api/generate\nGET     /ollama/api/tags\nGET     /ollama/api/api/tags\nGET     /ollama/api/v1/tags\nPOST    /ollama/api/show\nGET     /ollama/api/ps\nGET     /ollama/api/version\nHEAD    /ollama/\nHEAD    /ollama/api/version\n\n# Command Control\nPOST    /v1/cancel/{command_id}\n\n# Image Generation\nPOST    /v1/images/generations\nPOST    /bench/images/generations\nPOST    /v1/images/edits\nPOST    /bench/images/edits\nGET     /images\nGET     /images/{image_id}\n```\n\n## 8. Notes\n\n### API Compatibility\n\nEXO provides multiple API-compatible interfaces:\n\n* **OpenAI Chat Completions API** - Compatible with OpenAI clients and tools\n* **Claude Messages API** - Compatible with Anthropic's Claude API format\n* **OpenAI Responses API** - Compatible with OpenAI's Responses API format\n* **Ollama API** - Compatible with Ollama and tools like OpenWebUI\n\nExisting OpenAI, Claude, or Ollama clients can be pointed to EXO by changing the base URL.\n\n### Custom Models\n\nYou can add custom models from HuggingFace using the `/models/add` endpoint. Custom models are identified by the `is_custom` field in model list responses.\n\n**Security:** Models requiring `trust_remote_code` must be explicitly enabled (default is false) for security. Only enable this if you trust the model's remote code.\n\n### Usage Statistics\n\nChat completion responses include usage statistics with:\n\n* `prompt_tokens` - Number of tokens in the prompt\n* `completion_tokens` - Number of tokens generated\n* `total_tokens` - Sum of prompt and completion tokens\n\n### Request Cancellation\n\nYou can cancel active requests by:\n\n1. Closing the HTTP connection (for streaming requests)\n2. Calling `/v1/cancel/{command_id}` (for any request)\n\nThe server detects cancellation and stops processing immediately.\n\n### Instance Placement\n\nThe instance placement endpoints allow you to plan and preview cluster allocations before creating instances. This helps optimize resource usage across nodes.\n\n### Observability\n\nThe `/events` and `/state` endpoints are primarily intended for operational visibility and debugging.\n","RULES.md":"# Repository Rules\n\n* if you see any code that violates these rules, raise it with me directly rather than trying to fix.\n  * where applicable, file a GitHub Issue.\n* adhere to these rules strictly.\n\n## General Rules\n\n* if its possible to eliminate an extra try-catch or if-statement at runtime using type-level discipline, do it!\n* name your types, functions, and classes appropriately.\n  * no three-letter acronyms.\n  * no non-standard contractions.\n  * each data type has a meaning, pick a name which is accurate and descriptive.\n  * the average layman should be able to easily understand what your function does using the function signature alone!\n    * sometimes, there will be exceptions. eg, when you're using specific technical terms that are well understood (saga, event, etc).\n    * usually, you'll think that your code is an exception to the rules, but it won't be.\n\n## State, Functions and Classes\n\n* every function, given the same inputs, should produce the same outputs. ie, no hidden state.\n* use classes to prevent fixed state from being mutated arbitrarily (unsafely); methods provide a safe way of interfacing with state.\n* if your logic doesn't mutate fixed state, it probably belongs in a standalone function rather than a class.\n* functions shouldn't usually produce side-effects (they should be computationally pure).\n  * if, for example, you're updating a state using an event (computationally pure), and you want to trigger a saga (computational side-effect), store the logic for triggering the saga into an effect handler (a function, capable of producing side-effects, that you pass into an otherwise computationally pure function, so that it may trigger side-effects safely).\n\n## Pydantic\n\n* read the Pydantic docs.\n* respect the Pydantic docs.\n* pydantic is all you need.\n* declare and re-use a central `ConfigDict` for your use-case, you'll usually want `frozen` and `strict` to be `True`.\n\n## Unique ID (UUID) Generation\n\n* inherit from Pydantic's `UUID4` class to create your own UUID class.\n* use `uuid.uuid4()` to initialize your class with a fresh UUID where possible.\n* ensure that idempotency tags are generated by taking the salted hash of persisted state.\n  * rationale: if a node crashes and resumes from an older state, it should not accidentally re-publish the same event twice under different idempotency tags. \n  * every distinct function should feature a unique salt, so that there are no accidental collisions in idempotency tags.\n\n## Type Wrappers\n\n* reuse types that already exist in the Python standard library.\n* when two distinct data types are structurally identical (for example, different IDs which are both UUIDs but shouldn't never mixed up), make sure they can't be conflated by the type system.\n  * if you're working with a primitive data type (`str`, `int`, etc), use `NewType` (it has zero runtime overhead).\n  * if you're working with serializable data objects, consider adding a field (type `str`) that states its type.\n\n## Type Discipline\n\n* do not bypass the type-checker, preserve strict typing by any means necessary.\n* by default, use literal types (like `Literal['one', 'two']`) where an enum seems appropriate.\n\npro-tip: Python's type system is quite complex and feature-rich, so reading the documentation is often advisable; Matt discovered that Python `typing` library allows you to check that you've implemented a `match` exhaustively using `Literal` and `get_args(type)` after reading the docs.\n\n## Use of `@final`, Freezing\n\n* use wherever applicable.\n\n## Error Handling\n\n* don't try-catch for no reason.\n* make sure that you always know where and when the exceptions your code produces are meant to be handled, so that it's never a nasty surprise.\n  * always write the rationale for your error-handling down in the docstring!\n  * communicate the details to your colleagues when appropriate.\n\n## Dependencies\n\n* don't introduce any new dependencies without asking.\n* don't ask for any dependencies that aren't ubiquitous within production environments.\n\n## Commit Messages\n\n*   use the imperative mood in the subject line.\n*   prefix the subject line with a change type. our change types are:\n    *   `documentation`: documentation changes.\n    *   `feature`: a new feature.\n    *   `refactor`: a code change that neither fixes a bug nor adds a feature.\n    *   `bugfix`: a bug fix.\n    *   `chore`: routine tasks, maintenance, or tooling changes.\n    *   `test`: adding or correcting tests.\n*   restrict the subject line to fifty characters or less.\n*   capitalize the subject line.\n*   do not end the subject line with a period.\n*   separate subject from body with a blank line."},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/exo-explore/exo/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\nThis file provides guidance to AI coding agents when working with code in this repository.\n\n## Project Overview\n\nexo is a distributed AI inference system that connects multiple devices into a cluster. It enables running large language models across multiple machines using MLX as the inference backend and zenoh for peer-to-peer networking.\n\n## Build & Run Commands\n\n```bash\n# Build the dashboard (required before running exo)\ncd dashboard && npm install && npm run build && cd ..\n\n# Run exo (starts both master and worker with API at http://localhost:52415)\nuv run exo\n\n# Run with verbose logging\nuv run exo -v   # or -vv for more verbose\n\n# Run tests (excludes slow tests by default)\nuv run pytest\n\n# Run all tests including slow tests\nuv run pytest -m \"\"\n\n# Run a specific test file\nuv run pytest src/exo/shared/tests/test_election.py\n\n# Run a specific test function\nuv run pytest src/exo/shared/tests/test_election.py::test_function_name\n\n# Type checking (strict mode)\nuv run basedpyright\n\n# Linting\nuv run ruff check\n\n# Format code (using nix)\nnix fmt\n```\n\n## Pre-Commit Checks (REQUIRED)\n\n**IMPORTANT: Always run these checks before committing code. CI will fail if these don't pass.**\n\n```bash\n# 1. Type checking - MUST pass with 0 errors\nuv run basedpyright\n\n# 2. Linting - MUST pass\nuv run ruff check\n\n# 3. Formatting - MUST be applied\nnix fmt\n\n# 4. Tests - MUST pass\nuv run pytest\n```\n\nRun all checks in sequence:\n```bash\nuv run basedpyright && uv run ruff check && nix fmt && uv run pytest\n```\n\nIf `nix fmt` changes any files, stage them before committing. The CI runs `nix flake check` which verifies formatting, linting, and runs Rust tests.\n\n## Architecture\n\n### Node Composition\nA single exo `Node` (src/exo/main.py) runs multiple components:\n- **Router**: zenoh-based pub/sub messaging via Rust bindings (exo_rs)\n- **Worker**: Handles inference tasks, downloads models, manages runner processes\n- **Master**: Coordinates cluster state, places model instances across nodes\n- **Election**: Bully algorithm for master election\n- **API**: FastAPI server for OpenAI-compatible chat completions\n\n### Message Flow\nComponents communicate via typed pub/sub topics (src/exo/routing/topics.py):\n- `GLOBAL_EVENTS`: Master broadcasts indexed events to all workers\n- `LOCAL_EVENTS`: Workers send events to master for indexing\n- `COMMANDS`: Workers/API send commands to master\n- `ELECTION_MESSAGES`: Election protocol messages\n- `CONNECTION_MESSAGES`: zenoh connection updates\n\n### Event Sourcing\nThe system uses event sourcing for state management:\n- `State` (src/exo/shared/types/state.py): Immutable state object\n- `apply()` (src/exo/shared/apply.py): Pure function that applies events to state\n- Master indexes events and broadcasts; workers apply indexed events\n- The master only keeps recent events. A node that falls further behind (e.g. a late joiner) gets a `StateSnapshot` instead of a replay, so never assume a node has seen every event since the session started\n\n### Key Type Hierarchy\n- `src/exo/shared/types/`: Pydantic models for all shared types\n  - `events.py`: Event types (discriminated union)\n  - `commands.py`: Command types\n  - `tasks.py`: Task types for worker execution\n  - `state.py`: Cluster state model\n\n### Rust Components\nRust code in `rust/` provides:\n- `networking`: zenoh networking (gossipsub, peer discovery)\n- `exo_rs`: PyO3 bindings exposing Rust to Python\n- `system_custodian`: System-level operations\n\n### Dashboard\nSvelte 5 + TypeScript frontend in `dashboard/`. Build output goes to `dashboard/build/` and is served by the API.\n\n## Code Style Requirements\n\nFrom .cursorrules:\n- Strict, exhaustive typing - never bypass the type-checker\n- Use `Literal[...]` for enum-like sets, `typing.NewType` for primitives\n- Pydantic models with `frozen=True` and `strict=True`\n- Pure functions with injectable effect handlers for side-effects\n- Descriptive names - no abbreviations or 3-letter acronyms\n- Catch exceptions only where you can handle them meaningfully\n- Use `@final` and immutability wherever applicable\n\n## Testing\n\nTests use pytest-asyncio with `asyncio_mode = \"auto\"`. Tests are in `tests/` subdirectories alongside the code they test. The `EXO_TESTS=1` env var is set during tests.\n\n## Dashboard UI Testing & Screenshots\n\n### Building and Running the Dashboard\n```bash\n# Build the dashboard (must be done before running exo)\ncd dashboard && npm install && npm run build && cd ..\n\n# Start exo (serves the dashboard at http://localhost:52415)\nuv run exo &\nsleep 8  # Wait for server to start\n```\n\n### Taking Headless Screenshots with Playwright\nUse Playwright with headless Chromium for programmatic screenshots — no manual browser interaction needed.\n\n**Setup (one-time):**\n```bash\nnpx --yes playwright install chromium\ncd /tmp && npm init -y && npm install playwright\n```\n\n**Taking screenshots:**\n```javascript\n// Run from /tmp where playwright is installed: cd /tmp && node -e \"...\"\nconst { chromium } = require('playwright');\n(async () => {\n  const browser = await chromium.launch({ headless: true });\n  const page = await browser.newPage({ viewport: { width: 1280, height: 800 } });\n  await page.goto('http://localhost:52415', { waitUntil: 'networkidle' });\n  await page.waitForTimeout(2000);\n\n  // Inject test data into localStorage if needed (e.g., recent models)\n  await page.evaluate(() => {\n    localStorage.setItem('exo-recent-models', JSON.stringify([\n      { modelId: 'mlx-community/Qwen3-30B-A3B-4bit', launchedAt: Date.now() },\n    ]));\n  });\n  await page.reload({ waitUntil: 'networkidle' });\n  await page.waitForTimeout(2000);\n\n  // Interact with UI elements\n  await page.locator('text=SELECT MODEL').click();\n  await page.waitForTimeout(1000);\n\n  // Take screenshot\n  await page.screenshot({ path: '/tmp/screenshot.png', fullPage: false });\n  await browser.close();\n})();\n```\n\n### Uploading Images to GitHub PRs\nGitHub's API doesn't support direct image upload for PR comments. Workaround:\n\n1. **Commit images to the branch** (temporarily):\n   ```bash\n   cp /tmp/screenshot.png .\n   git add screenshot.png\n   git commit -m \"temp: add screenshots for PR\"\n   git push origin <branch>\n   COMMIT_SHA=$(git rev-parse HEAD)\n   ```\n\n2. **Post PR comment** referencing the raw image URL (uses permanent commit SHA so images survive deletion):\n   ```bash\n   gh pr comment <PR_NUMBER> --body \"![Screenshot](https://raw.githubusercontent.com/exo-explore/exo/${COMMIT_SHA}/screenshot.png)\"\n   ```\n\n3. **Remove the images** from the branch:\n   ```bash\n   git rm screenshot.png\n   git commit -m \"chore: remove temporary screenshot files\"\n   git push origin <branch>\n   ```\n   The images still render in the PR comment because they reference the permanent commit SHA.\n","isInternal":false,"tokens":1636,"sizeBytes":6730},{"name":"RULES.md","path":"RULES.md","rawUrl":"https://raw.githubusercontent.com/exo-explore/exo/HEAD/RULES.md","title":"Development & Architecture Rules","category":"root-instruction","format":"markdown","content":"# Repository Rules\n\n* if you see any code that violates these rules, raise it with me directly rather than trying to fix.\n  * where applicable, file a GitHub Issue.\n* adhere to these rules strictly.\n\n## General Rules\n\n* if its possible to eliminate an extra try-catch or if-statement at runtime using type-level discipline, do it!\n* name your types, functions, and classes appropriately.\n  * no three-letter acronyms.\n  * no non-standard contractions.\n  * each data type has a meaning, pick a name which is accurate and descriptive.\n  * the average layman should be able to easily understand what your function does using the function signature alone!\n    * sometimes, there will be exceptions. eg, when you're using specific technical terms that are well understood (saga, event, etc).\n    * usually, you'll think that your code is an exception to the rules, but it won't be.\n\n## State, Functions and Classes\n\n* every function, given the same inputs, should produce the same outputs. ie, no hidden state.\n* use classes to prevent fixed state from being mutated arbitrarily (unsafely); methods provide a safe way of interfacing with state.\n* if your logic doesn't mutate fixed state, it probably belongs in a standalone function rather than a class.\n* functions shouldn't usually produce side-effects (they should be computationally pure).\n  * if, for example, you're updating a state using an event (computationally pure), and you want to trigger a saga (computational side-effect), store the logic for triggering the saga into an effect handler (a function, capable of producing side-effects, that you pass into an otherwise computationally pure function, so that it may trigger side-effects safely).\n\n## Pydantic\n\n* read the Pydantic docs.\n* respect the Pydantic docs.\n* pydantic is all you need.\n* declare and re-use a central `ConfigDict` for your use-case, you'll usually want `frozen` and `strict` to be `True`.\n\n## Unique ID (UUID) Generation\n\n* inherit from Pydantic's `UUID4` class to create your own UUID class.\n* use `uuid.uuid4()` to initialize your class with a fresh UUID where possible.\n* ensure that idempotency tags are generated by taking the salted hash of persisted state.\n  * rationale: if a node crashes and resumes from an older state, it should not accidentally re-publish the same event twice under different idempotency tags. \n  * every distinct function should feature a unique salt, so that there are no accidental collisions in idempotency tags.\n\n## Type Wrappers\n\n* reuse types that already exist in the Python standard library.\n* when two distinct data types are structurally identical (for example, different IDs which are both UUIDs but shouldn't never mixed up), make sure they can't be conflated by the type system.\n  * if you're working with a primitive data type (`str`, `int`, etc), use `NewType` (it has zero runtime overhead).\n  * if you're working with serializable data objects, consider adding a field (type `str`) that states its type.\n\n## Type Discipline\n\n* do not bypass the type-checker, preserve strict typing by any means necessary.\n* by default, use literal types (like `Literal['one', 'two']`) where an enum seems appropriate.\n\npro-tip: Python's type system is quite complex and feature-rich, so reading the documentation is often advisable; Matt discovered that Python `typing` library allows you to check that you've implemented a `match` exhaustively using `Literal` and `get_args(type)` after reading the docs.\n\n## Use of `@final`, Freezing\n\n* use wherever applicable.\n\n## Error Handling\n\n* don't try-catch for no reason.\n* make sure that you always know where and when the exceptions your code produces are meant to be handled, so that it's never a nasty surprise.\n  * always write the rationale for your error-handling down in the docstring!\n  * communicate the details to your colleagues when appropriate.\n\n## Dependencies\n\n* don't introduce any new dependencies without asking.\n* don't ask for any dependencies that aren't ubiquitous within production environments.\n\n## Commit Messages\n\n*   use the imperative mood in the subject line.\n*   prefix the subject line with a change type. our change types are:\n    *   `documentation`: documentation changes.\n    *   `feature`: a new feature.\n    *   `refactor`: a code change that neither fixes a bug nor adds a feature.\n    *   `bugfix`: a bug fix.\n    *   `chore`: routine tasks, maintenance, or tooling changes.\n    *   `test`: adding or correcting tests.\n*   restrict the subject line to fifty characters or less.\n*   capitalize the subject line.\n*   do not end the subject line with a period.\n*   separate subject from body with a blank line.","isInternal":false,"tokens":1033,"sizeBytes":4653},{"name":"api.md","path":"docs/api.md","rawUrl":"https://raw.githubusercontent.com/exo-explore/exo/HEAD/docs/api.md","title":"docs - API Interface Contract & Specifications","category":"project-spec","format":"markdown","content":"# EXO API – Technical Reference\n\nThis document describes the REST API exposed by the **EXO** service, as implemented in:\n\n`src/exo/api/main.py` (request and response types are in `src/exo/api/types/`)\n\nThe API is used to manage model instances in the cluster, inspect cluster state, and perform inference using multiple API-compatible interfaces.\n\nBase URL example:\n\n```\nhttp://localhost:52415\n```\n\n## 1. General / Meta Endpoints\n\n### Get Master Node ID\n\n**GET** `/node_id`\n\nReturns the identifier of the current master node.\n\n**Response (example):**\n\n```json\n{\n  \"node_id\": \"node-1234\"\n}\n```\n\n### Get Cluster State\n\n**GET** `/state`\n\nReturns the current state of the cluster, including nodes and active instances.\n\n**Response:**\nJSON object describing topology, nodes, and instances.\n\n### Get Events\n\n**GET** `/events`\n\nReturns the most recent internal events this node has applied, up to 10,000 of them (mainly for debugging and observability). Every generated token is an event, so under load this covers the last few seconds to minutes. A node that joined late, or fell far behind, catches up from a state snapshot, so it only has the events after that point.\n\n**Response:**\nArray of event objects.\n\n## 2. Model Instance Management\n\n### Create Instance\n\n**POST** `/instance`\n\nCreates a new model instance in the cluster.\n\n**Request body (example):**\n\n```json\n{\n  \"instance\": { ... }\n}\n```\n\nThe `instance` is a placement as returned by `/instance/previews` (each preview's\n`instance` field), for example the first preview without an error.\n\n**Response:**\nCommand acknowledgement. Instance creation is asynchronous; clients should wait\nfor the model to appear through `/instance/await` before sending inference\nrequests for that model.\n\n### Delete Instance\n\n**DELETE** `/instance/{instance_id}`\n\nDeletes an existing instance by ID.\n\n**Path parameters:**\n\n* `instance_id`: string, ID of the instance to delete\n\n**Response:**\nStatus / confirmation JSON.\n\n### Get Instance\n\n**GET** `/instance/{instance_id}`\n\nReturns details of a specific instance.\n\n**Path parameters:**\n\n* `instance_id`: string\n\n**Response:**\nJSON description of the instance.\n\n### Await Instance\n\n**GET** `/instance/await?model_id=...&timeout_seconds=0`\n\nWaits until API state contains an instance for the requested model. The response\nis an SSE stream so clients receive keep-alive comments while waiting.\n\n**Query parameters:**\n\n* `model_id`: string, required\n* `timeout_seconds`: float, optional, default `0`. `0` waits indefinitely;\n  positive values time out after that many seconds. Maximum positive value:\n  `300`.\n\n**Stream messages:**\n\n```text\ndata: {\"type\": \"ready\", \"instance\": {...}}\n\ndata: {\"type\": \"timeout\", \"message\": \"No instance found for model ...\"}\n```\n\nThe HTTP status is `200` for both messages because the stream starts before the\nfinal result is known. The `type` field disambiguates the terminal message.\n\n### Preview Placements\n\n**GET** `/instance/previews?model_id=...`\n\nReturns possible placement previews for a given model.\n\n**Query parameters:**\n\n* `model_id`: string, required\n\n**Response:**\nArray of placement preview objects.\n\n### Compute Placement\n\n**GET** `/instance/placement`\n\nComputes a placement for a potential instance without creating it.\n\n**Query parameters (typical):**\n\n* `model_id`: string\n* `sharding`: string or config\n* `instance_meta`: JSON-encoded metadata\n* `min_nodes`: integer\n\n**Response:**\nJSON object describing the proposed placement / instance configuration.\n\n### Place Instance\n\n**POST** `/place_instance`\n\nPlaces an instance for a model using the server's placement logic.\n\n**Request body:**\nJSON describing the instance to be placed.\n\n**Response:**\nCommand acknowledgement. The instance may not be ready immediately; wait for it\nto appear through `/instance/await` before sending inference requests.\n\n## 3. Models\n\n### List Models\n\n**GET** `/models`\n**GET** `/v1/models` (alias)\n\nReturns the list of available models and their metadata.\n\n**Query parameters:**\n\n* `status`: string (optional) - Filter by `downloaded` to show only downloaded models\n\n**Response:**\nArray of model descriptors including `is_custom` field for custom HuggingFace models.\n\n### Add Custom Model\n\n**POST** `/models/add`\n\nAdd a custom model from HuggingFace hub.\n\n**Request body (example):**\n\n```json\n{\n  \"model_id\": \"mlx-community/my-custom-model\"\n}\n```\n\n**Response:**\nModel descriptor for the added model.\n\n**Security note:**\nModels with `trust_remote_code` enabled in their configuration require explicit opt-in (default is false) for security.\n\n### Delete Custom Model\n\n**DELETE** `/models/custom/{model_id}`\n\nDelete a user-added custom model card.\n\n**Path parameters:**\n\n* `model_id`: string, ID of the custom model to delete\n\n**Response:**\nConfirmation JSON with deleted model ID.\n\n### Search Models\n\n**GET** `/models/search`\n\nSearch HuggingFace Hub for mlx-community models.\n\n**Query parameters:**\n\n* `query`: string (optional) - Search query\n* `limit`: integer (default: 20) - Maximum number of results\n\n**Response:**\nArray of HuggingFace model search results.\n\n## 4. Inference / Chat Completions\n\n### OpenAI-Compatible Chat Completions\n\n**POST** `/v1/chat/completions`\n\nExecutes a chat completion request using an OpenAI-compatible schema. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"system\", \"content\": \"You are a helpful assistant.\" },\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Request parameters:**\n\n* `model`: string, required - Model ID to use\n* `messages`: array, required - Conversation messages\n* `stream`: boolean (default: false) - Enable streaming responses\n* `max_tokens`: integer (optional) - Maximum tokens to generate\n* `temperature`: float (optional) - Sampling temperature\n* `top_p`: float (optional) - Nucleus sampling parameter\n* `top_k`: integer (optional) - Top-k sampling parameter\n* `stop`: string or array (optional) - Stop sequences\n* `seed`: integer (optional) - Random seed for reproducibility\n* `enable_thinking`: boolean (optional) - Enable thinking mode for capable models (DeepSeek V3.1, Qwen3, GLM-4.7)\n* `tools`: array (optional) - Tool definitions for function calling\n* `logprobs`: boolean (optional) - Return log probabilities\n* `top_logprobs`: integer (optional) - Number of top log probabilities to return\n\n**Response:**\nOpenAI-compatible chat completion response.\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events (SSE) with format:\n\n```\ndata: {\"id\":\"...\",\"object\":\"chat.completion\",\"created\":...,\"model\":\"...\",\"choices\":[...]}\n\ndata: [DONE]\n```\n\n**Non-streaming response includes usage statistics:**\n\n```json\n{\n  \"id\": \"...\",\n  \"object\": \"chat.completion\",\n  \"created\": 1234567890,\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"choices\": [{\n    \"index\": 0,\n    \"message\": {\n      \"role\": \"assistant\",\n      \"content\": \"Hello! How can I help you?\"\n    },\n    \"finish_reason\": \"stop\"\n  }],\n  \"usage\": {\n    \"prompt_tokens\": 15,\n    \"completion_tokens\": 8,\n    \"total_tokens\": 23\n  }\n}\n```\n\n**Cancellation:**\nYou can cancel an active generation by closing the HTTP connection. The server detects the disconnection and stops processing.\n\n### Claude Messages API\n\n**POST** `/v1/messages`\n\nExecutes a chat completion request using the Claude Messages API format. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"max_tokens\": 1024,\n  \"stream\": false\n}\n```\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events with Claude-specific event types:\n\n* `message_start` - Message generation started\n* `content_block_start` - Content block started\n* `content_block_delta` - Incremental content chunk\n* `content_block_stop` - Content block completed\n* `message_delta` - Message metadata updates\n* `message_stop` - Message generation completed\n\n**Response:**\nClaude-compatible messages response.\n\n### OpenAI Responses API\n\n**POST** `/v1/responses`\n\nExecutes a chat completion request using the OpenAI Responses API format. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"input\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Request parameters:**\n\n* `model`: string, required - Model ID to use\n* `input`: string or array, required - The prompt, or a list of input items such as messages\n* `instructions`: string (optional) - System instructions\n* `max_output_tokens`: integer (optional) - Maximum tokens to generate\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events with response-specific event types:\n\n* `response.created` - Response generation started\n* `response.in_progress` - Response is being generated\n* `response.output_item.added` - New output item added\n* `response.output_text.delta` - Incremental text chunk\n* `response.output_item.done` - Output item completed\n* `response.completed` - Response generation completed\n\n**Response:**\nOpenAI Responses API-compatible response.\n\n### Benchmarked Chat Completions\n\n**POST** `/bench/chat/completions`\n\nSame as `/v1/chat/completions`, but also returns performance and generation statistics.\n\n**Request body:**\nSame schema as `/v1/chat/completions`.\n\n**Response:**\nChat completion plus benchmarking metrics including:\n\n* `prompt_tps` - Tokens per second during prompt processing\n* `generation_tps` - Tokens per second during generation\n* `prompt_tokens` - Number of prompt tokens\n* `generation_tokens` - Number of generated tokens\n* `peak_memory_usage` - Peak memory used during generation\n\n### Cancel Command\n\n**POST** `/v1/cancel/{command_id}`\n\nCancels an active generation command (text or image). Notifies workers and closes the stream.\n\n**Path parameters:**\n\n* `command_id`: string, ID of the command to cancel\n\n**Response (example):**\n\n```json\n{\n  \"message\": \"Command cancelled.\",\n  \"command_id\": \"cmd-abc-123\"\n}\n```\n\nReturns 404 if the command is not found or already completed.\n\n## 5. Ollama API Compatibility\n\nEXO provides Ollama API compatibility for tools like OpenWebUI.\n\n### Ollama Chat\n\n**POST** `/ollama/api/chat`\n**POST** `/ollama/api/api/chat` (alias)\n**POST** `/ollama/api/v1/chat` (alias)\n\nExecute a chat request using Ollama API format.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Response:**\nOllama-compatible chat response.\n\n### Ollama Generate\n\n**POST** `/ollama/api/generate`\n\nExecute a text generation request using Ollama API format.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"prompt\": \"Hello\",\n  \"stream\": false\n}\n```\n\n**Response:**\nOllama-compatible generation response.\n\n### Ollama Tags\n\n**GET** `/ollama/api/tags`\n**GET** `/ollama/api/api/tags` (alias)\n**GET** `/ollama/api/v1/tags` (alias)\n\nReturns list of downloaded models in Ollama tags format.\n\n**Response:**\nArray of model tags with metadata.\n\n### Ollama Show\n\n**POST** `/ollama/api/show`\n\nReturns model information in Ollama show format.\n\n**Request body:**\n\n```json\n{\n  \"name\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\"\n}\n```\n\n**Response:**\nModel details including modelfile and family.\n\n### Ollama PS\n\n**GET** `/ollama/api/ps`\n\nReturns list of running models (active instances).\n\n**Response:**\nArray of active model instances.\n\n### Ollama Version\n\n**GET** `/ollama/api/version`\n**HEAD** `/ollama/` (alias)\n**HEAD** `/ollama/api/version` (alias)\n\nReturns version information for Ollama API compatibility.\n\n**Response:**\n\n```json\n{\n  \"version\": \"exo v1.0\"\n}\n```\n\n## 6. Image Generation & Editing\n\n### Image Generation\n\n**POST** `/v1/images/generations`\n\nExecutes an image generation request using an OpenAI-compatible schema with additional advanced_params. Supports both streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"prompt\": \"a robot playing chess\",\n  \"model\": \"exolabs/FLUX.1-dev\",\n  \"n\": 1,\n  \"size\": \"1024x1024\",\n  \"stream\": false,\n  \"response_format\": \"b64_json\"\n}\n```\n\n**Request parameters:**\n\n* `prompt`: string, required - Text description of the image\n* `model`: string, required - Image model ID\n* `n`: integer (default: 1) - Number of images to generate\n* `size`: string (default: \"auto\") - Image dimensions. Supported sizes:\n  - `512x512`\n  - `768x768`\n  - `1024x768`\n  - `768x1024`\n  - `1024x1024`\n  - `1024x1536`\n  - `1536x1024`\n  - `1024x1365`\n  - `1365x1024`\n* `stream`: boolean (default: false) - Enable streaming for partial images\n* `partial_images`: integer (default: 0) - Number of partial images to stream during generation\n* `response_format`: string (default: \"b64_json\") - Either `url` or `b64_json`\n* `quality`: string (default: \"medium\") - Either `high`, `medium`, or `low`\n* `output_format`: string (default: \"png\") - Either `png`, `jpeg`, or `webp`\n* `advanced_params`: object (optional) - Advanced generation parameters\n\n**Advanced Parameters (`advanced_params`):**\n\n| Parameter | Type | Constraints | Description |\n|-----------|------|-------------|-------------|\n| `seed` | int | >= 0 | Random seed for reproducible generation |\n| `num_inference_steps` | int | 1-100 | Number of denoising steps |\n| `guidance` | float | 1.0-20.0 | Classifier-free guidance scale |\n| `negative_prompt` | string | - | Text describing what to avoid in the image |\n\n**Non-streaming response:**\n\n```json\n{\n  \"created\": 1234567890,\n  \"data\": [\n    {\n      \"b64_json\": \"iVBORw0KGgoAAAANSUhEUgAA...\",\n      \"url\": null\n    }\n  ]\n}\n```\n\n**Streaming response format:**\nWhen `stream=true` and `partial_images > 0`, returns Server-Sent Events:\n\n```\ndata: {\"type\":\"partial\",\"image_index\":0,\"partial_index\":1,\"total_partials\":5,\"format\":\"png\",\"data\":{\"b64_json\":\"...\"}}\n\ndata: {\"type\":\"final\",\"image_index\":0,\"format\":\"png\",\"data\":{\"b64_json\":\"...\"}}\n\ndata: [DONE]\n```\n\n### Image Editing\n\n**POST** `/v1/images/edits`\n\nExecutes an image editing request (img2img) using FLUX.1-Kontext-dev or similar models.\n\n**Request (multipart/form-data):**\n\n* `image`: file, required - Input image to edit\n* `prompt`: string, required - Text description of desired changes\n* `model`: string, required - Image editing model ID (e.g., `exolabs/FLUX.1-Kontext-dev`)\n* `n`: integer (default: 1) - Number of edited images to generate\n* `size`: string (optional) - Output image dimensions\n* `response_format`: string (default: \"b64_json\") - Either `url` or `b64_json`\n* `input_fidelity`: string (default: \"low\") - Either `low` or `high` - Controls how closely the output follows the input image\n* `stream`: string (default: \"false\") - Enable streaming\n* `partial_images`: string (default: \"0\") - Number of partial images to stream\n* `quality`: string (default: \"medium\") - Either `high`, `medium`, or `low`\n* `output_format`: string (default: \"png\") - Either `png`, `jpeg`, or `webp`\n* `advanced_params`: string (optional) - JSON-encoded advanced parameters\n\n**Response:**\nSame format as `/v1/images/generations`.\n\n### Benchmarked Image Generation\n\n**POST** `/bench/images/generations`\n\nSame as `/v1/images/generations`, but also returns generation statistics.\n\n**Request body:**\nSame schema as `/v1/images/generations`.\n\n**Response:**\nImage generation plus benchmarking metrics including:\n\n* `seconds_per_step` - Average time per denoising step\n* `total_generation_time` - Total generation time\n* `num_inference_steps` - Number of inference steps used\n* `num_images` - Number of images generated\n* `image_width` - Output image width\n* `image_height` - Output image height\n* `peak_memory_usage` - Peak memory used during generation\n\n### Benchmarked Image Editing\n\n**POST** `/bench/images/edits`\n\nSame as `/v1/images/edits`, but also returns generation statistics.\n\n**Request:**\nSame schema as `/v1/images/edits`.\n\n**Response:**\nSame format as `/bench/images/generations`, including `generation_stats`.\n\n### List Images\n\n**GET** `/images`\n\nList all stored images.\n\n**Response:**\nArray of image metadata including URLs and expiration times.\n\n### Get Image\n\n**GET** `/images/{image_id}`\n\nRetrieve a stored image by ID.\n\n**Path parameters:**\n\n* `image_id`: string, ID of the image\n\n**Response:**\nImage file with appropriate content type.\n\n## 7. Complete Endpoint Summary\n\n```\n# General\nGET     /node_id\nGET     /state\nGET     /events\n\n# Instance Management\nPOST    /instance\nGET     /instance/await\nGET     /instance/previews\nGET     /instance/placement\nGET     /instance/{instance_id}\nDELETE  /instance/{instance_id}\nPOST    /place_instance\n\n# Models\nGET     /models\nGET     /v1/models\nPOST    /models/add\nDELETE  /models/custom/{model_id}\nGET     /models/search\n\n# Text Generation (OpenAI Chat Completions)\nPOST    /v1/chat/completions\nPOST    /bench/chat/completions\n\n# Text Generation (Claude Messages API)\nPOST    /v1/messages\n\n# Text Generation (OpenAI Responses API)\nPOST    /v1/responses\n\n# Text Generation (Ollama API)\nPOST    /ollama/api/chat\nPOST    /ollama/api/api/chat\nPOST    /ollama/api/v1/chat\nPOST    /ollama/api/generate\nGET     /ollama/api/tags\nGET     /ollama/api/api/tags\nGET     /ollama/api/v1/tags\nPOST    /ollama/api/show\nGET     /ollama/api/ps\nGET     /ollama/api/version\nHEAD    /ollama/\nHEAD    /ollama/api/version\n\n# Command Control\nPOST    /v1/cancel/{command_id}\n\n# Image Generation\nPOST    /v1/images/generations\nPOST    /bench/images/generations\nPOST    /v1/images/edits\nPOST    /bench/images/edits\nGET     /images\nGET     /images/{image_id}\n```\n\n## 8. Notes\n\n### API Compatibility\n\nEXO provides multiple API-compatible interfaces:\n\n* **OpenAI Chat Completions API** - Compatible with OpenAI clients and tools\n* **Claude Messages API** - Compatible with Anthropic's Claude API format\n* **OpenAI Responses API** - Compatible with OpenAI's Responses API format\n* **Ollama API** - Compatible with Ollama and tools like OpenWebUI\n\nExisting OpenAI, Claude, or Ollama clients can be pointed to EXO by changing the base URL.\n\n### Custom Models\n\nYou can add custom models from HuggingFace using the `/models/add` endpoint. Custom models are identified by the `is_custom` field in model list responses.\n\n**Security:** Models requiring `trust_remote_code` must be explicitly enabled (default is false) for security. Only enable this if you trust the model's remote code.\n\n### Usage Statistics\n\nChat completion responses include usage statistics with:\n\n* `prompt_tokens` - Number of tokens in the prompt\n* `completion_tokens` - Number of tokens generated\n* `total_tokens` - Sum of prompt and completion tokens\n\n### Request Cancellation\n\nYou can cancel active requests by:\n\n1. Closing the HTTP connection (for streaming requests)\n2. Calling `/v1/cancel/{command_id}` (for any request)\n\nThe server detects cancellation and stops processing immediately.\n\n### Instance Placement\n\nThe instance placement endpoints allow you to plan and preview cluster allocations before creating instances. This helps optimize resource usage across nodes.\n\n### Observability\n\nThe `/events` and `/state` endpoints are primarily intended for operational visibility and debugging.\n","isInternal":false,"tokens":4889,"sizeBytes":19297},{"name":".cursorrules","path":".cursorrules","rawUrl":"https://raw.githubusercontent.com/exo-explore/exo/HEAD/.cursorrules","title":"Cursor IDE Native Rules","category":"cursor-rule","format":"markdown","content":"# follow **every** rule exactly; report any violation instead of silently fixing it.\n\nYou must prioritize straightforward code semantics, well-named types, clear function signatures, and robust, carefully-chosen abstractions. Think about how your decisions might impact these aspects of code quality before proposing any changes.\n\nYou can use the advanced features of `typing`. You have access to all of the new features from Python 3.13, 3.12, 3.11...\n\n**When you're done making your changes, remove any redundant comments that you may have left; the comments that remain should only apply to complex segments of code, adding relevant context.**\n\n## 1. Code Discipline\n\n* Eliminate superfluous `try` / `catch` and `if` branches through strict typing and static analysis.\n* Use pure functions unless you must mutate fixed state—then wrap that state in a class.\n* Every function is **referentially transparent**: same inputs ⇒ same outputs, no hidden state, no unintended I/O.\n* Put side-effects in injectable “effect handlers”; keep core logic pure.\n\n## 2. Naming\n\n* Choose descriptive, non-abbreviated names—no 3-letter acronyms or non-standard contractions.\n* Anyone reading a function’s type signature alone should grasp its purpose without extra context.\n\n## 3. Typing\n\n* Maintain **strict, exhaustive** typing; never bypass the type-checker.\n* Default to `Literal[...]` when an enum-like set is needed.\n* Prefer built-in types; when two values share structure but differ in meaning, enforce separation:\n  * Use `typing.NewType` for primitives (zero runtime cost).\n  * For serialisable objects, add a `type: str` field that states the object’s identity.\n\n## 4. Pydantic\n\n* Read, respect, and rely on Pydantic docs.\n* Centralise a common `ConfigDict` with `frozen=True` and `strict=True` (or stricter) and reuse it everywhere.\n* For hierarchies of `BaseModel` variants, declare a discriminated union with `typing.Annotated[Base, Field(discriminator='variant')]`; publish a single `TypeAdapter[Base]` so all variants share one strict validator.\n\n## 5. IDs & UUIDs\n\n* Subclass Pydantic’s `UUID4` for custom ID types.\n* Generate fresh IDs with `uuid.uuid4()`.\n* Create idempotency keys by hashing *persisted* state plus a **function-specific salt** to avoid collisions after crashes.\n\n## 6. Error Handling\n\n* Catch an exception **only** where you can handle or transform it meaningfully.\n* State in the docstring **where** each exception is expected to be handled and **why**.\n\n## 7. Dependencies\n\n* Introduce new external dependencies only after approval.\n* Request only libraries common in production environments.\n\n## 8. Use of `@final` & Freezing\n\n* Mark classes, methods, and variables as `@final` or otherwise immutable wherever applicable.\n\n## 9. Repository Workflow\n\nIf you spot a rule violation within code that you've not been asked to work on directly, inform the user rather than patching it ad-hoc.\n\n\n---\n\n### One-Sentence Summary\n\nWrite strictly-typed, pure, self-describing Python that uses Pydantic, well-scoped side-effects, immutable state, approved dependencies, and explicit error handling\n","isInternal":false,"tokens":710,"sizeBytes":3126}],"systemPromptSnippet":"<yakaai_skills repo=\"exo-explore/exo\">\n<!-- File: AGENTS.md (Tokens: ~1636 | Category: root-instruction) -->\n# AGENTS.md\n\nThis file provides guidance to AI coding agents when working with code in this repository.\n\n## Project Overview\n\nexo is a distributed AI inference system that connects multiple devices into a cluster. It enables running large language models across multiple machines using MLX as the inference backend and zenoh for peer-to-peer networking.\n\n## Build & Run Commands\n\n```bash\n# Build the dashboard (required before running exo)\ncd dashboard && npm install && npm run build && cd ..\n\n# Run exo (starts both master and worker with API at http://localhost:52415)\nuv run exo\n\n# Run with verbose logging\nuv run exo -v   # or -vv for more verbose\n\n# Run tests (excludes slow tests by default)\nuv run pytest\n\n# Run all tests including slow tests\nuv run pytest -m \"\"\n\n# Run a specific test file\nuv run pytest src/exo/shared/tests/test_election.py\n\n# Run a specific test function\nuv run pytest src/exo/shared/tests/test_election.py::test_function_name\n\n# Type checking (strict mode)\nuv run basedpyright\n\n# Linting\nuv run ruff check\n\n# Format code (using nix)\nnix fmt\n```\n\n## Pre-Commit Checks (REQUIRED)\n\n**IMPORTANT: Always run these checks before committing code. CI will fail if these don't pass.**\n\n```bash\n# 1. Type checking - MUST pass with 0 errors\nuv run basedpyright\n\n# 2. Linting - MUST pass\nuv run ruff check\n\n# 3. Formatting - MUST be applied\nnix fmt\n\n# 4. Tests - MUST pass\nuv run pytest\n```\n\nRun all checks in sequence:\n```bash\nuv run basedpyright && uv run ruff check && nix fmt && uv run pytest\n```\n\nIf `nix fmt` changes any files, stage them before committing. The CI runs `nix flake check` which verifies formatting, linting, and runs Rust tests.\n\n## Architecture\n\n### Node Composition\nA single exo `Node` (src/exo/main.py) runs multiple components:\n- **Router**: zenoh-based pub/sub messaging via Rust bindings (exo_rs)\n- **Worker**: Handles inference tasks, downloads models, manages runner processes\n- **Master**: Coordinates cluster state, places model instances across nodes\n- **Election**: Bully algorithm for master election\n- **API**: FastAPI server for OpenAI-compatible chat completions\n\n### Message Flow\nComponents communicate via typed pub/sub topics (src/exo/routing/topics.py):\n- `GLOBAL_EVENTS`: Master broadcasts indexed events to all workers\n- `LOCAL_EVENTS`: Workers send events to master for indexing\n- `COMMANDS`: Workers/API send commands to master\n- `ELECTION_MESSAGES`: Election protocol messages\n- `CONNECTION_MESSAGES`: zenoh connection updates\n\n### Event Sourcing\nThe system uses event sourcing for state management:\n- `State` (src/exo/shared/types/state.py): Immutable state object\n- `apply()` (src/exo/shared/apply.py): Pure function that applies events to state\n- Master indexes events and broadcasts; workers apply indexed events\n- The master only keeps recent events. A node that falls further behind (e.g. a late joiner) gets a `StateSnapshot` instead of a replay, so never assume a node has seen every event since the session started\n\n### Key Type Hierarchy\n- `src/exo/shared/types/`: Pydantic models for all shared types\n  - `events.py`: Event types (discriminated union)\n  - `commands.py`: Command types\n  - `tasks.py`: Task types for worker execution\n  - `state.py`: Cluster state model\n\n### Rust Components\nRust code in `rust/` provides:\n- `networking`: zenoh networking (gossipsub, peer discovery)\n- `exo_rs`: PyO3 bindings exposing Rust to Python\n- `system_custodian`: System-level operations\n\n### Dashboard\nSvelte 5 + TypeScript frontend in `dashboard/`. Build output goes to `dashboard/build/` and is served by the API.\n\n## Code Style Requirements\n\nFrom .cursorrules:\n- Strict, exhaustive typing - never bypass the type-checker\n- Use `Literal[...]` for enum-like sets, `typing.NewType` for primitives\n- Pydantic models with `frozen=True` and `strict=True`\n- Pure functions with injectable effect handlers for side-effects\n- Descriptive names - no abbreviations or 3-letter acronyms\n- Catch exceptions only where you can handle them meaningfully\n- Use `@final` and immutability wherever applicable\n\n## Testing\n\nTests use pytest-asyncio with `asyncio_mode = \"auto\"`. Tests are in `tests/` subdirectories alongside the code they test. The `EXO_TESTS=1` env var is set during tests.\n\n## Dashboard UI Testing & Screenshots\n\n### Building and Running the Dashboard\n```bash\n# Build the dashboard (must be done before running exo)\ncd dashboard && npm install && npm run build && cd ..\n\n# Start exo (serves the dashboard at http://localhost:52415)\nuv run exo &\nsleep 8  # Wait for server to start\n```\n\n### Taking Headless Screenshots with Playwright\nUse Playwright with headless Chromium for programmatic screenshots — no manual browser interaction needed.\n\n**Setup (one-time):**\n```bash\nnpx --yes playwright install chromium\ncd /tmp && npm init -y && npm install playwright\n```\n\n**Taking screenshots:**\n```javascript\n// Run from /tmp where playwright is installed: cd /tmp && node -e \"...\"\nconst { chromium } = require('playwright');\n(async () => {\n  const browser = await chromium.launch({ headless: true });\n  const page = await browser.newPage({ viewport: { width: 1280, height: 800 } });\n  await page.goto('http://localhost:52415', { waitUntil: 'networkidle' });\n  await page.waitForTimeout(2000);\n\n  // Inject test data into localStorage if needed (e.g., recent models)\n  await page.evaluate(() => {\n    localStorage.setItem('exo-recent-models', JSON.stringify([\n      { modelId: 'mlx-community/Qwen3-30B-A3B-4bit', launchedAt: Date.now() },\n    ]));\n  });\n  await page.reload({ waitUntil: 'networkidle' });\n  await page.waitForTimeout(2000);\n\n  // Interact with UI elements\n  await page.locator('text=SELECT MODEL').click();\n  await page.waitForTimeout(1000);\n\n  // Take screenshot\n  await page.screenshot({ path: '/tmp/screenshot.png', fullPage: false });\n  await browser.close();\n})();\n```\n\n### Uploading Images to GitHub PRs\nGitHub's API doesn't support direct image upload for PR comments. Workaround:\n\n1. **Commit images to the branch** (temporarily):\n   ```bash\n   cp /tmp/screenshot.png .\n   git add screenshot.png\n   git commit -m \"temp: add screenshots for PR\"\n   git push origin <branch>\n   COMMIT_SHA=$(git rev-parse HEAD)\n   ```\n\n2. **Post PR comment** referencing the raw image URL (uses permanent commit SHA so images survive deletion):\n   ```bash\n   gh pr comment <PR_NUMBER> --body \"![Screenshot](https://raw.githubusercontent.com/exo-explore/exo/${COMMIT_SHA}/screenshot.png)\"\n   ```\n\n3. **Remove the images** from the branch:\n   ```bash\n   git rm screenshot.png\n   git commit -m \"chore: remove temporary screenshot files\"\n   git push origin <branch>\n   ```\n   The images still render in the PR comment because they reference the permanent commit SHA.\n\n<!-- File: RULES.md (Tokens: ~1033 | Category: root-instruction) -->\n# Repository Rules\n\n* if you see any code that violates these rules, raise it with me directly rather than trying to fix.\n  * where applicable, file a GitHub Issue.\n* adhere to these rules strictly.\n\n## General Rules\n\n* if its possible to eliminate an extra try-catch or if-statement at runtime using type-level discipline, do it!\n* name your types, functions, and classes appropriately.\n  * no three-letter acronyms.\n  * no non-standard contractions.\n  * each data type has a meaning, pick a name which is accurate and descriptive.\n  * the average layman should be able to easily understand what your function does using the function signature alone!\n    * sometimes, there will be exceptions. eg, when you're using specific technical terms that are well understood (saga, event, etc).\n    * usually, you'll think that your code is an exception to the rules, but it won't be.\n\n## State, Functions and Classes\n\n* every function, given the same inputs, should produce the same outputs. ie, no hidden state.\n* use classes to prevent fixed state from being mutated arbitrarily (unsafely); methods provide a safe way of interfacing with state.\n* if your logic doesn't mutate fixed state, it probably belongs in a standalone function rather than a class.\n* functions shouldn't usually produce side-effects (they should be computationally pure).\n  * if, for example, you're updating a state using an event (computationally pure), and you want to trigger a saga (computational side-effect), store the logic for triggering the saga into an effect handler (a function, capable of producing side-effects, that you pass into an otherwise computationally pure function, so that it may trigger side-effects safely).\n\n## Pydantic\n\n* read the Pydantic docs.\n* respect the Pydantic docs.\n* pydantic is all you need.\n* declare and re-use a central `ConfigDict` for your use-case, you'll usually want `frozen` and `strict` to be `True`.\n\n## Unique ID (UUID) Generation\n\n* inherit from Pydantic's `UUID4` class to create your own UUID class.\n* use `uuid.uuid4()` to initialize your class with a fresh UUID where possible.\n* ensure that idempotency tags are generated by taking the salted hash of persisted state.\n  * rationale: if a node crashes and resumes from an older state, it should not accidentally re-publish the same event twice under different idempotency tags. \n  * every distinct function should feature a unique salt, so that there are no accidental collisions in idempotency tags.\n\n## Type Wrappers\n\n* reuse types that already exist in the Python standard library.\n* when two distinct data types are structurally identical (for example, different IDs which are both UUIDs but shouldn't never mixed up), make sure they can't be conflated by the type system.\n  * if you're working with a primitive data type (`str`, `int`, etc), use `NewType` (it has zero runtime overhead).\n  * if you're working with serializable data objects, consider adding a field (type `str`) that states its type.\n\n## Type Discipline\n\n* do not bypass the type-checker, preserve strict typing by any means necessary.\n* by default, use literal types (like `Literal['one', 'two']`) where an enum seems appropriate.\n\npro-tip: Python's type system is quite complex and feature-rich, so reading the documentation is often advisable; Matt discovered that Python `typing` library allows you to check that you've implemented a `match` exhaustively using `Literal` and `get_args(type)` after reading the docs.\n\n## Use of `@final`, Freezing\n\n* use wherever applicable.\n\n## Error Handling\n\n* don't try-catch for no reason.\n* make sure that you always know where and when the exceptions your code produces are meant to be handled, so that it's never a nasty surprise.\n  * always write the rationale for your error-handling down in the docstring!\n  * communicate the details to your colleagues when appropriate.\n\n## Dependencies\n\n* don't introduce any new dependencies without asking.\n* don't ask for any dependencies that aren't ubiquitous within production environments.\n\n## Commit Messages\n\n*   use the imperative mood in the subject line.\n*   prefix the subject line with a change type. our change types are:\n    *   `documentation`: documentation changes.\n    *   `feature`: a new feature.\n    *   `refactor`: a code change that neither fixes a bug nor adds a feature.\n    *   `bugfix`: a bug fix.\n    *   `chore`: routine tasks, maintenance, or tooling changes.\n    *   `test`: adding or correcting tests.\n*   restrict the subject line to fifty characters or less.\n*   capitalize the subject line.\n*   do not end the subject line with a period.\n*   separate subject from body with a blank line.\n\n<!-- File: docs/api.md (Tokens: ~4889 | Category: project-spec) -->\n# EXO API – Technical Reference\n\nThis document describes the REST API exposed by the **EXO** service, as implemented in:\n\n`src/exo/api/main.py` (request and response types are in `src/exo/api/types/`)\n\nThe API is used to manage model instances in the cluster, inspect cluster state, and perform inference using multiple API-compatible interfaces.\n\nBase URL example:\n\n```\nhttp://localhost:52415\n```\n\n## 1. General / Meta Endpoints\n\n### Get Master Node ID\n\n**GET** `/node_id`\n\nReturns the identifier of the current master node.\n\n**Response (example):**\n\n```json\n{\n  \"node_id\": \"node-1234\"\n}\n```\n\n### Get Cluster State\n\n**GET** `/state`\n\nReturns the current state of the cluster, including nodes and active instances.\n\n**Response:**\nJSON object describing topology, nodes, and instances.\n\n### Get Events\n\n**GET** `/events`\n\nReturns the most recent internal events this node has applied, up to 10,000 of them (mainly for debugging and observability). Every generated token is an event, so under load this covers the last few seconds to minutes. A node that joined late, or fell far behind, catches up from a state snapshot, so it only has the events after that point.\n\n**Response:**\nArray of event objects.\n\n## 2. Model Instance Management\n\n### Create Instance\n\n**POST** `/instance`\n\nCreates a new model instance in the cluster.\n\n**Request body (example):**\n\n```json\n{\n  \"instance\": { ... }\n}\n```\n\nThe `instance` is a placement as returned by `/instance/previews` (each preview's\n`instance` field), for example the first preview without an error.\n\n**Response:**\nCommand acknowledgement. Instance creation is asynchronous; clients should wait\nfor the model to appear through `/instance/await` before sending inference\nrequests for that model.\n\n### Delete Instance\n\n**DELETE** `/instance/{instance_id}`\n\nDeletes an existing instance by ID.\n\n**Path parameters:**\n\n* `instance_id`: string, ID of the instance to delete\n\n**Response:**\nStatus / confirmation JSON.\n\n### Get Instance\n\n**GET** `/instance/{instance_id}`\n\nReturns details of a specific instance.\n\n**Path parameters:**\n\n* `instance_id`: string\n\n**Response:**\nJSON description of the instance.\n\n### Await Instance\n\n**GET** `/instance/await?model_id=...&timeout_seconds=0`\n\nWaits until API state contains an instance for the requested model. The response\nis an SSE stream so clients receive keep-alive comments while waiting.\n\n**Query parameters:**\n\n* `model_id`: string, required\n* `timeout_seconds`: float, optional, default `0`. `0` waits indefinitely;\n  positive values time out after that many seconds. Maximum positive value:\n  `300`.\n\n**Stream messages:**\n\n```text\ndata: {\"type\": \"ready\", \"instance\": {...}}\n\ndata: {\"type\": \"timeout\", \"message\": \"No instance found for model ...\"}\n```\n\nThe HTTP status is `200` for both messages because the stream starts before the\nfinal result is known. The `type` field disambiguates the terminal message.\n\n### Preview Placements\n\n**GET** `/instance/previews?model_id=...`\n\nReturns possible placement previews for a given model.\n\n**Query parameters:**\n\n* `model_id`: string, required\n\n**Response:**\nArray of placement preview objects.\n\n### Compute Placement\n\n**GET** `/instance/placement`\n\nComputes a placement for a potential instance without creating it.\n\n**Query parameters (typical):**\n\n* `model_id`: string\n* `sharding`: string or config\n* `instance_meta`: JSON-encoded metadata\n* `min_nodes`: integer\n\n**Response:**\nJSON object describing the proposed placement / instance configuration.\n\n### Place Instance\n\n**POST** `/place_instance`\n\nPlaces an instance for a model using the server's placement logic.\n\n**Request body:**\nJSON describing the instance to be placed.\n\n**Response:**\nCommand acknowledgement. The instance may not be ready immediately; wait for it\nto appear through `/instance/await` before sending inference requests.\n\n## 3. Models\n\n### List Models\n\n**GET** `/models`\n**GET** `/v1/models` (alias)\n\nReturns the list of available models and their metadata.\n\n**Query parameters:**\n\n* `status`: string (optional) - Filter by `downloaded` to show only downloaded models\n\n**Response:**\nArray of model descriptors including `is_custom` field for custom HuggingFace models.\n\n### Add Custom Model\n\n**POST** `/models/add`\n\nAdd a custom model from HuggingFace hub.\n\n**Request body (example):**\n\n```json\n{\n  \"model_id\": \"mlx-community/my-custom-model\"\n}\n```\n\n**Response:**\nModel descriptor for the added model.\n\n**Security note:**\nModels with `trust_remote_code` enabled in their configuration require explicit opt-in (default is false) for security.\n\n### Delete Custom Model\n\n**DELETE** `/models/custom/{model_id}`\n\nDelete a user-added custom model card.\n\n**Path parameters:**\n\n* `model_id`: string, ID of the custom model to delete\n\n**Response:**\nConfirmation JSON with deleted model ID.\n\n### Search Models\n\n**GET** `/models/search`\n\nSearch HuggingFace Hub for mlx-community models.\n\n**Query parameters:**\n\n* `query`: string (optional) - Search query\n* `limit`: integer (default: 20) - Maximum number of results\n\n**Response:**\nArray of HuggingFace model search results.\n\n## 4. Inference / Chat Completions\n\n### OpenAI-Compatible Chat Completions\n\n**POST** `/v1/chat/completions`\n\nExecutes a chat completion request using an OpenAI-compatible schema. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"system\", \"content\": \"You are a helpful assistant.\" },\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Request parameters:**\n\n* `model`: string, required - Model ID to use\n* `messages`: array, required - Conversation messages\n* `stream`: boolean (default: false) - Enable streaming responses\n* `max_tokens`: integer (optional) - Maximum tokens to generate\n* `temperature`: float (optional) - Sampling temperature\n* `top_p`: float (optional) - Nucleus sampling parameter\n* `top_k`: integer (optional) - Top-k sampling parameter\n* `stop`: string or array (optional) - Stop sequences\n* `seed`: integer (optional) - Random seed for reproducibility\n* `enable_thinking`: boolean (optional) - Enable thinking mode for capable models (DeepSeek V3.1, Qwen3, GLM-4.7)\n* `tools`: array (optional) - Tool definitions for function calling\n* `logprobs`: boolean (optional) - Return log probabilities\n* `top_logprobs`: integer (optional) - Number of top log probabilities to return\n\n**Response:**\nOpenAI-compatible chat completion response.\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events (SSE) with format:\n\n```\ndata: {\"id\":\"...\",\"object\":\"chat.completion\",\"created\":...,\"model\":\"...\",\"choices\":[...]}\n\ndata: [DONE]\n```\n\n**Non-streaming response includes usage statistics:**\n\n```json\n{\n  \"id\": \"...\",\n  \"object\": \"chat.completion\",\n  \"created\": 1234567890,\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"choices\": [{\n    \"index\": 0,\n    \"message\": {\n      \"role\": \"assistant\",\n      \"content\": \"Hello! How can I help you?\"\n    },\n    \"finish_reason\": \"stop\"\n  }],\n  \"usage\": {\n    \"prompt_tokens\": 15,\n    \"completion_tokens\": 8,\n    \"total_tokens\": 23\n  }\n}\n```\n\n**Cancellation:**\nYou can cancel an active generation by closing the HTTP connection. The server detects the disconnection and stops processing.\n\n### Claude Messages API\n\n**POST** `/v1/messages`\n\nExecutes a chat completion request using the Claude Messages API format. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"max_tokens\": 1024,\n  \"stream\": false\n}\n```\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events with Claude-specific event types:\n\n* `message_start` - Message generation started\n* `content_block_start` - Content block started\n* `content_block_delta` - Incremental content chunk\n* `content_block_stop` - Content block completed\n* `message_delta` - Message metadata updates\n* `message_stop` - Message generation completed\n\n**Response:**\nClaude-compatible messages response.\n\n### OpenAI Responses API\n\n**POST** `/v1/responses`\n\nExecutes a chat completion request using the OpenAI Responses API format. Supports streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"input\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Request parameters:**\n\n* `model`: string, required - Model ID to use\n* `input`: string or array, required - The prompt, or a list of input items such as messages\n* `instructions`: string (optional) - System instructions\n* `max_output_tokens`: integer (optional) - Maximum tokens to generate\n\n**Streaming response format:**\nWhen `stream=true`, returns Server-Sent Events with response-specific event types:\n\n* `response.created` - Response generation started\n* `response.in_progress` - Response is being generated\n* `response.output_item.added` - New output item added\n* `response.output_text.delta` - Incremental text chunk\n* `response.output_item.done` - Output item completed\n* `response.completed` - Response generation completed\n\n**Response:**\nOpenAI Responses API-compatible response.\n\n### Benchmarked Chat Completions\n\n**POST** `/bench/chat/completions`\n\nSame as `/v1/chat/completions`, but also returns performance and generation statistics.\n\n**Request body:**\nSame schema as `/v1/chat/completions`.\n\n**Response:**\nChat completion plus benchmarking metrics including:\n\n* `prompt_tps` - Tokens per second during prompt processing\n* `generation_tps` - Tokens per second during generation\n* `prompt_tokens` - Number of prompt tokens\n* `generation_tokens` - Number of generated tokens\n* `peak_memory_usage` - Peak memory used during generation\n\n### Cancel Command\n\n**POST** `/v1/cancel/{command_id}`\n\nCancels an active generation command (text or image). Notifies workers and closes the stream.\n\n**Path parameters:**\n\n* `command_id`: string, ID of the command to cancel\n\n**Response (example):**\n\n```json\n{\n  \"message\": \"Command cancelled.\",\n  \"command_id\": \"cmd-abc-123\"\n}\n```\n\nReturns 404 if the command is not found or already completed.\n\n## 5. Ollama API Compatibility\n\nEXO provides Ollama API compatibility for tools like OpenWebUI.\n\n### Ollama Chat\n\n**POST** `/ollama/api/chat`\n**POST** `/ollama/api/api/chat` (alias)\n**POST** `/ollama/api/v1/chat` (alias)\n\nExecute a chat request using Ollama API format.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"messages\": [\n    { \"role\": \"user\", \"content\": \"Hello\" }\n  ],\n  \"stream\": false\n}\n```\n\n**Response:**\nOllama-compatible chat response.\n\n### Ollama Generate\n\n**POST** `/ollama/api/generate`\n\nExecute a text generation request using Ollama API format.\n\n**Request body (example):**\n\n```json\n{\n  \"model\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\",\n  \"prompt\": \"Hello\",\n  \"stream\": false\n}\n```\n\n**Response:**\nOllama-compatible generation response.\n\n### Ollama Tags\n\n**GET** `/ollama/api/tags`\n**GET** `/ollama/api/api/tags` (alias)\n**GET** `/ollama/api/v1/tags` (alias)\n\nReturns list of downloaded models in Ollama tags format.\n\n**Response:**\nArray of model tags with metadata.\n\n### Ollama Show\n\n**POST** `/ollama/api/show`\n\nReturns model information in Ollama show format.\n\n**Request body:**\n\n```json\n{\n  \"name\": \"mlx-community/Llama-3.2-1B-Instruct-4bit\"\n}\n```\n\n**Response:**\nModel details including modelfile and family.\n\n### Ollama PS\n\n**GET** `/ollama/api/ps`\n\nReturns list of running models (active instances).\n\n**Response:**\nArray of active model instances.\n\n### Ollama Version\n\n**GET** `/ollama/api/version`\n**HEAD** `/ollama/` (alias)\n**HEAD** `/ollama/api/version` (alias)\n\nReturns version information for Ollama API compatibility.\n\n**Response:**\n\n```json\n{\n  \"version\": \"exo v1.0\"\n}\n```\n\n## 6. Image Generation & Editing\n\n### Image Generation\n\n**POST** `/v1/images/generations`\n\nExecutes an image generation request using an OpenAI-compatible schema with additional advanced_params. Supports both streaming and non-streaming modes.\n\n**Request body (example):**\n\n```json\n{\n  \"prompt\": \"a robot playing chess\",\n  \"model\": \"exolabs/FLUX.1-dev\",\n  \"n\": 1,\n  \"size\": \"1024x1024\",\n  \"stream\": false,\n  \"response_format\": \"b64_json\"\n}\n```\n\n**Request parameters:**\n\n* `prompt`: string, required - Text description of the image\n* `model`: string, required - Image model ID\n* `n`: integer (default: 1) - Number of images to generate\n* `size`: string (default: \"auto\") - Image dimensions. Supported sizes:\n  - `512x512`\n  - `768x768`\n  - `1024x768`\n  - `768x1024`\n  - `1024x1024`\n  - `1024x1536`\n  - `1536x1024`\n  - `1024x1365`\n  - `1365x1024`\n* `stream`: boolean (default: false) - Enable streaming for partial images\n* `partial_images`: integer (default: 0) - Number of partial images to stream during generation\n* `response_format`: string (default: \"b64_json\") - Either `url` or `b64_json`\n* `quality`: string (default: \"medium\") - Either `high`, `medium`, or `low`\n* `output_format`: string (default: \"png\") - Either `png`, `jpeg`, or `webp`\n* `advanced_params`: object (optional) - Advanced generation parameters\n\n**Advanced Parameters (`advanced_params`):**\n\n| Parameter | Type | Constraints | Description |\n|-----------|------|-------------|-------------|\n| `seed` | int | >= 0 | Random seed for reproducible generation |\n| `num_inference_steps` | int | 1-100 | Number of denoising steps |\n| `guidance` | float | 1.0-20.0 | Classifier-free guidance scale |\n| `negative_prompt` | string | - | Text describing what to avoid in the image |\n\n**Non-streaming response:**\n\n```json\n{\n  \"created\": 1234567890,\n  \"data\": [\n    {\n      \"b64_json\": \"iVBORw0KGgoAAAANSUhEUgAA...\",\n      \"url\": null\n    }\n  ]\n}\n```\n\n**Streaming response format:**\nWhen `stream=true` and `partial_images > 0`, returns Server-Sent Events:\n\n```\ndata: {\"type\":\"partial\",\"image_index\":0,\"partial_index\":1,\"total_partials\":5,\"format\":\"png\",\"data\":{\"b64_json\":\"...\"}}\n\ndata: {\"type\":\"final\",\"image_index\":0,\"format\":\"png\",\"data\":{\"b64_json\":\"...\"}}\n\ndata: [DONE]\n```\n\n### Image Editing\n\n**POST** `/v1/images/edits`\n\nExecutes an image editing request (img2img) using FLUX.1-Kontext-dev or similar models.\n\n**Request (multipart/form-data):**\n\n* `image`: file, required - Input image to edit\n* `prompt`: string, required - Text description of desired changes\n* `model`: string, required - Image editing model ID (e.g., `exolabs/FLUX.1-Kontext-dev`)\n* `n`: integer (default: 1) - Number of edited images to generate\n* `size`: string (optional) - Output image dimensions\n* `response_format`: string (default: \"b64_json\") - Either `url` or `b64_json`\n* `input_fidelity`: string (default: \"low\") - Either `low` or `high` - Controls how closely the output follows the input image\n* `stream`: string (default: \"false\") - Enable streaming\n* `partial_images`: string (default: \"0\") - Number of partial images to stream\n* `quality`: string (default: \"medium\") - Either `high`, `medium`, or `low`\n* `output_format`: string (default: \"png\") - Either `png`, `jpeg`, or `webp`\n* `advanced_params`: string (optional) - JSON-encoded advanced parameters\n\n**Response:**\nSame format as `/v1/images/generations`.\n\n### Benchmarked Image Generation\n\n**POST** `/bench/images/generations`\n\nSame as `/v1/images/generations`, but also returns generation statistics.\n\n**Request body:**\nSame schema as `/v1/images/generations`.\n\n**Response:**\nImage generation plus benchmarking metrics including:\n\n* `seconds_per_step` - Average time per denoising step\n* `total_generation_time` - Total generation time\n* `num_inference_steps` - Number of inference steps used\n* `num_images` - Number of images generated\n* `image_width` - Output image width\n* `image_height` - Output image height\n* `peak_memory_usage` - Peak memory used during generation\n\n### Benchmarked Image Editing\n\n**POST** `/bench/images/edits`\n\nSame as `/v1/images/edits`, but also returns generation statistics.\n\n**Request:**\nSame schema as `/v1/images/edits`.\n\n**Response:**\nSame format as `/bench/images/generations`, including `generation_stats`.\n\n### List Images\n\n**GET** `/images`\n\nList all stored images.\n\n**Response:**\nArray of image metadata including URLs and expiration times.\n\n### Get Image\n\n**GET** `/images/{image_id}`\n\nRetrieve a stored image by ID.\n\n**Path parameters:**\n\n* `image_id`: string, ID of the image\n\n**Response:**\nImage file with appropriate content type.\n\n## 7. Complete Endpoint Summary\n\n```\n# General\nGET     /node_id\nGET     /state\nGET     /events\n\n# Instance Management\nPOST    /instance\nGET     /instance/await\nGET     /instance/previews\nGET     /instance/placement\nGET     /instance/{instance_id}\nDELETE  /instance/{instance_id}\nPOST    /place_instance\n\n# Models\nGET     /models\nGET     /v1/models\nPOST    /models/add\nDELETE  /models/custom/{model_id}\nGET     /models/search\n\n# Text Generation (OpenAI Chat Completions)\nPOST    /v1/chat/completions\nPOST    /bench/chat/completions\n\n# Text Generation (Claude Messages API)\nPOST    /v1/messages\n\n# Text Generation (OpenAI Responses API)\nPOST    /v1/responses\n\n# Text Generation (Ollama API)\nPOST    /ollama/api/chat\nPOST    /ollama/api/api/chat\nPOST    /ollama/api/v1/chat\nPOST    /ollama/api/generate\nGET     /ollama/api/tags\nGET     /ollama/api/api/tags\nGET     /ollama/api/v1/tags\nPOST    /ollama/api/show\nGET     /ollama/api/ps\nGET     /ollama/api/version\nHEAD    /ollama/\nHEAD    /ollama/api/version\n\n# Command Control\nPOST    /v1/cancel/{command_id}\n\n# Image Generation\nPOST    /v1/images/generations\nPOST    /bench/images/generations\nPOST    /v1/images/edits\nPOST    /bench/images/edits\nGET     /images\nGET     /images/{image_id}\n```\n\n## 8. Notes\n\n### API Compatibility\n\nEXO provides multiple API-compatible interfaces:\n\n* **OpenAI Chat Completions API** - Compatible with OpenAI clients and tools\n* **Claude Messages API** - Compatible with Anthropic's Claude API format\n* **OpenAI Responses API** - Compatible with OpenAI's Responses API format\n* **Ollama API** - Compatible with Ollama and tools like OpenWebUI\n\nExisting OpenAI, Claude, or Ollama clients can be pointed to EXO by changing the base URL.\n\n### Custom Models\n\nYou can add custom models from HuggingFace using the `/models/add` endpoint. Custom models are identified by the `is_custom` field in model list responses.\n\n**Security:** Models requiring `trust_remote_code` must be explicitly enabled (default is false) for security. Only enable this if you trust the model's remote code.\n\n### Usage Statistics\n\nChat completion responses include usage statistics with:\n\n* `prompt_tokens` - Number of tokens in the prompt\n* `completion_tokens` - Number of tokens generated\n* `total_tokens` - Sum of prompt and completion tokens\n\n### Request Cancellation\n\nYou can cancel active requests by:\n\n1. Closing the HTTP connection (for streaming requests)\n2. Calling `/v1/cancel/{command_id}` (for any request)\n\nThe server detects cancellation and stops processing immediately.\n\n### Instance Placement\n\nThe instance placement endpoints allow you to plan and preview cluster allocations before creating instances. This helps optimize resource usage across nodes.\n\n### Observability\n\nThe `/events` and `/state` endpoints are primarily intended for operational visibility and debugging.\n\n<!-- File: .cursorrules (Tokens: ~710 | Category: cursor-rule) -->\n# follow **every** rule exactly; report any violation instead of silently fixing it.\n\nYou must prioritize straightforward code semantics, well-named types, clear function signatures, and robust, carefully-chosen abstractions. Think about how your decisions might impact these aspects of code quality before proposing any changes.\n\nYou can use the advanced features of `typing`. You have access to all of the new features from Python 3.13, 3.12, 3.11...\n\n**When you're done making your changes, remove any redundant comments that you may have left; the comments that remain should only apply to complex segments of code, adding relevant context.**\n\n## 1. Code Discipline\n\n* Eliminate superfluous `try` / `catch` and `if` branches through strict typing and static analysis.\n* Use pure functions unless you must mutate fixed state—then wrap that state in a class.\n* Every function is **referentially transparent**: same inputs ⇒ same outputs, no hidden state, no unintended I/O.\n* Put side-effects in injectable “effect handlers”; keep core logic pure.\n\n## 2. Naming\n\n* Choose descriptive, non-abbreviated names—no 3-letter acronyms or non-standard contractions.\n* Anyone reading a function’s type signature alone should grasp its purpose without extra context.\n\n## 3. Typing\n\n* Maintain **strict, exhaustive** typing; never bypass the type-checker.\n* Default to `Literal[...]` when an enum-like set is needed.\n* Prefer built-in types; when two values share structure but differ in meaning, enforce separation:\n  * Use `typing.NewType` for primitives (zero runtime cost).\n  * For serialisable objects, add a `type: str` field that states the object’s identity.\n\n## 4. Pydantic\n\n* Read, respect, and rely on Pydantic docs.\n* Centralise a common `ConfigDict` with `frozen=True` and `strict=True` (or stricter) and reuse it everywhere.\n* For hierarchies of `BaseModel` variants, declare a discriminated union with `typing.Annotated[Base, Field(discriminator='variant')]`; publish a single `TypeAdapter[Base]` so all variants share one strict validator.\n\n## 5. IDs & UUIDs\n\n* Subclass Pydantic’s `UUID4` for custom ID types.\n* Generate fresh IDs with `uuid.uuid4()`.\n* Create idempotency keys by hashing *persisted* state plus a **function-specific salt** to avoid collisions after crashes.\n\n## 6. Error Handling\n\n* Catch an exception **only** where you can handle or transform it meaningfully.\n* State in the docstring **where** each exception is expected to be handled and **why**.\n\n## 7. Dependencies\n\n* Introduce new external dependencies only after approval.\n* Request only libraries common in production environments.\n\n## 8. Use of `@final` & Freezing\n\n* Mark classes, methods, and variables as `@final` or otherwise immutable wherever applicable.\n\n## 9. Repository Workflow\n\nIf you spot a rule violation within code that you've not been asked to work on directly, inform the user rather than patching it ad-hoc.\n\n\n---\n\n### One-Sentence Summary\n\nWrite strictly-typed, pure, self-describing Python that uses Pydantic, well-scoped side-effects, immutable state, approved dependencies, and explicit error handling\n</yakaai_skills>"}