{"owner":"Michael-A-Kuykendall","repo":"shimmy","hasSkills":true,"totalSkillsCount":2,"totalTokensCount":2690,"categories":["project-spec"],"hasMcp":false,"mcpConfig":null,"found":["docs/ARCHITECTURE.md","docs/API.md"],"skills":{"docs/ARCHITECTURE.md":"# Shimmy v2.0 Architecture\r\n\r\n## Overview\r\n\r\nShimmy is a single-binary OpenAI-compatible HTTP server for local GGUF inference.\r\nIn v2.0 the default inference backend is **Airframe** — a pure-Rust WebGPU (WGSL)\r\ntransformer runtime that requires no C++ toolchain, no CUDA SDK, and no Vulkan SDK.\r\n\r\n---\r\n\r\n## Component Map\r\n\r\n```\r\nshimmy (binary)\r\n ├── HTTP layer (axum / tokio)\r\n │    ├── POST /v1/chat/completions\r\n │    ├── POST /v1/completions\r\n │    └── GET  /v1/models\r\n ├── Engine bridge (src/engine/airframe.rs)\r\n │    └── converts OpenAI request → GenOptions → SamplingParams\r\n └── Airframe runtime  (../airframe crate, path dep)\r\n      ├── GGUF loader (weights + metadata)\r\n      ├── Tokenizer (shimmytok)\r\n      ├── GPU pipeline (wgpu + WGSL shaders)\r\n      │    ├── Embedding lookup\r\n      │    ├── Per-layer transformer blocks\r\n      │    │    ├── Attention (Q/K/V matmul + RoPE + softmax)\r\n      │    │    └── FFN (gate / up / down projections)\r\n      │    ├── Output head (dequant + logit projection)\r\n      │    └── KV cache (GPU buffers, reset per request)\r\n      └── Sampler (repeat penalty → temperature → top-p)\r\n```\r\n\r\n---\r\n\r\n## Airframe Engine\r\n\r\n### WGSL / WebGPU\r\n\r\nAirframe uses **wgpu** as the WebGPU implementation layer.  wgpu translates WGSL\r\nshaders to the best available native API:\r\n\r\n| Platform | Native API used by wgpu |\r\n|---|---|\r\n| Windows | Direct3D 12 (primary) or Vulkan |\r\n| Linux   | Vulkan (primary) or OpenGL |\r\n| macOS   | Metal |\r\n\r\nNo Vulkan SDK or CUDA toolkit is required — wgpu drivers ship with the binary.\r\nWGSL shaders are compiled at startup using `wgpu::Device::create_shader_module`.\r\n\r\n### Quantization\r\n\r\nWeights are stored in GGUF quantized format.  The GPU dequantization shaders\r\ndecode each block on the fly during the matrix multiply:\r\n\r\n| GGUF type | GPU shader |\r\n|---|---|\r\n| F32 / F16 | passthrough / fp16 unpack |\r\n| Q4_0      | 32-element block, 4-bit signed |\r\n| Q8_0      | 32-element block, 8-bit signed |\r\n| Q4_K_M    | super-block with row scales |\r\n| Q5_K_M    | super-block, 5-bit with high-bit plane |\r\n| Q6_K      | super-block, 6-bit |\r\n\r\nEvery GPU shader result is validated against the CPU reference before a model\r\nis considered supported (`quant_verify` tool).\r\n\r\n### KV Cache\r\n\r\nThe KV cache is a pair of GPU buffers (keys + values) allocated at server start\r\nwith capacity `n_layers × n_kv_heads × head_dim × max_ctx × sizeof(f32)`.\r\nThe cache is **fully reset between requests** — Shimmy is stateless.\r\n\r\n### Extended Context (YaRN)\r\n\r\nWhen `SHIMMY_MAX_CTX` exceeds the model's native context window, Airframe:\r\n1. Recomputes the RoPE theta scale using the YaRN formula\r\n2. Reallocates the KV cache to the requested length\r\n3. Applies the scale to all position-dependent attention computations\r\n\r\n---\r\n\r\n## Request Lifecycle\r\n\r\n```\r\nHTTP POST /v1/chat/completions\r\n  │\r\n  ├─ serde_json deserialization (ChatCompletionRequest)\r\n  ├─ Input validation (empty messages, max_tokens bounds)\r\n  ├─ Chat template application → prompt string\r\n  ├─ Engine bridge: build GenOptions + SamplingParams\r\n  │    ├─ stop_tokens from chat template (e.g. <|eot_id|> for Llama-3)\r\n  │    ├─ extra_stop_tokens propagated to SamplingParams\r\n  │    └─ penalty fields mapped: repeat_penalty = 1 + max(freq, presence) * 0.5\r\n  ├─ Airframe generate()\r\n  │    ├─ Tokenize prompt\r\n  │    ├─ Chunked prefill (512-token chunks)\r\n  │    ├─ Decode loop (one token per step)\r\n  │    │    ├─ GPU forward pass\r\n  │    │    ├─ Sample (penalty → temperature → top-p)\r\n  │    │    └─ Stop on EOS | extra_stop_tokens | max_tokens\r\n  │    └─ Detokenize\r\n  └─ OpenAI response JSON\r\n```\r\n\r\n---\r\n\r\n## Model Discovery\r\n\r\nOn startup Shimmy scans the following paths for GGUF files:\r\n\r\n1. `SHIMMY_BASE_GGUF` (explicit env var — highest priority)\r\n2. `SHIMMY_MODEL_PATHS` (colon-separated list)\r\n3. `~/.cache/huggingface/hub/` (HuggingFace cache)\r\n4. `~/.ollama/models/` (Ollama model store)\r\n5. `~/.cache/lm-studio/models/` (LM Studio)\r\n6. `~/models/` (generic local dir)\r\n\r\n---\r\n\r\n## Known Limitations (v2.0)\r\n\r\n| Limitation | Detail |\r\n|---|---|\r\n| 2 GB WebGPU buffer cap | Single tensor buffer ≤ 2 GB; output heads > 2 GB (e.g. Gemma-2-2B) require chunking (not yet implemented) |\r\n| Fused QKV tensors | Phi-3 / Phi-3.5 pack Q+K+V into one weight; Airframe expects separate tensors — server will crash |\r\n| Stateless only | No multi-turn KV cache persistence; each request starts fresh |\r\n| Streaming | Token streaming via SSE is on the roadmap; `stream: true` is accepted but returns full completion |\r\n| MoE models | CPU offloading for 70B+ MoE is on the roadmap; current architecture loads entire model on GPU |\r\n","docs/API.md":"# API Reference\n\nShimmy exposes three interface families:\n\n- **OpenAI-compatible** (`/v1/*`) — drop-in replacement for OpenAI SDKs and tools\n- **Shimmy native** (`/api/*`) — the original REST + WebSocket API\n- **Ollama-compatible** (`/api/tags`) and **Anthropic-compatible** (`/v1/messages`) — for existing tooling\n\nAll endpoints serve the same inference engine. See\n[docs/OPENAI_COMPAT.md](OPENAI_COMPAT.md) for the exact OpenAI parameter\ncompatibility matrix.\n\nInteractive documentation is available from a running server at\n`GET /docs`; the machine-readable contract is `GET /openapi.json`.\nSee [OpenAPI and Swagger UI](OPENAPI.md) for usage and implementation details.\n\n## OpenAI-Compatible Endpoints\n\n### `GET /v1/models`\nLists available models (also served as `GET /api/tags` for Ollama clients).\n\n```json\n{\n  \"object\": \"list\",\n  \"data\": [\n    { \"id\": \"tinyllama-1.1b\", \"object\": \"model\", \"owned_by\": \"shimmy\" }\n  ]\n}\n```\n\n### `POST /v1/chat/completions`\nOpenAI-compatible chat completions. Streaming supported via SSE when\n`\"stream\": true`.\n\n```json\n{\n  \"model\": \"tinyllama-1.1b\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Say hi in 5 words.\"}],\n  \"max_tokens\": 32,\n  \"temperature\": 0.7\n}\n```\n\n### `POST /v1/completions`\nLegacy text completions (non-chat). Same request shape minus `messages`.\n\n### `POST /v1/messages`\nAnthropic Messages API compatibility (`x-api-key` optional, ignored).\n\n## Shimmy Native Endpoints\n\n### `POST /api/generate`\nText generation (streaming + non-streaming).\n\n**Request:**\n```json\n{\n  \"model\": \"string\",\n  \"prompt\": \"string\",\n  \"max_tokens\": 100,\n  \"temperature\": 0.7,\n  \"stream\": false\n}\n```\n\n**Non-streaming response:**\n```json\n{\n  \"choices\": [{\"text\": \"Generated text\", \"index\": 0, \"finish_reason\": \"length\"}],\n  \"usage\": {\"prompt_tokens\": 10, \"completion_tokens\": 20, \"total_tokens\": 30}\n}\n```\n\n**Streaming response** (SSE):\n```\ndata: {\"choices\":[{\"text\":\"Hello\",\"index\":0}]}\ndata: {\"choices\":[{\"text\":\" world\",\"index\":0}]}\ndata: [DONE]\n```\n\n### `GET /api/models`\nList loaded models.\n\n### `POST /api/models/discover`\nTrigger model re-discovery.\n\n### `POST /api/models/:name/load` / `POST /api/models/:name/unload`\nLoad or unload a specific model by name.\n\n### `GET /api/models/:name/status`\nModel status (loaded, VRAM, context).\n\n### `GET /api/tags`\nOllama-compatible model list.\n\n### `GET /health`\nHealth check.\n\n```json\n{\"status\": \"healthy\", \"models_loaded\": 1, \"memory_usage\": \"2.1GB\"}\n```\n\n### `GET /metrics`\nPrometheus-style metrics endpoint.\n\n### `GET /diag`\nDiagnostics endpoint.\n\n## WebSocket API\n\n**Endpoint:** `ws://localhost:11435/ws/generate`\n\nSend a single JSON `GenerateRequest` frame:\n```json\n{\"model\": \"default\", \"prompt\": \"Hello world\", \"max_tokens\": 50, \"temperature\": 0.7}\n```\n\nReceive token frames:\n```json\n{\"token\": \"Hello\"}\n{\"token\": \" world\"}\n{\"done\": true}\n```\n\nThe WebSocket upgrade is documented in the OpenAPI contract as `GET\n/ws/generate`, but its token frames are not REST JSON responses.\n\n## CLI\n\n```bash\nshimmy serve                              # Start server (auto port allocation)\nshimmy serve --bind 127.0.0.1:8080        # Manual port binding\nshimmy serve --kv-quant int4              # Enable TurboShimmy INT4 KV\nshimmy list                               # Show available models\nshimmy discover                           # Refresh model discovery\nshimmy generate --name X --prompt \"Hi\"    # Test generation\nshimmy probe model-name                   # Verify model loads\nshimmy gpu-info                           # Show selected WebGPU adapter\n```\n\nGlobal flags: `--verbose/-v`, `--help/-h`, `--version/-V`.\n\n## Environment Variables\n\n| Variable | Default | Description |\n|---|---|---|\n| `SHIMMY_BASE_GGUF` | *(auto-discover)* | Path to GGUF model loaded as default |\n| `SHIMMY_PORT` | `8080` | Listen port |\n| `SHIMMY_BIND_ADDRESS` | `0.0.0.0:8080` | Full bind address (overrides port) |\n| `SHIMMY_MAX_CTX` | *(from GGUF)* | Override context window; activates YaRN RoPE scaling when above native |\n| `SHIMMY_MODEL_PATHS` | *(auto)* | Colon-separated extra model search paths |\n| `SHIMMY_ENGINE_BACKEND` | `airframe` | Inference engine (airframe is the only engine) |\n| `SHIMMY_ROPE_SCALE` | *(auto)* | Override computed YaRN scale factor |\n| `SHIMMY_KV_QUANT` | `f32` | `f32` (default) or `int4` ([TurboShimmy](turboshimmy.md)) |\n| `SHIMMY_PREFILL_CHUNK` | `64` | Tokens per prefill dispatch; set to `8` on Windows to avoid GPU TDR resets |\n| `RUST_BACKTRACE` | *(off)* | Set to `1` for crash backtraces |\n\nSee [docs/CONFIGURATION.md](CONFIGURATION.md) for the complete configuration\nreference.\n\n## Error Responses\n\nAll endpoints return a consistent error format:\n\n```json\n{\n  \"error\": {\n    \"code\": \"model_not_found\",\n    \"message\": \"The specified model was not found\",\n    \"details\": \"Model 'invalid-model' is not available\"\n  }\n}\n```\n\nCommon error codes:\n- `model_not_found` — requested model is not available\n- `invalid_request` — request format is invalid\n- `generation_failed` — text generation failed\n- `server_error` — internal server error\n\n## Rate Limiting\n\nNo rate limiting is currently implemented. For production use, place Shimmy\nbehind a reverse proxy with rate-limiting capabilities.\n"},"files":{"docs/ARCHITECTURE.md":"# Shimmy v2.0 Architecture\r\n\r\n## Overview\r\n\r\nShimmy is a single-binary OpenAI-compatible HTTP server for local GGUF inference.\r\nIn v2.0 the default inference backend is **Airframe** — a pure-Rust WebGPU (WGSL)\r\ntransformer runtime that requires no C++ toolchain, no CUDA SDK, and no Vulkan SDK.\r\n\r\n---\r\n\r\n## Component Map\r\n\r\n```\r\nshimmy (binary)\r\n ├── HTTP layer (axum / tokio)\r\n │    ├── POST /v1/chat/completions\r\n │    ├── POST /v1/completions\r\n │    └── GET  /v1/models\r\n ├── Engine bridge (src/engine/airframe.rs)\r\n │    └── converts OpenAI request → GenOptions → SamplingParams\r\n └── Airframe runtime  (../airframe crate, path dep)\r\n      ├── GGUF loader (weights + metadata)\r\n      ├── Tokenizer (shimmytok)\r\n      ├── GPU pipeline (wgpu + WGSL shaders)\r\n      │    ├── Embedding lookup\r\n      │    ├── Per-layer transformer blocks\r\n      │    │    ├── Attention (Q/K/V matmul + RoPE + softmax)\r\n      │    │    └── FFN (gate / up / down projections)\r\n      │    ├── Output head (dequant + logit projection)\r\n      │    └── KV cache (GPU buffers, reset per request)\r\n      └── Sampler (repeat penalty → temperature → top-p)\r\n```\r\n\r\n---\r\n\r\n## Airframe Engine\r\n\r\n### WGSL / WebGPU\r\n\r\nAirframe uses **wgpu** as the WebGPU implementation layer.  wgpu translates WGSL\r\nshaders to the best available native API:\r\n\r\n| Platform | Native API used by wgpu |\r\n|---|---|\r\n| Windows | Direct3D 12 (primary) or Vulkan |\r\n| Linux   | Vulkan (primary) or OpenGL |\r\n| macOS   | Metal |\r\n\r\nNo Vulkan SDK or CUDA toolkit is required — wgpu drivers ship with the binary.\r\nWGSL shaders are compiled at startup using `wgpu::Device::create_shader_module`.\r\n\r\n### Quantization\r\n\r\nWeights are stored in GGUF quantized format.  The GPU dequantization shaders\r\ndecode each block on the fly during the matrix multiply:\r\n\r\n| GGUF type | GPU shader |\r\n|---|---|\r\n| F32 / F16 | passthrough / fp16 unpack |\r\n| Q4_0      | 32-element block, 4-bit signed |\r\n| Q8_0      | 32-element block, 8-bit signed |\r\n| Q4_K_M    | super-block with row scales |\r\n| Q5_K_M    | super-block, 5-bit with high-bit plane |\r\n| Q6_K      | super-block, 6-bit |\r\n\r\nEvery GPU shader result is validated against the CPU reference before a model\r\nis considered supported (`quant_verify` tool).\r\n\r\n### KV Cache\r\n\r\nThe KV cache is a pair of GPU buffers (keys + values) allocated at server start\r\nwith capacity `n_layers × n_kv_heads × head_dim × max_ctx × sizeof(f32)`.\r\nThe cache is **fully reset between requests** — Shimmy is stateless.\r\n\r\n### Extended Context (YaRN)\r\n\r\nWhen `SHIMMY_MAX_CTX` exceeds the model's native context window, Airframe:\r\n1. Recomputes the RoPE theta scale using the YaRN formula\r\n2. Reallocates the KV cache to the requested length\r\n3. Applies the scale to all position-dependent attention computations\r\n\r\n---\r\n\r\n## Request Lifecycle\r\n\r\n```\r\nHTTP POST /v1/chat/completions\r\n  │\r\n  ├─ serde_json deserialization (ChatCompletionRequest)\r\n  ├─ Input validation (empty messages, max_tokens bounds)\r\n  ├─ Chat template application → prompt string\r\n  ├─ Engine bridge: build GenOptions + SamplingParams\r\n  │    ├─ stop_tokens from chat template (e.g. <|eot_id|> for Llama-3)\r\n  │    ├─ extra_stop_tokens propagated to SamplingParams\r\n  │    └─ penalty fields mapped: repeat_penalty = 1 + max(freq, presence) * 0.5\r\n  ├─ Airframe generate()\r\n  │    ├─ Tokenize prompt\r\n  │    ├─ Chunked prefill (512-token chunks)\r\n  │    ├─ Decode loop (one token per step)\r\n  │    │    ├─ GPU forward pass\r\n  │    │    ├─ Sample (penalty → temperature → top-p)\r\n  │    │    └─ Stop on EOS | extra_stop_tokens | max_tokens\r\n  │    └─ Detokenize\r\n  └─ OpenAI response JSON\r\n```\r\n\r\n---\r\n\r\n## Model Discovery\r\n\r\nOn startup Shimmy scans the following paths for GGUF files:\r\n\r\n1. `SHIMMY_BASE_GGUF` (explicit env var — highest priority)\r\n2. `SHIMMY_MODEL_PATHS` (colon-separated list)\r\n3. `~/.cache/huggingface/hub/` (HuggingFace cache)\r\n4. `~/.ollama/models/` (Ollama model store)\r\n5. `~/.cache/lm-studio/models/` (LM Studio)\r\n6. `~/models/` (generic local dir)\r\n\r\n---\r\n\r\n## Known Limitations (v2.0)\r\n\r\n| Limitation | Detail |\r\n|---|---|\r\n| 2 GB WebGPU buffer cap | Single tensor buffer ≤ 2 GB; output heads > 2 GB (e.g. Gemma-2-2B) require chunking (not yet implemented) |\r\n| Fused QKV tensors | Phi-3 / Phi-3.5 pack Q+K+V into one weight; Airframe expects separate tensors — server will crash |\r\n| Stateless only | No multi-turn KV cache persistence; each request starts fresh |\r\n| Streaming | Token streaming via SSE is on the roadmap; `stream: true` is accepted but returns full completion |\r\n| MoE models | CPU offloading for 70B+ MoE is on the roadmap; current architecture loads entire model on GPU |\r\n","docs/API.md":"# API Reference\n\nShimmy exposes three interface families:\n\n- **OpenAI-compatible** (`/v1/*`) — drop-in replacement for OpenAI SDKs and tools\n- **Shimmy native** (`/api/*`) — the original REST + WebSocket API\n- **Ollama-compatible** (`/api/tags`) and **Anthropic-compatible** (`/v1/messages`) — for existing tooling\n\nAll endpoints serve the same inference engine. See\n[docs/OPENAI_COMPAT.md](OPENAI_COMPAT.md) for the exact OpenAI parameter\ncompatibility matrix.\n\nInteractive documentation is available from a running server at\n`GET /docs`; the machine-readable contract is `GET /openapi.json`.\nSee [OpenAPI and Swagger UI](OPENAPI.md) for usage and implementation details.\n\n## OpenAI-Compatible Endpoints\n\n### `GET /v1/models`\nLists available models (also served as `GET /api/tags` for Ollama clients).\n\n```json\n{\n  \"object\": \"list\",\n  \"data\": [\n    { \"id\": \"tinyllama-1.1b\", \"object\": \"model\", \"owned_by\": \"shimmy\" }\n  ]\n}\n```\n\n### `POST /v1/chat/completions`\nOpenAI-compatible chat completions. Streaming supported via SSE when\n`\"stream\": true`.\n\n```json\n{\n  \"model\": \"tinyllama-1.1b\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Say hi in 5 words.\"}],\n  \"max_tokens\": 32,\n  \"temperature\": 0.7\n}\n```\n\n### `POST /v1/completions`\nLegacy text completions (non-chat). Same request shape minus `messages`.\n\n### `POST /v1/messages`\nAnthropic Messages API compatibility (`x-api-key` optional, ignored).\n\n## Shimmy Native Endpoints\n\n### `POST /api/generate`\nText generation (streaming + non-streaming).\n\n**Request:**\n```json\n{\n  \"model\": \"string\",\n  \"prompt\": \"string\",\n  \"max_tokens\": 100,\n  \"temperature\": 0.7,\n  \"stream\": false\n}\n```\n\n**Non-streaming response:**\n```json\n{\n  \"choices\": [{\"text\": \"Generated text\", \"index\": 0, \"finish_reason\": \"length\"}],\n  \"usage\": {\"prompt_tokens\": 10, \"completion_tokens\": 20, \"total_tokens\": 30}\n}\n```\n\n**Streaming response** (SSE):\n```\ndata: {\"choices\":[{\"text\":\"Hello\",\"index\":0}]}\ndata: {\"choices\":[{\"text\":\" world\",\"index\":0}]}\ndata: [DONE]\n```\n\n### `GET /api/models`\nList loaded models.\n\n### `POST /api/models/discover`\nTrigger model re-discovery.\n\n### `POST /api/models/:name/load` / `POST /api/models/:name/unload`\nLoad or unload a specific model by name.\n\n### `GET /api/models/:name/status`\nModel status (loaded, VRAM, context).\n\n### `GET /api/tags`\nOllama-compatible model list.\n\n### `GET /health`\nHealth check.\n\n```json\n{\"status\": \"healthy\", \"models_loaded\": 1, \"memory_usage\": \"2.1GB\"}\n```\n\n### `GET /metrics`\nPrometheus-style metrics endpoint.\n\n### `GET /diag`\nDiagnostics endpoint.\n\n## WebSocket API\n\n**Endpoint:** `ws://localhost:11435/ws/generate`\n\nSend a single JSON `GenerateRequest` frame:\n```json\n{\"model\": \"default\", \"prompt\": \"Hello world\", \"max_tokens\": 50, \"temperature\": 0.7}\n```\n\nReceive token frames:\n```json\n{\"token\": \"Hello\"}\n{\"token\": \" world\"}\n{\"done\": true}\n```\n\nThe WebSocket upgrade is documented in the OpenAPI contract as `GET\n/ws/generate`, but its token frames are not REST JSON responses.\n\n## CLI\n\n```bash\nshimmy serve                              # Start server (auto port allocation)\nshimmy serve --bind 127.0.0.1:8080        # Manual port binding\nshimmy serve --kv-quant int4              # Enable TurboShimmy INT4 KV\nshimmy list                               # Show available models\nshimmy discover                           # Refresh model discovery\nshimmy generate --name X --prompt \"Hi\"    # Test generation\nshimmy probe model-name                   # Verify model loads\nshimmy gpu-info                           # Show selected WebGPU adapter\n```\n\nGlobal flags: `--verbose/-v`, `--help/-h`, `--version/-V`.\n\n## Environment Variables\n\n| Variable | Default | Description |\n|---|---|---|\n| `SHIMMY_BASE_GGUF` | *(auto-discover)* | Path to GGUF model loaded as default |\n| `SHIMMY_PORT` | `8080` | Listen port |\n| `SHIMMY_BIND_ADDRESS` | `0.0.0.0:8080` | Full bind address (overrides port) |\n| `SHIMMY_MAX_CTX` | *(from GGUF)* | Override context window; activates YaRN RoPE scaling when above native |\n| `SHIMMY_MODEL_PATHS` | *(auto)* | Colon-separated extra model search paths |\n| `SHIMMY_ENGINE_BACKEND` | `airframe` | Inference engine (airframe is the only engine) |\n| `SHIMMY_ROPE_SCALE` | *(auto)* | Override computed YaRN scale factor |\n| `SHIMMY_KV_QUANT` | `f32` | `f32` (default) or `int4` ([TurboShimmy](turboshimmy.md)) |\n| `SHIMMY_PREFILL_CHUNK` | `64` | Tokens per prefill dispatch; set to `8` on Windows to avoid GPU TDR resets |\n| `RUST_BACKTRACE` | *(off)* | Set to `1` for crash backtraces |\n\nSee [docs/CONFIGURATION.md](CONFIGURATION.md) for the complete configuration\nreference.\n\n## Error Responses\n\nAll endpoints return a consistent error format:\n\n```json\n{\n  \"error\": {\n    \"code\": \"model_not_found\",\n    \"message\": \"The specified model was not found\",\n    \"details\": \"Model 'invalid-model' is not available\"\n  }\n}\n```\n\nCommon error codes:\n- `model_not_found` — requested model is not available\n- `invalid_request` — request format is invalid\n- `generation_failed` — text generation failed\n- `server_error` — internal server error\n\n## Rate Limiting\n\nNo rate limiting is currently implemented. For production use, place Shimmy\nbehind a reverse proxy with rate-limiting capabilities.\n"},"items":[{"name":"API.md","path":"docs/API.md","rawUrl":"https://raw.githubusercontent.com/Michael-A-Kuykendall/shimmy/HEAD/docs/API.md","title":"docs - API Interface Contract & Specifications","category":"project-spec","format":"markdown","content":"# API Reference\n\nShimmy exposes three interface families:\n\n- **OpenAI-compatible** (`/v1/*`) — drop-in replacement for OpenAI SDKs and tools\n- **Shimmy native** (`/api/*`) — the original REST + WebSocket API\n- **Ollama-compatible** (`/api/tags`) and **Anthropic-compatible** (`/v1/messages`) — for existing tooling\n\nAll endpoints serve the same inference engine. See\n[docs/OPENAI_COMPAT.md](OPENAI_COMPAT.md) for the exact OpenAI parameter\ncompatibility matrix.\n\nInteractive documentation is available from a running server at\n`GET /docs`; the machine-readable contract is `GET /openapi.json`.\nSee [OpenAPI and Swagger UI](OPENAPI.md) for usage and implementation details.\n\n## OpenAI-Compatible Endpoints\n\n### `GET /v1/models`\nLists available models (also served as `GET /api/tags` for Ollama clients).\n\n```json\n{\n  \"object\": \"list\",\n  \"data\": [\n    { \"id\": \"tinyllama-1.1b\", \"object\": \"model\", \"owned_by\": \"shimmy\" }\n  ]\n}\n```\n\n### `POST /v1/chat/completions`\nOpenAI-compatible chat completions. Streaming supported via SSE when\n`\"stream\": true`.\n\n```json\n{\n  \"model\": \"tinyllama-1.1b\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Say hi in 5 words.\"}],\n  \"max_tokens\": 32,\n  \"temperature\": 0.7\n}\n```\n\n### `POST /v1/completions`\nLegacy text completions (non-chat). Same request shape minus `messages`.\n\n### `POST /v1/messages`\nAnthropic Messages API compatibility (`x-api-key` optional, ignored).\n\n## Shimmy Native Endpoints\n\n### `POST /api/generate`\nText generation (streaming + non-streaming).\n\n**Request:**\n```json\n{\n  \"model\": \"string\",\n  \"prompt\": \"string\",\n  \"max_tokens\": 100,\n  \"temperature\": 0.7,\n  \"stream\": false\n}\n```\n\n**Non-streaming response:**\n```json\n{\n  \"choices\": [{\"text\": \"Generated text\", \"index\": 0, \"finish_reason\": \"length\"}],\n  \"usage\": {\"prompt_tokens\": 10, \"completion_tokens\": 20, \"total_tokens\": 30}\n}\n```\n\n**Streaming response** (SSE):\n```\ndata: {\"choices\":[{\"text\":\"Hello\",\"index\":0}]}\ndata: {\"choices\":[{\"text\":\" world\",\"index\":0}]}\ndata: [DONE]\n```\n\n### `GET /api/models`\nList loaded models.\n\n### `POST /api/models/discover`\nTrigger model re-discovery.\n\n### `POST /api/models/:name/load` / `POST /api/models/:name/unload`\nLoad or unload a specific model by name.\n\n### `GET /api/models/:name/status`\nModel status (loaded, VRAM, context).\n\n### `GET /api/tags`\nOllama-compatible model list.\n\n### `GET /health`\nHealth check.\n\n```json\n{\"status\": \"healthy\", \"models_loaded\": 1, \"memory_usage\": \"2.1GB\"}\n```\n\n### `GET /metrics`\nPrometheus-style metrics endpoint.\n\n### `GET /diag`\nDiagnostics endpoint.\n\n## WebSocket API\n\n**Endpoint:** `ws://localhost:11435/ws/generate`\n\nSend a single JSON `GenerateRequest` frame:\n```json\n{\"model\": \"default\", \"prompt\": \"Hello world\", \"max_tokens\": 50, \"temperature\": 0.7}\n```\n\nReceive token frames:\n```json\n{\"token\": \"Hello\"}\n{\"token\": \" world\"}\n{\"done\": true}\n```\n\nThe WebSocket upgrade is documented in the OpenAPI contract as `GET\n/ws/generate`, but its token frames are not REST JSON responses.\n\n## CLI\n\n```bash\nshimmy serve                              # Start server (auto port allocation)\nshimmy serve --bind 127.0.0.1:8080        # Manual port binding\nshimmy serve --kv-quant int4              # Enable TurboShimmy INT4 KV\nshimmy list                               # Show available models\nshimmy discover                           # Refresh model discovery\nshimmy generate --name X --prompt \"Hi\"    # Test generation\nshimmy probe model-name                   # Verify model loads\nshimmy gpu-info                           # Show selected WebGPU adapter\n```\n\nGlobal flags: `--verbose/-v`, `--help/-h`, `--version/-V`.\n\n## Environment Variables\n\n| Variable | Default | Description |\n|---|---|---|\n| `SHIMMY_BASE_GGUF` | *(auto-discover)* | Path to GGUF model loaded as default |\n| `SHIMMY_PORT` | `8080` | Listen port |\n| `SHIMMY_BIND_ADDRESS` | `0.0.0.0:8080` | Full bind address (overrides port) |\n| `SHIMMY_MAX_CTX` | *(from GGUF)* | Override context window; activates YaRN RoPE scaling when above native |\n| `SHIMMY_MODEL_PATHS` | *(auto)* | Colon-separated extra model search paths |\n| `SHIMMY_ENGINE_BACKEND` | `airframe` | Inference engine (airframe is the only engine) |\n| `SHIMMY_ROPE_SCALE` | *(auto)* | Override computed YaRN scale factor |\n| `SHIMMY_KV_QUANT` | `f32` | `f32` (default) or `int4` ([TurboShimmy](turboshimmy.md)) |\n| `SHIMMY_PREFILL_CHUNK` | `64` | Tokens per prefill dispatch; set to `8` on Windows to avoid GPU TDR resets |\n| `RUST_BACKTRACE` | *(off)* | Set to `1` for crash backtraces |\n\nSee [docs/CONFIGURATION.md](CONFIGURATION.md) for the complete configuration\nreference.\n\n## Error Responses\n\nAll endpoints return a consistent error format:\n\n```json\n{\n  \"error\": {\n    \"code\": \"model_not_found\",\n    \"message\": \"The specified model was not found\",\n    \"details\": \"Model 'invalid-model' is not available\"\n  }\n}\n```\n\nCommon error codes:\n- `model_not_found` — requested model is not available\n- `invalid_request` — request format is invalid\n- `generation_failed` — text generation failed\n- `server_error` — internal server error\n\n## Rate Limiting\n\nNo rate limiting is currently implemented. For production use, place Shimmy\nbehind a reverse proxy with rate-limiting capabilities.\n","isInternal":false,"tokens":1428,"sizeBytes":5194},{"name":"ARCHITECTURE.md","path":"docs/ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/Michael-A-Kuykendall/shimmy/HEAD/docs/ARCHITECTURE.md","title":"docs - System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# Shimmy v2.0 Architecture\r\n\r\n## Overview\r\n\r\nShimmy is a single-binary OpenAI-compatible HTTP server for local GGUF inference.\r\nIn v2.0 the default inference backend is **Airframe** — a pure-Rust WebGPU (WGSL)\r\ntransformer runtime that requires no C++ toolchain, no CUDA SDK, and no Vulkan SDK.\r\n\r\n---\r\n\r\n## Component Map\r\n\r\n```\r\nshimmy (binary)\r\n ├── HTTP layer (axum / tokio)\r\n │    ├── POST /v1/chat/completions\r\n │    ├── POST /v1/completions\r\n │    └── GET  /v1/models\r\n ├── Engine bridge (src/engine/airframe.rs)\r\n │    └── converts OpenAI request → GenOptions → SamplingParams\r\n └── Airframe runtime  (../airframe crate, path dep)\r\n      ├── GGUF loader (weights + metadata)\r\n      ├── Tokenizer (shimmytok)\r\n      ├── GPU pipeline (wgpu + WGSL shaders)\r\n      │    ├── Embedding lookup\r\n      │    ├── Per-layer transformer blocks\r\n      │    │    ├── Attention (Q/K/V matmul + RoPE + softmax)\r\n      │    │    └── FFN (gate / up / down projections)\r\n      │    ├── Output head (dequant + logit projection)\r\n      │    └── KV cache (GPU buffers, reset per request)\r\n      └── Sampler (repeat penalty → temperature → top-p)\r\n```\r\n\r\n---\r\n\r\n## Airframe Engine\r\n\r\n### WGSL / WebGPU\r\n\r\nAirframe uses **wgpu** as the WebGPU implementation layer.  wgpu translates WGSL\r\nshaders to the best available native API:\r\n\r\n| Platform | Native API used by wgpu |\r\n|---|---|\r\n| Windows | Direct3D 12 (primary) or Vulkan |\r\n| Linux   | Vulkan (primary) or OpenGL |\r\n| macOS   | Metal |\r\n\r\nNo Vulkan SDK or CUDA toolkit is required — wgpu drivers ship with the binary.\r\nWGSL shaders are compiled at startup using `wgpu::Device::create_shader_module`.\r\n\r\n### Quantization\r\n\r\nWeights are stored in GGUF quantized format.  The GPU dequantization shaders\r\ndecode each block on the fly during the matrix multiply:\r\n\r\n| GGUF type | GPU shader |\r\n|---|---|\r\n| F32 / F16 | passthrough / fp16 unpack |\r\n| Q4_0      | 32-element block, 4-bit signed |\r\n| Q8_0      | 32-element block, 8-bit signed |\r\n| Q4_K_M    | super-block with row scales |\r\n| Q5_K_M    | super-block, 5-bit with high-bit plane |\r\n| Q6_K      | super-block, 6-bit |\r\n\r\nEvery GPU shader result is validated against the CPU reference before a model\r\nis considered supported (`quant_verify` tool).\r\n\r\n### KV Cache\r\n\r\nThe KV cache is a pair of GPU buffers (keys + values) allocated at server start\r\nwith capacity `n_layers × n_kv_heads × head_dim × max_ctx × sizeof(f32)`.\r\nThe cache is **fully reset between requests** — Shimmy is stateless.\r\n\r\n### Extended Context (YaRN)\r\n\r\nWhen `SHIMMY_MAX_CTX` exceeds the model's native context window, Airframe:\r\n1. Recomputes the RoPE theta scale using the YaRN formula\r\n2. Reallocates the KV cache to the requested length\r\n3. Applies the scale to all position-dependent attention computations\r\n\r\n---\r\n\r\n## Request Lifecycle\r\n\r\n```\r\nHTTP POST /v1/chat/completions\r\n  │\r\n  ├─ serde_json deserialization (ChatCompletionRequest)\r\n  ├─ Input validation (empty messages, max_tokens bounds)\r\n  ├─ Chat template application → prompt string\r\n  ├─ Engine bridge: build GenOptions + SamplingParams\r\n  │    ├─ stop_tokens from chat template (e.g. <|eot_id|> for Llama-3)\r\n  │    ├─ extra_stop_tokens propagated to SamplingParams\r\n  │    └─ penalty fields mapped: repeat_penalty = 1 + max(freq, presence) * 0.5\r\n  ├─ Airframe generate()\r\n  │    ├─ Tokenize prompt\r\n  │    ├─ Chunked prefill (512-token chunks)\r\n  │    ├─ Decode loop (one token per step)\r\n  │    │    ├─ GPU forward pass\r\n  │    │    ├─ Sample (penalty → temperature → top-p)\r\n  │    │    └─ Stop on EOS | extra_stop_tokens | max_tokens\r\n  │    └─ Detokenize\r\n  └─ OpenAI response JSON\r\n```\r\n\r\n---\r\n\r\n## Model Discovery\r\n\r\nOn startup Shimmy scans the following paths for GGUF files:\r\n\r\n1. `SHIMMY_BASE_GGUF` (explicit env var — highest priority)\r\n2. `SHIMMY_MODEL_PATHS` (colon-separated list)\r\n3. `~/.cache/huggingface/hub/` (HuggingFace cache)\r\n4. `~/.ollama/models/` (Ollama model store)\r\n5. `~/.cache/lm-studio/models/` (LM Studio)\r\n6. `~/models/` (generic local dir)\r\n\r\n---\r\n\r\n## Known Limitations (v2.0)\r\n\r\n| Limitation | Detail |\r\n|---|---|\r\n| 2 GB WebGPU buffer cap | Single tensor buffer ≤ 2 GB; output heads > 2 GB (e.g. Gemma-2-2B) require chunking (not yet implemented) |\r\n| Fused QKV tensors | Phi-3 / Phi-3.5 pack Q+K+V into one weight; Airframe expects separate tensors — server will crash |\r\n| Stateless only | No multi-turn KV cache persistence; each request starts fresh |\r\n| Streaming | Token streaming via SSE is on the roadmap; `stream: true` is accepted but returns full completion |\r\n| MoE models | CPU offloading for 70B+ MoE is on the roadmap; current architecture loads entire model on GPU |\r\n","isInternal":false,"tokens":1262,"sizeBytes":4907}],"systemPromptSnippet":"<yakaai_skills repo=\"Michael-A-Kuykendall/shimmy\">\n<!-- File: docs/API.md (Tokens: ~1428 | Category: project-spec) -->\n# API Reference\n\nShimmy exposes three interface families:\n\n- **OpenAI-compatible** (`/v1/*`) — drop-in replacement for OpenAI SDKs and tools\n- **Shimmy native** (`/api/*`) — the original REST + WebSocket API\n- **Ollama-compatible** (`/api/tags`) and **Anthropic-compatible** (`/v1/messages`) — for existing tooling\n\nAll endpoints serve the same inference engine. See\n[docs/OPENAI_COMPAT.md](OPENAI_COMPAT.md) for the exact OpenAI parameter\ncompatibility matrix.\n\nInteractive documentation is available from a running server at\n`GET /docs`; the machine-readable contract is `GET /openapi.json`.\nSee [OpenAPI and Swagger UI](OPENAPI.md) for usage and implementation details.\n\n## OpenAI-Compatible Endpoints\n\n### `GET /v1/models`\nLists available models (also served as `GET /api/tags` for Ollama clients).\n\n```json\n{\n  \"object\": \"list\",\n  \"data\": [\n    { \"id\": \"tinyllama-1.1b\", \"object\": \"model\", \"owned_by\": \"shimmy\" }\n  ]\n}\n```\n\n### `POST /v1/chat/completions`\nOpenAI-compatible chat completions. Streaming supported via SSE when\n`\"stream\": true`.\n\n```json\n{\n  \"model\": \"tinyllama-1.1b\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Say hi in 5 words.\"}],\n  \"max_tokens\": 32,\n  \"temperature\": 0.7\n}\n```\n\n### `POST /v1/completions`\nLegacy text completions (non-chat). Same request shape minus `messages`.\n\n### `POST /v1/messages`\nAnthropic Messages API compatibility (`x-api-key` optional, ignored).\n\n## Shimmy Native Endpoints\n\n### `POST /api/generate`\nText generation (streaming + non-streaming).\n\n**Request:**\n```json\n{\n  \"model\": \"string\",\n  \"prompt\": \"string\",\n  \"max_tokens\": 100,\n  \"temperature\": 0.7,\n  \"stream\": false\n}\n```\n\n**Non-streaming response:**\n```json\n{\n  \"choices\": [{\"text\": \"Generated text\", \"index\": 0, \"finish_reason\": \"length\"}],\n  \"usage\": {\"prompt_tokens\": 10, \"completion_tokens\": 20, \"total_tokens\": 30}\n}\n```\n\n**Streaming response** (SSE):\n```\ndata: {\"choices\":[{\"text\":\"Hello\",\"index\":0}]}\ndata: {\"choices\":[{\"text\":\" world\",\"index\":0}]}\ndata: [DONE]\n```\n\n### `GET /api/models`\nList loaded models.\n\n### `POST /api/models/discover`\nTrigger model re-discovery.\n\n### `POST /api/models/:name/load` / `POST /api/models/:name/unload`\nLoad or unload a specific model by name.\n\n### `GET /api/models/:name/status`\nModel status (loaded, VRAM, context).\n\n### `GET /api/tags`\nOllama-compatible model list.\n\n### `GET /health`\nHealth check.\n\n```json\n{\"status\": \"healthy\", \"models_loaded\": 1, \"memory_usage\": \"2.1GB\"}\n```\n\n### `GET /metrics`\nPrometheus-style metrics endpoint.\n\n### `GET /diag`\nDiagnostics endpoint.\n\n## WebSocket API\n\n**Endpoint:** `ws://localhost:11435/ws/generate`\n\nSend a single JSON `GenerateRequest` frame:\n```json\n{\"model\": \"default\", \"prompt\": \"Hello world\", \"max_tokens\": 50, \"temperature\": 0.7}\n```\n\nReceive token frames:\n```json\n{\"token\": \"Hello\"}\n{\"token\": \" world\"}\n{\"done\": true}\n```\n\nThe WebSocket upgrade is documented in the OpenAPI contract as `GET\n/ws/generate`, but its token frames are not REST JSON responses.\n\n## CLI\n\n```bash\nshimmy serve                              # Start server (auto port allocation)\nshimmy serve --bind 127.0.0.1:8080        # Manual port binding\nshimmy serve --kv-quant int4              # Enable TurboShimmy INT4 KV\nshimmy list                               # Show available models\nshimmy discover                           # Refresh model discovery\nshimmy generate --name X --prompt \"Hi\"    # Test generation\nshimmy probe model-name                   # Verify model loads\nshimmy gpu-info                           # Show selected WebGPU adapter\n```\n\nGlobal flags: `--verbose/-v`, `--help/-h`, `--version/-V`.\n\n## Environment Variables\n\n| Variable | Default | Description |\n|---|---|---|\n| `SHIMMY_BASE_GGUF` | *(auto-discover)* | Path to GGUF model loaded as default |\n| `SHIMMY_PORT` | `8080` | Listen port |\n| `SHIMMY_BIND_ADDRESS` | `0.0.0.0:8080` | Full bind address (overrides port) |\n| `SHIMMY_MAX_CTX` | *(from GGUF)* | Override context window; activates YaRN RoPE scaling when above native |\n| `SHIMMY_MODEL_PATHS` | *(auto)* | Colon-separated extra model search paths |\n| `SHIMMY_ENGINE_BACKEND` | `airframe` | Inference engine (airframe is the only engine) |\n| `SHIMMY_ROPE_SCALE` | *(auto)* | Override computed YaRN scale factor |\n| `SHIMMY_KV_QUANT` | `f32` | `f32` (default) or `int4` ([TurboShimmy](turboshimmy.md)) |\n| `SHIMMY_PREFILL_CHUNK` | `64` | Tokens per prefill dispatch; set to `8` on Windows to avoid GPU TDR resets |\n| `RUST_BACKTRACE` | *(off)* | Set to `1` for crash backtraces |\n\nSee [docs/CONFIGURATION.md](CONFIGURATION.md) for the complete configuration\nreference.\n\n## Error Responses\n\nAll endpoints return a consistent error format:\n\n```json\n{\n  \"error\": {\n    \"code\": \"model_not_found\",\n    \"message\": \"The specified model was not found\",\n    \"details\": \"Model 'invalid-model' is not available\"\n  }\n}\n```\n\nCommon error codes:\n- `model_not_found` — requested model is not available\n- `invalid_request` — request format is invalid\n- `generation_failed` — text generation failed\n- `server_error` — internal server error\n\n## Rate Limiting\n\nNo rate limiting is currently implemented. For production use, place Shimmy\nbehind a reverse proxy with rate-limiting capabilities.\n\n<!-- File: docs/ARCHITECTURE.md (Tokens: ~1262 | Category: project-spec) -->\n# Shimmy v2.0 Architecture\r\n\r\n## Overview\r\n\r\nShimmy is a single-binary OpenAI-compatible HTTP server for local GGUF inference.\r\nIn v2.0 the default inference backend is **Airframe** — a pure-Rust WebGPU (WGSL)\r\ntransformer runtime that requires no C++ toolchain, no CUDA SDK, and no Vulkan SDK.\r\n\r\n---\r\n\r\n## Component Map\r\n\r\n```\r\nshimmy (binary)\r\n ├── HTTP layer (axum / tokio)\r\n │    ├── POST /v1/chat/completions\r\n │    ├── POST /v1/completions\r\n │    └── GET  /v1/models\r\n ├── Engine bridge (src/engine/airframe.rs)\r\n │    └── converts OpenAI request → GenOptions → SamplingParams\r\n └── Airframe runtime  (../airframe crate, path dep)\r\n      ├── GGUF loader (weights + metadata)\r\n      ├── Tokenizer (shimmytok)\r\n      ├── GPU pipeline (wgpu + WGSL shaders)\r\n      │    ├── Embedding lookup\r\n      │    ├── Per-layer transformer blocks\r\n      │    │    ├── Attention (Q/K/V matmul + RoPE + softmax)\r\n      │    │    └── FFN (gate / up / down projections)\r\n      │    ├── Output head (dequant + logit projection)\r\n      │    └── KV cache (GPU buffers, reset per request)\r\n      └── Sampler (repeat penalty → temperature → top-p)\r\n```\r\n\r\n---\r\n\r\n## Airframe Engine\r\n\r\n### WGSL / WebGPU\r\n\r\nAirframe uses **wgpu** as the WebGPU implementation layer.  wgpu translates WGSL\r\nshaders to the best available native API:\r\n\r\n| Platform | Native API used by wgpu |\r\n|---|---|\r\n| Windows | Direct3D 12 (primary) or Vulkan |\r\n| Linux   | Vulkan (primary) or OpenGL |\r\n| macOS   | Metal |\r\n\r\nNo Vulkan SDK or CUDA toolkit is required — wgpu drivers ship with the binary.\r\nWGSL shaders are compiled at startup using `wgpu::Device::create_shader_module`.\r\n\r\n### Quantization\r\n\r\nWeights are stored in GGUF quantized format.  The GPU dequantization shaders\r\ndecode each block on the fly during the matrix multiply:\r\n\r\n| GGUF type | GPU shader |\r\n|---|---|\r\n| F32 / F16 | passthrough / fp16 unpack |\r\n| Q4_0      | 32-element block, 4-bit signed |\r\n| Q8_0      | 32-element block, 8-bit signed |\r\n| Q4_K_M    | super-block with row scales |\r\n| Q5_K_M    | super-block, 5-bit with high-bit plane |\r\n| Q6_K      | super-block, 6-bit |\r\n\r\nEvery GPU shader result is validated against the CPU reference before a model\r\nis considered supported (`quant_verify` tool).\r\n\r\n### KV Cache\r\n\r\nThe KV cache is a pair of GPU buffers (keys + values) allocated at server start\r\nwith capacity `n_layers × n_kv_heads × head_dim × max_ctx × sizeof(f32)`.\r\nThe cache is **fully reset between requests** — Shimmy is stateless.\r\n\r\n### Extended Context (YaRN)\r\n\r\nWhen `SHIMMY_MAX_CTX` exceeds the model's native context window, Airframe:\r\n1. Recomputes the RoPE theta scale using the YaRN formula\r\n2. Reallocates the KV cache to the requested length\r\n3. Applies the scale to all position-dependent attention computations\r\n\r\n---\r\n\r\n## Request Lifecycle\r\n\r\n```\r\nHTTP POST /v1/chat/completions\r\n  │\r\n  ├─ serde_json deserialization (ChatCompletionRequest)\r\n  ├─ Input validation (empty messages, max_tokens bounds)\r\n  ├─ Chat template application → prompt string\r\n  ├─ Engine bridge: build GenOptions + SamplingParams\r\n  │    ├─ stop_tokens from chat template (e.g. <|eot_id|> for Llama-3)\r\n  │    ├─ extra_stop_tokens propagated to SamplingParams\r\n  │    └─ penalty fields mapped: repeat_penalty = 1 + max(freq, presence) * 0.5\r\n  ├─ Airframe generate()\r\n  │    ├─ Tokenize prompt\r\n  │    ├─ Chunked prefill (512-token chunks)\r\n  │    ├─ Decode loop (one token per step)\r\n  │    │    ├─ GPU forward pass\r\n  │    │    ├─ Sample (penalty → temperature → top-p)\r\n  │    │    └─ Stop on EOS | extra_stop_tokens | max_tokens\r\n  │    └─ Detokenize\r\n  └─ OpenAI response JSON\r\n```\r\n\r\n---\r\n\r\n## Model Discovery\r\n\r\nOn startup Shimmy scans the following paths for GGUF files:\r\n\r\n1. `SHIMMY_BASE_GGUF` (explicit env var — highest priority)\r\n2. `SHIMMY_MODEL_PATHS` (colon-separated list)\r\n3. `~/.cache/huggingface/hub/` (HuggingFace cache)\r\n4. `~/.ollama/models/` (Ollama model store)\r\n5. `~/.cache/lm-studio/models/` (LM Studio)\r\n6. `~/models/` (generic local dir)\r\n\r\n---\r\n\r\n## Known Limitations (v2.0)\r\n\r\n| Limitation | Detail |\r\n|---|---|\r\n| 2 GB WebGPU buffer cap | Single tensor buffer ≤ 2 GB; output heads > 2 GB (e.g. Gemma-2-2B) require chunking (not yet implemented) |\r\n| Fused QKV tensors | Phi-3 / Phi-3.5 pack Q+K+V into one weight; Airframe expects separate tensors — server will crash |\r\n| Stateless only | No multi-turn KV cache persistence; each request starts fresh |\r\n| Streaming | Token streaming via SSE is on the roadmap; `stream: true` is accepted but returns full completion |\r\n| MoE models | CPU offloading for 70B+ MoE is on the roadmap; current architecture loads entire model on GPU |\n</yakaai_skills>"}