{"owner":"AlexsJones","repo":"llmfit","hasSkills":true,"totalSkillsCount":3,"totalTokensCount":10029,"categories":["project-spec","anthropic-skill","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["API.md","skills/llmfit-advisor/SKILL.md","AGENTS.md"],"skills":{"API.md":"# llmfit REST API Guide\n\nThis document is for agent/client builders integrating with `llmfit serve`.\n\n## Purpose\n\n`llmfit serve` exposes node-local model fit analysis (same core data used by TUI/CLI) over HTTP and serves a local web dashboard.\n\nPrimary use case:\n- Query each node in a cluster for top runnable models.\n- Aggregate externally (scheduler/controller/UI) for placement decisions.\n\n## Start the server\n\n```sh\nllmfit serve --port 8787\n```\n\nGlobal flags still apply:\n\n```sh\nllmfit --memory 24G --ram 64G --cpu-cores 16 --max-context 8192 serve --port 8787\n```\n\nHardware overrides (`--memory`, `--ram`, `--cpu-cores`) are reflected in API responses, making the server report the overridden values instead of the detected hardware.\n\n## Base URL\n\nDefault local base URL:\n\n```text\nhttp://127.0.0.1:8787\n```\n\nTo expose outside localhost, pass `--host 0.0.0.0`.\n\n### Unix domain socket\n\nFor same-host consumers that should not touch the network at all (e.g. a\nsidecar in a `hostNetwork` Kubernetes pod, where a TCP bind would land on the\nnode's loopback), listen on a Unix socket instead:\n\n```sh\nllmfit serve --unix-socket /run/llmfit/llmfit.sock\n```\n\nThe socket is created with mode `0660`; a stale socket file from a previous\ninstance is replaced automatically. All HTTP endpoints are identical:\n\n```sh\ncurl --unix-socket /run/llmfit/llmfit.sock http://localhost/api/v1/system\n```\n\n`--unix-socket` conflicts with `--host`/`--port` and is unix-platforms only.\n\nIf you are building from source and want the dashboard embedded in `llmfit`, build web assets first:\n\n```sh\ncd llmfit-web && npm ci && npm run build\n```\n\n## Endpoints\n\n### `GET /`\nWeb dashboard entrypoint (same-origin UI for fit exploration).\n\n### `GET /health`\nLiveness probe.\n\nExample response:\n\n```json\n{\n  \"status\": \"ok\",\n  \"node\": {\n    \"name\": \"worker-1\",\n    \"os\": \"linux\"\n  }\n}\n```\n\n---\n\n### `GET /api/v1/system`\nReturns node identity + detected hardware.\n\nExample response shape:\n\n```json\n{\n  \"node\": {\n    \"name\": \"worker-1\",\n    \"os\": \"linux\"\n  },\n  \"system\": {\n    \"total_ram_gb\": 62.23,\n    \"available_ram_gb\": 41.08,\n    \"cpu_cores\": 14,\n    \"cpu_name\": \"Intel(R) Core(TM) Ultra 7 165U\",\n    \"has_gpu\": false,\n    \"gpu_vram_gb\": null,\n    \"gpu_available_gb\": null,\n    \"gpu_name\": null,\n    \"gpu_count\": 0,\n    \"unified_memory\": false,\n    \"backend\": \"CPU (x86)\",\n    \"gpus\": []\n  },\n  \"profile\": null\n}\n```\n\n`profile` is the name of the hardware profile the server was started with (`llmfit serve --profile <name>`), or `null` when it describes the machine it runs on.\n\n`gpu_available_gb` is the VRAM free right now, pooled across discrete GPUs (from `nvidia-smi memory.free` and amdgpu's `mem_info_vram_used`); each `gpus[]` entry carries its own `free_vram_gb`. Both are `null` when a backend does not report it (Intel, Windows, older drivers) or after a hardware override. On Apple Silicon `gpu_available_gb` is instead Metal's wiring cap for the unified pool. `plan` grades GPU run paths against the free figure when it is known and against total VRAM otherwise.\n\n---\n\n### `GET /api/v1/models`\nReturns filtered/sorted model-fit rows for this node.\n\nEnvelope shape:\n\n```json\n{\n  \"node\": { \"name\": \"worker-1\", \"os\": \"linux\" },\n  \"system\": { \"...\": \"...\" },\n  \"total_models\": 23,\n  \"returned_models\": 10,\n  \"filters\": { \"...\": \"echo of query state\" },\n  \"models\": [\n    {\n      \"name\": \"Qwen/Qwen2.5-Coder-7B-Instruct\",\n      \"provider\": \"Qwen\",\n      \"parameter_count\": \"7B\",\n      \"params_b\": 7.0,\n      \"context_length\": 32768,\n      \"usable_context\": 32768,\n      \"effective_context_length\": 8192,\n      \"use_case\": \"Coding\",\n      \"category\": \"Coding\",\n      \"release_date\": \"2025-03-14\",\n      \"is_moe\": false,\n      \"fit_level\": \"good\",\n      \"fit_label\": \"Good\",\n      \"run_mode\": \"gpu\",\n      \"run_mode_label\": \"GPU\",\n      \"score\": 86.5,\n      \"score_components\": {\n        \"quality\": 87.0,\n        \"speed\": 81.2,\n        \"fit\": 90.1,\n        \"context\": 88.0\n      },\n      \"estimated_tps\": 42.5,\n      \"estimate_confidence\": \"estimated\",\n      \"estimate_confidence_label\": \"estimated\",\n      \"prefill_tps\": 812.3,\n      \"ttft_ms\": 10.1,\n      \"runtime\": \"llamacpp\",\n      \"runtime_label\": \"llama.cpp\",\n      \"best_quant\": \"Q5_K_M\",\n      \"memory_required_gb\": 5.8,\n      \"memory_available_gb\": 12.0,\n      \"utilization_pct\": 48.3,\n      \"notes\": [],\n      \"gguf_sources\": [],\n      \"capabilities\": [\"tool_use\"],\n      \"capability_ids\": [\"tool_use\"],\n      \"license\": \"apache-2.0\",\n      \"supports_tp\": [1, 2, 4],\n      \"installed\": false,\n      \"disk_size_gb\": 5.1,\n      \"ollama_name\": \"qwen2.5-coder:7b-instruct\",\n      \"estimate_basis\": {\n        \"method\": \"roofline\",\n        \"gpu_bandwidth_gbps\": 320.0,\n        \"ddr_bandwidth_gbps\": null,\n        \"local_calibration\": null,\n        \"efficiency\": 0.85,\n        \"assumed_context\": 8192\n      },\n      \"verify_command\": \"llama-bench -m <path-to-Q5_K_M-gguf> -ngl 99 -p 512 -n 128\",\n      \"measured_tps\": null\n    }\n  ]\n}\n```\n\nThe three context fields answer different questions:\n\n- `context_length` — the model's native window, as advertised upstream.\n- `usable_context` — how much of that window actually fits in this node's\n  available memory alongside the weights. Use this one to pick a runtime `-c`.\n- `effective_context_length` — the context the `memory_required_gb` and\n  `estimated_tps` figures on this row were computed at. Defaults to\n  `min(context_length, 8192)`; set by `max_context` when supplied.\n\nThe envelope also carries these fields, now at parity with `llmfit fit --json`\n(both frontends serialize through one shared function):\n\n- `installed` — whether the model was found in a local runtime provider. `llmfit serve` probes providers in the background (at startup, after a finished download, and when a request finds the result older than 30 seconds), so a model pulled or removed outside the server shows up within one refresh after that; it is `false` everywhere until the first probe finishes.\n- `disk_size_gb` — estimated on-disk size at `best_quant`.\n- `capability_ids` — machine-readable capability ids (snake_case); mirrors\n  `capabilities` here. Note `llmfit fit --json` overloads its `capabilities`\n  key with human labels (e.g. `\"Tool Use\"`) — that overload is CLI-only and\n  slated for deprecation.\n- `ollama_name` — the `ollama pull` tag for this model, when derivable.\n- `estimate_basis` — how `memory_required_gb`/`estimated_tps` were derived\n  (bandwidths, efficiency, assumed context), for reproducibility.\n- `estimate_confidence` / `estimate_confidence_label` — how much to trust\n  `estimated_tps`, most to least trustworthy: `measured_local` (your own\n  `llmfit bench` runs), `measured_community` (llmfit community submissions\n  or localmaxxing.com data on matching hardware), `calibrated` (a formula\n  estimate scaled by a correction factor from benchmark runs on this exact\n  hardware), `estimated` (a bare formula estimate), `unsupported` (no\n  estimate — the model needs a runtime llmfit can't model). See\n  `measured_tps`/`estimate_basis.local_calibration` for the data each level\n  is derived from.\n- `prefill_tps` / `ttft_ms` — estimated prompt-processing throughput\n  (tok/s) and time-to-first-token (ms) for a prompt of\n  `effective_context_length` tokens. Prefill is compute-bound, not\n  bandwidth-bound, so both are `null` — not `0.0` — unless the system's GPU\n  compute throughput is known. A `null` here means \"not estimated\", never\n  \"instant\" or \"stalled\".\n- `verify_command` — a `llama-bench` invocation measuring the same throughput\n  this row estimates (llama.cpp GPU / CPU-only runs; `null` otherwise).\n- `measured_tps` — a recorded benchmark result if one exists, else `null`.\n\nNote on vocabulary: `fit_level`, `run_mode`, and `runtime` here are stable\nmachine codes (e.g. `\"good\"`, `\"gpu\"`, `\"llamacpp\"`), with the human string\nunder the paired `*_label` key. `llmfit fit --json` emits the human string\ndirectly under those same keys — a CLI-only legacy overload.\n`estimate_confidence` follows the same convention (machine code, paired\n`estimate_confidence_label`) in both frontends.\n\n### A note on `best_quant`\n\n`best_quant` is normally the llama.cpp/GGUF quant llmfit picked for this\nhardware (e.g. `\"Q5_K_M\"`). It is `null` when the model's own repo name\ndeclares a native low-precision format (NVFP4/MXFP4) that a GGUF quant\nlabel doesn't apply to — the row's `notes` array carries the explanation\nin that case. It's never a GGUF label misattributed to a non-GGUF repo.\n\n---\n\n### Catalog sanitization\n\n`/api/v1/models` never returns speculative-decoding draft heads\n(EAGLE/DFlash/DSpark naming), entries whose name implies a wildly different\nparameter count than their declared size, or other catalog rows llmfit\ncan't score honestly — they're demoted out of fit ranking, not deleted, so\nthis is invisible unless you were expecting a specific draft-head repo to\nshow up as a standalone model. If you maintain your own filtering on top of\n`llmfit fit --json` output today, you can likely delete that workaround.\n\n---\n\n### `GET /api/v1/models/top`\nKey scheduling endpoint. Same schema as `/api/v1/models`, but defaults to top 5 runnable entries.\n\nImportant behavior:\n- Defaults `limit=5`.\n- Excludes `too_tight` rows unless explicitly overridden (and top endpoint still keeps runnable semantics).\n\n---\n\n### `GET /api/v1/models/{name}`\nPath-constrained search. Equivalent to a text search scoped by `{name}`.\n\nUseful for:\n- Client-side drilldown after selecting a model family.\n\n---\n\n### `GET /api/v1/concurrency`\nConcurrent-session capacity for one model: the same estimate as\n`llmfit concurrency <model> --json`. It is a memory-capacity ceiling (how many\nsessions fit resident at once), not a throughput figure under load.\n\nQuery parameters:\n- `model` (required): exact catalog name, case-insensitive.\n- `quant`: weight quantization (e.g. `Q4_K_M`); defaults to the model's best fit.\n- `kv_quant`: `fp16|fp8|q8_0|q4_0|tq` (default `fp16`).\n- `context`: report one context instead of the 4k-256k ladder (tokens).\n- `users`: also report the largest context that fits this many sessions.\n- `ram_gb`, `vram_gb`, `cpu_cores`: hardware overrides, as on `/api/v1/system`.\n\n```json\n{\n  \"model\": \"Qwen/Qwen2.5-7B-Instruct\",\n  \"run_mode\": \"gpu\",\n  \"fit_level\": \"perfect\",\n  \"target_users\": 8,\n  \"max_context_for_target\": 32768,\n  \"estimate\": {\n    \"pool_gb\": 24.0,\n    \"weights_resident_gb\": 5.1,\n    \"kv_budget_gb\": 18.9,\n    \"quant\": \"Q4_K_M\",\n    \"kv_quant\": \"fp16\",\n    \"native_context\": 32768,\n    \"ladder\": [\n      { \"requested_context\": 4096, \"effective_context\": 4096, \"clamped\": false, \"per_session_kv_gb\": 0.22, \"max_sessions\": 85 }\n    ]\n  }\n}\n```\n\n`run_mode` and `fit_level` use the same codes as `/api/v1/models`.\n\n---\n\n### `GET /api/v1/storage`\nDisk needed to keep a library of runnable models, used one at a time: the same\nestimate as `llmfit storage --json`, returned as `{ \"system\": ..., \"storage\": ... }`.\n\nQuery parameters (defaults match the CLI):\n- `keep`: number of distinct runnable models to keep (default `3`).\n- `selection`: `score` (highest-ranked, default) or `largest` (largest weight storage).\n- `os_reserve`: space for the OS, apps and other files (default `100G`; decimal GB, `GiB`/`TiB` are binary).\n- `scratch`: `auto` (default, the largest selected model) or a size; `0` disables it.\n- `headroom`: percent of the suggested SSD to keep free, `0`-`99` (default `15`).\n- `perfect`: `true` to select only Perfect fits (otherwise Perfect, Good and Marginal).\n- `search`: narrow the candidates by name, provider or parameter size.\n- `max_context`, `ram_gb`, `vram_gb`, `cpu_cores`: as on `/api/v1/models`.\n\nThe `storage` object carries `models` (name, best quant, fit level, runtime,\nscore, disk size, context), `library_gb`, `download_scratch_gb`, `need_gb`,\n`target_capacity_gb`, `minimum_ssd_gb`, `suggested_ssd_gb` and `warnings`.\n\n## Query parameters\n\nSupported on `/api/v1/models` and `/api/v1/models/top` (also `/api/v1/models/{name}`):\n\n- `limit` (or alias `n`): max rows returned.\n- `perfect`: `true|false` (when `true`, only perfect fits).\n- `min_fit`: `perfect|good|marginal|too_tight`.\n- `runtime`: `any|mlx|llamacpp|vllm|bitnetcpp`.\n- `use_case`: `general|coding|reasoning|chat|multimodal|embedding`.\n- `provider`: provider substring filter.\n- `search`: free-text filter (name/provider/params/use-case/category).\n- `sort`: `score|tps|params|mem|ctx|date|use_case`.\n- `include_too_tight`: `true` includes unrunnable (`too_tight`) rows even when `min_fit` is not given; `false` drops them even with `min_fit=too_tight`. With neither parameter, `/models` returns rows from `marginal` up; `/models/top` never returns `too_tight` rows.\n- `max_context`: per-request context cap used by memory estimation.\n- `force_runtime`: `mlx|llamacpp|vllm|bitnetcpp` — override automatic runtime selection during analysis (e.g. get llama.cpp recommendations on Apple Silicon instead of MLX).\n\n## Error handling\n\nInvalid filter values return HTTP 400:\n\n```json\n{\n  \"error\": \"invalid min_fit value: use perfect|good|marginal|too_tight\"\n}\n```\n\nServer errors return HTTP 500 with `{\"error\": \"...\"}`.\n\n## Client integration recommendations\n\n### 1) Polling pattern for schedulers\nFor each node agent:\n1. Call `/health`.\n2. Call `/api/v1/system`.\n3. Call `/api/v1/models/top?limit=K&min_fit=good`.\n4. Attach node metadata and forward to your central scheduler.\n\n### 2) Conservative placement defaults\nFor production placement, prefer:\n\n```text\nmin_fit=good\ninclude_too_tight=false\nsort=score\nlimit=5..20\n```\n\n### 3) Per-workload targeting\nExamples:\n- Coding workloads: `use_case=coding`\n- Embedding workloads: `use_case=embedding`\n- Runtime constrained to llama.cpp fleet: `runtime=llamacpp`\n\n### 4) Stable parsing\nTreat unknown fields as forward-compatible additions:\n- Parse required fields you depend on.\n- Ignore unknown fields.\n\n## Curl examples\n\n```sh\ncurl http://127.0.0.1:8787/health\ncurl http://127.0.0.1:8787/api/v1/system\ncurl \"http://127.0.0.1:8787/api/v1/models?limit=20&min_fit=marginal&sort=score\"\ncurl \"http://127.0.0.1:8787/api/v1/models/top?limit=5&min_fit=good&use_case=coding\"\ncurl \"http://127.0.0.1:8787/api/v1/models/Mistral?runtime=any\"\ncurl \"http://127.0.0.1:8787/api/v1/concurrency?model=Qwen/Qwen2.5-7B-Instruct&users=8\"\ncurl \"http://127.0.0.1:8787/api/v1/storage?keep=5&selection=largest&os_reserve=200G\"\n```\n\n---\n\n## MCP Server Mode\n\nllmfit can run as an MCP (Model Context Protocol) server over stdio, making it discoverable by AI agents (Claude, Cursor, etc.).\n\n### Start the MCP server\n\n```sh\nllmfit serve --mcp\n```\n\nGlobal hardware overrides still apply:\n\n```sh\nllmfit --memory 24G --ram 64G serve --mcp\n```\n\n### MCP client configuration\n\nAdd to your MCP client config (e.g. `claude_desktop_config.json`):\n\n```json\n{\n  \"mcpServers\": {\n    \"llmfit\": {\n      \"command\": \"llmfit\",\n      \"args\": [\"serve\", \"--mcp\"]\n    }\n  }\n}\n```\n\n### Available tools\n\n| Tool | Description | Parameters |\n|------|-------------|------------|\n| `get_system_specs` | Node hardware info (RAM, GPU, CPU) | None |\n| `recommend_models` | Top models for this hardware | `limit?`, `use_case?`, `min_fit?`, `runtime?`, `license?`, `sort?` |\n| `search_models` | Free-text model search | `query`, `limit?` |\n| `plan_hardware` | Hardware requirements for a model | `model`, `context?`, `quant?`, `target_tps?` |\n| `get_runtimes` | Installed inference runtimes | None |\n| `get_installed_models` | Models in local runtimes | None |\n\n`plan_hardware` and `POST /api/v1/plan` return the shared plan estimate,\nincluding `disk_size_gb`: estimated weight storage in decimal GB at the\nresolved `quantization`. This excludes KV cache, inference buffers, and\ndownload scratch. `llmfit plan --json` returns the same plan fields.\n\n---\n\n## NATS Event Publishing\n\nWhen built with the `nats` feature, llmfit can publish hardware and model events to NATS for integration with coordination systems (e.g. Sympozium membrane).\n\n### Build with NATS support\n\n```sh\ncargo build --features nats\n```\n\n### Enable event publishing\n\n```sh\nllmfit serve --send-events --nats-url nats://localhost:4222\nllmfit serve --mcp --send-events  # also works with MCP mode\n```\n\nThe `NATS_URL` environment variable is also supported.\n\n### Event subjects\n\nEvents are published to `llmfit.{event_type}.{hostname}`:\n\n| Subject | Trigger | Payload |\n|---------|---------|---------|\n| `llmfit.system.{hostname}` | Startup + every 60s | System hardware specs |\n| `llmfit.fit.{hostname}` | After fit analysis | Model fit summary |\n| `llmfit.plan.{hostname}` | After plan estimate | Plan estimate |\n| `llmfit.runtimes.{hostname}` | Startup + on query | Runtime availability |\n| `llmfit.installed.{hostname}` | Startup + on query | Installed models |\n\n### Event envelope\n\nAll events are wrapped in a common envelope:\n\n```json\n{\n  \"timestamp\": \"1747058400\",\n  \"hostname\": \"worker-1\",\n  \"event_type\": \"system\",\n  \"version\": \"1\",\n  \"data\": { ... }\n}\n```\n\n### Subscribe to events\n\n```sh\nnats sub 'llmfit.>'                    # all events from all nodes\nnats sub 'llmfit.system.>'             # system specs from all nodes\nnats sub 'llmfit.system.worker-1'      # system specs from specific node\n```\n\n---\n\n## Versioning notes\n\nCurrent API prefix is `v1`.\n\nIf you build long-lived clients, pin to `/api/v1/...` and validate behavior with the local test script in `scripts/test_api.py`.\n","skills/llmfit-advisor/SKILL.md":"---\nname: llmfit-advisor\ndescription: Detect local hardware (RAM, CPU, GPU/VRAM) and recommend the best-fit local LLM models with optimal quantization, speed estimates, and fit scoring.\nmetadata:\n  {\n    \"openclaw\":\n      {\n        \"emoji\": \"🧠\",\n        \"requires\": { \"bins\": [\"llmfit\"] },\n        \"install\":\n          [\n            {\n              \"id\": \"brew\",\n              \"kind\": \"brew\",\n              \"formula\": \"llmfit\",\n              \"bins\": [\"llmfit\"],\n              \"label\": \"Install llmfit (brew)\",\n            },\n            {\n              \"id\": \"cargo\",\n              \"kind\": \"node\",\n              \"bins\": [\"llmfit\"],\n              \"label\": \"Install llmfit (cargo install llmfit)\",\n            },\n          ],\n      },\n  }\n---\n\n# llmfit-advisor\n\nHardware-aware local LLM advisor. Detects your system specs (RAM, CPU, GPU/VRAM) and recommends models that actually fit, with optimal quantization and speed estimates.\n\n## When to use (trigger phrases)\n\nUse this skill immediately when the user asks any of:\n\n- \"what local models can I run?\"\n- \"which LLMs fit my hardware?\"\n- \"recommend a local model\"\n- \"what's the best model for my GPU?\"\n- \"can I run Llama 70B locally?\"\n- \"configure local models\"\n- \"set up Ollama models\"\n- \"what models fit my VRAM?\"\n- \"help me pick a local model for coding\"\n\nAlso use this skill when:\n\n- The user wants to configure `models.providers.ollama` or `models.providers.lmstudio`\n- The user mentions running models locally and you need to know what fits\n- A model recommendation is needed and the user has local inference capability (Ollama, vLLM, LM Studio)\n\n## Quick start\n\n### Detect hardware\n\n```bash\nllmfit --json system\n```\n\nReturns JSON with CPU, RAM, GPU name, VRAM, multi-GPU info, and whether memory is unified (Apple Silicon).\n\n### Get top recommendations\n\n```bash\nllmfit recommend --json --limit 5\n```\n\nReturns the top 5 models ranked by a composite score (quality, speed, fit, context) with optimal quantization for the detected hardware.\n\n### Filter by use case\n\n```bash\nllmfit recommend --json --use-case coding --limit 3\nllmfit recommend --json --use-case reasoning --limit 3\nllmfit recommend --json --use-case chat --limit 3\n```\n\nValid use cases: `general`, `coding`, `reasoning`, `chat`, `multimodal`, `embedding`.\n\n### Filter by minimum fit level\n\n```bash\nllmfit recommend --json --min-fit good --limit 10\n```\n\nValid fit levels (best to worst): `perfect`, `good`, `marginal`.\n\n## Understanding the output\n\n### System JSON\n\n```json\n{\n  \"system\": {\n    \"cpu_name\": \"Apple M2 Max\",\n    \"cpu_cores\": 12,\n    \"total_ram_gb\": 32.0,\n    \"available_ram_gb\": 24.5,\n    \"has_gpu\": true,\n    \"gpu_name\": \"Apple M2 Max\",\n    \"gpu_vram_gb\": 32.0,\n    \"gpu_count\": 1,\n    \"backend\": \"Metal\",\n    \"unified_memory\": true\n  }\n}\n```\n\n### Recommendation JSON\n\nEach model in the `models` array includes:\n\n| Field | Meaning |\n|---|---|\n| `name` | HuggingFace model ID (e.g. `meta-llama/Llama-3.1-8B-Instruct`) |\n| `provider` | Model provider (Meta, Alibaba, Google, etc.) |\n| `params_b` | Parameter count in billions |\n| `score` | Composite score 0–100 (higher is better) |\n| `score_components` | Breakdown: `quality`, `speed`, `fit`, `context` (each 0–100) |\n| `fit_level` | `Perfect`, `Good`, `Marginal`, or `TooTight` |\n| `run_mode` | `GPU`, `CPU+GPU Offload`, or `CPU` |\n| `category` | Model category (e.g. `Reasoning`, `Coding`, `Chat`, `Embedding`) |\n| `is_moe` | Whether the model uses Mixture of Experts architecture |\n| `parameter_count` | Human-readable param count string (e.g. `\"7.6B\"`) |\n| `notes` | Array of human-readable notes about the recommendation |\n| `best_quant` | Optimal quantization for the hardware (e.g. `Q5_K_M`, `Q4_K_M`) |\n| `estimated_tps` | Estimated tokens per second |\n| `memory_required_gb` | VRAM/RAM needed at this quantization |\n| `memory_available_gb` | Available VRAM/RAM detected |\n| `utilization_pct` | How much of available memory the model uses |\n| `use_case` | What the model is designed for |\n| `context_length` | Maximum context window |\n\n### Fit levels explained\n\n- **Perfect**: Model fits comfortably with room to spare. Ideal choice.\n- **Good**: Model fits but uses most available memory. Will work well.\n- **Marginal**: Model barely fits. May work but expect slower performance or reduced context.\n- **TooTight**: Model does not fit. Do not recommend.\n\n### Run modes explained\n\n- **GPU**: Full GPU inference. Fastest. Model weights loaded entirely into VRAM.\n- **CPU+GPU Offload**: Some layers on GPU, rest in system RAM. Slower than pure GPU.\n- **CPU**: All inference on CPU using system RAM. Slowest but works without GPU.\n\n## Configuring OpenClaw with results\n\nAfter getting recommendations, configure the user's local model provider.\n\n### For Ollama\n\nMap the HuggingFace model name to its Ollama tag. Common mappings:\n\n| llmfit name | Ollama tag |\n|---|---|\n| `meta-llama/Llama-3.1-8B-Instruct` | `llama3.1:8b` |\n| `meta-llama/Llama-3.3-70B-Instruct` | `llama3.3:70b` |\n| `Qwen/Qwen2.5-Coder-7B-Instruct` | `qwen2.5-coder:7b` |\n| `Qwen/Qwen2.5-72B-Instruct` | `qwen2.5:72b` |\n| `deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct` | `deepseek-coder-v2:16b` |\n| `deepseek-ai/DeepSeek-R1-Distill-Qwen-32B` | `deepseek-r1:32b` |\n| `google/gemma-2-9b-it` | `gemma2:9b` |\n| `mistralai/Mistral-7B-Instruct-v0.3` | `mistral:7b` |\n| `microsoft/Phi-3-mini-4k-instruct` | `phi3:mini` |\n| `microsoft/Phi-4-mini-instruct` | `phi4-mini` |\n\nThen update `openclaw.json`:\n\n```json\n{\n  \"models\": {\n    \"providers\": {\n      \"ollama\": {\n        \"models\": [\"ollama/<ollama-tag>\"]\n      }\n    }\n  }\n}\n```\n\nAnd optionally set as default:\n\n```json\n{\n  \"agents\": {\n    \"defaults\": {\n      \"model\": {\n        \"primary\": \"ollama/<ollama-tag>\"\n      }\n    }\n  }\n}\n```\n\n### For vLLM / LM Studio\n\nUse the HuggingFace model name directly as the model identifier with the appropriate provider prefix (`vllm/` or `lmstudio/`).\n\n## Workflow example\n\nWhen a user asks \"what local models can I run?\":\n\n1. Run `llmfit --json system` to show hardware summary\n2. Run `llmfit recommend --json --limit 5` to get top picks\n3. Present the recommendations with scores and fit levels\n4. If the user wants to configure one, map it to the appropriate Ollama/vLLM/LM Studio tag\n5. Offer to update `openclaw.json` with the chosen model\n\nWhen a user asks for a specific use case like \"recommend a coding model\":\n\n1. Run `llmfit recommend --json --use-case coding --limit 3`\n2. Present the coding-specific recommendations\n3. Offer to pull via Ollama and configure\n\n## Notes\n\n- llmfit detects NVIDIA GPUs (via nvidia-smi), AMD GPUs (via rocm-smi), and Apple Silicon (unified memory).\n- Multi-GPU setups aggregate VRAM across cards automatically.\n- The `best_quant` field tells you the optimal quantization — higher quant (Q6_K, Q8_0) means better quality if VRAM allows.\n- Speed estimates (`estimated_tps`) are approximate and vary by hardware and quantization.\n- Models with `fit_level: \"TooTight\"` should never be recommended to users.\n","AGENTS.md":"# AGENTS.md\n\nInstructions for AI agents contributing to this codebase.\n\n---\n\n## Project overview\n\n`llmfit` is a Rust CLI/TUI tool that matches LLM models against local system hardware (RAM, CPU, GPU). It detects system specs, loads a model database from embedded JSON, scores each model's fit, and presents results in an interactive terminal UI or classic table output.\n\n## Language and toolchain\n\n- Rust, edition 2024.\n- Build with `cargo build`. Run with `cargo run`.\n- No nightly features required. Stable toolchain only.\n- Minimum supported Rust version: whatever edition 2024 requires (1.85+).\n\n## Architecture\n\n```\nllmfit-core/      Shared Rust library. It owns hardware detection, model data,\n                 fit analysis, planning, providers, benchmarks, quality checks,\n                 model updates, diagnostics, claims, and result sharing.\n\nllmfit-tui/       Main `llmfit` binary. It provides the CLI, ratatui TUI,\n                 Axum HTTP API, embedded Web dashboard, and stdio MCP server.\n                 `main.rs` parses all clap flags and selects an interface.\n\nllmfit-desktop/   Tauri desktop application. Tauri commands call llmfit-core\n                 for hardware detection, fit analysis, and Ollama downloads.\n\nllmfit-web/       React 18 and Vite dashboard. It calls `/api/v1/*` endpoints\n                 from llmfit-tui. The llmfit-tui build script embeds `dist/`.\n                 This directory is not a Cargo workspace member.\n\nllmfit-python/    Python package wrapper. Its wheel includes the compiled Rust\n                 binary. `python -m llmfit` forwards arguments to that binary.\n                 It does not expose llmfit-core through a native Python API.\n```\n\nThe Cargo workspace contains `llmfit-core`, `llmfit-tui`, and\n`llmfit-desktop`. The default members are `llmfit-core` and `llmfit-tui`.\n\nSource modules in `llmfit-core/src/`:\n\n- `analysis.rs`: Builds model-fit results. It marks installed models and applies\n  local, community, and measured benchmark calibration.\n- `bench.rs`: Runs throughput benchmarks against Ollama and OpenAI-compatible\n  endpoints. It also discovers available benchmark targets.\n- `benchmarks.rs`: Loads embedded and remote benchmark data. It builds measured\n  throughput indexes and hardware leaderboard queries.\n- `claim.rs`: Calculates model resource bounds. It renders Kubernetes DRA\n  `ResourceClaim` and `ResourceClaimTemplate` manifests.\n- `doctor.rs`: Collects installation, hardware, runtime, and model diagnostics.\n- `fit.rs`: Calculates fit level, run mode, runtime, quantization, score, and\n  estimated throughput.\n- `hardware.rs`: Detects RAM, CPU, GPUs, unified memory, clusters, and memory\n  bandwidth.\n- `hwprofile.rs`: Loads embedded and user hardware profiles (schema v1). Applies\n  capacity to `SystemSpecs` and bandwidth/efficiency overrides to `CalcConfig`.\n- `models.rs`: Defines model metadata. It loads embedded HF and ONNX catalogs,\n  custom models, and the update cache.\n- `plan.rs`: Estimates memory, throughput, run paths, and hardware upgrade needs\n  for a requested model setup.\n- `providers.rs`: Integrates Ollama, MLX, llama.cpp, Docker Model Runner,\n  LM Studio, vLLM, and RamaLama.\n- `quality.rs`: Runs response quality tests. It scores roles, builds routing\n  recommendations, and compares results with baselines.\n- `share.rs`: Stores local benchmark results. It handles GitHub authentication\n  and submits community benchmark data.\n- `task_bench.rs`: Provides task benchmark scores for model and task pairs.\n- `update.rs`: Fetches model metadata and manages the local model update cache.\n\nSource modules in `llmfit-tui/src/`:\n\n- `main.rs`: Owns CLI parsing, hardware overrides, command execution, and\n  interface dispatch.\n- `display.rs`: Renders classic CLI tables, model plans, JSON, and CSV output.\n- `download_history.rs`: Stores persistent model download records.\n- `events.rs`: Publishes optional NATS system events and periodic snapshots.\n- `filter_config.rs`: Loads and saves persistent TUI filter settings.\n- `mcp_server.rs`: Exposes hardware, model, runtime, and planning MCP tools.\n- `serve_api.rs`: Serves the embedded Web dashboard and JSON API with Axum.\n- `serve_shared.rs`: Converts shared core types into API and MCP JSON values.\n- `theme.rs`: Defines TUI color themes and stores the selected theme.\n- `tui_app.rs`: Owns TUI state, model results, filters, downloads, and selection.\n- `tui_events.rs`: Handles crossterm input and mutates TUI state.\n- `tui_ui.rs`: Renders TUI views, tables, details, plans, and popups with\n  ratatui.\n\n## Data flow\n\nAll interfaces use the same core analysis flow:\n\n1. `SystemSpecs::detect()` detects CPU, RAM, GPU, unified-memory, and cluster\n   information. CLI hardware overrides can replace detected values.\n2. `ModelDatabase::new()` loads the embedded HF and ONNX catalogs.\n3. Custom models replace matching embedded models. The update cache appends\n   models that are not already present.\n4. `build_model_fits()` removes backend-incompatible models. It calls\n   `ModelFit::analyze_with_forced_runtime()` for each remaining model.\n5. Fit analysis selects a runtime, quantization, and run mode. It calculates\n   memory use, throughput, fit level, score components, and notes.\n6. Local benchmark results, community results, and measured presets can replace\n   or calibrate formula-based throughput estimates.\n7. Each interface applies its own filters, sorting, limits, and presentation.\n\n`ModelFit::analyze()` is the default analysis wrapper. Use\n`analyze_with_context_limit()` for a context cap. Use\n`analyze_with_forced_runtime()` for runtime selection. Use\n`analyze_with_config()` for custom calculation parameters. These methods share\nthe private `analyze_inner()` implementation.\n\nInterface-specific flow:\n\n- CLI: `main.rs` dispatches a subcommand. The command calls llmfit-core and\n  writes a table, JSON, or CSV result.\n- TUI: `App` owns model and filter state. `tui_events` changes that state.\n  `apply_filters()` updates visible indices. `tui_ui` renders the current state.\n- Web: React calls `/api/v1/*`. Axum handlers in `serve_api.rs` call llmfit-core\n  and return JSON. The same server returns the embedded React assets.\n- MCP: `LlmfitMcpServer` receives stdio tool calls. Each tool calls shared core\n  analysis or planning logic and returns JSON text.\n- Desktop: Tauri commands call llmfit-core and serialize results for the desktop\n  UI. Ollama pull state stays in the Tauri application state.\n- Python: The Python entry point locates the installed `llmfit` binary. It then\n  replaces the process on Unix or starts a subprocess on Windows.\n\n## Model database\n\n- Source: `llmfit-core/data/hf_models.json` (33 models).\n- Generated by `scripts/scrape_hf_models.py` (Python, stdlib only, no pip deps).\n- Embedded at compile time via `include_str!(\"../data/hf_models.json\")`.\n- Schema per entry: name, provider, parameter_count, min_ram_gb, recommended_ram_gb, min_vram_gb, quantization, context_length, use_case.\n- `min_vram_gb` is VRAM needed for GPU inference. `min_ram_gb` is system RAM needed for CPU inference. Both are derived from the same parameter count.\n- RAM formula: `params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.2 overhead`.\n- VRAM formula: `params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.1 activation overhead`.\n- Recommended RAM: `model_size * 2.0`.\n\nDo not manually edit `hf_models.json`. Regenerate it by running the scraper:\n\n```sh\npython3 scripts/scrape_hf_models.py\n```\n\nThe scraper has hardcoded fallback entries for gated models that require authentication.\n\n## Conventions\n\n- No `unsafe` code.\n- No `.unwrap()` on user-facing paths. Use proper error handling or `expect()` with a descriptive message for internal invariants only.\n- Fit levels are ordered: Perfect > Good > Marginal > TooTight. Do not add levels without updating `rank_models_by_fit()` sort logic.\n- Fit is VRAM-first. `RunMode` has five execution paths: `Gpu`, `MoeOffload`,\n  `CpuOffload`, `CpuOnly`, and `TensorParallel`.\n- `Gpu` keeps the model in VRAM. `MoeOffload` keeps active experts in VRAM and\n  inactive experts in RAM. `CpuOffload` splits work between VRAM and RAM.\n  `CpuOnly` uses system RAM. `TensorParallel` distributes work across nodes.\n- `min_vram_gb` is the VRAM needed to load model weights on GPU. `min_ram_gb` is the system RAM needed for CPU-only inference (same weights, loaded into RAM instead). They represent the same workload on different hardware paths.\n- On Apple Silicon (unified memory), VRAM = system RAM. The `CpuOffload` path is skipped because there is no separate RAM pool to spill to. `SystemSpecs::unified_memory` tracks this.\n- TUI rendering is stateless. `tui_ui::draw()` must not mutate `App`. Pass `&mut App` only for `TableState` widget requirements -- do not use it to change application state.\n- Event handling in `tui_events.rs` is the sole place that mutates `App` in the TUI loop.\n- Keep `display.rs` and `tui_*.rs` independent. The CLI path must work without initializing any TUI state.\n\n## Adding a new model to the database\n\n1. Add the model's HuggingFace repo ID to `TARGET_MODELS` in `scripts/scrape_hf_models.py`.\n2. If the model is gated (requires HF auth), add a fallback entry to the `FALLBACK` dict in the same script.\n3. Run `python3 scripts/scrape_hf_models.py`.\n4. Verify the output in `llmfit-core/data/hf_models.json`.\n5. Run `cargo build` to verify compilation.\n\n## Adding a new filter\n\n1. Add the filter state to `App` in `tui_app.rs`.\n2. Add filtering logic inside `apply_filters()`.\n3. Add the keybinding in `tui_events.rs` (Normal mode handler).\n4. Add the UI widget in `tui_ui.rs` (`draw_search_and_filters()` function).\n5. Update the status bar help text in `draw_status_bar()`.\n\n## Adding a new CLI subcommand\n\n1. Add a variant to the `Commands` enum in `main.rs`.\n2. Add the match arm in the `main()` function's command dispatch.\n3. Use `display.rs` functions for output, or add new ones as needed.\n\n## Testing\n\nThe project has Rust, Web, and Python test suites.\n\n- Rust unit tests live beside code in `llmfit-core/src/` and\n  `llmfit-tui/src/`.\n- Core integration tests in `llmfit-core/tests/` validate catalog schemas and\n  ONNX model data.\n- CLI integration tests in `llmfit-tui/tests/` use `assert_cmd` against the\n  compiled `llmfit` binary.\n- HTTP API tests exercise Axum routers and JSON responses in `serve_api.rs`.\n- TUI tests focus on state transitions, filters, event handling, and render\n  output. Keep production rendering stateless.\n- Web tests use Vitest, jsdom, and Testing Library. They cover API query\n  construction, localization, filtering, and dashboard interactions.\n- Python tests use pytest. They cover binary discovery, package versioning, and\n  invocation of the packaged Rust binary.\n\nRun the default Rust test set:\n\n```sh\ncargo test\n```\n\nRun all Rust workspace members, including the desktop crate:\n\n```sh\ncargo test --workspace\n```\n\nRun one Rust package:\n\n```sh\ncargo test -p llmfit-core\ncargo test -p llmfit\n```\n\nRun the Web tests:\n\n```sh\nnpm --prefix llmfit-web test\n```\n\nRun the Python tests and quality checks:\n\n```sh\nuv run --project llmfit-python pytest llmfit-python/tests\nmake -C llmfit-python check\n```\n\n## Dependencies policy\n\n- Prefer crates that are well-maintained and have minimal transitive dependencies.\n- `sysinfo` is the system detection crate. Do not replace it with raw platform calls.\n- `ureq` is the blocking HTTP client for providers, benchmarks, updates, quality\n  tests, and sharing. Do not add a second core HTTP client without a concrete need.\n- `which` locates installed runtime binaries. Keep runtime discovery in\n  `providers.rs` instead of adding manual `PATH` parsing.\n- `regex` supports response scoring and text parsing. `serde_yml` parses quality\n  test configuration. `base64` encodes benchmark submissions for GitHub.\n- `objc2-metal` reads the effective Metal working-set limit on macOS. Keep it a\n  macOS-only dependency. Do not replace it with raw platform calls.\n- `ratatui` + `crossterm` is the TUI stack. Do not mix in `termion` or `ncurses`.\n- `clap` with derive feature for CLI parsing. Do not use manual arg parsing.\n- The Python scraper uses only stdlib (`urllib`, `json`). Do not add pip dependencies.\n\n## Common tasks\n\n```sh\n# Build\ncargo build\n\n# Run TUI\ncargo run\n\n# Run CLI mode\ncargo run -- --cli\n\n# Run specific subcommand\ncargo run -- system\ncargo run -- fit --perfect -n 5\ncargo run -- search \"llama\"\n\n# Refresh model database\npython3 scripts/scrape_hf_models.py && cargo build\n\n# Check for compilation issues\ncargo check\n\n# Format code\ncargo fmt\n\n# Lint\ncargo clippy\n```\n\n## Platform notes\n\n- GPU detection shells out to `nvidia-smi` (NVIDIA) and `rocm-smi` (AMD). These are best-effort and fail silently if unavailable.\n- Apple Silicon detection uses `system_profiler SPDisplaysDataType`. On unified memory Macs, VRAM is reported as available system RAM (same pool).\n- `sysinfo` handles cross-platform RAM/CPU. No conditional compilation needed.\n- The TUI uses crossterm which works on Linux, macOS, and Windows terminals.\n"},"files":{"API.md":"# llmfit REST API Guide\n\nThis document is for agent/client builders integrating with `llmfit serve`.\n\n## Purpose\n\n`llmfit serve` exposes node-local model fit analysis (same core data used by TUI/CLI) over HTTP and serves a local web dashboard.\n\nPrimary use case:\n- Query each node in a cluster for top runnable models.\n- Aggregate externally (scheduler/controller/UI) for placement decisions.\n\n## Start the server\n\n```sh\nllmfit serve --port 8787\n```\n\nGlobal flags still apply:\n\n```sh\nllmfit --memory 24G --ram 64G --cpu-cores 16 --max-context 8192 serve --port 8787\n```\n\nHardware overrides (`--memory`, `--ram`, `--cpu-cores`) are reflected in API responses, making the server report the overridden values instead of the detected hardware.\n\n## Base URL\n\nDefault local base URL:\n\n```text\nhttp://127.0.0.1:8787\n```\n\nTo expose outside localhost, pass `--host 0.0.0.0`.\n\n### Unix domain socket\n\nFor same-host consumers that should not touch the network at all (e.g. a\nsidecar in a `hostNetwork` Kubernetes pod, where a TCP bind would land on the\nnode's loopback), listen on a Unix socket instead:\n\n```sh\nllmfit serve --unix-socket /run/llmfit/llmfit.sock\n```\n\nThe socket is created with mode `0660`; a stale socket file from a previous\ninstance is replaced automatically. All HTTP endpoints are identical:\n\n```sh\ncurl --unix-socket /run/llmfit/llmfit.sock http://localhost/api/v1/system\n```\n\n`--unix-socket` conflicts with `--host`/`--port` and is unix-platforms only.\n\nIf you are building from source and want the dashboard embedded in `llmfit`, build web assets first:\n\n```sh\ncd llmfit-web && npm ci && npm run build\n```\n\n## Endpoints\n\n### `GET /`\nWeb dashboard entrypoint (same-origin UI for fit exploration).\n\n### `GET /health`\nLiveness probe.\n\nExample response:\n\n```json\n{\n  \"status\": \"ok\",\n  \"node\": {\n    \"name\": \"worker-1\",\n    \"os\": \"linux\"\n  }\n}\n```\n\n---\n\n### `GET /api/v1/system`\nReturns node identity + detected hardware.\n\nExample response shape:\n\n```json\n{\n  \"node\": {\n    \"name\": \"worker-1\",\n    \"os\": \"linux\"\n  },\n  \"system\": {\n    \"total_ram_gb\": 62.23,\n    \"available_ram_gb\": 41.08,\n    \"cpu_cores\": 14,\n    \"cpu_name\": \"Intel(R) Core(TM) Ultra 7 165U\",\n    \"has_gpu\": false,\n    \"gpu_vram_gb\": null,\n    \"gpu_available_gb\": null,\n    \"gpu_name\": null,\n    \"gpu_count\": 0,\n    \"unified_memory\": false,\n    \"backend\": \"CPU (x86)\",\n    \"gpus\": []\n  },\n  \"profile\": null\n}\n```\n\n`profile` is the name of the hardware profile the server was started with (`llmfit serve --profile <name>`), or `null` when it describes the machine it runs on.\n\n`gpu_available_gb` is the VRAM free right now, pooled across discrete GPUs (from `nvidia-smi memory.free` and amdgpu's `mem_info_vram_used`); each `gpus[]` entry carries its own `free_vram_gb`. Both are `null` when a backend does not report it (Intel, Windows, older drivers) or after a hardware override. On Apple Silicon `gpu_available_gb` is instead Metal's wiring cap for the unified pool. `plan` grades GPU run paths against the free figure when it is known and against total VRAM otherwise.\n\n---\n\n### `GET /api/v1/models`\nReturns filtered/sorted model-fit rows for this node.\n\nEnvelope shape:\n\n```json\n{\n  \"node\": { \"name\": \"worker-1\", \"os\": \"linux\" },\n  \"system\": { \"...\": \"...\" },\n  \"total_models\": 23,\n  \"returned_models\": 10,\n  \"filters\": { \"...\": \"echo of query state\" },\n  \"models\": [\n    {\n      \"name\": \"Qwen/Qwen2.5-Coder-7B-Instruct\",\n      \"provider\": \"Qwen\",\n      \"parameter_count\": \"7B\",\n      \"params_b\": 7.0,\n      \"context_length\": 32768,\n      \"usable_context\": 32768,\n      \"effective_context_length\": 8192,\n      \"use_case\": \"Coding\",\n      \"category\": \"Coding\",\n      \"release_date\": \"2025-03-14\",\n      \"is_moe\": false,\n      \"fit_level\": \"good\",\n      \"fit_label\": \"Good\",\n      \"run_mode\": \"gpu\",\n      \"run_mode_label\": \"GPU\",\n      \"score\": 86.5,\n      \"score_components\": {\n        \"quality\": 87.0,\n        \"speed\": 81.2,\n        \"fit\": 90.1,\n        \"context\": 88.0\n      },\n      \"estimated_tps\": 42.5,\n      \"estimate_confidence\": \"estimated\",\n      \"estimate_confidence_label\": \"estimated\",\n      \"prefill_tps\": 812.3,\n      \"ttft_ms\": 10.1,\n      \"runtime\": \"llamacpp\",\n      \"runtime_label\": \"llama.cpp\",\n      \"best_quant\": \"Q5_K_M\",\n      \"memory_required_gb\": 5.8,\n      \"memory_available_gb\": 12.0,\n      \"utilization_pct\": 48.3,\n      \"notes\": [],\n      \"gguf_sources\": [],\n      \"capabilities\": [\"tool_use\"],\n      \"capability_ids\": [\"tool_use\"],\n      \"license\": \"apache-2.0\",\n      \"supports_tp\": [1, 2, 4],\n      \"installed\": false,\n      \"disk_size_gb\": 5.1,\n      \"ollama_name\": \"qwen2.5-coder:7b-instruct\",\n      \"estimate_basis\": {\n        \"method\": \"roofline\",\n        \"gpu_bandwidth_gbps\": 320.0,\n        \"ddr_bandwidth_gbps\": null,\n        \"local_calibration\": null,\n        \"efficiency\": 0.85,\n        \"assumed_context\": 8192\n      },\n      \"verify_command\": \"llama-bench -m <path-to-Q5_K_M-gguf> -ngl 99 -p 512 -n 128\",\n      \"measured_tps\": null\n    }\n  ]\n}\n```\n\nThe three context fields answer different questions:\n\n- `context_length` — the model's native window, as advertised upstream.\n- `usable_context` — how much of that window actually fits in this node's\n  available memory alongside the weights. Use this one to pick a runtime `-c`.\n- `effective_context_length` — the context the `memory_required_gb` and\n  `estimated_tps` figures on this row were computed at. Defaults to\n  `min(context_length, 8192)`; set by `max_context` when supplied.\n\nThe envelope also carries these fields, now at parity with `llmfit fit --json`\n(both frontends serialize through one shared function):\n\n- `installed` — whether the model was found in a local runtime provider. `llmfit serve` probes providers in the background (at startup, after a finished download, and when a request finds the result older than 30 seconds), so a model pulled or removed outside the server shows up within one refresh after that; it is `false` everywhere until the first probe finishes.\n- `disk_size_gb` — estimated on-disk size at `best_quant`.\n- `capability_ids` — machine-readable capability ids (snake_case); mirrors\n  `capabilities` here. Note `llmfit fit --json` overloads its `capabilities`\n  key with human labels (e.g. `\"Tool Use\"`) — that overload is CLI-only and\n  slated for deprecation.\n- `ollama_name` — the `ollama pull` tag for this model, when derivable.\n- `estimate_basis` — how `memory_required_gb`/`estimated_tps` were derived\n  (bandwidths, efficiency, assumed context), for reproducibility.\n- `estimate_confidence` / `estimate_confidence_label` — how much to trust\n  `estimated_tps`, most to least trustworthy: `measured_local` (your own\n  `llmfit bench` runs), `measured_community` (llmfit community submissions\n  or localmaxxing.com data on matching hardware), `calibrated` (a formula\n  estimate scaled by a correction factor from benchmark runs on this exact\n  hardware), `estimated` (a bare formula estimate), `unsupported` (no\n  estimate — the model needs a runtime llmfit can't model). See\n  `measured_tps`/`estimate_basis.local_calibration` for the data each level\n  is derived from.\n- `prefill_tps` / `ttft_ms` — estimated prompt-processing throughput\n  (tok/s) and time-to-first-token (ms) for a prompt of\n  `effective_context_length` tokens. Prefill is compute-bound, not\n  bandwidth-bound, so both are `null` — not `0.0` — unless the system's GPU\n  compute throughput is known. A `null` here means \"not estimated\", never\n  \"instant\" or \"stalled\".\n- `verify_command` — a `llama-bench` invocation measuring the same throughput\n  this row estimates (llama.cpp GPU / CPU-only runs; `null` otherwise).\n- `measured_tps` — a recorded benchmark result if one exists, else `null`.\n\nNote on vocabulary: `fit_level`, `run_mode`, and `runtime` here are stable\nmachine codes (e.g. `\"good\"`, `\"gpu\"`, `\"llamacpp\"`), with the human string\nunder the paired `*_label` key. `llmfit fit --json` emits the human string\ndirectly under those same keys — a CLI-only legacy overload.\n`estimate_confidence` follows the same convention (machine code, paired\n`estimate_confidence_label`) in both frontends.\n\n### A note on `best_quant`\n\n`best_quant` is normally the llama.cpp/GGUF quant llmfit picked for this\nhardware (e.g. `\"Q5_K_M\"`). It is `null` when the model's own repo name\ndeclares a native low-precision format (NVFP4/MXFP4) that a GGUF quant\nlabel doesn't apply to — the row's `notes` array carries the explanation\nin that case. It's never a GGUF label misattributed to a non-GGUF repo.\n\n---\n\n### Catalog sanitization\n\n`/api/v1/models` never returns speculative-decoding draft heads\n(EAGLE/DFlash/DSpark naming), entries whose name implies a wildly different\nparameter count than their declared size, or other catalog rows llmfit\ncan't score honestly — they're demoted out of fit ranking, not deleted, so\nthis is invisible unless you were expecting a specific draft-head repo to\nshow up as a standalone model. If you maintain your own filtering on top of\n`llmfit fit --json` output today, you can likely delete that workaround.\n\n---\n\n### `GET /api/v1/models/top`\nKey scheduling endpoint. Same schema as `/api/v1/models`, but defaults to top 5 runnable entries.\n\nImportant behavior:\n- Defaults `limit=5`.\n- Excludes `too_tight` rows unless explicitly overridden (and top endpoint still keeps runnable semantics).\n\n---\n\n### `GET /api/v1/models/{name}`\nPath-constrained search. Equivalent to a text search scoped by `{name}`.\n\nUseful for:\n- Client-side drilldown after selecting a model family.\n\n---\n\n### `GET /api/v1/concurrency`\nConcurrent-session capacity for one model: the same estimate as\n`llmfit concurrency <model> --json`. It is a memory-capacity ceiling (how many\nsessions fit resident at once), not a throughput figure under load.\n\nQuery parameters:\n- `model` (required): exact catalog name, case-insensitive.\n- `quant`: weight quantization (e.g. `Q4_K_M`); defaults to the model's best fit.\n- `kv_quant`: `fp16|fp8|q8_0|q4_0|tq` (default `fp16`).\n- `context`: report one context instead of the 4k-256k ladder (tokens).\n- `users`: also report the largest context that fits this many sessions.\n- `ram_gb`, `vram_gb`, `cpu_cores`: hardware overrides, as on `/api/v1/system`.\n\n```json\n{\n  \"model\": \"Qwen/Qwen2.5-7B-Instruct\",\n  \"run_mode\": \"gpu\",\n  \"fit_level\": \"perfect\",\n  \"target_users\": 8,\n  \"max_context_for_target\": 32768,\n  \"estimate\": {\n    \"pool_gb\": 24.0,\n    \"weights_resident_gb\": 5.1,\n    \"kv_budget_gb\": 18.9,\n    \"quant\": \"Q4_K_M\",\n    \"kv_quant\": \"fp16\",\n    \"native_context\": 32768,\n    \"ladder\": [\n      { \"requested_context\": 4096, \"effective_context\": 4096, \"clamped\": false, \"per_session_kv_gb\": 0.22, \"max_sessions\": 85 }\n    ]\n  }\n}\n```\n\n`run_mode` and `fit_level` use the same codes as `/api/v1/models`.\n\n---\n\n### `GET /api/v1/storage`\nDisk needed to keep a library of runnable models, used one at a time: the same\nestimate as `llmfit storage --json`, returned as `{ \"system\": ..., \"storage\": ... }`.\n\nQuery parameters (defaults match the CLI):\n- `keep`: number of distinct runnable models to keep (default `3`).\n- `selection`: `score` (highest-ranked, default) or `largest` (largest weight storage).\n- `os_reserve`: space for the OS, apps and other files (default `100G`; decimal GB, `GiB`/`TiB` are binary).\n- `scratch`: `auto` (default, the largest selected model) or a size; `0` disables it.\n- `headroom`: percent of the suggested SSD to keep free, `0`-`99` (default `15`).\n- `perfect`: `true` to select only Perfect fits (otherwise Perfect, Good and Marginal).\n- `search`: narrow the candidates by name, provider or parameter size.\n- `max_context`, `ram_gb`, `vram_gb`, `cpu_cores`: as on `/api/v1/models`.\n\nThe `storage` object carries `models` (name, best quant, fit level, runtime,\nscore, disk size, context), `library_gb`, `download_scratch_gb`, `need_gb`,\n`target_capacity_gb`, `minimum_ssd_gb`, `suggested_ssd_gb` and `warnings`.\n\n## Query parameters\n\nSupported on `/api/v1/models` and `/api/v1/models/top` (also `/api/v1/models/{name}`):\n\n- `limit` (or alias `n`): max rows returned.\n- `perfect`: `true|false` (when `true`, only perfect fits).\n- `min_fit`: `perfect|good|marginal|too_tight`.\n- `runtime`: `any|mlx|llamacpp|vllm|bitnetcpp`.\n- `use_case`: `general|coding|reasoning|chat|multimodal|embedding`.\n- `provider`: provider substring filter.\n- `search`: free-text filter (name/provider/params/use-case/category).\n- `sort`: `score|tps|params|mem|ctx|date|use_case`.\n- `include_too_tight`: `true` includes unrunnable (`too_tight`) rows even when `min_fit` is not given; `false` drops them even with `min_fit=too_tight`. With neither parameter, `/models` returns rows from `marginal` up; `/models/top` never returns `too_tight` rows.\n- `max_context`: per-request context cap used by memory estimation.\n- `force_runtime`: `mlx|llamacpp|vllm|bitnetcpp` — override automatic runtime selection during analysis (e.g. get llama.cpp recommendations on Apple Silicon instead of MLX).\n\n## Error handling\n\nInvalid filter values return HTTP 400:\n\n```json\n{\n  \"error\": \"invalid min_fit value: use perfect|good|marginal|too_tight\"\n}\n```\n\nServer errors return HTTP 500 with `{\"error\": \"...\"}`.\n\n## Client integration recommendations\n\n### 1) Polling pattern for schedulers\nFor each node agent:\n1. Call `/health`.\n2. Call `/api/v1/system`.\n3. Call `/api/v1/models/top?limit=K&min_fit=good`.\n4. Attach node metadata and forward to your central scheduler.\n\n### 2) Conservative placement defaults\nFor production placement, prefer:\n\n```text\nmin_fit=good\ninclude_too_tight=false\nsort=score\nlimit=5..20\n```\n\n### 3) Per-workload targeting\nExamples:\n- Coding workloads: `use_case=coding`\n- Embedding workloads: `use_case=embedding`\n- Runtime constrained to llama.cpp fleet: `runtime=llamacpp`\n\n### 4) Stable parsing\nTreat unknown fields as forward-compatible additions:\n- Parse required fields you depend on.\n- Ignore unknown fields.\n\n## Curl examples\n\n```sh\ncurl http://127.0.0.1:8787/health\ncurl http://127.0.0.1:8787/api/v1/system\ncurl \"http://127.0.0.1:8787/api/v1/models?limit=20&min_fit=marginal&sort=score\"\ncurl \"http://127.0.0.1:8787/api/v1/models/top?limit=5&min_fit=good&use_case=coding\"\ncurl \"http://127.0.0.1:8787/api/v1/models/Mistral?runtime=any\"\ncurl \"http://127.0.0.1:8787/api/v1/concurrency?model=Qwen/Qwen2.5-7B-Instruct&users=8\"\ncurl \"http://127.0.0.1:8787/api/v1/storage?keep=5&selection=largest&os_reserve=200G\"\n```\n\n---\n\n## MCP Server Mode\n\nllmfit can run as an MCP (Model Context Protocol) server over stdio, making it discoverable by AI agents (Claude, Cursor, etc.).\n\n### Start the MCP server\n\n```sh\nllmfit serve --mcp\n```\n\nGlobal hardware overrides still apply:\n\n```sh\nllmfit --memory 24G --ram 64G serve --mcp\n```\n\n### MCP client configuration\n\nAdd to your MCP client config (e.g. `claude_desktop_config.json`):\n\n```json\n{\n  \"mcpServers\": {\n    \"llmfit\": {\n      \"command\": \"llmfit\",\n      \"args\": [\"serve\", \"--mcp\"]\n    }\n  }\n}\n```\n\n### Available tools\n\n| Tool | Description | Parameters |\n|------|-------------|------------|\n| `get_system_specs` | Node hardware info (RAM, GPU, CPU) | None |\n| `recommend_models` | Top models for this hardware | `limit?`, `use_case?`, `min_fit?`, `runtime?`, `license?`, `sort?` |\n| `search_models` | Free-text model search | `query`, `limit?` |\n| `plan_hardware` | Hardware requirements for a model | `model`, `context?`, `quant?`, `target_tps?` |\n| `get_runtimes` | Installed inference runtimes | None |\n| `get_installed_models` | Models in local runtimes | None |\n\n`plan_hardware` and `POST /api/v1/plan` return the shared plan estimate,\nincluding `disk_size_gb`: estimated weight storage in decimal GB at the\nresolved `quantization`. This excludes KV cache, inference buffers, and\ndownload scratch. `llmfit plan --json` returns the same plan fields.\n\n---\n\n## NATS Event Publishing\n\nWhen built with the `nats` feature, llmfit can publish hardware and model events to NATS for integration with coordination systems (e.g. Sympozium membrane).\n\n### Build with NATS support\n\n```sh\ncargo build --features nats\n```\n\n### Enable event publishing\n\n```sh\nllmfit serve --send-events --nats-url nats://localhost:4222\nllmfit serve --mcp --send-events  # also works with MCP mode\n```\n\nThe `NATS_URL` environment variable is also supported.\n\n### Event subjects\n\nEvents are published to `llmfit.{event_type}.{hostname}`:\n\n| Subject | Trigger | Payload |\n|---------|---------|---------|\n| `llmfit.system.{hostname}` | Startup + every 60s | System hardware specs |\n| `llmfit.fit.{hostname}` | After fit analysis | Model fit summary |\n| `llmfit.plan.{hostname}` | After plan estimate | Plan estimate |\n| `llmfit.runtimes.{hostname}` | Startup + on query | Runtime availability |\n| `llmfit.installed.{hostname}` | Startup + on query | Installed models |\n\n### Event envelope\n\nAll events are wrapped in a common envelope:\n\n```json\n{\n  \"timestamp\": \"1747058400\",\n  \"hostname\": \"worker-1\",\n  \"event_type\": \"system\",\n  \"version\": \"1\",\n  \"data\": { ... }\n}\n```\n\n### Subscribe to events\n\n```sh\nnats sub 'llmfit.>'                    # all events from all nodes\nnats sub 'llmfit.system.>'             # system specs from all nodes\nnats sub 'llmfit.system.worker-1'      # system specs from specific node\n```\n\n---\n\n## Versioning notes\n\nCurrent API prefix is `v1`.\n\nIf you build long-lived clients, pin to `/api/v1/...` and validate behavior with the local test script in `scripts/test_api.py`.\n","skills/llmfit-advisor/SKILL.md":"---\nname: llmfit-advisor\ndescription: Detect local hardware (RAM, CPU, GPU/VRAM) and recommend the best-fit local LLM models with optimal quantization, speed estimates, and fit scoring.\nmetadata:\n  {\n    \"openclaw\":\n      {\n        \"emoji\": \"🧠\",\n        \"requires\": { \"bins\": [\"llmfit\"] },\n        \"install\":\n          [\n            {\n              \"id\": \"brew\",\n              \"kind\": \"brew\",\n              \"formula\": \"llmfit\",\n              \"bins\": [\"llmfit\"],\n              \"label\": \"Install llmfit (brew)\",\n            },\n            {\n              \"id\": \"cargo\",\n              \"kind\": \"node\",\n              \"bins\": [\"llmfit\"],\n              \"label\": \"Install llmfit (cargo install llmfit)\",\n            },\n          ],\n      },\n  }\n---\n\n# llmfit-advisor\n\nHardware-aware local LLM advisor. Detects your system specs (RAM, CPU, GPU/VRAM) and recommends models that actually fit, with optimal quantization and speed estimates.\n\n## When to use (trigger phrases)\n\nUse this skill immediately when the user asks any of:\n\n- \"what local models can I run?\"\n- \"which LLMs fit my hardware?\"\n- \"recommend a local model\"\n- \"what's the best model for my GPU?\"\n- \"can I run Llama 70B locally?\"\n- \"configure local models\"\n- \"set up Ollama models\"\n- \"what models fit my VRAM?\"\n- \"help me pick a local model for coding\"\n\nAlso use this skill when:\n\n- The user wants to configure `models.providers.ollama` or `models.providers.lmstudio`\n- The user mentions running models locally and you need to know what fits\n- A model recommendation is needed and the user has local inference capability (Ollama, vLLM, LM Studio)\n\n## Quick start\n\n### Detect hardware\n\n```bash\nllmfit --json system\n```\n\nReturns JSON with CPU, RAM, GPU name, VRAM, multi-GPU info, and whether memory is unified (Apple Silicon).\n\n### Get top recommendations\n\n```bash\nllmfit recommend --json --limit 5\n```\n\nReturns the top 5 models ranked by a composite score (quality, speed, fit, context) with optimal quantization for the detected hardware.\n\n### Filter by use case\n\n```bash\nllmfit recommend --json --use-case coding --limit 3\nllmfit recommend --json --use-case reasoning --limit 3\nllmfit recommend --json --use-case chat --limit 3\n```\n\nValid use cases: `general`, `coding`, `reasoning`, `chat`, `multimodal`, `embedding`.\n\n### Filter by minimum fit level\n\n```bash\nllmfit recommend --json --min-fit good --limit 10\n```\n\nValid fit levels (best to worst): `perfect`, `good`, `marginal`.\n\n## Understanding the output\n\n### System JSON\n\n```json\n{\n  \"system\": {\n    \"cpu_name\": \"Apple M2 Max\",\n    \"cpu_cores\": 12,\n    \"total_ram_gb\": 32.0,\n    \"available_ram_gb\": 24.5,\n    \"has_gpu\": true,\n    \"gpu_name\": \"Apple M2 Max\",\n    \"gpu_vram_gb\": 32.0,\n    \"gpu_count\": 1,\n    \"backend\": \"Metal\",\n    \"unified_memory\": true\n  }\n}\n```\n\n### Recommendation JSON\n\nEach model in the `models` array includes:\n\n| Field | Meaning |\n|---|---|\n| `name` | HuggingFace model ID (e.g. `meta-llama/Llama-3.1-8B-Instruct`) |\n| `provider` | Model provider (Meta, Alibaba, Google, etc.) |\n| `params_b` | Parameter count in billions |\n| `score` | Composite score 0–100 (higher is better) |\n| `score_components` | Breakdown: `quality`, `speed`, `fit`, `context` (each 0–100) |\n| `fit_level` | `Perfect`, `Good`, `Marginal`, or `TooTight` |\n| `run_mode` | `GPU`, `CPU+GPU Offload`, or `CPU` |\n| `category` | Model category (e.g. `Reasoning`, `Coding`, `Chat`, `Embedding`) |\n| `is_moe` | Whether the model uses Mixture of Experts architecture |\n| `parameter_count` | Human-readable param count string (e.g. `\"7.6B\"`) |\n| `notes` | Array of human-readable notes about the recommendation |\n| `best_quant` | Optimal quantization for the hardware (e.g. `Q5_K_M`, `Q4_K_M`) |\n| `estimated_tps` | Estimated tokens per second |\n| `memory_required_gb` | VRAM/RAM needed at this quantization |\n| `memory_available_gb` | Available VRAM/RAM detected |\n| `utilization_pct` | How much of available memory the model uses |\n| `use_case` | What the model is designed for |\n| `context_length` | Maximum context window |\n\n### Fit levels explained\n\n- **Perfect**: Model fits comfortably with room to spare. Ideal choice.\n- **Good**: Model fits but uses most available memory. Will work well.\n- **Marginal**: Model barely fits. May work but expect slower performance or reduced context.\n- **TooTight**: Model does not fit. Do not recommend.\n\n### Run modes explained\n\n- **GPU**: Full GPU inference. Fastest. Model weights loaded entirely into VRAM.\n- **CPU+GPU Offload**: Some layers on GPU, rest in system RAM. Slower than pure GPU.\n- **CPU**: All inference on CPU using system RAM. Slowest but works without GPU.\n\n## Configuring OpenClaw with results\n\nAfter getting recommendations, configure the user's local model provider.\n\n### For Ollama\n\nMap the HuggingFace model name to its Ollama tag. Common mappings:\n\n| llmfit name | Ollama tag |\n|---|---|\n| `meta-llama/Llama-3.1-8B-Instruct` | `llama3.1:8b` |\n| `meta-llama/Llama-3.3-70B-Instruct` | `llama3.3:70b` |\n| `Qwen/Qwen2.5-Coder-7B-Instruct` | `qwen2.5-coder:7b` |\n| `Qwen/Qwen2.5-72B-Instruct` | `qwen2.5:72b` |\n| `deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct` | `deepseek-coder-v2:16b` |\n| `deepseek-ai/DeepSeek-R1-Distill-Qwen-32B` | `deepseek-r1:32b` |\n| `google/gemma-2-9b-it` | `gemma2:9b` |\n| `mistralai/Mistral-7B-Instruct-v0.3` | `mistral:7b` |\n| `microsoft/Phi-3-mini-4k-instruct` | `phi3:mini` |\n| `microsoft/Phi-4-mini-instruct` | `phi4-mini` |\n\nThen update `openclaw.json`:\n\n```json\n{\n  \"models\": {\n    \"providers\": {\n      \"ollama\": {\n        \"models\": [\"ollama/<ollama-tag>\"]\n      }\n    }\n  }\n}\n```\n\nAnd optionally set as default:\n\n```json\n{\n  \"agents\": {\n    \"defaults\": {\n      \"model\": {\n        \"primary\": \"ollama/<ollama-tag>\"\n      }\n    }\n  }\n}\n```\n\n### For vLLM / LM Studio\n\nUse the HuggingFace model name directly as the model identifier with the appropriate provider prefix (`vllm/` or `lmstudio/`).\n\n## Workflow example\n\nWhen a user asks \"what local models can I run?\":\n\n1. Run `llmfit --json system` to show hardware summary\n2. Run `llmfit recommend --json --limit 5` to get top picks\n3. Present the recommendations with scores and fit levels\n4. If the user wants to configure one, map it to the appropriate Ollama/vLLM/LM Studio tag\n5. Offer to update `openclaw.json` with the chosen model\n\nWhen a user asks for a specific use case like \"recommend a coding model\":\n\n1. Run `llmfit recommend --json --use-case coding --limit 3`\n2. Present the coding-specific recommendations\n3. Offer to pull via Ollama and configure\n\n## Notes\n\n- llmfit detects NVIDIA GPUs (via nvidia-smi), AMD GPUs (via rocm-smi), and Apple Silicon (unified memory).\n- Multi-GPU setups aggregate VRAM across cards automatically.\n- The `best_quant` field tells you the optimal quantization — higher quant (Q6_K, Q8_0) means better quality if VRAM allows.\n- Speed estimates (`estimated_tps`) are approximate and vary by hardware and quantization.\n- Models with `fit_level: \"TooTight\"` should never be recommended to users.\n","AGENTS.md":"# AGENTS.md\n\nInstructions for AI agents contributing to this codebase.\n\n---\n\n## Project overview\n\n`llmfit` is a Rust CLI/TUI tool that matches LLM models against local system hardware (RAM, CPU, GPU). It detects system specs, loads a model database from embedded JSON, scores each model's fit, and presents results in an interactive terminal UI or classic table output.\n\n## Language and toolchain\n\n- Rust, edition 2024.\n- Build with `cargo build`. Run with `cargo run`.\n- No nightly features required. Stable toolchain only.\n- Minimum supported Rust version: whatever edition 2024 requires (1.85+).\n\n## Architecture\n\n```\nllmfit-core/      Shared Rust library. It owns hardware detection, model data,\n                 fit analysis, planning, providers, benchmarks, quality checks,\n                 model updates, diagnostics, claims, and result sharing.\n\nllmfit-tui/       Main `llmfit` binary. It provides the CLI, ratatui TUI,\n                 Axum HTTP API, embedded Web dashboard, and stdio MCP server.\n                 `main.rs` parses all clap flags and selects an interface.\n\nllmfit-desktop/   Tauri desktop application. Tauri commands call llmfit-core\n                 for hardware detection, fit analysis, and Ollama downloads.\n\nllmfit-web/       React 18 and Vite dashboard. It calls `/api/v1/*` endpoints\n                 from llmfit-tui. The llmfit-tui build script embeds `dist/`.\n                 This directory is not a Cargo workspace member.\n\nllmfit-python/    Python package wrapper. Its wheel includes the compiled Rust\n                 binary. `python -m llmfit` forwards arguments to that binary.\n                 It does not expose llmfit-core through a native Python API.\n```\n\nThe Cargo workspace contains `llmfit-core`, `llmfit-tui`, and\n`llmfit-desktop`. The default members are `llmfit-core` and `llmfit-tui`.\n\nSource modules in `llmfit-core/src/`:\n\n- `analysis.rs`: Builds model-fit results. It marks installed models and applies\n  local, community, and measured benchmark calibration.\n- `bench.rs`: Runs throughput benchmarks against Ollama and OpenAI-compatible\n  endpoints. It also discovers available benchmark targets.\n- `benchmarks.rs`: Loads embedded and remote benchmark data. It builds measured\n  throughput indexes and hardware leaderboard queries.\n- `claim.rs`: Calculates model resource bounds. It renders Kubernetes DRA\n  `ResourceClaim` and `ResourceClaimTemplate` manifests.\n- `doctor.rs`: Collects installation, hardware, runtime, and model diagnostics.\n- `fit.rs`: Calculates fit level, run mode, runtime, quantization, score, and\n  estimated throughput.\n- `hardware.rs`: Detects RAM, CPU, GPUs, unified memory, clusters, and memory\n  bandwidth.\n- `hwprofile.rs`: Loads embedded and user hardware profiles (schema v1). Applies\n  capacity to `SystemSpecs` and bandwidth/efficiency overrides to `CalcConfig`.\n- `models.rs`: Defines model metadata. It loads embedded HF and ONNX catalogs,\n  custom models, and the update cache.\n- `plan.rs`: Estimates memory, throughput, run paths, and hardware upgrade needs\n  for a requested model setup.\n- `providers.rs`: Integrates Ollama, MLX, llama.cpp, Docker Model Runner,\n  LM Studio, vLLM, and RamaLama.\n- `quality.rs`: Runs response quality tests. It scores roles, builds routing\n  recommendations, and compares results with baselines.\n- `share.rs`: Stores local benchmark results. It handles GitHub authentication\n  and submits community benchmark data.\n- `task_bench.rs`: Provides task benchmark scores for model and task pairs.\n- `update.rs`: Fetches model metadata and manages the local model update cache.\n\nSource modules in `llmfit-tui/src/`:\n\n- `main.rs`: Owns CLI parsing, hardware overrides, command execution, and\n  interface dispatch.\n- `display.rs`: Renders classic CLI tables, model plans, JSON, and CSV output.\n- `download_history.rs`: Stores persistent model download records.\n- `events.rs`: Publishes optional NATS system events and periodic snapshots.\n- `filter_config.rs`: Loads and saves persistent TUI filter settings.\n- `mcp_server.rs`: Exposes hardware, model, runtime, and planning MCP tools.\n- `serve_api.rs`: Serves the embedded Web dashboard and JSON API with Axum.\n- `serve_shared.rs`: Converts shared core types into API and MCP JSON values.\n- `theme.rs`: Defines TUI color themes and stores the selected theme.\n- `tui_app.rs`: Owns TUI state, model results, filters, downloads, and selection.\n- `tui_events.rs`: Handles crossterm input and mutates TUI state.\n- `tui_ui.rs`: Renders TUI views, tables, details, plans, and popups with\n  ratatui.\n\n## Data flow\n\nAll interfaces use the same core analysis flow:\n\n1. `SystemSpecs::detect()` detects CPU, RAM, GPU, unified-memory, and cluster\n   information. CLI hardware overrides can replace detected values.\n2. `ModelDatabase::new()` loads the embedded HF and ONNX catalogs.\n3. Custom models replace matching embedded models. The update cache appends\n   models that are not already present.\n4. `build_model_fits()` removes backend-incompatible models. It calls\n   `ModelFit::analyze_with_forced_runtime()` for each remaining model.\n5. Fit analysis selects a runtime, quantization, and run mode. It calculates\n   memory use, throughput, fit level, score components, and notes.\n6. Local benchmark results, community results, and measured presets can replace\n   or calibrate formula-based throughput estimates.\n7. Each interface applies its own filters, sorting, limits, and presentation.\n\n`ModelFit::analyze()` is the default analysis wrapper. Use\n`analyze_with_context_limit()` for a context cap. Use\n`analyze_with_forced_runtime()` for runtime selection. Use\n`analyze_with_config()` for custom calculation parameters. These methods share\nthe private `analyze_inner()` implementation.\n\nInterface-specific flow:\n\n- CLI: `main.rs` dispatches a subcommand. The command calls llmfit-core and\n  writes a table, JSON, or CSV result.\n- TUI: `App` owns model and filter state. `tui_events` changes that state.\n  `apply_filters()` updates visible indices. `tui_ui` renders the current state.\n- Web: React calls `/api/v1/*`. Axum handlers in `serve_api.rs` call llmfit-core\n  and return JSON. The same server returns the embedded React assets.\n- MCP: `LlmfitMcpServer` receives stdio tool calls. Each tool calls shared core\n  analysis or planning logic and returns JSON text.\n- Desktop: Tauri commands call llmfit-core and serialize results for the desktop\n  UI. Ollama pull state stays in the Tauri application state.\n- Python: The Python entry point locates the installed `llmfit` binary. It then\n  replaces the process on Unix or starts a subprocess on Windows.\n\n## Model database\n\n- Source: `llmfit-core/data/hf_models.json` (33 models).\n- Generated by `scripts/scrape_hf_models.py` (Python, stdlib only, no pip deps).\n- Embedded at compile time via `include_str!(\"../data/hf_models.json\")`.\n- Schema per entry: name, provider, parameter_count, min_ram_gb, recommended_ram_gb, min_vram_gb, quantization, context_length, use_case.\n- `min_vram_gb` is VRAM needed for GPU inference. `min_ram_gb` is system RAM needed for CPU inference. Both are derived from the same parameter count.\n- RAM formula: `params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.2 overhead`.\n- VRAM formula: `params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.1 activation overhead`.\n- Recommended RAM: `model_size * 2.0`.\n\nDo not manually edit `hf_models.json`. Regenerate it by running the scraper:\n\n```sh\npython3 scripts/scrape_hf_models.py\n```\n\nThe scraper has hardcoded fallback entries for gated models that require authentication.\n\n## Conventions\n\n- No `unsafe` code.\n- No `.unwrap()` on user-facing paths. Use proper error handling or `expect()` with a descriptive message for internal invariants only.\n- Fit levels are ordered: Perfect > Good > Marginal > TooTight. Do not add levels without updating `rank_models_by_fit()` sort logic.\n- Fit is VRAM-first. `RunMode` has five execution paths: `Gpu`, `MoeOffload`,\n  `CpuOffload`, `CpuOnly`, and `TensorParallel`.\n- `Gpu` keeps the model in VRAM. `MoeOffload` keeps active experts in VRAM and\n  inactive experts in RAM. `CpuOffload` splits work between VRAM and RAM.\n  `CpuOnly` uses system RAM. `TensorParallel` distributes work across nodes.\n- `min_vram_gb` is the VRAM needed to load model weights on GPU. `min_ram_gb` is the system RAM needed for CPU-only inference (same weights, loaded into RAM instead). They represent the same workload on different hardware paths.\n- On Apple Silicon (unified memory), VRAM = system RAM. The `CpuOffload` path is skipped because there is no separate RAM pool to spill to. `SystemSpecs::unified_memory` tracks this.\n- TUI rendering is stateless. `tui_ui::draw()` must not mutate `App`. Pass `&mut App` only for `TableState` widget requirements -- do not use it to change application state.\n- Event handling in `tui_events.rs` is the sole place that mutates `App` in the TUI loop.\n- Keep `display.rs` and `tui_*.rs` independent. The CLI path must work without initializing any TUI state.\n\n## Adding a new model to the database\n\n1. Add the model's HuggingFace repo ID to `TARGET_MODELS` in `scripts/scrape_hf_models.py`.\n2. If the model is gated (requires HF auth), add a fallback entry to the `FALLBACK` dict in the same script.\n3. Run `python3 scripts/scrape_hf_models.py`.\n4. Verify the output in `llmfit-core/data/hf_models.json`.\n5. Run `cargo build` to verify compilation.\n\n## Adding a new filter\n\n1. Add the filter state to `App` in `tui_app.rs`.\n2. Add filtering logic inside `apply_filters()`.\n3. Add the keybinding in `tui_events.rs` (Normal mode handler).\n4. Add the UI widget in `tui_ui.rs` (`draw_search_and_filters()` function).\n5. Update the status bar help text in `draw_status_bar()`.\n\n## Adding a new CLI subcommand\n\n1. Add a variant to the `Commands` enum in `main.rs`.\n2. Add the match arm in the `main()` function's command dispatch.\n3. Use `display.rs` functions for output, or add new ones as needed.\n\n## Testing\n\nThe project has Rust, Web, and Python test suites.\n\n- Rust unit tests live beside code in `llmfit-core/src/` and\n  `llmfit-tui/src/`.\n- Core integration tests in `llmfit-core/tests/` validate catalog schemas and\n  ONNX model data.\n- CLI integration tests in `llmfit-tui/tests/` use `assert_cmd` against the\n  compiled `llmfit` binary.\n- HTTP API tests exercise Axum routers and JSON responses in `serve_api.rs`.\n- TUI tests focus on state transitions, filters, event handling, and render\n  output. Keep production rendering stateless.\n- Web tests use Vitest, jsdom, and Testing Library. They cover API query\n  construction, localization, filtering, and dashboard interactions.\n- Python tests use pytest. They cover binary discovery, package versioning, and\n  invocation of the packaged Rust binary.\n\nRun the default Rust test set:\n\n```sh\ncargo test\n```\n\nRun all Rust workspace members, including the desktop crate:\n\n```sh\ncargo test --workspace\n```\n\nRun one Rust package:\n\n```sh\ncargo test -p llmfit-core\ncargo test -p llmfit\n```\n\nRun the Web tests:\n\n```sh\nnpm --prefix llmfit-web test\n```\n\nRun the Python tests and quality checks:\n\n```sh\nuv run --project llmfit-python pytest llmfit-python/tests\nmake -C llmfit-python check\n```\n\n## Dependencies policy\n\n- Prefer crates that are well-maintained and have minimal transitive dependencies.\n- `sysinfo` is the system detection crate. Do not replace it with raw platform calls.\n- `ureq` is the blocking HTTP client for providers, benchmarks, updates, quality\n  tests, and sharing. Do not add a second core HTTP client without a concrete need.\n- `which` locates installed runtime binaries. Keep runtime discovery in\n  `providers.rs` instead of adding manual `PATH` parsing.\n- `regex` supports response scoring and text parsing. `serde_yml` parses quality\n  test configuration. `base64` encodes benchmark submissions for GitHub.\n- `objc2-metal` reads the effective Metal working-set limit on macOS. Keep it a\n  macOS-only dependency. Do not replace it with raw platform calls.\n- `ratatui` + `crossterm` is the TUI stack. Do not mix in `termion` or `ncurses`.\n- `clap` with derive feature for CLI parsing. Do not use manual arg parsing.\n- The Python scraper uses only stdlib (`urllib`, `json`). Do not add pip dependencies.\n\n## Common tasks\n\n```sh\n# Build\ncargo build\n\n# Run TUI\ncargo run\n\n# Run CLI mode\ncargo run -- --cli\n\n# Run specific subcommand\ncargo run -- system\ncargo run -- fit --perfect -n 5\ncargo run -- search \"llama\"\n\n# Refresh model database\npython3 scripts/scrape_hf_models.py && cargo build\n\n# Check for compilation issues\ncargo check\n\n# Format code\ncargo fmt\n\n# Lint\ncargo clippy\n```\n\n## Platform notes\n\n- GPU detection shells out to `nvidia-smi` (NVIDIA) and `rocm-smi` (AMD). These are best-effort and fail silently if unavailable.\n- Apple Silicon detection uses `system_profiler SPDisplaysDataType`. On unified memory Macs, VRAM is reported as available system RAM (same pool).\n- `sysinfo` handles cross-platform RAM/CPU. No conditional compilation needed.\n- The TUI uses crossterm which works on Linux, macOS, and Windows terminals.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/AlexsJones/llmfit/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\nInstructions for AI agents contributing to this codebase.\n\n---\n\n## Project overview\n\n`llmfit` is a Rust CLI/TUI tool that matches LLM models against local system hardware (RAM, CPU, GPU). It detects system specs, loads a model database from embedded JSON, scores each model's fit, and presents results in an interactive terminal UI or classic table output.\n\n## Language and toolchain\n\n- Rust, edition 2024.\n- Build with `cargo build`. Run with `cargo run`.\n- No nightly features required. Stable toolchain only.\n- Minimum supported Rust version: whatever edition 2024 requires (1.85+).\n\n## Architecture\n\n```\nllmfit-core/      Shared Rust library. It owns hardware detection, model data,\n                 fit analysis, planning, providers, benchmarks, quality checks,\n                 model updates, diagnostics, claims, and result sharing.\n\nllmfit-tui/       Main `llmfit` binary. It provides the CLI, ratatui TUI,\n                 Axum HTTP API, embedded Web dashboard, and stdio MCP server.\n                 `main.rs` parses all clap flags and selects an interface.\n\nllmfit-desktop/   Tauri desktop application. Tauri commands call llmfit-core\n                 for hardware detection, fit analysis, and Ollama downloads.\n\nllmfit-web/       React 18 and Vite dashboard. It calls `/api/v1/*` endpoints\n                 from llmfit-tui. The llmfit-tui build script embeds `dist/`.\n                 This directory is not a Cargo workspace member.\n\nllmfit-python/    Python package wrapper. Its wheel includes the compiled Rust\n                 binary. `python -m llmfit` forwards arguments to that binary.\n                 It does not expose llmfit-core through a native Python API.\n```\n\nThe Cargo workspace contains `llmfit-core`, `llmfit-tui`, and\n`llmfit-desktop`. The default members are `llmfit-core` and `llmfit-tui`.\n\nSource modules in `llmfit-core/src/`:\n\n- `analysis.rs`: Builds model-fit results. It marks installed models and applies\n  local, community, and measured benchmark calibration.\n- `bench.rs`: Runs throughput benchmarks against Ollama and OpenAI-compatible\n  endpoints. It also discovers available benchmark targets.\n- `benchmarks.rs`: Loads embedded and remote benchmark data. It builds measured\n  throughput indexes and hardware leaderboard queries.\n- `claim.rs`: Calculates model resource bounds. It renders Kubernetes DRA\n  `ResourceClaim` and `ResourceClaimTemplate` manifests.\n- `doctor.rs`: Collects installation, hardware, runtime, and model diagnostics.\n- `fit.rs`: Calculates fit level, run mode, runtime, quantization, score, and\n  estimated throughput.\n- `hardware.rs`: Detects RAM, CPU, GPUs, unified memory, clusters, and memory\n  bandwidth.\n- `hwprofile.rs`: Loads embedded and user hardware profiles (schema v1). Applies\n  capacity to `SystemSpecs` and bandwidth/efficiency overrides to `CalcConfig`.\n- `models.rs`: Defines model metadata. It loads embedded HF and ONNX catalogs,\n  custom models, and the update cache.\n- `plan.rs`: Estimates memory, throughput, run paths, and hardware upgrade needs\n  for a requested model setup.\n- `providers.rs`: Integrates Ollama, MLX, llama.cpp, Docker Model Runner,\n  LM Studio, vLLM, and RamaLama.\n- `quality.rs`: Runs response quality tests. It scores roles, builds routing\n  recommendations, and compares results with baselines.\n- `share.rs`: Stores local benchmark results. It handles GitHub authentication\n  and submits community benchmark data.\n- `task_bench.rs`: Provides task benchmark scores for model and task pairs.\n- `update.rs`: Fetches model metadata and manages the local model update cache.\n\nSource modules in `llmfit-tui/src/`:\n\n- `main.rs`: Owns CLI parsing, hardware overrides, command execution, and\n  interface dispatch.\n- `display.rs`: Renders classic CLI tables, model plans, JSON, and CSV output.\n- `download_history.rs`: Stores persistent model download records.\n- `events.rs`: Publishes optional NATS system events and periodic snapshots.\n- `filter_config.rs`: Loads and saves persistent TUI filter settings.\n- `mcp_server.rs`: Exposes hardware, model, runtime, and planning MCP tools.\n- `serve_api.rs`: Serves the embedded Web dashboard and JSON API with Axum.\n- `serve_shared.rs`: Converts shared core types into API and MCP JSON values.\n- `theme.rs`: Defines TUI color themes and stores the selected theme.\n- `tui_app.rs`: Owns TUI state, model results, filters, downloads, and selection.\n- `tui_events.rs`: Handles crossterm input and mutates TUI state.\n- `tui_ui.rs`: Renders TUI views, tables, details, plans, and popups with\n  ratatui.\n\n## Data flow\n\nAll interfaces use the same core analysis flow:\n\n1. `SystemSpecs::detect()` detects CPU, RAM, GPU, unified-memory, and cluster\n   information. CLI hardware overrides can replace detected values.\n2. `ModelDatabase::new()` loads the embedded HF and ONNX catalogs.\n3. Custom models replace matching embedded models. The update cache appends\n   models that are not already present.\n4. `build_model_fits()` removes backend-incompatible models. It calls\n   `ModelFit::analyze_with_forced_runtime()` for each remaining model.\n5. Fit analysis selects a runtime, quantization, and run mode. It calculates\n   memory use, throughput, fit level, score components, and notes.\n6. Local benchmark results, community results, and measured presets can replace\n   or calibrate formula-based throughput estimates.\n7. Each interface applies its own filters, sorting, limits, and presentation.\n\n`ModelFit::analyze()` is the default analysis wrapper. Use\n`analyze_with_context_limit()` for a context cap. Use\n`analyze_with_forced_runtime()` for runtime selection. Use\n`analyze_with_config()` for custom calculation parameters. These methods share\nthe private `analyze_inner()` implementation.\n\nInterface-specific flow:\n\n- CLI: `main.rs` dispatches a subcommand. The command calls llmfit-core and\n  writes a table, JSON, or CSV result.\n- TUI: `App` owns model and filter state. `tui_events` changes that state.\n  `apply_filters()` updates visible indices. `tui_ui` renders the current state.\n- Web: React calls `/api/v1/*`. Axum handlers in `serve_api.rs` call llmfit-core\n  and return JSON. The same server returns the embedded React assets.\n- MCP: `LlmfitMcpServer` receives stdio tool calls. Each tool calls shared core\n  analysis or planning logic and returns JSON text.\n- Desktop: Tauri commands call llmfit-core and serialize results for the desktop\n  UI. Ollama pull state stays in the Tauri application state.\n- Python: The Python entry point locates the installed `llmfit` binary. It then\n  replaces the process on Unix or starts a subprocess on Windows.\n\n## Model database\n\n- Source: `llmfit-core/data/hf_models.json` (33 models).\n- Generated by `scripts/scrape_hf_models.py` (Python, stdlib only, no pip deps).\n- Embedded at compile time via `include_str!(\"../data/hf_models.json\")`.\n- Schema per entry: name, provider, parameter_count, min_ram_gb, recommended_ram_gb, min_vram_gb, quantization, context_length, use_case.\n- `min_vram_gb` is VRAM needed for GPU inference. `min_ram_gb` is system RAM needed for CPU inference. Both are derived from the same parameter count.\n- RAM formula: `params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.2 overhead`.\n- VRAM formula: `params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.1 activation overhead`.\n- Recommended RAM: `model_size * 2.0`.\n\nDo not manually edit `hf_models.json`. Regenerate it by running the scraper:\n\n```sh\npython3 scripts/scrape_hf_models.py\n```\n\nThe scraper has hardcoded fallback entries for gated models that require authentication.\n\n## Conventions\n\n- No `unsafe` code.\n- No `.unwrap()` on user-facing paths. Use proper error handling or `expect()` with a descriptive message for internal invariants only.\n- Fit levels are ordered: Perfect > Good > Marginal > TooTight. Do not add levels without updating `rank_models_by_fit()` sort logic.\n- Fit is VRAM-first. `RunMode` has five execution paths: `Gpu`, `MoeOffload`,\n  `CpuOffload`, `CpuOnly`, and `TensorParallel`.\n- `Gpu` keeps the model in VRAM. `MoeOffload` keeps active experts in VRAM and\n  inactive experts in RAM. `CpuOffload` splits work between VRAM and RAM.\n  `CpuOnly` uses system RAM. `TensorParallel` distributes work across nodes.\n- `min_vram_gb` is the VRAM needed to load model weights on GPU. `min_ram_gb` is the system RAM needed for CPU-only inference (same weights, loaded into RAM instead). They represent the same workload on different hardware paths.\n- On Apple Silicon (unified memory), VRAM = system RAM. The `CpuOffload` path is skipped because there is no separate RAM pool to spill to. `SystemSpecs::unified_memory` tracks this.\n- TUI rendering is stateless. `tui_ui::draw()` must not mutate `App`. Pass `&mut App` only for `TableState` widget requirements -- do not use it to change application state.\n- Event handling in `tui_events.rs` is the sole place that mutates `App` in the TUI loop.\n- Keep `display.rs` and `tui_*.rs` independent. The CLI path must work without initializing any TUI state.\n\n## Adding a new model to the database\n\n1. Add the model's HuggingFace repo ID to `TARGET_MODELS` in `scripts/scrape_hf_models.py`.\n2. If the model is gated (requires HF auth), add a fallback entry to the `FALLBACK` dict in the same script.\n3. Run `python3 scripts/scrape_hf_models.py`.\n4. Verify the output in `llmfit-core/data/hf_models.json`.\n5. Run `cargo build` to verify compilation.\n\n## Adding a new filter\n\n1. Add the filter state to `App` in `tui_app.rs`.\n2. Add filtering logic inside `apply_filters()`.\n3. Add the keybinding in `tui_events.rs` (Normal mode handler).\n4. Add the UI widget in `tui_ui.rs` (`draw_search_and_filters()` function).\n5. Update the status bar help text in `draw_status_bar()`.\n\n## Adding a new CLI subcommand\n\n1. Add a variant to the `Commands` enum in `main.rs`.\n2. Add the match arm in the `main()` function's command dispatch.\n3. Use `display.rs` functions for output, or add new ones as needed.\n\n## Testing\n\nThe project has Rust, Web, and Python test suites.\n\n- Rust unit tests live beside code in `llmfit-core/src/` and\n  `llmfit-tui/src/`.\n- Core integration tests in `llmfit-core/tests/` validate catalog schemas and\n  ONNX model data.\n- CLI integration tests in `llmfit-tui/tests/` use `assert_cmd` against the\n  compiled `llmfit` binary.\n- HTTP API tests exercise Axum routers and JSON responses in `serve_api.rs`.\n- TUI tests focus on state transitions, filters, event handling, and render\n  output. Keep production rendering stateless.\n- Web tests use Vitest, jsdom, and Testing Library. They cover API query\n  construction, localization, filtering, and dashboard interactions.\n- Python tests use pytest. They cover binary discovery, package versioning, and\n  invocation of the packaged Rust binary.\n\nRun the default Rust test set:\n\n```sh\ncargo test\n```\n\nRun all Rust workspace members, including the desktop crate:\n\n```sh\ncargo test --workspace\n```\n\nRun one Rust package:\n\n```sh\ncargo test -p llmfit-core\ncargo test -p llmfit\n```\n\nRun the Web tests:\n\n```sh\nnpm --prefix llmfit-web test\n```\n\nRun the Python tests and quality checks:\n\n```sh\nuv run --project llmfit-python pytest llmfit-python/tests\nmake -C llmfit-python check\n```\n\n## Dependencies policy\n\n- Prefer crates that are well-maintained and have minimal transitive dependencies.\n- `sysinfo` is the system detection crate. Do not replace it with raw platform calls.\n- `ureq` is the blocking HTTP client for providers, benchmarks, updates, quality\n  tests, and sharing. Do not add a second core HTTP client without a concrete need.\n- `which` locates installed runtime binaries. Keep runtime discovery in\n  `providers.rs` instead of adding manual `PATH` parsing.\n- `regex` supports response scoring and text parsing. `serde_yml` parses quality\n  test configuration. `base64` encodes benchmark submissions for GitHub.\n- `objc2-metal` reads the effective Metal working-set limit on macOS. Keep it a\n  macOS-only dependency. Do not replace it with raw platform calls.\n- `ratatui` + `crossterm` is the TUI stack. Do not mix in `termion` or `ncurses`.\n- `clap` with derive feature for CLI parsing. Do not use manual arg parsing.\n- The Python scraper uses only stdlib (`urllib`, `json`). Do not add pip dependencies.\n\n## Common tasks\n\n```sh\n# Build\ncargo build\n\n# Run TUI\ncargo run\n\n# Run CLI mode\ncargo run -- --cli\n\n# Run specific subcommand\ncargo run -- system\ncargo run -- fit --perfect -n 5\ncargo run -- search \"llama\"\n\n# Refresh model database\npython3 scripts/scrape_hf_models.py && cargo build\n\n# Check for compilation issues\ncargo check\n\n# Format code\ncargo fmt\n\n# Lint\ncargo clippy\n```\n\n## Platform notes\n\n- GPU detection shells out to `nvidia-smi` (NVIDIA) and `rocm-smi` (AMD). These are best-effort and fail silently if unavailable.\n- Apple Silicon detection uses `system_profiler SPDisplaysDataType`. On unified memory Macs, VRAM is reported as available system RAM (same pool).\n- `sysinfo` handles cross-platform RAM/CPU. No conditional compilation needed.\n- The TUI uses crossterm which works on Linux, macOS, and Windows terminals.\n","isInternal":false,"tokens":3199,"sizeBytes":13031},{"name":"API.md","path":"API.md","rawUrl":"https://raw.githubusercontent.com/AlexsJones/llmfit/HEAD/API.md","title":"API Interface Contract & Specifications","category":"project-spec","format":"markdown","content":"# llmfit REST API Guide\n\nThis document is for agent/client builders integrating with `llmfit serve`.\n\n## Purpose\n\n`llmfit serve` exposes node-local model fit analysis (same core data used by TUI/CLI) over HTTP and serves a local web dashboard.\n\nPrimary use case:\n- Query each node in a cluster for top runnable models.\n- Aggregate externally (scheduler/controller/UI) for placement decisions.\n\n## Start the server\n\n```sh\nllmfit serve --port 8787\n```\n\nGlobal flags still apply:\n\n```sh\nllmfit --memory 24G --ram 64G --cpu-cores 16 --max-context 8192 serve --port 8787\n```\n\nHardware overrides (`--memory`, `--ram`, `--cpu-cores`) are reflected in API responses, making the server report the overridden values instead of the detected hardware.\n\n## Base URL\n\nDefault local base URL:\n\n```text\nhttp://127.0.0.1:8787\n```\n\nTo expose outside localhost, pass `--host 0.0.0.0`.\n\n### Unix domain socket\n\nFor same-host consumers that should not touch the network at all (e.g. a\nsidecar in a `hostNetwork` Kubernetes pod, where a TCP bind would land on the\nnode's loopback), listen on a Unix socket instead:\n\n```sh\nllmfit serve --unix-socket /run/llmfit/llmfit.sock\n```\n\nThe socket is created with mode `0660`; a stale socket file from a previous\ninstance is replaced automatically. All HTTP endpoints are identical:\n\n```sh\ncurl --unix-socket /run/llmfit/llmfit.sock http://localhost/api/v1/system\n```\n\n`--unix-socket` conflicts with `--host`/`--port` and is unix-platforms only.\n\nIf you are building from source and want the dashboard embedded in `llmfit`, build web assets first:\n\n```sh\ncd llmfit-web && npm ci && npm run build\n```\n\n## Endpoints\n\n### `GET /`\nWeb dashboard entrypoint (same-origin UI for fit exploration).\n\n### `GET /health`\nLiveness probe.\n\nExample response:\n\n```json\n{\n  \"status\": \"ok\",\n  \"node\": {\n    \"name\": \"worker-1\",\n    \"os\": \"linux\"\n  }\n}\n```\n\n---\n\n### `GET /api/v1/system`\nReturns node identity + detected hardware.\n\nExample response shape:\n\n```json\n{\n  \"node\": {\n    \"name\": \"worker-1\",\n    \"os\": \"linux\"\n  },\n  \"system\": {\n    \"total_ram_gb\": 62.23,\n    \"available_ram_gb\": 41.08,\n    \"cpu_cores\": 14,\n    \"cpu_name\": \"Intel(R) Core(TM) Ultra 7 165U\",\n    \"has_gpu\": false,\n    \"gpu_vram_gb\": null,\n    \"gpu_available_gb\": null,\n    \"gpu_name\": null,\n    \"gpu_count\": 0,\n    \"unified_memory\": false,\n    \"backend\": \"CPU (x86)\",\n    \"gpus\": []\n  },\n  \"profile\": null\n}\n```\n\n`profile` is the name of the hardware profile the server was started with (`llmfit serve --profile <name>`), or `null` when it describes the machine it runs on.\n\n`gpu_available_gb` is the VRAM free right now, pooled across discrete GPUs (from `nvidia-smi memory.free` and amdgpu's `mem_info_vram_used`); each `gpus[]` entry carries its own `free_vram_gb`. Both are `null` when a backend does not report it (Intel, Windows, older drivers) or after a hardware override. On Apple Silicon `gpu_available_gb` is instead Metal's wiring cap for the unified pool. `plan` grades GPU run paths against the free figure when it is known and against total VRAM otherwise.\n\n---\n\n### `GET /api/v1/models`\nReturns filtered/sorted model-fit rows for this node.\n\nEnvelope shape:\n\n```json\n{\n  \"node\": { \"name\": \"worker-1\", \"os\": \"linux\" },\n  \"system\": { \"...\": \"...\" },\n  \"total_models\": 23,\n  \"returned_models\": 10,\n  \"filters\": { \"...\": \"echo of query state\" },\n  \"models\": [\n    {\n      \"name\": \"Qwen/Qwen2.5-Coder-7B-Instruct\",\n      \"provider\": \"Qwen\",\n      \"parameter_count\": \"7B\",\n      \"params_b\": 7.0,\n      \"context_length\": 32768,\n      \"usable_context\": 32768,\n      \"effective_context_length\": 8192,\n      \"use_case\": \"Coding\",\n      \"category\": \"Coding\",\n      \"release_date\": \"2025-03-14\",\n      \"is_moe\": false,\n      \"fit_level\": \"good\",\n      \"fit_label\": \"Good\",\n      \"run_mode\": \"gpu\",\n      \"run_mode_label\": \"GPU\",\n      \"score\": 86.5,\n      \"score_components\": {\n        \"quality\": 87.0,\n        \"speed\": 81.2,\n        \"fit\": 90.1,\n        \"context\": 88.0\n      },\n      \"estimated_tps\": 42.5,\n      \"estimate_confidence\": \"estimated\",\n      \"estimate_confidence_label\": \"estimated\",\n      \"prefill_tps\": 812.3,\n      \"ttft_ms\": 10.1,\n      \"runtime\": \"llamacpp\",\n      \"runtime_label\": \"llama.cpp\",\n      \"best_quant\": \"Q5_K_M\",\n      \"memory_required_gb\": 5.8,\n      \"memory_available_gb\": 12.0,\n      \"utilization_pct\": 48.3,\n      \"notes\": [],\n      \"gguf_sources\": [],\n      \"capabilities\": [\"tool_use\"],\n      \"capability_ids\": [\"tool_use\"],\n      \"license\": \"apache-2.0\",\n      \"supports_tp\": [1, 2, 4],\n      \"installed\": false,\n      \"disk_size_gb\": 5.1,\n      \"ollama_name\": \"qwen2.5-coder:7b-instruct\",\n      \"estimate_basis\": {\n        \"method\": \"roofline\",\n        \"gpu_bandwidth_gbps\": 320.0,\n        \"ddr_bandwidth_gbps\": null,\n        \"local_calibration\": null,\n        \"efficiency\": 0.85,\n        \"assumed_context\": 8192\n      },\n      \"verify_command\": \"llama-bench -m <path-to-Q5_K_M-gguf> -ngl 99 -p 512 -n 128\",\n      \"measured_tps\": null\n    }\n  ]\n}\n```\n\nThe three context fields answer different questions:\n\n- `context_length` — the model's native window, as advertised upstream.\n- `usable_context` — how much of that window actually fits in this node's\n  available memory alongside the weights. Use this one to pick a runtime `-c`.\n- `effective_context_length` — the context the `memory_required_gb` and\n  `estimated_tps` figures on this row were computed at. Defaults to\n  `min(context_length, 8192)`; set by `max_context` when supplied.\n\nThe envelope also carries these fields, now at parity with `llmfit fit --json`\n(both frontends serialize through one shared function):\n\n- `installed` — whether the model was found in a local runtime provider. `llmfit serve` probes providers in the background (at startup, after a finished download, and when a request finds the result older than 30 seconds), so a model pulled or removed outside the server shows up within one refresh after that; it is `false` everywhere until the first probe finishes.\n- `disk_size_gb` — estimated on-disk size at `best_quant`.\n- `capability_ids` — machine-readable capability ids (snake_case); mirrors\n  `capabilities` here. Note `llmfit fit --json` overloads its `capabilities`\n  key with human labels (e.g. `\"Tool Use\"`) — that overload is CLI-only and\n  slated for deprecation.\n- `ollama_name` — the `ollama pull` tag for this model, when derivable.\n- `estimate_basis` — how `memory_required_gb`/`estimated_tps` were derived\n  (bandwidths, efficiency, assumed context), for reproducibility.\n- `estimate_confidence` / `estimate_confidence_label` — how much to trust\n  `estimated_tps`, most to least trustworthy: `measured_local` (your own\n  `llmfit bench` runs), `measured_community` (llmfit community submissions\n  or localmaxxing.com data on matching hardware), `calibrated` (a formula\n  estimate scaled by a correction factor from benchmark runs on this exact\n  hardware), `estimated` (a bare formula estimate), `unsupported` (no\n  estimate — the model needs a runtime llmfit can't model). See\n  `measured_tps`/`estimate_basis.local_calibration` for the data each level\n  is derived from.\n- `prefill_tps` / `ttft_ms` — estimated prompt-processing throughput\n  (tok/s) and time-to-first-token (ms) for a prompt of\n  `effective_context_length` tokens. Prefill is compute-bound, not\n  bandwidth-bound, so both are `null` — not `0.0` — unless the system's GPU\n  compute throughput is known. A `null` here means \"not estimated\", never\n  \"instant\" or \"stalled\".\n- `verify_command` — a `llama-bench` invocation measuring the same throughput\n  this row estimates (llama.cpp GPU / CPU-only runs; `null` otherwise).\n- `measured_tps` — a recorded benchmark result if one exists, else `null`.\n\nNote on vocabulary: `fit_level`, `run_mode`, and `runtime` here are stable\nmachine codes (e.g. `\"good\"`, `\"gpu\"`, `\"llamacpp\"`), with the human string\nunder the paired `*_label` key. `llmfit fit --json` emits the human string\ndirectly under those same keys — a CLI-only legacy overload.\n`estimate_confidence` follows the same convention (machine code, paired\n`estimate_confidence_label`) in both frontends.\n\n### A note on `best_quant`\n\n`best_quant` is normally the llama.cpp/GGUF quant llmfit picked for this\nhardware (e.g. `\"Q5_K_M\"`). It is `null` when the model's own repo name\ndeclares a native low-precision format (NVFP4/MXFP4) that a GGUF quant\nlabel doesn't apply to — the row's `notes` array carries the explanation\nin that case. It's never a GGUF label misattributed to a non-GGUF repo.\n\n---\n\n### Catalog sanitization\n\n`/api/v1/models` never returns speculative-decoding draft heads\n(EAGLE/DFlash/DSpark naming), entries whose name implies a wildly different\nparameter count than their declared size, or other catalog rows llmfit\ncan't score honestly — they're demoted out of fit ranking, not deleted, so\nthis is invisible unless you were expecting a specific draft-head repo to\nshow up as a standalone model. If you maintain your own filtering on top of\n`llmfit fit --json` output today, you can likely delete that workaround.\n\n---\n\n### `GET /api/v1/models/top`\nKey scheduling endpoint. Same schema as `/api/v1/models`, but defaults to top 5 runnable entries.\n\nImportant behavior:\n- Defaults `limit=5`.\n- Excludes `too_tight` rows unless explicitly overridden (and top endpoint still keeps runnable semantics).\n\n---\n\n### `GET /api/v1/models/{name}`\nPath-constrained search. Equivalent to a text search scoped by `{name}`.\n\nUseful for:\n- Client-side drilldown after selecting a model family.\n\n---\n\n### `GET /api/v1/concurrency`\nConcurrent-session capacity for one model: the same estimate as\n`llmfit concurrency <model> --json`. It is a memory-capacity ceiling (how many\nsessions fit resident at once), not a throughput figure under load.\n\nQuery parameters:\n- `model` (required): exact catalog name, case-insensitive.\n- `quant`: weight quantization (e.g. `Q4_K_M`); defaults to the model's best fit.\n- `kv_quant`: `fp16|fp8|q8_0|q4_0|tq` (default `fp16`).\n- `context`: report one context instead of the 4k-256k ladder (tokens).\n- `users`: also report the largest context that fits this many sessions.\n- `ram_gb`, `vram_gb`, `cpu_cores`: hardware overrides, as on `/api/v1/system`.\n\n```json\n{\n  \"model\": \"Qwen/Qwen2.5-7B-Instruct\",\n  \"run_mode\": \"gpu\",\n  \"fit_level\": \"perfect\",\n  \"target_users\": 8,\n  \"max_context_for_target\": 32768,\n  \"estimate\": {\n    \"pool_gb\": 24.0,\n    \"weights_resident_gb\": 5.1,\n    \"kv_budget_gb\": 18.9,\n    \"quant\": \"Q4_K_M\",\n    \"kv_quant\": \"fp16\",\n    \"native_context\": 32768,\n    \"ladder\": [\n      { \"requested_context\": 4096, \"effective_context\": 4096, \"clamped\": false, \"per_session_kv_gb\": 0.22, \"max_sessions\": 85 }\n    ]\n  }\n}\n```\n\n`run_mode` and `fit_level` use the same codes as `/api/v1/models`.\n\n---\n\n### `GET /api/v1/storage`\nDisk needed to keep a library of runnable models, used one at a time: the same\nestimate as `llmfit storage --json`, returned as `{ \"system\": ..., \"storage\": ... }`.\n\nQuery parameters (defaults match the CLI):\n- `keep`: number of distinct runnable models to keep (default `3`).\n- `selection`: `score` (highest-ranked, default) or `largest` (largest weight storage).\n- `os_reserve`: space for the OS, apps and other files (default `100G`; decimal GB, `GiB`/`TiB` are binary).\n- `scratch`: `auto` (default, the largest selected model) or a size; `0` disables it.\n- `headroom`: percent of the suggested SSD to keep free, `0`-`99` (default `15`).\n- `perfect`: `true` to select only Perfect fits (otherwise Perfect, Good and Marginal).\n- `search`: narrow the candidates by name, provider or parameter size.\n- `max_context`, `ram_gb`, `vram_gb`, `cpu_cores`: as on `/api/v1/models`.\n\nThe `storage` object carries `models` (name, best quant, fit level, runtime,\nscore, disk size, context), `library_gb`, `download_scratch_gb`, `need_gb`,\n`target_capacity_gb`, `minimum_ssd_gb`, `suggested_ssd_gb` and `warnings`.\n\n## Query parameters\n\nSupported on `/api/v1/models` and `/api/v1/models/top` (also `/api/v1/models/{name}`):\n\n- `limit` (or alias `n`): max rows returned.\n- `perfect`: `true|false` (when `true`, only perfect fits).\n- `min_fit`: `perfect|good|marginal|too_tight`.\n- `runtime`: `any|mlx|llamacpp|vllm|bitnetcpp`.\n- `use_case`: `general|coding|reasoning|chat|multimodal|embedding`.\n- `provider`: provider substring filter.\n- `search`: free-text filter (name/provider/params/use-case/category).\n- `sort`: `score|tps|params|mem|ctx|date|use_case`.\n- `include_too_tight`: `true` includes unrunnable (`too_tight`) rows even when `min_fit` is not given; `false` drops them even with `min_fit=too_tight`. With neither parameter, `/models` returns rows from `marginal` up; `/models/top` never returns `too_tight` rows.\n- `max_context`: per-request context cap used by memory estimation.\n- `force_runtime`: `mlx|llamacpp|vllm|bitnetcpp` — override automatic runtime selection during analysis (e.g. get llama.cpp recommendations on Apple Silicon instead of MLX).\n\n## Error handling\n\nInvalid filter values return HTTP 400:\n\n```json\n{\n  \"error\": \"invalid min_fit value: use perfect|good|marginal|too_tight\"\n}\n```\n\nServer errors return HTTP 500 with `{\"error\": \"...\"}`.\n\n## Client integration recommendations\n\n### 1) Polling pattern for schedulers\nFor each node agent:\n1. Call `/health`.\n2. Call `/api/v1/system`.\n3. Call `/api/v1/models/top?limit=K&min_fit=good`.\n4. Attach node metadata and forward to your central scheduler.\n\n### 2) Conservative placement defaults\nFor production placement, prefer:\n\n```text\nmin_fit=good\ninclude_too_tight=false\nsort=score\nlimit=5..20\n```\n\n### 3) Per-workload targeting\nExamples:\n- Coding workloads: `use_case=coding`\n- Embedding workloads: `use_case=embedding`\n- Runtime constrained to llama.cpp fleet: `runtime=llamacpp`\n\n### 4) Stable parsing\nTreat unknown fields as forward-compatible additions:\n- Parse required fields you depend on.\n- Ignore unknown fields.\n\n## Curl examples\n\n```sh\ncurl http://127.0.0.1:8787/health\ncurl http://127.0.0.1:8787/api/v1/system\ncurl \"http://127.0.0.1:8787/api/v1/models?limit=20&min_fit=marginal&sort=score\"\ncurl \"http://127.0.0.1:8787/api/v1/models/top?limit=5&min_fit=good&use_case=coding\"\ncurl \"http://127.0.0.1:8787/api/v1/models/Mistral?runtime=any\"\ncurl \"http://127.0.0.1:8787/api/v1/concurrency?model=Qwen/Qwen2.5-7B-Instruct&users=8\"\ncurl \"http://127.0.0.1:8787/api/v1/storage?keep=5&selection=largest&os_reserve=200G\"\n```\n\n---\n\n## MCP Server Mode\n\nllmfit can run as an MCP (Model Context Protocol) server over stdio, making it discoverable by AI agents (Claude, Cursor, etc.).\n\n### Start the MCP server\n\n```sh\nllmfit serve --mcp\n```\n\nGlobal hardware overrides still apply:\n\n```sh\nllmfit --memory 24G --ram 64G serve --mcp\n```\n\n### MCP client configuration\n\nAdd to your MCP client config (e.g. `claude_desktop_config.json`):\n\n```json\n{\n  \"mcpServers\": {\n    \"llmfit\": {\n      \"command\": \"llmfit\",\n      \"args\": [\"serve\", \"--mcp\"]\n    }\n  }\n}\n```\n\n### Available tools\n\n| Tool | Description | Parameters |\n|------|-------------|------------|\n| `get_system_specs` | Node hardware info (RAM, GPU, CPU) | None |\n| `recommend_models` | Top models for this hardware | `limit?`, `use_case?`, `min_fit?`, `runtime?`, `license?`, `sort?` |\n| `search_models` | Free-text model search | `query`, `limit?` |\n| `plan_hardware` | Hardware requirements for a model | `model`, `context?`, `quant?`, `target_tps?` |\n| `get_runtimes` | Installed inference runtimes | None |\n| `get_installed_models` | Models in local runtimes | None |\n\n`plan_hardware` and `POST /api/v1/plan` return the shared plan estimate,\nincluding `disk_size_gb`: estimated weight storage in decimal GB at the\nresolved `quantization`. This excludes KV cache, inference buffers, and\ndownload scratch. `llmfit plan --json` returns the same plan fields.\n\n---\n\n## NATS Event Publishing\n\nWhen built with the `nats` feature, llmfit can publish hardware and model events to NATS for integration with coordination systems (e.g. Sympozium membrane).\n\n### Build with NATS support\n\n```sh\ncargo build --features nats\n```\n\n### Enable event publishing\n\n```sh\nllmfit serve --send-events --nats-url nats://localhost:4222\nllmfit serve --mcp --send-events  # also works with MCP mode\n```\n\nThe `NATS_URL` environment variable is also supported.\n\n### Event subjects\n\nEvents are published to `llmfit.{event_type}.{hostname}`:\n\n| Subject | Trigger | Payload |\n|---------|---------|---------|\n| `llmfit.system.{hostname}` | Startup + every 60s | System hardware specs |\n| `llmfit.fit.{hostname}` | After fit analysis | Model fit summary |\n| `llmfit.plan.{hostname}` | After plan estimate | Plan estimate |\n| `llmfit.runtimes.{hostname}` | Startup + on query | Runtime availability |\n| `llmfit.installed.{hostname}` | Startup + on query | Installed models |\n\n### Event envelope\n\nAll events are wrapped in a common envelope:\n\n```json\n{\n  \"timestamp\": \"1747058400\",\n  \"hostname\": \"worker-1\",\n  \"event_type\": \"system\",\n  \"version\": \"1\",\n  \"data\": { ... }\n}\n```\n\n### Subscribe to events\n\n```sh\nnats sub 'llmfit.>'                    # all events from all nodes\nnats sub 'llmfit.system.>'             # system specs from all nodes\nnats sub 'llmfit.system.worker-1'      # system specs from specific node\n```\n\n---\n\n## Versioning notes\n\nCurrent API prefix is `v1`.\n\nIf you build long-lived clients, pin to `/api/v1/...` and validate behavior with the local test script in `scripts/test_api.py`.\n","isInternal":false,"tokens":4843,"sizeBytes":17383},{"name":"SKILL.md","path":"skills/llmfit-advisor/SKILL.md","rawUrl":"https://raw.githubusercontent.com/AlexsJones/llmfit/HEAD/skills/llmfit-advisor/SKILL.md","title":"llmfit-advisor","category":"anthropic-skill","format":"markdown","content":"---\nname: llmfit-advisor\ndescription: Detect local hardware (RAM, CPU, GPU/VRAM) and recommend the best-fit local LLM models with optimal quantization, speed estimates, and fit scoring.\nmetadata:\n  {\n    \"openclaw\":\n      {\n        \"emoji\": \"🧠\",\n        \"requires\": { \"bins\": [\"llmfit\"] },\n        \"install\":\n          [\n            {\n              \"id\": \"brew\",\n              \"kind\": \"brew\",\n              \"formula\": \"llmfit\",\n              \"bins\": [\"llmfit\"],\n              \"label\": \"Install llmfit (brew)\",\n            },\n            {\n              \"id\": \"cargo\",\n              \"kind\": \"node\",\n              \"bins\": [\"llmfit\"],\n              \"label\": \"Install llmfit (cargo install llmfit)\",\n            },\n          ],\n      },\n  }\n---\n\n# llmfit-advisor\n\nHardware-aware local LLM advisor. Detects your system specs (RAM, CPU, GPU/VRAM) and recommends models that actually fit, with optimal quantization and speed estimates.\n\n## When to use (trigger phrases)\n\nUse this skill immediately when the user asks any of:\n\n- \"what local models can I run?\"\n- \"which LLMs fit my hardware?\"\n- \"recommend a local model\"\n- \"what's the best model for my GPU?\"\n- \"can I run Llama 70B locally?\"\n- \"configure local models\"\n- \"set up Ollama models\"\n- \"what models fit my VRAM?\"\n- \"help me pick a local model for coding\"\n\nAlso use this skill when:\n\n- The user wants to configure `models.providers.ollama` or `models.providers.lmstudio`\n- The user mentions running models locally and you need to know what fits\n- A model recommendation is needed and the user has local inference capability (Ollama, vLLM, LM Studio)\n\n## Quick start\n\n### Detect hardware\n\n```bash\nllmfit --json system\n```\n\nReturns JSON with CPU, RAM, GPU name, VRAM, multi-GPU info, and whether memory is unified (Apple Silicon).\n\n### Get top recommendations\n\n```bash\nllmfit recommend --json --limit 5\n```\n\nReturns the top 5 models ranked by a composite score (quality, speed, fit, context) with optimal quantization for the detected hardware.\n\n### Filter by use case\n\n```bash\nllmfit recommend --json --use-case coding --limit 3\nllmfit recommend --json --use-case reasoning --limit 3\nllmfit recommend --json --use-case chat --limit 3\n```\n\nValid use cases: `general`, `coding`, `reasoning`, `chat`, `multimodal`, `embedding`.\n\n### Filter by minimum fit level\n\n```bash\nllmfit recommend --json --min-fit good --limit 10\n```\n\nValid fit levels (best to worst): `perfect`, `good`, `marginal`.\n\n## Understanding the output\n\n### System JSON\n\n```json\n{\n  \"system\": {\n    \"cpu_name\": \"Apple M2 Max\",\n    \"cpu_cores\": 12,\n    \"total_ram_gb\": 32.0,\n    \"available_ram_gb\": 24.5,\n    \"has_gpu\": true,\n    \"gpu_name\": \"Apple M2 Max\",\n    \"gpu_vram_gb\": 32.0,\n    \"gpu_count\": 1,\n    \"backend\": \"Metal\",\n    \"unified_memory\": true\n  }\n}\n```\n\n### Recommendation JSON\n\nEach model in the `models` array includes:\n\n| Field | Meaning |\n|---|---|\n| `name` | HuggingFace model ID (e.g. `meta-llama/Llama-3.1-8B-Instruct`) |\n| `provider` | Model provider (Meta, Alibaba, Google, etc.) |\n| `params_b` | Parameter count in billions |\n| `score` | Composite score 0–100 (higher is better) |\n| `score_components` | Breakdown: `quality`, `speed`, `fit`, `context` (each 0–100) |\n| `fit_level` | `Perfect`, `Good`, `Marginal`, or `TooTight` |\n| `run_mode` | `GPU`, `CPU+GPU Offload`, or `CPU` |\n| `category` | Model category (e.g. `Reasoning`, `Coding`, `Chat`, `Embedding`) |\n| `is_moe` | Whether the model uses Mixture of Experts architecture |\n| `parameter_count` | Human-readable param count string (e.g. `\"7.6B\"`) |\n| `notes` | Array of human-readable notes about the recommendation |\n| `best_quant` | Optimal quantization for the hardware (e.g. `Q5_K_M`, `Q4_K_M`) |\n| `estimated_tps` | Estimated tokens per second |\n| `memory_required_gb` | VRAM/RAM needed at this quantization |\n| `memory_available_gb` | Available VRAM/RAM detected |\n| `utilization_pct` | How much of available memory the model uses |\n| `use_case` | What the model is designed for |\n| `context_length` | Maximum context window |\n\n### Fit levels explained\n\n- **Perfect**: Model fits comfortably with room to spare. Ideal choice.\n- **Good**: Model fits but uses most available memory. Will work well.\n- **Marginal**: Model barely fits. May work but expect slower performance or reduced context.\n- **TooTight**: Model does not fit. Do not recommend.\n\n### Run modes explained\n\n- **GPU**: Full GPU inference. Fastest. Model weights loaded entirely into VRAM.\n- **CPU+GPU Offload**: Some layers on GPU, rest in system RAM. Slower than pure GPU.\n- **CPU**: All inference on CPU using system RAM. Slowest but works without GPU.\n\n## Configuring OpenClaw with results\n\nAfter getting recommendations, configure the user's local model provider.\n\n### For Ollama\n\nMap the HuggingFace model name to its Ollama tag. Common mappings:\n\n| llmfit name | Ollama tag |\n|---|---|\n| `meta-llama/Llama-3.1-8B-Instruct` | `llama3.1:8b` |\n| `meta-llama/Llama-3.3-70B-Instruct` | `llama3.3:70b` |\n| `Qwen/Qwen2.5-Coder-7B-Instruct` | `qwen2.5-coder:7b` |\n| `Qwen/Qwen2.5-72B-Instruct` | `qwen2.5:72b` |\n| `deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct` | `deepseek-coder-v2:16b` |\n| `deepseek-ai/DeepSeek-R1-Distill-Qwen-32B` | `deepseek-r1:32b` |\n| `google/gemma-2-9b-it` | `gemma2:9b` |\n| `mistralai/Mistral-7B-Instruct-v0.3` | `mistral:7b` |\n| `microsoft/Phi-3-mini-4k-instruct` | `phi3:mini` |\n| `microsoft/Phi-4-mini-instruct` | `phi4-mini` |\n\nThen update `openclaw.json`:\n\n```json\n{\n  \"models\": {\n    \"providers\": {\n      \"ollama\": {\n        \"models\": [\"ollama/<ollama-tag>\"]\n      }\n    }\n  }\n}\n```\n\nAnd optionally set as default:\n\n```json\n{\n  \"agents\": {\n    \"defaults\": {\n      \"model\": {\n        \"primary\": \"ollama/<ollama-tag>\"\n      }\n    }\n  }\n}\n```\n\n### For vLLM / LM Studio\n\nUse the HuggingFace model name directly as the model identifier with the appropriate provider prefix (`vllm/` or `lmstudio/`).\n\n## Workflow example\n\nWhen a user asks \"what local models can I run?\":\n\n1. Run `llmfit --json system` to show hardware summary\n2. Run `llmfit recommend --json --limit 5` to get top picks\n3. Present the recommendations with scores and fit levels\n4. If the user wants to configure one, map it to the appropriate Ollama/vLLM/LM Studio tag\n5. Offer to update `openclaw.json` with the chosen model\n\nWhen a user asks for a specific use case like \"recommend a coding model\":\n\n1. Run `llmfit recommend --json --use-case coding --limit 3`\n2. Present the coding-specific recommendations\n3. Offer to pull via Ollama and configure\n\n## Notes\n\n- llmfit detects NVIDIA GPUs (via nvidia-smi), AMD GPUs (via rocm-smi), and Apple Silicon (unified memory).\n- Multi-GPU setups aggregate VRAM across cards automatically.\n- The `best_quant` field tells you the optimal quantization — higher quant (Q6_K, Q8_0) means better quality if VRAM allows.\n- Speed estimates (`estimated_tps`) are approximate and vary by hardware and quantization.\n- Models with `fit_level: \"TooTight\"` should never be recommended to users.\n","frontmatter":{"name":"llmfit-advisor","description":"Detect local hardware (RAM, CPU, GPU/VRAM) and recommend the best-fit local LLM models with optimal quantization, speed estimates, and fit scoring.","metadata":{"openclaw":{"emoji":"🧠","requires":{"bins":["llmfit"]},"install":[{"id":"brew","kind":"brew","formula":"llmfit","bins":["llmfit"],"label":"Install llmfit (brew)"},{"id":"cargo","kind":"node","bins":["llmfit"],"label":"Install llmfit (cargo install llmfit)"}]}}},"isInternal":false,"tokens":1987,"sizeBytes":6983}],"systemPromptSnippet":"<yakaai_skills repo=\"AlexsJones/llmfit\">\n<!-- File: AGENTS.md (Tokens: ~3199 | Category: root-instruction) -->\n# AGENTS.md\n\nInstructions for AI agents contributing to this codebase.\n\n---\n\n## Project overview\n\n`llmfit` is a Rust CLI/TUI tool that matches LLM models against local system hardware (RAM, CPU, GPU). It detects system specs, loads a model database from embedded JSON, scores each model's fit, and presents results in an interactive terminal UI or classic table output.\n\n## Language and toolchain\n\n- Rust, edition 2024.\n- Build with `cargo build`. Run with `cargo run`.\n- No nightly features required. Stable toolchain only.\n- Minimum supported Rust version: whatever edition 2024 requires (1.85+).\n\n## Architecture\n\n```\nllmfit-core/      Shared Rust library. It owns hardware detection, model data,\n                 fit analysis, planning, providers, benchmarks, quality checks,\n                 model updates, diagnostics, claims, and result sharing.\n\nllmfit-tui/       Main `llmfit` binary. It provides the CLI, ratatui TUI,\n                 Axum HTTP API, embedded Web dashboard, and stdio MCP server.\n                 `main.rs` parses all clap flags and selects an interface.\n\nllmfit-desktop/   Tauri desktop application. Tauri commands call llmfit-core\n                 for hardware detection, fit analysis, and Ollama downloads.\n\nllmfit-web/       React 18 and Vite dashboard. It calls `/api/v1/*` endpoints\n                 from llmfit-tui. The llmfit-tui build script embeds `dist/`.\n                 This directory is not a Cargo workspace member.\n\nllmfit-python/    Python package wrapper. Its wheel includes the compiled Rust\n                 binary. `python -m llmfit` forwards arguments to that binary.\n                 It does not expose llmfit-core through a native Python API.\n```\n\nThe Cargo workspace contains `llmfit-core`, `llmfit-tui`, and\n`llmfit-desktop`. The default members are `llmfit-core` and `llmfit-tui`.\n\nSource modules in `llmfit-core/src/`:\n\n- `analysis.rs`: Builds model-fit results. It marks installed models and applies\n  local, community, and measured benchmark calibration.\n- `bench.rs`: Runs throughput benchmarks against Ollama and OpenAI-compatible\n  endpoints. It also discovers available benchmark targets.\n- `benchmarks.rs`: Loads embedded and remote benchmark data. It builds measured\n  throughput indexes and hardware leaderboard queries.\n- `claim.rs`: Calculates model resource bounds. It renders Kubernetes DRA\n  `ResourceClaim` and `ResourceClaimTemplate` manifests.\n- `doctor.rs`: Collects installation, hardware, runtime, and model diagnostics.\n- `fit.rs`: Calculates fit level, run mode, runtime, quantization, score, and\n  estimated throughput.\n- `hardware.rs`: Detects RAM, CPU, GPUs, unified memory, clusters, and memory\n  bandwidth.\n- `hwprofile.rs`: Loads embedded and user hardware profiles (schema v1). Applies\n  capacity to `SystemSpecs` and bandwidth/efficiency overrides to `CalcConfig`.\n- `models.rs`: Defines model metadata. It loads embedded HF and ONNX catalogs,\n  custom models, and the update cache.\n- `plan.rs`: Estimates memory, throughput, run paths, and hardware upgrade needs\n  for a requested model setup.\n- `providers.rs`: Integrates Ollama, MLX, llama.cpp, Docker Model Runner,\n  LM Studio, vLLM, and RamaLama.\n- `quality.rs`: Runs response quality tests. It scores roles, builds routing\n  recommendations, and compares results with baselines.\n- `share.rs`: Stores local benchmark results. It handles GitHub authentication\n  and submits community benchmark data.\n- `task_bench.rs`: Provides task benchmark scores for model and task pairs.\n- `update.rs`: Fetches model metadata and manages the local model update cache.\n\nSource modules in `llmfit-tui/src/`:\n\n- `main.rs`: Owns CLI parsing, hardware overrides, command execution, and\n  interface dispatch.\n- `display.rs`: Renders classic CLI tables, model plans, JSON, and CSV output.\n- `download_history.rs`: Stores persistent model download records.\n- `events.rs`: Publishes optional NATS system events and periodic snapshots.\n- `filter_config.rs`: Loads and saves persistent TUI filter settings.\n- `mcp_server.rs`: Exposes hardware, model, runtime, and planning MCP tools.\n- `serve_api.rs`: Serves the embedded Web dashboard and JSON API with Axum.\n- `serve_shared.rs`: Converts shared core types into API and MCP JSON values.\n- `theme.rs`: Defines TUI color themes and stores the selected theme.\n- `tui_app.rs`: Owns TUI state, model results, filters, downloads, and selection.\n- `tui_events.rs`: Handles crossterm input and mutates TUI state.\n- `tui_ui.rs`: Renders TUI views, tables, details, plans, and popups with\n  ratatui.\n\n## Data flow\n\nAll interfaces use the same core analysis flow:\n\n1. `SystemSpecs::detect()` detects CPU, RAM, GPU, unified-memory, and cluster\n   information. CLI hardware overrides can replace detected values.\n2. `ModelDatabase::new()` loads the embedded HF and ONNX catalogs.\n3. Custom models replace matching embedded models. The update cache appends\n   models that are not already present.\n4. `build_model_fits()` removes backend-incompatible models. It calls\n   `ModelFit::analyze_with_forced_runtime()` for each remaining model.\n5. Fit analysis selects a runtime, quantization, and run mode. It calculates\n   memory use, throughput, fit level, score components, and notes.\n6. Local benchmark results, community results, and measured presets can replace\n   or calibrate formula-based throughput estimates.\n7. Each interface applies its own filters, sorting, limits, and presentation.\n\n`ModelFit::analyze()` is the default analysis wrapper. Use\n`analyze_with_context_limit()` for a context cap. Use\n`analyze_with_forced_runtime()` for runtime selection. Use\n`analyze_with_config()` for custom calculation parameters. These methods share\nthe private `analyze_inner()` implementation.\n\nInterface-specific flow:\n\n- CLI: `main.rs` dispatches a subcommand. The command calls llmfit-core and\n  writes a table, JSON, or CSV result.\n- TUI: `App` owns model and filter state. `tui_events` changes that state.\n  `apply_filters()` updates visible indices. `tui_ui` renders the current state.\n- Web: React calls `/api/v1/*`. Axum handlers in `serve_api.rs` call llmfit-core\n  and return JSON. The same server returns the embedded React assets.\n- MCP: `LlmfitMcpServer` receives stdio tool calls. Each tool calls shared core\n  analysis or planning logic and returns JSON text.\n- Desktop: Tauri commands call llmfit-core and serialize results for the desktop\n  UI. Ollama pull state stays in the Tauri application state.\n- Python: The Python entry point locates the installed `llmfit` binary. It then\n  replaces the process on Unix or starts a subprocess on Windows.\n\n## Model database\n\n- Source: `llmfit-core/data/hf_models.json` (33 models).\n- Generated by `scripts/scrape_hf_models.py` (Python, stdlib only, no pip deps).\n- Embedded at compile time via `include_str!(\"../data/hf_models.json\")`.\n- Schema per entry: name, provider, parameter_count, min_ram_gb, recommended_ram_gb, min_vram_gb, quantization, context_length, use_case.\n- `min_vram_gb` is VRAM needed for GPU inference. `min_ram_gb` is system RAM needed for CPU inference. Both are derived from the same parameter count.\n- RAM formula: `params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.2 overhead`.\n- VRAM formula: `params * 0.5 bytes (Q4_K_M) / 1024^3 * 1.1 activation overhead`.\n- Recommended RAM: `model_size * 2.0`.\n\nDo not manually edit `hf_models.json`. Regenerate it by running the scraper:\n\n```sh\npython3 scripts/scrape_hf_models.py\n```\n\nThe scraper has hardcoded fallback entries for gated models that require authentication.\n\n## Conventions\n\n- No `unsafe` code.\n- No `.unwrap()` on user-facing paths. Use proper error handling or `expect()` with a descriptive message for internal invariants only.\n- Fit levels are ordered: Perfect > Good > Marginal > TooTight. Do not add levels without updating `rank_models_by_fit()` sort logic.\n- Fit is VRAM-first. `RunMode` has five execution paths: `Gpu`, `MoeOffload`,\n  `CpuOffload`, `CpuOnly`, and `TensorParallel`.\n- `Gpu` keeps the model in VRAM. `MoeOffload` keeps active experts in VRAM and\n  inactive experts in RAM. `CpuOffload` splits work between VRAM and RAM.\n  `CpuOnly` uses system RAM. `TensorParallel` distributes work across nodes.\n- `min_vram_gb` is the VRAM needed to load model weights on GPU. `min_ram_gb` is the system RAM needed for CPU-only inference (same weights, loaded into RAM instead). They represent the same workload on different hardware paths.\n- On Apple Silicon (unified memory), VRAM = system RAM. The `CpuOffload` path is skipped because there is no separate RAM pool to spill to. `SystemSpecs::unified_memory` tracks this.\n- TUI rendering is stateless. `tui_ui::draw()` must not mutate `App`. Pass `&mut App` only for `TableState` widget requirements -- do not use it to change application state.\n- Event handling in `tui_events.rs` is the sole place that mutates `App` in the TUI loop.\n- Keep `display.rs` and `tui_*.rs` independent. The CLI path must work without initializing any TUI state.\n\n## Adding a new model to the database\n\n1. Add the model's HuggingFace repo ID to `TARGET_MODELS` in `scripts/scrape_hf_models.py`.\n2. If the model is gated (requires HF auth), add a fallback entry to the `FALLBACK` dict in the same script.\n3. Run `python3 scripts/scrape_hf_models.py`.\n4. Verify the output in `llmfit-core/data/hf_models.json`.\n5. Run `cargo build` to verify compilation.\n\n## Adding a new filter\n\n1. Add the filter state to `App` in `tui_app.rs`.\n2. Add filtering logic inside `apply_filters()`.\n3. Add the keybinding in `tui_events.rs` (Normal mode handler).\n4. Add the UI widget in `tui_ui.rs` (`draw_search_and_filters()` function).\n5. Update the status bar help text in `draw_status_bar()`.\n\n## Adding a new CLI subcommand\n\n1. Add a variant to the `Commands` enum in `main.rs`.\n2. Add the match arm in the `main()` function's command dispatch.\n3. Use `display.rs` functions for output, or add new ones as needed.\n\n## Testing\n\nThe project has Rust, Web, and Python test suites.\n\n- Rust unit tests live beside code in `llmfit-core/src/` and\n  `llmfit-tui/src/`.\n- Core integration tests in `llmfit-core/tests/` validate catalog schemas and\n  ONNX model data.\n- CLI integration tests in `llmfit-tui/tests/` use `assert_cmd` against the\n  compiled `llmfit` binary.\n- HTTP API tests exercise Axum routers and JSON responses in `serve_api.rs`.\n- TUI tests focus on state transitions, filters, event handling, and render\n  output. Keep production rendering stateless.\n- Web tests use Vitest, jsdom, and Testing Library. They cover API query\n  construction, localization, filtering, and dashboard interactions.\n- Python tests use pytest. They cover binary discovery, package versioning, and\n  invocation of the packaged Rust binary.\n\nRun the default Rust test set:\n\n```sh\ncargo test\n```\n\nRun all Rust workspace members, including the desktop crate:\n\n```sh\ncargo test --workspace\n```\n\nRun one Rust package:\n\n```sh\ncargo test -p llmfit-core\ncargo test -p llmfit\n```\n\nRun the Web tests:\n\n```sh\nnpm --prefix llmfit-web test\n```\n\nRun the Python tests and quality checks:\n\n```sh\nuv run --project llmfit-python pytest llmfit-python/tests\nmake -C llmfit-python check\n```\n\n## Dependencies policy\n\n- Prefer crates that are well-maintained and have minimal transitive dependencies.\n- `sysinfo` is the system detection crate. Do not replace it with raw platform calls.\n- `ureq` is the blocking HTTP client for providers, benchmarks, updates, quality\n  tests, and sharing. Do not add a second core HTTP client without a concrete need.\n- `which` locates installed runtime binaries. Keep runtime discovery in\n  `providers.rs` instead of adding manual `PATH` parsing.\n- `regex` supports response scoring and text parsing. `serde_yml` parses quality\n  test configuration. `base64` encodes benchmark submissions for GitHub.\n- `objc2-metal` reads the effective Metal working-set limit on macOS. Keep it a\n  macOS-only dependency. Do not replace it with raw platform calls.\n- `ratatui` + `crossterm` is the TUI stack. Do not mix in `termion` or `ncurses`.\n- `clap` with derive feature for CLI parsing. Do not use manual arg parsing.\n- The Python scraper uses only stdlib (`urllib`, `json`). Do not add pip dependencies.\n\n## Common tasks\n\n```sh\n# Build\ncargo build\n\n# Run TUI\ncargo run\n\n# Run CLI mode\ncargo run -- --cli\n\n# Run specific subcommand\ncargo run -- system\ncargo run -- fit --perfect -n 5\ncargo run -- search \"llama\"\n\n# Refresh model database\npython3 scripts/scrape_hf_models.py && cargo build\n\n# Check for compilation issues\ncargo check\n\n# Format code\ncargo fmt\n\n# Lint\ncargo clippy\n```\n\n## Platform notes\n\n- GPU detection shells out to `nvidia-smi` (NVIDIA) and `rocm-smi` (AMD). These are best-effort and fail silently if unavailable.\n- Apple Silicon detection uses `system_profiler SPDisplaysDataType`. On unified memory Macs, VRAM is reported as available system RAM (same pool).\n- `sysinfo` handles cross-platform RAM/CPU. No conditional compilation needed.\n- The TUI uses crossterm which works on Linux, macOS, and Windows terminals.\n\n<!-- File: API.md (Tokens: ~4843 | Category: project-spec) -->\n# llmfit REST API Guide\n\nThis document is for agent/client builders integrating with `llmfit serve`.\n\n## Purpose\n\n`llmfit serve` exposes node-local model fit analysis (same core data used by TUI/CLI) over HTTP and serves a local web dashboard.\n\nPrimary use case:\n- Query each node in a cluster for top runnable models.\n- Aggregate externally (scheduler/controller/UI) for placement decisions.\n\n## Start the server\n\n```sh\nllmfit serve --port 8787\n```\n\nGlobal flags still apply:\n\n```sh\nllmfit --memory 24G --ram 64G --cpu-cores 16 --max-context 8192 serve --port 8787\n```\n\nHardware overrides (`--memory`, `--ram`, `--cpu-cores`) are reflected in API responses, making the server report the overridden values instead of the detected hardware.\n\n## Base URL\n\nDefault local base URL:\n\n```text\nhttp://127.0.0.1:8787\n```\n\nTo expose outside localhost, pass `--host 0.0.0.0`.\n\n### Unix domain socket\n\nFor same-host consumers that should not touch the network at all (e.g. a\nsidecar in a `hostNetwork` Kubernetes pod, where a TCP bind would land on the\nnode's loopback), listen on a Unix socket instead:\n\n```sh\nllmfit serve --unix-socket /run/llmfit/llmfit.sock\n```\n\nThe socket is created with mode `0660`; a stale socket file from a previous\ninstance is replaced automatically. All HTTP endpoints are identical:\n\n```sh\ncurl --unix-socket /run/llmfit/llmfit.sock http://localhost/api/v1/system\n```\n\n`--unix-socket` conflicts with `--host`/`--port` and is unix-platforms only.\n\nIf you are building from source and want the dashboard embedded in `llmfit`, build web assets first:\n\n```sh\ncd llmfit-web && npm ci && npm run build\n```\n\n## Endpoints\n\n### `GET /`\nWeb dashboard entrypoint (same-origin UI for fit exploration).\n\n### `GET /health`\nLiveness probe.\n\nExample response:\n\n```json\n{\n  \"status\": \"ok\",\n  \"node\": {\n    \"name\": \"worker-1\",\n    \"os\": \"linux\"\n  }\n}\n```\n\n---\n\n### `GET /api/v1/system`\nReturns node identity + detected hardware.\n\nExample response shape:\n\n```json\n{\n  \"node\": {\n    \"name\": \"worker-1\",\n    \"os\": \"linux\"\n  },\n  \"system\": {\n    \"total_ram_gb\": 62.23,\n    \"available_ram_gb\": 41.08,\n    \"cpu_cores\": 14,\n    \"cpu_name\": \"Intel(R) Core(TM) Ultra 7 165U\",\n    \"has_gpu\": false,\n    \"gpu_vram_gb\": null,\n    \"gpu_available_gb\": null,\n    \"gpu_name\": null,\n    \"gpu_count\": 0,\n    \"unified_memory\": false,\n    \"backend\": \"CPU (x86)\",\n    \"gpus\": []\n  },\n  \"profile\": null\n}\n```\n\n`profile` is the name of the hardware profile the server was started with (`llmfit serve --profile <name>`), or `null` when it describes the machine it runs on.\n\n`gpu_available_gb` is the VRAM free right now, pooled across discrete GPUs (from `nvidia-smi memory.free` and amdgpu's `mem_info_vram_used`); each `gpus[]` entry carries its own `free_vram_gb`. Both are `null` when a backend does not report it (Intel, Windows, older drivers) or after a hardware override. On Apple Silicon `gpu_available_gb` is instead Metal's wiring cap for the unified pool. `plan` grades GPU run paths against the free figure when it is known and against total VRAM otherwise.\n\n---\n\n### `GET /api/v1/models`\nReturns filtered/sorted model-fit rows for this node.\n\nEnvelope shape:\n\n```json\n{\n  \"node\": { \"name\": \"worker-1\", \"os\": \"linux\" },\n  \"system\": { \"...\": \"...\" },\n  \"total_models\": 23,\n  \"returned_models\": 10,\n  \"filters\": { \"...\": \"echo of query state\" },\n  \"models\": [\n    {\n      \"name\": \"Qwen/Qwen2.5-Coder-7B-Instruct\",\n      \"provider\": \"Qwen\",\n      \"parameter_count\": \"7B\",\n      \"params_b\": 7.0,\n      \"context_length\": 32768,\n      \"usable_context\": 32768,\n      \"effective_context_length\": 8192,\n      \"use_case\": \"Coding\",\n      \"category\": \"Coding\",\n      \"release_date\": \"2025-03-14\",\n      \"is_moe\": false,\n      \"fit_level\": \"good\",\n      \"fit_label\": \"Good\",\n      \"run_mode\": \"gpu\",\n      \"run_mode_label\": \"GPU\",\n      \"score\": 86.5,\n      \"score_components\": {\n        \"quality\": 87.0,\n        \"speed\": 81.2,\n        \"fit\": 90.1,\n        \"context\": 88.0\n      },\n      \"estimated_tps\": 42.5,\n      \"estimate_confidence\": \"estimated\",\n      \"estimate_confidence_label\": \"estimated\",\n      \"prefill_tps\": 812.3,\n      \"ttft_ms\": 10.1,\n      \"runtime\": \"llamacpp\",\n      \"runtime_label\": \"llama.cpp\",\n      \"best_quant\": \"Q5_K_M\",\n      \"memory_required_gb\": 5.8,\n      \"memory_available_gb\": 12.0,\n      \"utilization_pct\": 48.3,\n      \"notes\": [],\n      \"gguf_sources\": [],\n      \"capabilities\": [\"tool_use\"],\n      \"capability_ids\": [\"tool_use\"],\n      \"license\": \"apache-2.0\",\n      \"supports_tp\": [1, 2, 4],\n      \"installed\": false,\n      \"disk_size_gb\": 5.1,\n      \"ollama_name\": \"qwen2.5-coder:7b-instruct\",\n      \"estimate_basis\": {\n        \"method\": \"roofline\",\n        \"gpu_bandwidth_gbps\": 320.0,\n        \"ddr_bandwidth_gbps\": null,\n        \"local_calibration\": null,\n        \"efficiency\": 0.85,\n        \"assumed_context\": 8192\n      },\n      \"verify_command\": \"llama-bench -m <path-to-Q5_K_M-gguf> -ngl 99 -p 512 -n 128\",\n      \"measured_tps\": null\n    }\n  ]\n}\n```\n\nThe three context fields answer different questions:\n\n- `context_length` — the model's native window, as advertised upstream.\n- `usable_context` — how much of that window actually fits in this node's\n  available memory alongside the weights. Use this one to pick a runtime `-c`.\n- `effective_context_length` — the context the `memory_required_gb` and\n  `estimated_tps` figures on this row were computed at. Defaults to\n  `min(context_length, 8192)`; set by `max_context` when supplied.\n\nThe envelope also carries these fields, now at parity with `llmfit fit --json`\n(both frontends serialize through one shared function):\n\n- `installed` — whether the model was found in a local runtime provider. `llmfit serve` probes providers in the background (at startup, after a finished download, and when a request finds the result older than 30 seconds), so a model pulled or removed outside the server shows up within one refresh after that; it is `false` everywhere until the first probe finishes.\n- `disk_size_gb` — estimated on-disk size at `best_quant`.\n- `capability_ids` — machine-readable capability ids (snake_case); mirrors\n  `capabilities` here. Note `llmfit fit --json` overloads its `capabilities`\n  key with human labels (e.g. `\"Tool Use\"`) — that overload is CLI-only and\n  slated for deprecation.\n- `ollama_name` — the `ollama pull` tag for this model, when derivable.\n- `estimate_basis` — how `memory_required_gb`/`estimated_tps` were derived\n  (bandwidths, efficiency, assumed context), for reproducibility.\n- `estimate_confidence` / `estimate_confidence_label` — how much to trust\n  `estimated_tps`, most to least trustworthy: `measured_local` (your own\n  `llmfit bench` runs), `measured_community` (llmfit community submissions\n  or localmaxxing.com data on matching hardware), `calibrated` (a formula\n  estimate scaled by a correction factor from benchmark runs on this exact\n  hardware), `estimated` (a bare formula estimate), `unsupported` (no\n  estimate — the model needs a runtime llmfit can't model). See\n  `measured_tps`/`estimate_basis.local_calibration` for the data each level\n  is derived from.\n- `prefill_tps` / `ttft_ms` — estimated prompt-processing throughput\n  (tok/s) and time-to-first-token (ms) for a prompt of\n  `effective_context_length` tokens. Prefill is compute-bound, not\n  bandwidth-bound, so both are `null` — not `0.0` — unless the system's GPU\n  compute throughput is known. A `null` here means \"not estimated\", never\n  \"instant\" or \"stalled\".\n- `verify_command` — a `llama-bench` invocation measuring the same throughput\n  this row estimates (llama.cpp GPU / CPU-only runs; `null` otherwise).\n- `measured_tps` — a recorded benchmark result if one exists, else `null`.\n\nNote on vocabulary: `fit_level`, `run_mode`, and `runtime` here are stable\nmachine codes (e.g. `\"good\"`, `\"gpu\"`, `\"llamacpp\"`), with the human string\nunder the paired `*_label` key. `llmfit fit --json` emits the human string\ndirectly under those same keys — a CLI-only legacy overload.\n`estimate_confidence` follows the same convention (machine code, paired\n`estimate_confidence_label`) in both frontends.\n\n### A note on `best_quant`\n\n`best_quant` is normally the llama.cpp/GGUF quant llmfit picked for this\nhardware (e.g. `\"Q5_K_M\"`). It is `null` when the model's own repo name\ndeclares a native low-precision format (NVFP4/MXFP4) that a GGUF quant\nlabel doesn't apply to — the row's `notes` array carries the explanation\nin that case. It's never a GGUF label misattributed to a non-GGUF repo.\n\n---\n\n### Catalog sanitization\n\n`/api/v1/models` never returns speculative-decoding draft heads\n(EAGLE/DFlash/DSpark naming), entries whose name implies a wildly different\nparameter count than their declared size, or other catalog rows llmfit\ncan't score honestly — they're demoted out of fit ranking, not deleted, so\nthis is invisible unless you were expecting a specific draft-head repo to\nshow up as a standalone model. If you maintain your own filtering on top of\n`llmfit fit --json` output today, you can likely delete that workaround.\n\n---\n\n### `GET /api/v1/models/top`\nKey scheduling endpoint. Same schema as `/api/v1/models`, but defaults to top 5 runnable entries.\n\nImportant behavior:\n- Defaults `limit=5`.\n- Excludes `too_tight` rows unless explicitly overridden (and top endpoint still keeps runnable semantics).\n\n---\n\n### `GET /api/v1/models/{name}`\nPath-constrained search. Equivalent to a text search scoped by `{name}`.\n\nUseful for:\n- Client-side drilldown after selecting a model family.\n\n---\n\n### `GET /api/v1/concurrency`\nConcurrent-session capacity for one model: the same estimate as\n`llmfit concurrency <model> --json`. It is a memory-capacity ceiling (how many\nsessions fit resident at once), not a throughput figure under load.\n\nQuery parameters:\n- `model` (required): exact catalog name, case-insensitive.\n- `quant`: weight quantization (e.g. `Q4_K_M`); defaults to the model's best fit.\n- `kv_quant`: `fp16|fp8|q8_0|q4_0|tq` (default `fp16`).\n- `context`: report one context instead of the 4k-256k ladder (tokens).\n- `users`: also report the largest context that fits this many sessions.\n- `ram_gb`, `vram_gb`, `cpu_cores`: hardware overrides, as on `/api/v1/system`.\n\n```json\n{\n  \"model\": \"Qwen/Qwen2.5-7B-Instruct\",\n  \"run_mode\": \"gpu\",\n  \"fit_level\": \"perfect\",\n  \"target_users\": 8,\n  \"max_context_for_target\": 32768,\n  \"estimate\": {\n    \"pool_gb\": 24.0,\n    \"weights_resident_gb\": 5.1,\n    \"kv_budget_gb\": 18.9,\n    \"quant\": \"Q4_K_M\",\n    \"kv_quant\": \"fp16\",\n    \"native_context\": 32768,\n    \"ladder\": [\n      { \"requested_context\": 4096, \"effective_context\": 4096, \"clamped\": false, \"per_session_kv_gb\": 0.22, \"max_sessions\": 85 }\n    ]\n  }\n}\n```\n\n`run_mode` and `fit_level` use the same codes as `/api/v1/models`.\n\n---\n\n### `GET /api/v1/storage`\nDisk needed to keep a library of runnable models, used one at a time: the same\nestimate as `llmfit storage --json`, returned as `{ \"system\": ..., \"storage\": ... }`.\n\nQuery parameters (defaults match the CLI):\n- `keep`: number of distinct runnable models to keep (default `3`).\n- `selection`: `score` (highest-ranked, default) or `largest` (largest weight storage).\n- `os_reserve`: space for the OS, apps and other files (default `100G`; decimal GB, `GiB`/`TiB` are binary).\n- `scratch`: `auto` (default, the largest selected model) or a size; `0` disables it.\n- `headroom`: percent of the suggested SSD to keep free, `0`-`99` (default `15`).\n- `perfect`: `true` to select only Perfect fits (otherwise Perfect, Good and Marginal).\n- `search`: narrow the candidates by name, provider or parameter size.\n- `max_context`, `ram_gb`, `vram_gb`, `cpu_cores`: as on `/api/v1/models`.\n\nThe `storage` object carries `models` (name, best quant, fit level, runtime,\nscore, disk size, context), `library_gb`, `download_scratch_gb`, `need_gb`,\n`target_capacity_gb`, `minimum_ssd_gb`, `suggested_ssd_gb` and `warnings`.\n\n## Query parameters\n\nSupported on `/api/v1/models` and `/api/v1/models/top` (also `/api/v1/models/{name}`):\n\n- `limit` (or alias `n`): max rows returned.\n- `perfect`: `true|false` (when `true`, only perfect fits).\n- `min_fit`: `perfect|good|marginal|too_tight`.\n- `runtime`: `any|mlx|llamacpp|vllm|bitnetcpp`.\n- `use_case`: `general|coding|reasoning|chat|multimodal|embedding`.\n- `provider`: provider substring filter.\n- `search`: free-text filter (name/provider/params/use-case/category).\n- `sort`: `score|tps|params|mem|ctx|date|use_case`.\n- `include_too_tight`: `true` includes unrunnable (`too_tight`) rows even when `min_fit` is not given; `false` drops them even with `min_fit=too_tight`. With neither parameter, `/models` returns rows from `marginal` up; `/models/top` never returns `too_tight` rows.\n- `max_context`: per-request context cap used by memory estimation.\n- `force_runtime`: `mlx|llamacpp|vllm|bitnetcpp` — override automatic runtime selection during analysis (e.g. get llama.cpp recommendations on Apple Silicon instead of MLX).\n\n## Error handling\n\nInvalid filter values return HTTP 400:\n\n```json\n{\n  \"error\": \"invalid min_fit value: use perfect|good|marginal|too_tight\"\n}\n```\n\nServer errors return HTTP 500 with `{\"error\": \"...\"}`.\n\n## Client integration recommendations\n\n### 1) Polling pattern for schedulers\nFor each node agent:\n1. Call `/health`.\n2. Call `/api/v1/system`.\n3. Call `/api/v1/models/top?limit=K&min_fit=good`.\n4. Attach node metadata and forward to your central scheduler.\n\n### 2) Conservative placement defaults\nFor production placement, prefer:\n\n```text\nmin_fit=good\ninclude_too_tight=false\nsort=score\nlimit=5..20\n```\n\n### 3) Per-workload targeting\nExamples:\n- Coding workloads: `use_case=coding`\n- Embedding workloads: `use_case=embedding`\n- Runtime constrained to llama.cpp fleet: `runtime=llamacpp`\n\n### 4) Stable parsing\nTreat unknown fields as forward-compatible additions:\n- Parse required fields you depend on.\n- Ignore unknown fields.\n\n## Curl examples\n\n```sh\ncurl http://127.0.0.1:8787/health\ncurl http://127.0.0.1:8787/api/v1/system\ncurl \"http://127.0.0.1:8787/api/v1/models?limit=20&min_fit=marginal&sort=score\"\ncurl \"http://127.0.0.1:8787/api/v1/models/top?limit=5&min_fit=good&use_case=coding\"\ncurl \"http://127.0.0.1:8787/api/v1/models/Mistral?runtime=any\"\ncurl \"http://127.0.0.1:8787/api/v1/concurrency?model=Qwen/Qwen2.5-7B-Instruct&users=8\"\ncurl \"http://127.0.0.1:8787/api/v1/storage?keep=5&selection=largest&os_reserve=200G\"\n```\n\n---\n\n## MCP Server Mode\n\nllmfit can run as an MCP (Model Context Protocol) server over stdio, making it discoverable by AI agents (Claude, Cursor, etc.).\n\n### Start the MCP server\n\n```sh\nllmfit serve --mcp\n```\n\nGlobal hardware overrides still apply:\n\n```sh\nllmfit --memory 24G --ram 64G serve --mcp\n```\n\n### MCP client configuration\n\nAdd to your MCP client config (e.g. `claude_desktop_config.json`):\n\n```json\n{\n  \"mcpServers\": {\n    \"llmfit\": {\n      \"command\": \"llmfit\",\n      \"args\": [\"serve\", \"--mcp\"]\n    }\n  }\n}\n```\n\n### Available tools\n\n| Tool | Description | Parameters |\n|------|-------------|------------|\n| `get_system_specs` | Node hardware info (RAM, GPU, CPU) | None |\n| `recommend_models` | Top models for this hardware | `limit?`, `use_case?`, `min_fit?`, `runtime?`, `license?`, `sort?` |\n| `search_models` | Free-text model search | `query`, `limit?` |\n| `plan_hardware` | Hardware requirements for a model | `model`, `context?`, `quant?`, `target_tps?` |\n| `get_runtimes` | Installed inference runtimes | None |\n| `get_installed_models` | Models in local runtimes | None |\n\n`plan_hardware` and `POST /api/v1/plan` return the shared plan estimate,\nincluding `disk_size_gb`: estimated weight storage in decimal GB at the\nresolved `quantization`. This excludes KV cache, inference buffers, and\ndownload scratch. `llmfit plan --json` returns the same plan fields.\n\n---\n\n## NATS Event Publishing\n\nWhen built with the `nats` feature, llmfit can publish hardware and model events to NATS for integration with coordination systems (e.g. Sympozium membrane).\n\n### Build with NATS support\n\n```sh\ncargo build --features nats\n```\n\n### Enable event publishing\n\n```sh\nllmfit serve --send-events --nats-url nats://localhost:4222\nllmfit serve --mcp --send-events  # also works with MCP mode\n```\n\nThe `NATS_URL` environment variable is also supported.\n\n### Event subjects\n\nEvents are published to `llmfit.{event_type}.{hostname}`:\n\n| Subject | Trigger | Payload |\n|---------|---------|---------|\n| `llmfit.system.{hostname}` | Startup + every 60s | System hardware specs |\n| `llmfit.fit.{hostname}` | After fit analysis | Model fit summary |\n| `llmfit.plan.{hostname}` | After plan estimate | Plan estimate |\n| `llmfit.runtimes.{hostname}` | Startup + on query | Runtime availability |\n| `llmfit.installed.{hostname}` | Startup + on query | Installed models |\n\n### Event envelope\n\nAll events are wrapped in a common envelope:\n\n```json\n{\n  \"timestamp\": \"1747058400\",\n  \"hostname\": \"worker-1\",\n  \"event_type\": \"system\",\n  \"version\": \"1\",\n  \"data\": { ... }\n}\n```\n\n### Subscribe to events\n\n```sh\nnats sub 'llmfit.>'                    # all events from all nodes\nnats sub 'llmfit.system.>'             # system specs from all nodes\nnats sub 'llmfit.system.worker-1'      # system specs from specific node\n```\n\n---\n\n## Versioning notes\n\nCurrent API prefix is `v1`.\n\nIf you build long-lived clients, pin to `/api/v1/...` and validate behavior with the local test script in `scripts/test_api.py`.\n\n<!-- File: skills/llmfit-advisor/SKILL.md (Tokens: ~1987 | Category: anthropic-skill) -->\n---\nname: llmfit-advisor\ndescription: Detect local hardware (RAM, CPU, GPU/VRAM) and recommend the best-fit local LLM models with optimal quantization, speed estimates, and fit scoring.\nmetadata:\n  {\n    \"openclaw\":\n      {\n        \"emoji\": \"🧠\",\n        \"requires\": { \"bins\": [\"llmfit\"] },\n        \"install\":\n          [\n            {\n              \"id\": \"brew\",\n              \"kind\": \"brew\",\n              \"formula\": \"llmfit\",\n              \"bins\": [\"llmfit\"],\n              \"label\": \"Install llmfit (brew)\",\n            },\n            {\n              \"id\": \"cargo\",\n              \"kind\": \"node\",\n              \"bins\": [\"llmfit\"],\n              \"label\": \"Install llmfit (cargo install llmfit)\",\n            },\n          ],\n      },\n  }\n---\n\n# llmfit-advisor\n\nHardware-aware local LLM advisor. Detects your system specs (RAM, CPU, GPU/VRAM) and recommends models that actually fit, with optimal quantization and speed estimates.\n\n## When to use (trigger phrases)\n\nUse this skill immediately when the user asks any of:\n\n- \"what local models can I run?\"\n- \"which LLMs fit my hardware?\"\n- \"recommend a local model\"\n- \"what's the best model for my GPU?\"\n- \"can I run Llama 70B locally?\"\n- \"configure local models\"\n- \"set up Ollama models\"\n- \"what models fit my VRAM?\"\n- \"help me pick a local model for coding\"\n\nAlso use this skill when:\n\n- The user wants to configure `models.providers.ollama` or `models.providers.lmstudio`\n- The user mentions running models locally and you need to know what fits\n- A model recommendation is needed and the user has local inference capability (Ollama, vLLM, LM Studio)\n\n## Quick start\n\n### Detect hardware\n\n```bash\nllmfit --json system\n```\n\nReturns JSON with CPU, RAM, GPU name, VRAM, multi-GPU info, and whether memory is unified (Apple Silicon).\n\n### Get top recommendations\n\n```bash\nllmfit recommend --json --limit 5\n```\n\nReturns the top 5 models ranked by a composite score (quality, speed, fit, context) with optimal quantization for the detected hardware.\n\n### Filter by use case\n\n```bash\nllmfit recommend --json --use-case coding --limit 3\nllmfit recommend --json --use-case reasoning --limit 3\nllmfit recommend --json --use-case chat --limit 3\n```\n\nValid use cases: `general`, `coding`, `reasoning`, `chat`, `multimodal`, `embedding`.\n\n### Filter by minimum fit level\n\n```bash\nllmfit recommend --json --min-fit good --limit 10\n```\n\nValid fit levels (best to worst): `perfect`, `good`, `marginal`.\n\n## Understanding the output\n\n### System JSON\n\n```json\n{\n  \"system\": {\n    \"cpu_name\": \"Apple M2 Max\",\n    \"cpu_cores\": 12,\n    \"total_ram_gb\": 32.0,\n    \"available_ram_gb\": 24.5,\n    \"has_gpu\": true,\n    \"gpu_name\": \"Apple M2 Max\",\n    \"gpu_vram_gb\": 32.0,\n    \"gpu_count\": 1,\n    \"backend\": \"Metal\",\n    \"unified_memory\": true\n  }\n}\n```\n\n### Recommendation JSON\n\nEach model in the `models` array includes:\n\n| Field | Meaning |\n|---|---|\n| `name` | HuggingFace model ID (e.g. `meta-llama/Llama-3.1-8B-Instruct`) |\n| `provider` | Model provider (Meta, Alibaba, Google, etc.) |\n| `params_b` | Parameter count in billions |\n| `score` | Composite score 0–100 (higher is better) |\n| `score_components` | Breakdown: `quality`, `speed`, `fit`, `context` (each 0–100) |\n| `fit_level` | `Perfect`, `Good`, `Marginal`, or `TooTight` |\n| `run_mode` | `GPU`, `CPU+GPU Offload`, or `CPU` |\n| `category` | Model category (e.g. `Reasoning`, `Coding`, `Chat`, `Embedding`) |\n| `is_moe` | Whether the model uses Mixture of Experts architecture |\n| `parameter_count` | Human-readable param count string (e.g. `\"7.6B\"`) |\n| `notes` | Array of human-readable notes about the recommendation |\n| `best_quant` | Optimal quantization for the hardware (e.g. `Q5_K_M`, `Q4_K_M`) |\n| `estimated_tps` | Estimated tokens per second |\n| `memory_required_gb` | VRAM/RAM needed at this quantization |\n| `memory_available_gb` | Available VRAM/RAM detected |\n| `utilization_pct` | How much of available memory the model uses |\n| `use_case` | What the model is designed for |\n| `context_length` | Maximum context window |\n\n### Fit levels explained\n\n- **Perfect**: Model fits comfortably with room to spare. Ideal choice.\n- **Good**: Model fits but uses most available memory. Will work well.\n- **Marginal**: Model barely fits. May work but expect slower performance or reduced context.\n- **TooTight**: Model does not fit. Do not recommend.\n\n### Run modes explained\n\n- **GPU**: Full GPU inference. Fastest. Model weights loaded entirely into VRAM.\n- **CPU+GPU Offload**: Some layers on GPU, rest in system RAM. Slower than pure GPU.\n- **CPU**: All inference on CPU using system RAM. Slowest but works without GPU.\n\n## Configuring OpenClaw with results\n\nAfter getting recommendations, configure the user's local model provider.\n\n### For Ollama\n\nMap the HuggingFace model name to its Ollama tag. Common mappings:\n\n| llmfit name | Ollama tag |\n|---|---|\n| `meta-llama/Llama-3.1-8B-Instruct` | `llama3.1:8b` |\n| `meta-llama/Llama-3.3-70B-Instruct` | `llama3.3:70b` |\n| `Qwen/Qwen2.5-Coder-7B-Instruct` | `qwen2.5-coder:7b` |\n| `Qwen/Qwen2.5-72B-Instruct` | `qwen2.5:72b` |\n| `deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct` | `deepseek-coder-v2:16b` |\n| `deepseek-ai/DeepSeek-R1-Distill-Qwen-32B` | `deepseek-r1:32b` |\n| `google/gemma-2-9b-it` | `gemma2:9b` |\n| `mistralai/Mistral-7B-Instruct-v0.3` | `mistral:7b` |\n| `microsoft/Phi-3-mini-4k-instruct` | `phi3:mini` |\n| `microsoft/Phi-4-mini-instruct` | `phi4-mini` |\n\nThen update `openclaw.json`:\n\n```json\n{\n  \"models\": {\n    \"providers\": {\n      \"ollama\": {\n        \"models\": [\"ollama/<ollama-tag>\"]\n      }\n    }\n  }\n}\n```\n\nAnd optionally set as default:\n\n```json\n{\n  \"agents\": {\n    \"defaults\": {\n      \"model\": {\n        \"primary\": \"ollama/<ollama-tag>\"\n      }\n    }\n  }\n}\n```\n\n### For vLLM / LM Studio\n\nUse the HuggingFace model name directly as the model identifier with the appropriate provider prefix (`vllm/` or `lmstudio/`).\n\n## Workflow example\n\nWhen a user asks \"what local models can I run?\":\n\n1. Run `llmfit --json system` to show hardware summary\n2. Run `llmfit recommend --json --limit 5` to get top picks\n3. Present the recommendations with scores and fit levels\n4. If the user wants to configure one, map it to the appropriate Ollama/vLLM/LM Studio tag\n5. Offer to update `openclaw.json` with the chosen model\n\nWhen a user asks for a specific use case like \"recommend a coding model\":\n\n1. Run `llmfit recommend --json --use-case coding --limit 3`\n2. Present the coding-specific recommendations\n3. Offer to pull via Ollama and configure\n\n## Notes\n\n- llmfit detects NVIDIA GPUs (via nvidia-smi), AMD GPUs (via rocm-smi), and Apple Silicon (unified memory).\n- Multi-GPU setups aggregate VRAM across cards automatically.\n- The `best_quant` field tells you the optimal quantization — higher quant (Q6_K, Q8_0) means better quality if VRAM allows.\n- Speed estimates (`estimated_tps`) are approximate and vary by hardware and quantization.\n- Models with `fit_level: \"TooTight\"` should never be recommended to users.\n</yakaai_skills>"}