{"owner":"JustVugg","repo":"colibri","hasSkills":true,"totalSkillsCount":5,"totalTokensCount":15142,"categories":["plugin-manifest","anthropic-skill","project-spec"],"hasMcp":false,"mcpConfig":null,"found":["c/tools/expert_atlas/README.md","c/tools/iq3xxs_grid.json","c/tools/expert_atlas/probes.json","c/tools/README.md","docs/api.md"],"skills":{"c/tools/expert_atlas/README.md":"# Expert Atlas — what does each of the 19,456 experts actually do?\n\nProbe harness for #175. Runs a set of topic-tagged prompts, dumps each run's expert-routing\nhistogram, and turns them into a per-expert topic-affinity vector.\n\n```bash\ncd c\nexport COLI_MODEL=/path/to/glm52_i4\n./tools/expert_atlas/sweep.sh # 30 probes (10 topics x 3 prompts)\npython3 tools/expert_atlas/analyze.py --stats atlas_out/stats --out atlas_out/experts.json \\\n --web web/dist/experts.json # optional: feed the web dashboard Atlas\npython3 tools/expert_atlas/validate.py atlas_out/stats 200 # leave-one-prompt-out check\n```\n\n`--web` writes the same atlas in the shape the web dashboard consumes (the Atlas galaxy and the\nBrain hover tooltips): keyed `\"layer:expert\"` with `affinity`/`entropy`/`top`/`label`. It replaces\nthe retired `tools/expert_atlas.py`, whose API-driven probing ran through a live server and was\nexposed to exactly the traps above (server-side `--topp`, speculative drafts, shared `.coli_usage`).\n\n## Read this before you trust any atlas\n\nFour things silently corrupt this measurement. The sweep script controls all of them; if you\nroll your own, don't skip them.\n\n| trap | effect | control |\n|---|---|---|\n| **`--topp`** | prunes experts by cumulative probability — measured: it hides **38% of the distinct experts** (7,587 → 4,687). It is also the *recommended speed setting*. | `TOPP=0` |\n| **speculative drafts** | `eusage` is incremented inside `moe()`, *before* verification, so **rejected** drafts count. Those are experts routed for text the model never emitted. | `MTP=0 DRAFT=0` |\n| **`.coli_usage`** | is loaded at startup and accumulates, so a naive `STATS` dump contains **all prior history**, not this run. | remove per run (script backs it up and restores) |\n| **autocorrelation** | routing inside one run is highly correlated — the same context routes to the same experts token after token. An expert firing 38 times during one prompt is **one** observation, not 38. Chi-square/entropy on raw selections will certify single-prompt flukes as perfect specialists. | `analyze.py` requires the affinity to **replicate across a category's independent prompts** |\n\nThe CUDA expert tier is also not run-to-run deterministic, so the sweep uses `--gpu none`. Tier\nconfig only decides *where weights live*, not what the router picks, so this costs nothing.\n\n## Method\n\n`analyze.py`:\n\n1. `n[e][c]` — selections of expert *e* while running category *c*\n2. `f[e][c] = n[e][c] / N[c]` — normalise by **category size** (prefill routes the prompt too, so a\n verbose category would otherwise look busier)\n3. `p(c|e)` — renormalise into a topic distribution per expert, i.e. base-rate corrected. Ranking\n by raw count instead just rediscovers which experts are popular in general.\n4. `spec(e) = 1 − H(p(c|e)) / log C` — 0 = generalist, 1 = fires for exactly one topic\n5. **replication gate** — an expert is only a candidate specialist for *c* if it fires in ≥2 of *c*'s\n independent prompts\n\n`validate.py` — leave-one-prompt-out. Build each category's top-K specialist set from its *other*\nprompts, then check which set the held-out prompt's routing actually lands in. If specialisation\nwere an artifact of prompt wording, the held-out prompt would not prefer its own category.\n\n## Result on GLM-5.2 744B int4 (Zen5, CPU routing path)\n\n- **Leave-one-prompt-out: 29/30 = 96.7%** (chance 10%). Specialisation is a property of the topic,\n not of prompt wording.\n- The single miss is instructive: `写一首关于秋天的短诗` (\"write a short poem about autumn\") is\n classified **poetry**, not Chinese — routing follows the **task** over the **language**.\n- **Only 7.9% of experts are strong specialists** (spec ≥ 0.5). The \"one expert = one topic\"\n picture is wrong for ~92% of them.\n- Specialisation **rises with depth**: layer 3 ≈ 0.07 (generalist, token/syntax level) → layers\n 18–58 ≈ 0.19–0.27.\n- The replication gate removed **587** experts that looked like flawless specialists on one prompt.\n\n## Extending it\n\n`probes.json` is the whole probe set — add categories and prompts. Use **3+ prompts per category\nwith varied phrasing**, or the replication gate has nothing to check and you are back to measuring\none prompt. Keep prompt lengths in a similar band across categories.\n","c/tools/iq3xxs_grid.json":"[[4, 4, 4, 4], [20, 4, 4, 4], [36, 4, 4, 4], [12, 12, 4, 4], [28, 12, 4, 4], [62, 12, 4, 4], [4, 20, 4, 4], [20, 20, 4, 4], [12, 28, 4, 4], [20, 36, 4, 4], [28, 62, 4, 4], [44, 62, 4, 4], [12, 4, 12, 4], [28, 4, 12, 4], [4, 12, 12, 4], [20, 12, 12, 4], [12, 20, 12, 4], [44, 20, 12, 4], [4, 28, 12, 4], [20, 28, 12, 4], [12, 36, 12, 4], [36, 44, 12, 4], [4, 62, 12, 4], [4, 4, 20, 4], [20, 4, 20, 4], [36, 4, 20, 4], [12, 12, 20, 4], [4, 20, 20, 4], [20, 20, 20, 4], [12, 28, 20, 4], [28, 28, 20, 4], [62, 28, 20, 4], [12, 44, 20, 4], [62, 44, 20, 4], [44, 62, 20, 4], [12, 4, 28, 4], [62, 4, 28, 4], [4, 12, 28, 4], [20, 12, 28, 4], [44, 20, 28, 4], [4, 62, 28, 4], [28, 12, 36, 4], [62, 28, 36, 4], [36, 36, 36, 4], [62, 44, 36, 4], [28, 62, 36, 4], [44, 62, 36, 4], [12, 4, 44, 4], [62, 4, 44, 4], [20, 28, 44, 4], [20, 44, 44, 4], [44, 28, 52, 4], [36, 52, 52, 4], [4, 12, 62, 4], [36, 12, 62, 4], [52, 12, 62, 4], [28, 36, 62, 4], [12, 52, 62, 4], [12, 4, 4, 12], [28, 4, 4, 12], [4, 12, 4, 12], [20, 12, 4, 12], [12, 20, 4, 12], [28, 20, 4, 12], [4, 28, 4, 12], [20, 28, 4, 12], [36, 28, 4, 12], [62, 36, 4, 12], [4, 44, 4, 12], [4, 4, 12, 12], [20, 4, 12, 12], [12, 12, 12, 12], [4, 20, 12, 12], [20, 20, 12, 12], [12, 4, 20, 12], [28, 4, 20, 12], [4, 12, 20, 12], [20, 12, 20, 12], [12, 20, 20, 12], [4, 28, 20, 12], [20, 62, 20, 12], [4, 4, 28, 12], [20, 4, 28, 12], [4, 20, 28, 12], [12, 28, 28, 12], [52, 36, 28, 12], [52, 52, 28, 12], [12, 4, 36, 12], [44, 4, 36, 12], [4, 44, 36, 12], [4, 20, 44, 12], [36, 20, 44, 12], [52, 36, 44, 12], [12, 62, 44, 12], [44, 4, 52, 12], [20, 20, 62, 12], [4, 36, 62, 12], [4, 4, 4, 20], [20, 4, 4, 20], [12, 12, 4, 20], [28, 12, 4, 20], [4, 20, 4, 20], [20, 20, 4, 20], [52, 20, 4, 20], [12, 28, 4, 20], [20, 36, 4, 20], [12, 4, 12, 20], [28, 4, 12, 20], [44, 4, 12, 20], [4, 12, 12, 20], [20, 12, 12, 20], [12, 20, 12, 20], [4, 28, 12, 20], [28, 52, 12, 20], [62, 52, 12, 20], [4, 62, 12, 20], [4, 4, 20, 20], [20, 4, 20, 20], [12, 12, 20, 20], [62, 12, 20, 20], [4, 20, 20, 20], [20, 20, 20, 20], [62, 28, 20, 20], [4, 36, 20, 20], [44, 44, 20, 20], [12, 4, 28, 20], [4, 12, 28, 20], [36, 12, 28, 20], [4, 62, 28, 20], [36, 62, 28, 20], [44, 28, 36, 20], [28, 44, 36, 20], [28, 4, 44, 20], [62, 20, 44, 20], [12, 36, 44, 20], [36, 62, 44, 20], [12, 4, 62, 20], [28, 4, 62, 20], [52, 12, 62, 20], [44, 36, 62, 20], [12, 4, 4, 28], [4, 12, 4, 28], [20, 12, 4, 28], [12, 20, 4, 28], [28, 20, 4, 28], [4, 44, 4, 28], [44, 52, 4, 28], [20, 62, 4, 28], [4, 4, 12, 28], [20, 4, 12, 28], [4, 20, 12, 28], [12, 28, 12, 28], [36, 36, 12, 28], [52, 36, 12, 28], [12, 4, 20, 28], [28, 4, 20, 28], [4, 12, 20, 28], [44, 20, 20, 28], [20, 44, 20, 28], [20, 62, 20, 28], [12, 12, 28, 28], [28, 28, 28, 28], [4, 28, 36, 28], [62, 36, 36, 28], [20, 62, 36, 28], [4, 4, 44, 28], [52, 4, 44, 28], [20, 20, 44, 28], [44, 44, 44, 28], [36, 12, 52, 28], [52, 28, 52, 28], [28, 52, 52, 28], [28, 28, 62, 28], [4, 52, 62, 28], [36, 4, 4, 36], [62, 12, 4, 36], [44, 28, 4, 36], [62, 28, 4, 36], [28, 44, 4, 36], [62, 44, 4, 36], [36, 62, 12, 36], [4, 20, 20, 36], [62, 28, 20, 36], [4, 36, 20, 36], [4, 52, 20, 36], [52, 52, 20, 36], [62, 4, 28, 36], [44, 36, 28, 36], [36, 4, 36, 36], [12, 44, 36, 36], [36, 52, 36, 36], [44, 20, 44, 36], [28, 36, 44, 36], [4, 62, 44, 36], [44, 4, 62, 36], [4, 12, 62, 36], [20, 12, 62, 36], [4, 28, 62, 36], [20, 12, 4, 44], [12, 36, 4, 44], [4, 62, 4, 44], [4, 4, 12, 44], [52, 4, 12, 44], [52, 20, 12, 44], [44, 44, 12, 44], [36, 12, 20, 44], [20, 28, 20, 44], [20, 62, 20, 44], [20, 4, 28, 44], [28, 44, 28, 44], [4, 12, 36, 44], [28, 20, 36, 44], [62, 20, 36, 44], [20, 62, 36, 44], [20, 4, 44, 44], [12, 28, 44, 44], [4, 44, 52, 44], [36, 20, 62, 44], [20, 36, 62, 44], [36, 20, 4, 52], [36, 36, 4, 52], [52, 36, 4, 52], [36, 52, 4, 52], [12, 20, 12, 52], [12, 52, 12, 52], [62, 12, 20, 52], [36, 52, 20, 52], [4, 28, 28, 52], [52, 28, 28, 52], [36, 36, 36, 52], [44, 4, 44, 52], [20, 44, 44, 52], [28, 28, 52, 52], [28, 4, 62, 52], [12, 20, 62, 52], [28, 4, 4, 62], [44, 4, 4, 62], [62, 4, 4, 62], [4, 12, 4, 62], [20, 28, 4, 62], [20, 44, 4, 62], [52, 20, 12, 62], [4, 36, 12, 62], [20, 12, 20, 62], [44, 36, 20, 62], [20, 44, 20, 62], [4, 4, 28, 62], [44, 12, 28, 62], [28, 28, 28, 62], [4, 52, 28, 62], [12, 20, 36, 62], [12, 36, 36, 62], [4, 4, 44, 62], [20, 4, 44, 62], [36, 20, 44, 62], [4, 28, 52, 62]]","c/tools/expert_atlas/probes.json":"{\n \"_comment\": \"Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.\",\n\n \"code_python\": [\n \"Write a Python function that merges two sorted lists into one sorted list.\",\n \"In Python, explain the difference between a generator and a list comprehension.\",\n \"Refactor this Python snippet to avoid a nested loop: for a in xs: for b in ys: if a==b: out.append(a)\"\n ],\n \"code_sql\": [\n \"Write a SQL query that returns the top 5 customers by total order value.\",\n \"Explain what a LEFT JOIN does differently from an INNER JOIN in SQL.\",\n \"Write SQL to add an index on the email column of a users table and explain when it helps.\"\n ],\n \"math_proof\": [\n \"Prove that the square root of 2 is irrational.\",\n \"Show that the sum of the first n odd numbers equals n squared.\",\n \"Explain why the derivative of e^x is e^x.\"\n ],\n \"chinese\": [\n \"请用三句话解释什么是机器学习。\",\n \"写一首关于秋天的短诗。\",\n \"中国的四大发明是什么？请简要说明。\"\n ],\n \"german\": [\n \"Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.\",\n \"Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.\",\n \"Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?\"\n ],\n \"poetry\": [\n \"Write a short poem about the sea at night.\",\n \"Compose four lines of verse about an empty train station.\",\n \"Write a haiku about the first snow of winter.\"\n ],\n \"law\": [\n \"Explain in plain terms what consideration means in contract law.\",\n \"What is the difference between a misdemeanor and a felony?\",\n \"Summarize what the term 'force majeure' covers in a commercial contract.\"\n ],\n \"medicine\": [\n \"Explain the difference between type 1 and type 2 diabetes.\",\n \"What are the common symptoms of iron deficiency anemia?\",\n \"Describe how a vaccine produces immunity.\"\n ],\n \"json_format\": [\n \"Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.\",\n \"Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.\",\n \"Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'.\"\n ],\n \"casual_chat\": [\n \"Hey, what's a good way to spend a rainy Sunday afternoon?\",\n \"I'm feeling pretty tired today. Any tips to get through the afternoon?\",\n \"What's your favourite kind of weather, and why?\"\n ]\n}\n","c/tools/README.md":"# Tools\n\nThese scripts support model preparation and offline engineering work. They are\nnot runtime dependencies of the C engine.\n\n- `convert_fp8_to_int4.py`, `download_glm52.py`: model preparation\n- `qpack_install_policy.py`: source-bound resume decisions and manifest-last\n publication primitives for future qpack download/repack frontends. Hold its\n install session for the full transfer. Qpack v1 has sizes but no content\n hashes, so frontends must also enforce immutable revisions and any available\n transport checksum or ETag. It covers manifest-declared container artifacts,\n not required runtime auxiliaries such as `config.json`. Swiftlet manifests\n vary: the 35B container declares `packed_experts/layout.json` and its\n tokenizer files, the production Qwen3-Next-80B container declares only the\n weights, so `layout.json` is verified when declared and never required\n here. Directory-fsync crash durability is POSIX-only.\n- `qpack_http_install.py`: dependency-free Hugging Face frontend for that\n policy. It resolves a branch or tag to an immutable commit before opening the\n install session, streams manifest-declared files directly into `.part`\n files, resumes only exact HTTP ranges, and publishes `manifest.json` last.\n `config.json` is required, and so is `packed_experts/layout.json`, which\n the reader opens directly: when the manifest does not declare it, it is\n fetched as a required sidecar. Manifest-declared tokenizer/configuration\n files use the same resumable artifact path, while undeclared optional files\n are copied as verified sidecars when present. Strong ETags, documented server\n SHA-256 values, and a locally computed full-file SHA-256 are persisted in\n `.qpack-http.json` and checked on resume. `HF_TOKEN` is sent only to the Hub\n origin and is stripped on cross-origin HTTPS redirects.\n- `qpack_mirror_install.py`: static HTTPS/R2 frontend over the same transfer\n engine. A strict `hashes.json` is required and fetched before the manifest;\n it must cover `manifest.json`, `config.json`, `packed_experts/layout.json`,\n and every manifest artifact.\n The legacy Swiftlet `{\"files\":{\"path\":\"sha256\"}}` schema and the sized\n `qpack.hashes.v1` schema are accepted. The normalized mirror URL plus the\n digest of the exact raw index bytes bind resume state; indexed runtime files receive\n authoritative full-object SHA-256 values, while other valid indexed\n sidecars are ignored. The exact raw index is persisted beside the installed\n container. Mirror installs never inherit\n `HF_TOKEN`; an explicit bearer token can be read from a named environment\n variable and is stripped on cross-origin redirects.\n- `convert_fmt4_to_fmt2.py`: fmt=4 (grouped int4) -> fmt=2 (per-row int4)\n re-quant of a GLM-5.2 container, for Metal-backend compatibility\n (see `docs/METAL-M1ULTRA-FMT2-REPORT.md`)\n- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8) minting a\n standalone-loadable model directory: routed experts plus the full resident\n family (attention projections including `kv_b_proj`, shared-expert and\n dense-MLP weights), norms/router/embed/lm_head copied byte-identically in\n their original BF16/F32, and the non-tensor files the loader opens at\n runtime (`config.json`, `tokenizer.json`; `generation_config.json`\n best-effort). `--mtp` repacks the multi-token-prediction head as a separate\n pass into the same outdir. Caveat: the minted `kv_b_proj` loads today but\n must not serve batched-path decode until the engine's fmt=8 absorb support\n lands (branch `f8/absorb-fmt8`); the failure is a loud crash, not silent\n corruption -- see the module docstring and `docs/FORMATS.md`.\n Synthetic-fixture-tested only, no real-shard runs yet.\n- `make_glm_oracle.py`, `make_glm_bench_model.py`: deterministic fixtures\n- `benchmark_cuda_fixture.py`, `eval_glm.py`, `fetch_benchmarks.py`: benchmarks\n- `gen_unicode.py`: tokenizer table generation\n\nRun them from `c/`, for example:\n\n```sh\npython3 tools/convert_fp8_to_int4.py --selftest\npython3 tools/make_glm_bench_model.py --output /tmp/colibri-bench\npython3 tools/qpack_http_install.py OWNER/REPO \\\n --revision main --output /path/to/model.qpack\npython3 tools/qpack_mirror_install.py \\\n https://models.example/qwen/model.qpack \\\n --output /path/to/model.qpack \\\n --hashes-sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\n```\n\nAn interrupted qpack install can be restarted with the same command. Keep the\nsource repository and revision unchanged. The installer rejects HTTP range or\nvalidator inconsistencies instead of appending uncertain bytes. It requires\n`Content-Length` for large artifacts and never stages the original BF16/FP8\ncheckpoint, but it does not convert a raw checkpoint into qpack.\n\nStatic mirrors must use HTTPS and publish objects first, `manifest.json` next,\nand `hashes.json` last as the remote completion marker. Without\n`--hashes-sha256`, the first valid TLS response is trust on first use: the\nURL-and-index identity prevents a later snapshot from resuming into those\npartials, but it cannot authenticate a mirror that was already compromised on\nthe first request. Prefer a versioned mirror prefix and an out-of-band index\ndigest for production distribution. The installer uses exact byte ranges and\n`If-Match` when a strong ETag is available; final SHA-256 verification remains\nthe authority.\n\n`make_glm_oracle.py` also produces the quantized routed-expert fixtures for the\nfmt=6 (E8/IQ3, rotation-bearing) and fmt=4 (grouped int4, no-rotation control)\nparity gate (#3/#7). Only the routed experts are quantized; shared/dense/attn\nstay f32, and the reference (`ref_glm.json` inside each fixture dir) is computed\nfrom the dequantized weights so the engine reproduces it token-exactly:\n\n```sh\npython3 tools/make_glm_oracle.py --fmt6 # -> glm_tiny_fmt6/\npython3 tools/make_glm_oracle.py --fmt4 # -> glm_tiny_fmt4/\n# verify the engine loads the formats directly (32/32 expected):\nSNAP=./glm_tiny_fmt6 REF=./glm_tiny_fmt6/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16\nSNAP=./glm_tiny_fmt4 REF=./glm_tiny_fmt4/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16\n```\n","docs/api.md":"# OpenAI-compatible API, KV contexts & web UI\n\n## `coli serve`\n\n`coli serve` keeps one model process loaded and exposes a text-only\nOpenAI-compatible HTTP API. The gateway uses only the Python standard library;\ninference still runs in the same dependency-free C engine.\n\n```bash\ncd c\nCOLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \\\n --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri\n\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Authorization: Bearer local-secret' \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"glm-5.2-colibri\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"stream\": true\n }'\n```\n\nImplemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,\n`POST /v1/chat/completions`, legacy `POST /v1/completions` and\n`POST /v1/systemone`, colibri's one decision API ([systemone.md](systemone.md)):\nthe request and reply of TypeSafe's Jev API, scored by a language model through\nits logprob channel or answered natively by a decision model such as\n[Laya](laya.md),\n[GLiNER2.5-Decide](gliner_decide.md) or [Clef](clef.md). Chat and\ncompletion requests support JSON responses, SSE streaming, usage counts,\n`max_tokens`/`max_completion_tokens`, `temperature`, `top_p`, and up to four\ncustom `stop` sequences. `max_tokens` is a ceiling, not a target: when the\nprompt leaves less room than the budget asks for, every engine clamps the\nbudget to what the context holds and the reply ends with `finish_reason:\n\"length\"`; only a prompt that does not fit is refused, with\n`context_length_exceeded` (#260, #1641). Stop sequences are removed from the response and end\ngeneration early in both JSON and streaming modes.\n\nA trailing `assistant` message *continues* that turn instead of starting a new\none: the prompt ends inside it, which is the official template rendered with\n`add_generation_prompt=False`. This is on by default — a message list ending in a\nnon-empty `assistant` turn continues, the same contract as Anthropic's API — and\nthere is no request field for it, because a trailing assistant turn already says\n\"continue me\" and a body extension would only be reachable by hand-written JSON\nrather than from the clients that want it. The server-side switch\n`COLI_CONTINUE_ASSISTANT=0` restores the old behaviour (fold the turn into a\ncompleted one and append a fresh cue). Continuation is refused together with\n`tools`/`tool_calls`, because the tool-call parsers read an assistant turn from\nits start, and the turn must carry text not ending in whitespace: the template\nstrips trailing whitespace, so the model would resume from different bytes than\nthe ones sent. A family whose renderer has no open-turn shape yet falls through\nto the old behaviour rather than erroring — though every shipped family supports\ncontinuation today, Kimi K3 included (its open turn is framed engine-side, in\n`kimi_k3.c`, not derived in the gateway renderer).\n\nA continuation resumes from the exact bytes you send, which makes the split\npoint part of the prompt. Splitting mid-word puts the model at a token boundary\nit would not have produced itself, and the first generated token is conditioned\non that split: measured on GLM-5.3-Flash, `The capital of France is Par`\ncompletes to `París`, not `Paris` — deterministically, across every effort level\nand both endpoints. The continuation is real (the model finished the partial\nword rather than restarting, which is the behaviour this feature exists for);\nthe spelling is an artefact of where the split fell. This is inherent to\nresuming from an arbitrary byte offset rather than specific to this engine, and\nit is the same hazard as the trailing whitespace above — in the one form that\ncannot be refused, because splitting mid-word is sometimes exactly what the\ncaller wants. Split at a token-ish boundary when the spelling matters. The extension\n`x_colibri_ignore_leading_stop: true` discards leading stop sequences until\nthe first non-whitespace response content, which is useful for local templates\nthat occasionally emit a role marker before the answer; strict OpenAI stop\nbehavior remains the default for client-provided sequences. GLM chat requests\nwith no client `stop` automatically use the template's `<|user|>` and\n`<|observation|>` role markers, patiently ignoring only leading markers; this\nprevents a model-completed turn from silently generating a new user or tool\nturn. Inkling chat and legacy completion requests receive no implicit GLM stop\nsequences. The extension\n`enable_thinking: true` enables GLM-5.2's reasoning block; the standard\n`reasoning_effort` field also enables it unless set to `none`.\n\nOn Qwen3.6 the extension `preserve_thinking` is the official template's kwarg of\nthe same name: past assistant turns keep their `` block, with the\n`reasoning_content` the client sends back, empty when it sends none. It\ndefaults to `true` when thinking is off and to `false` when it is on. With\nthinking off, the empty block is the one each past turn was generated after,\nso a client that resends only `content` sends the history the engine already\nholds and the KV prefix is reused instead of prefilled again (#1759). With\nthinking on a standard client drops the reasoning, the history cannot match\neither way, and the template's default applies; a client that sends\n`reasoning_content` back can set `preserve_thinking: true` to get the reuse\ntoo. Both values render byte for byte like the official `chat_template.jinja`.\n\nThe server serves one generation at a time: the model stays in one persistent\nprocess, so concurrent HTTP requests queue instead of loading duplicate model\ncopies. Tool calling depends on the active engine; see the support matrix below.\nImages and token penalties return an explicit error rather than being silently\nignored. The OpenAI-compatible endpoints request log probabilities only from a\nglm engine (see below); on every other engine such a request is refused with a\nnamed error, never silently ignored. `seed` is accepted and ignored (see\nbelow). Audio is accepted only by Inkling checkpoints with audio support. The\ndefault bind address is localhost; set `COLI_API_KEY` before exposing the\nserver beyond the machine.\n\nThe hosted-platform bookkeeping fields `store`, `metadata`, `service_tier`,\n`user`, `safety_identifier`, `parallel_tool_calls`, `prompt_cache_key`,\n`verbosity`, `web_search_options`, `moderation`, and\n`stream_options.include_obfuscation` are accepted and intentionally ignored:\nthey have no local equivalent and do not affect generation. Unsupported\nresult-shaping requests are refused explicitly: `best_of` values above 1, a\nnon-empty `logit_bias`, and `suffix` infill. The optional `modalities` array\naccepts text output only; malformed values and requests for other output\nmodalities receive a named 400 rather than silently returning text.\n\n### `seed`\n\n`seed` is accepted (not rejected) for OpenAI-API request-shape compatibility;\nthe value is not validated. This server sends no per-request seed on the\nwire, so the value has no effect at any temperature.\n\n### Log probabilities and prompt echo\n\n`/v1/completions` accepts the legacy integer `logprobs` (**0–32**; 0 means no log\nprobabilities at all, see below; the upper bound is the engine's top-32 read-out\ninterface, and anything above 32 is a named 400) and boolean `echo`;\n`/v1/chat/completions` accepts boolean `logprobs` plus integer `top_logprobs` (0–32) and\nreturns `choices[].logprobs.content[]` (`{token, logprob, bytes, top_logprobs}` per\ngenerated token) — chat has no `echo` concept and rejects one with a 400. Each endpoint\ntakes the OpenAI request shape for it and refuses the other's by name. A non-boolean\n`echo` is a named 400 (`invalid_value`) on both endpoints, independent of whether\n`logprobs` is requested at all. On chat, `top_logprobs` is type- and range-checked even\nwhen `logprobs` is false or absent, so a malformed `top_logprobs` is a named 400 whether\nor not the gate it would feed is open; a valid `top_logprobs` with `logprobs` off remains\na documented no-op.\n\nThe zero semantics are explicit, not a truthiness accident: on `/v1/completions`,\n`logprobs: 0`, `false` and `null` all mean **no log probabilities** (the request succeeds\nwith `choices[].logprobs: null`, exactly as if the field were omitted), while boolean\n`true` is a named 400 — the legacy field is an integer count, and a boolean carries no\ncount. On `/v1/chat/completions` the field is a boolean gate (`null` behaves like `false`;\nany integer is a named 400). `echo: null` normalises to absent on both endpoints, so a\nclient that serialises its whole request model with nulls is never refused for a field it\ndid not mean to set. `logprobs` together with `stream` is a named 400: per-delta log\nprobabilities are not built.\n\n`/v1/completions` with `echo: true` returns the full legacy `logprobs` object (`tokens`,\n`token_logprobs`, `top_logprobs`, `text_offset`) covering the echoed prompt plus any\ngenerated tokens, and `text` itself is the reconstructed prompt followed by the completion\n(the standard OpenAI legacy behavior for `echo: true`) rather than the completion alone;\n`echo: true` without an active `logprobs` request is a named 400 (`echo` requires\n`logprobs`): the prompt echo is built out of the engine's per-token records, so without\nthem there is nothing to echo, and a request that asks for one is told so rather than\nserved without it. Echoing a prompt with no logprobs at all is offered as a separate\nproposal. `text_offset` is a character offset into\nthat same returned `text` string, always counted from 0 — including when `echo` is false,\nwhere `text` holds only the completion and the offsets describe only that text, not a\nposition within the (unreturned) prompt.\n\n`\"\".join(tokens)` is the returned `text`: where a `stop` sequence matches partway through\na token, that token's `tokens` entry is truncated to the characters actually emitted (a\n`stop` of `\"lo\"` against a token decoding to `\"Hello\"` reports `\"Hel\"`), rather than being\ndropped or reporting characters the client did not receive. On chat, a truncated entry's\n`bytes` — and its `top_logprobs` entries' `bytes` — are truncated with it, to the UTF-8 of\nthe characters that entry reports. An entry that was not truncated keeps its frame's own\npayload, which need not be the encoding of its `token`: a frame carrying only part of a\nmulti-byte codepoint reports `token: \"\"` with that frame's bytes.\n\nThe requested top-k table is **unsorted** on the wire — do not assume the first entry is\nthe argmax. Per-token values are printed by the engine to six decimal digits of precision.\nNon-finite values (a degenerate all-`-inf` logit row, say) serialize as JSON `null`, never\na clamped number.\n\nThese endpoints request the numeric per-token channel only from a glm or a MiMo engine,\nthe two that read out every prompt position `echo` needs; on every other engine the\nrequest returns a named 400 rather than being silently ignored. That is a\nstatement about what these endpoints request, and about nothing else.\n\nKnown limitations, current build:\n\n- **Cost.** Requesting `logprobs` at all — completions or chat, `echo` or not — asks the\n engine for a full read-out pass over the whole prompt, because the request carries one\n opt-in and not a separate \"echo\" one. Such a request normally forfeits prefix-cache\n reuse. There is no cap on prompt length for it, so a very long prompt pays a\n correspondingly large one-shot cost.\n- **Waiting.** A logprobs request waits for the engine exactly as any other request waits.\n An engine that never acknowledges the submission — one older than this extension, or one\n whose parser disagrees about the header's shape — leaves the request waiting, as it\n would through a cold prefill. There is no separate bounded wait for the opt-in. A request\n whose per-token frames could not be read waits the same way: the turn belongs to the\n engine until it sends a terminal frame, so the refusal below is delivered then rather\n than the moment the unreadable frame arrives. The engine is sent one stop when that\n happens, so the turn does not spend the rest of its budget on a response nobody receives.\n- **Cancellation.** A cancel may not take effect until the read-out this request asked for\n has finished, so the window in which a disconnect goes unnoticed is wider on a long\n prompt than it is without `logprobs`.\n- **Alternative-token labels.** `top_logprobs` entries for candidate token ids other than\n the position's own actual token are not decoded text (no server-side tokenizer exists, by\n design) — they are labeled ``. Only the position's own token, identified by\n an exact logprob match rather than by id, gets its real decoded text.\n- **The sampled token is not guaranteed to appear in its own `top_logprobs` table.** The\n engine's numeric channel reports the top-k candidates by its own read-out; if the chosen\n token falls outside that table, no entry represents it, and the response's\n `token_logprobs`/`logprob` field is still the chosen token's own value read from the\n frame directly, not looked up in the table.\n- **The arrays describe the text the client received, exactly, and nothing beyond it.**\n Every generated token is located in the returned text by its span in the raw engine\n stream, so the stage that consumed a character decides how much of that token's entry\n reaches the array: a token a matched `stop` cut partway through reports the characters\n that were emitted, and generated text that does not reach `message.content` — reasoning,\n template markers, tool-call syntax, a swallowed role marker — is not described there.\n Reasoning, template and tool-call tokens are therefore not described by\n `logprobs.content`; a separate field for the full stream is proposed separately. On\n every response that carries logprobs, `\"\".join(tokens)` is the returned `text` and\n `\"\".join` over `logprobs.content` is `message.content`. When the engine's records cannot\n support that — they stop short of the generated text, or none arrive at all — the\n request is refused by name rather than answered with arrays describing a prefix.\n- **Engine-side faults are named, not generic.** When the engine's per-token records cannot\n answer the request, the 5xx body carries a `code` saying what the engine did:\n `engine_logprob_tail_malformed` (a numeric tail this server cannot read — it fails that\n one request and no other), `engine_echo_position_malformed` (an unreadable prompt-echo\n position, which is a different field of the same frame),\n `engine_duplicate_logprob_candidate` (a top-k table repeating a candidate at one\n position), `engine_logprob_records_incomplete` (records that stop\n short of the generated text, none at all, or — under `echo` — no prompt-echo records,\n any of which would leave the arrays describing a prefix), and\n `engine_pinned_prefix_not_echoed` (a KV slot holding a pin snapshot, so the echoed\n positions would not start at 0). No logprobs-bearing response is ever a truncated or\n half-described 200.\n- **Server-side buffering.** A request that asks to see the echoed prompt (`echo: true` on\n `/v1/completions`) has its full echo table held in memory for the request's lifetime; no\n streaming is allowed together with `logprobs`, so that hold is bounded by one\n non-streaming response. A logprobs request that does not ask for the echo — all of chat,\n and every `echo: false` completion — retains nothing: the engine still sends an ECHO frame\n for every prompt position, and they are dropped as they arrive rather than held.\n\n### Tool-calling support\n\n| Engine | OpenAI `tools` | Anthropic `tool_use` | Native format |\n|---|---|---|---|\n| GLM-5.2 (`colibri`) | yes | yes | `` blocks |\n| GLM-5.3-Flash | yes | yes | `` blocks, with the 5.3 declaration block |\n| DeepSeek V4 | yes | yes | native DSML tool-call blocks |\n| Inkling | no | no | active tool declarations/choices return HTTP 400 |\n| Kimi K3 | yes | yes | native XTML `tools`/`call`/`argument` blocks (#1143) |\n| MiMo-V2.6 | yes | yes | native `` / `` blocks |\n| Qwen3.6 | yes | yes | native ``/`` blocks, the same XML-ish form as Qwen3.8 |\n| Qwen3.8-Flash-Next | yes | yes | native ``/`` blocks |\n| OLMoE | no | no | active tool declarations/choices return HTTP 400 |\n\nOn supported engines, pass OpenAI `tools` and optionally `tool_choice` to\n`/v1/chat/completions`. The Anthropic endpoint translates `tools`,\n`tool_use`/`tool_result`, and the `auto`, `any`, `none`, and forced-tool choice\nmodes into the active engine's native prompt and back into protocol responses.\nProtocol support does not guarantee that every quantized model emits valid\ntool syntax; `COLI_TOOL_SALVAGE=1` is an opt-in recovery path for malformed GLM\nint4 tool calls. DeepSeek V4 uses its strict native DSML parser instead.\n\n`tool_choice: \"required\"` is a prompt-level instruction, not a sampling\nconstraint. Every renderer that offers a tool block appends the same one line to\nit, and no renderer filters tokens or forces the sampler, so a model that\nanswers in prose anyway has done nothing the API said was impossible.\nGrammar forcing is not a remedy: that path feeds a draft the engine then\nverifies, so a schema the engine cannot compile costs the speedup and nothing\nelse. An engine with no tool block to attach the instruction to (Inkling, OLMoE\nwithout `COLI_TOOL_FALLBACK=1`) answers HTTP 400 rather than accept the choice\nand ignore it.\n\nA forced choice, `tool_choice: {\"type\": \"function\", \"function\": {\"name\": …}}`,\nis applied per engine and the engines do not agree on how: some narrow the\noffered tools to the named one, some keep the full list and name the tool in\nprose instead, and some (Qwen3.6, Qwen3.8) do neither. Read the rendered prompt\nrather than assuming the request was honoured.\n\nFor GLM calls that will execute tools, an OpenAI chat request may set\n`\"strict_tool_calls\": true`. This opt-in accepts only complete ``\nblocks whose function and arguments match the declared tool schema. It returns\nHTTP 502 with code `invalid_model_tool_call` for incomplete, duplicate, unknown,\nor invalid arguments and for a tool call cut off by the generation limit. It\nnever recovers or salvages a malformed call. The default recovery behavior is\nunchanged. Strict mode currently requires `stream: false` and does not support\n`logprobs`; these combinations receive HTTP 400 before generation.\n\nWhen a reverse proxy or MagicDNS hostname preserves a public `Host` header,\ntrust that exact hostname with repeatable `--allowed-host` options. The\ncomma-separated `COLI_ALLOWED_HOSTS` environment variable is equivalent:\n\n```bash\nCOLI_ALLOWED_HOSTS=llm.example.com ./coli serve --model /nvme/glm52_i4\n# or: ./coli serve --model /nvme/glm52_i4 --allowed-host llm.example.com\n```\n\nOnly configure hostnames or IP addresses you control; there is no wildcard.\nThis setting extends the DNS-rebinding allowlist and is independent of CORS and\nAPI-key authentication.\n\nBrowser access from the Vite development server and Tauri local origins is\nenabled by default. Repeat `--cors-origin https://your-ui.example` to allow\nanother exact origin, or use `--cors-origin '*'` only on a trusted local\nnetwork.\n\nThe engine owns its KV contexts, so HTTP generation uses a bounded FIFO\nadmission queue instead of pretending to run unsafe parallel sequences.\nConfigure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS`\n(default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment\nvariables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429\nerrors before streaming headers are sent. With `--max-queue 0`, a request\npinned to an occupied KV slot is rejected immediately even if another slot\nis free. Queue deadlines are checked before slot assignment: an expired\nwaiter receives `queue_timeout` even if a slot is now available. `GET /health` exposes\nactive/queued/completed/rejected counters, and successful generation responses\ninclude `x-colibri-queue-wait-ms`.\nRequests targeting any slot may use a free slot not reserved by earlier waiters.\nAn earlier pinned request keeps priority for its target; an earlier any-slot\nrequest keeps priority across all slots. A full waiting queue does not reject\na request that can immediately take an unreserved free slot; the queue limit\nbounds waiting requests, independently of active capacity.\n\n## Prometheus metrics\n\n`GET /metrics` returns Prometheus text exposition (version 0.0.4). When an\nAPI key is configured, supply the same `Authorization: Bearer ...` or\n`x-api-key` header used for generation; missing or invalid credentials return\n401. Without an API key, the endpoint follows the server's usual unauthenticated\naccess policy. Metrics contain no prompts, model paths, or request-ID labels.\n\nAll names start with `colibri_scheduler_`:\n\n| Suffix | Type | Meaning |\n|---|---|---|\n| `active`, `queued`, `capacity`, `max_queue` | gauge | Admitted requests, waiters, KV slot capacity, and queue limit |\n| `admitted_total` | counter | Requests admitted to a KV slot |\n| `completed_total` | counter | Admitted requests that returned normally |\n| `failed_total` | counter | Admitted requests that raised an error, excluding `ClientCancelled` |\n| `rejected_total`, `timed_out_total` | counter | Queue-full refusals and queue timeouts |\n| `cancelled_total` | counter | Cancellations detected before admission or during admitted work |\n| `queue_wait_seconds` | histogram | Wait until admission, for admitted requests only |\n| `slot_duration_seconds` | histogram | Slot occupancy until completion, failure, or cancellation |\n| `first_output_seconds` | histogram | Engine-call start to first nonempty text or tool-output callback |\n| `engine_call_seconds` | histogram | Duration of each finished engine generation call, including failure/cancellation |\n\nHistogram buckets are 0.001, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30, 60, 300 seconds,\nand `+Inf`; each histogram exposes `_bucket`, `_sum`, and `_count`.\nCounters reset when the gateway restarts. Collection does not call the engine\nor consume a generation slot. `failed` is also included in `/health`'s\nauthenticated scheduler snapshot; failures no longer increment `completed`.\n\nCancellation is checked before acquiring an available KV slot, including when a\nwaiting request wakes as capacity becomes free. A request cancelled at this\npoint increments `cancelled_total`, but not `admitted_total`, and contributes\nno admission-wait or slot-duration sample. Queue-full and scheduler-closed\nchecks can still reject a request before the cancellation check is reached.\n\n\nThe engine-call histograms exclude admission queue wait and prompt rendering.\nFirst output is observed before the gateway's stop filtering, reasoning split,\nor HTTP serialization: it can be reasoning or tool data, not necessarily\nuser-visible answer text. Empty callbacks, ACCEPT frames, and SSE keepalives do\nnot count. Calls that finish or fail without output add no first-output sample;\na failure after output retains that sample. Engine-call duration includes callback\nprocessing and response writes during generation. One request can invoke the\nengine multiple times (for example System One scoring), so these histogram counts are\nengine calls, not HTTP request counts.\n\nThese are gateway observations, not end-to-end client TTFT, per-token latency,\nor GPU kernel measurements. Output callbacks need not correspond one-to-one to\ntokens. Slot occupancy includes any response handling while the slot is held. Validation/authentication failures before admission are not counted.\n`completed` means the admitted handler returned normally, not that the client\nreceived every response byte. Request exceptions can include client input or\ntransport errors as well as engine failures.\n\nExample PromQL for the admitted-request queue-wait p95:\n\n```promql\nhistogram_quantile(0.95, sum by (le) (rate(colibri_scheduler_queue_wait_seconds_bucket[5m])))\n```\n\n## Anthropic-protocol endpoint (`/v1/messages`)\n\nThe same server also speaks the **Anthropic Messages API**, so clients that only talk\nto Anthropic endpoints — Claude Code, the Anthropic SDKs — work against colibri\nwithout a shim. Nothing to enable: `/v1/messages` is served alongside\n`/v1/chat/completions` on the same port.\n\n```bash\ncurl http://127.0.0.1:8000/v1/messages \\\n -H 'x-api-key: local' -H 'content-type: application/json' \\\n -d '{\"model\":\"glm-5.2-colibri\",\"max_tokens\":128,\n \"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}'\n```\n\nFor Claude Code, point it at the server and give it any non-empty key:\n\n```bash\nexport ANTHROPIC_BASE_URL=http://localhost:8000\nexport ANTHROPIC_API_KEY=local # only enforced if you set COLI_API_KEY\nexport ANTHROPIC_MODEL=glm-5.2-colibri\nclaude\n```\n\nSupported on every served architecture: system prompts (string or text blocks),\nmulti-turn `user`/`assistant` messages, streaming with the full named-event\nsequence (`message_start` → `content_block_*` → `message_delta` → `message_stop`,\nplus protocol `ping` keepalives during long prefills), `stop_reason`, Anthropic\n`usage` field names, and `x-api-key` authentication (`Authorization: Bearer`\nalso works). The gateway renders each request with the active engine's native\nchat template; GLM, Inkling, Kimi K3, Qwen3.8, OLMoE, and DeepSeek V4 prompts are not\ninterchangeable. Where the engine exposes a reasoning mode, extended thinking\nis enabled with `{\"thinking\": {\"type\": \"enabled\"}}` and translated to that\narchitecture's reasoning protocol; OLMoE disables it explicitly.\n\nTool use follows the per-engine matrix above. Unsupported engines reject active\ntool declarations and choices explicitly instead of feeding another\narchitecture's markers to an incompatible tokenizer.\n\nNot supported, and refused explicitly rather than ignored: `stop_sequences`,\n`top_k`, and non-text content blocks (images, documents). Errors use Anthropic's own\n`{\"type\":\"error\",\"error\":{...}}` envelope on this path. Architecture-local\nfeatures that have not been wired to this protocol are likewise rejected with\nan explicit error.\n\nA trailing `assistant` message continues that turn by default on both the\nAnthropic- and OpenAI-compatible endpoints (`COLI_CONTINUE_ASSISTANT=0` restores\nthe old behavior, where this endpoint appended a fresh cue). Note this changes\nwhat an existing Anthropic client sees on `/v1/messages`: a trailing assistant\nturn now continues rather than starting fresh — which is the real Anthropic\ncontract — and the off-switch is the escape hatch for anyone relying on the old\nbehavior.\n\n> The prefill warning below applies here too, and applies *hardest* to Claude Code:\n> its system prompt and tool catalog are large, and on a disk-streaming CPU path\n> that is a long silent wait before the first token. Read it before you connect.\n\n## Connect a coding CLI or editor\n\nThe API is OpenAI-compatible, so most coding CLIs and editor extensions work by\npointing them at Colibri as an *OpenAI-compatible* provider. Three settings:\n\n- **Base URL** — `http://localhost:8000/v1`\n- **Model** — `glm-5.2-colibri` (or whatever you pass to `--model-id`)\n- **API key** — any non-empty string, e.g. `local`\n\nColibri needs **no** API key by default, but many clients refuse to start without\none — give them any dummy value. The key is only enforced if you set `COLI_API_KEY`.\n\nSmoke-test the endpoint first (no key needed unless you set one):\n\n```bash\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\"model\":\"glm-5.2-colibri\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\n```\n\n**aider**\n\n```bash\nexport OPENAI_API_BASE=http://localhost:8000/v1\nexport OPENAI_API_KEY=local\naider --model openai/glm-5.2-colibri # the openai/ prefix routes to OPENAI_API_BASE\n```\n\n**crush** — add a provider to `crush.json` (`~/.config/crush/crush.json`, or\n`%USERPROFILE%\\AppData\\Local\\crush\\crush.json` on Windows):\n\n```json\n{\n \"$schema\": \"https://charm.land/crush.json\",\n \"providers\": {\n \"colibri\": {\n \"name\": \"Colibri\",\n \"type\": \"openai-compat\",\n \"base_url\": \"http://localhost:8000/v1/\",\n \"api_key\": \"local\",\n \"models\": [\n { \"name\": \"GLM-5.2 (Colibri)\", \"id\": \"glm-5.2-colibri\",\n \"context_window\": 131072, \"default_max_tokens\": 1024 }\n ]\n }\n }\n}\n```\n\nThe `\"api_key\": \"local\"` dummy is what satisfies clients that demand a key.\n`context_window` is only the client's budget display — set it to whatever your\nKV configuration actually allows.\n\n**pi** — add a custom provider to `~/.pi/agent/models.json` ([pi](https://github.com/earendil-works/pi-coding-agent) loads every OpenAI-compatible server through its `openai-completions` API):\n\n```json\n{\n \"providers\": {\n \"colibri\": {\n \"baseUrl\": \"http://localhost:8000/v1\",\n \"api\": \"openai-completions\",\n \"apiKey\": \"local\",\n \"compat\": {\n \"supportsDeveloperRole\": false,\n \"supportsReasoningEffort\": false\n },\n \"models\": [\n {\n \"id\": \"glm-5.2-colibri\",\n \"name\": \"GLM-5.2 (Colibri)\",\n \"contextWindow\": 131072,\n \"maxTokens\": 1024\n }\n ]\n }\n }\n}\n```\n\nThe `apiKey` dummy satisfies pi's auth requirement; colibri only enforces a\nkey if you set `COLI_API_KEY`. The `compat` flags tell pi to send the system\nprompt as a plain `system` message and to omit `reasoning_effort`, which keeps\nthe request inside what the gateway accepts by default. If you serve GLM-5.2\nand want its reasoning block, set `\"reasoning\": true` on the model and drop\n`supportsReasoningEffort` — the standard `reasoning_effort` field enables\nthinking on that engine. Then select the model with `pi --list-models` or the\n`/model` picker (`colibri / GLM-5.2 (Colibri)`).\n\n`contextWindow` is only the client's budget display — set it to whatever your\nKV configuration actually allows. Tool calling in pi works on the engines in\nthe [tool-calling matrix](#tool-calling-support) above; on unsupported engines\npi's tool calls fail the same way any OpenAI `tools` request does.\n\n**Continue, Cline / Roo, `llm`, the OpenAI SDKs, …** — set the provider's base\nURL to `http://localhost:8000/v1`, the model to `glm-5.2-colibri`, and any dummy\nkey (`OPENAI_API_KEY` / `OPENAI_BASE_URL` for env-based tools).\n\n> **Set your expectations before connecting an agentic CLI.** Two costs dominate,\n> and the first one is invisible until you know it's there:\n>\n> 1. **Prefill.** Coding agents (crush, aider in repo-map mode, Cline, …) send a\n> large system prompt plus tool definitions — often 10–20k tokens — *before\n> your first word*. Prefill on the CPU-streaming path runs at a few tokens per\n> second (it is attention-bound, see #153), so a 15k-token agent preamble is\n> **an hour of silent \"thinking\" before the first output token**. The client\n> looks hung; it isn't. Smoke-test with the tiny `curl` above first — if that\n> answers in about a minute, the pipeline works and what you're paying for is\n> prompt size.\n> 2. **Decode.** Roughly 1 tok/s for a large model, so multi-turn agent loops\n> (which re-pay the growing context every turn) compound the cost.\n>\n> Practical guidance: single surgical asks with a short context work; iterative\n> agent sessions against a disk-streaming 744B model do not resemble a hosted\n> API and mostly won't be worth the wait. If your client lets you trim or disable\n> its system preamble and tool catalog, do it.\n\n## Isolated KV contexts\n\n`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.\nRequests select one with the optional integer `cache_slot` field; ordinary\nOpenAI clients omit it and keep the original slot 0 behavior.\n\n```json\n{\n \"model\": \"glm-5.2-colibri\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Continue this conversation\"}],\n \"cache_slot\": 1\n}\n```\n\nEach slot owns its token history and its conversation's state: the KV cache, and\non the engines that have them the DeltaNet, KDA and convolution states and the\ncompressed DeepSeek attention. The engine matches each request's tokenized prompt\nagainst the slot's history and reuses the common prefix, so stateless HTTP turns\nkeep their cache across requests. On GLM-5.2 each slot also has its MTP window and\na crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...), so the cache\nsurvives an engine restart too. Use `COLI_KV_SLOTS=N` as the environment\nequivalent. Start small: at the default 4096-token context, every slot costs\nhundreds of MB, and `coli plan` counts them.\n\nEvery text engine serves the slots at the same time. Requests on different slots\nare decoded together: each step takes the next token of every active\nconversation as one batch, so the weights and the routed experts a step reads\nserve all of them, while each row's attention and recurrent state stay its own\nconversation's. A request gets the tokens it would get alone (greedy on the CPU,\nthe same bytes; `tests/serve_mux_check.py` checks it on every engine). A prompt is\nprefilled when it arrives, between two steps. With more than one slot, the\nengines other than GLM-5.2 and GLM-5.3-Flash draft nothing (MTP, DSpark and\nprompt lookup follow one conversation), and their Vulkan dense chain, DeltaNet on\nthe GPU and Metal paths stay off; the Vulkan expert tier serves every\nconversation's experts.\n\n## Web dashboard\n\nOne command serves the OpenAI-compatible API **and** the web console on the\nsame port, then opens your browser when the engine is ready:\n\n```bash\ncd web && npm install && npm run build # once\n./coli web --model \n```\n\n`coli web` differs from `coli serve` only in opening a browser — both serve the\ndashboard on the same port. On a headless host (no display, often no GPU at all)\nuse `coli serve`, or `coli web --no-browser`, and point a browser at it from\nanother machine. Nothing in the dashboard needs a desktop session on the host.\n\nWhat you get is one workspace with a dock to switch page:\n\n- **Chat**: streaming answers, a reasoning toggle, image input where the engine\n supports it, the KV slot to answer in, and the conversation exported as a file;\n- **System One**: closed-set answers. A document, a question and the only answers\n allowed; the engine reads the probability of each answer, generates nothing,\n and reports an entropy that says when it is not sure. Same thing as\n `POST /v1/systemone` (see [systemone.md](systemone.md));\n- **Brain**, two views. *Explore* draws the\n [measured expert atlas](https://github.com/JustVugg/colibri/issues/175) of GLM-5.2 as a cortex with ten regions to\n enter (publish `experts.json` from `tools/expert_atlas/analyze.py --web`).\n *Live routing* shows the model actually running: one cell per expert,\n colour = tier, brightness = routing heat, and the experts routed in each\n turn flash white and decay;\n- **Profiling**: where the engine spends each turn, by phase (I/O wait, expert\n matmul, attention, LM head, other), the disk service overlapped with compute,\n and the last 30 turns as a trend;\n- a light and a dark theme.\n\nThe dashboard talks to the engine over a small line protocol and plain JSON\nendpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API\nclient (React + TypeScript) and also works against any other compatible\nendpoint; the terminal `coli chat` remains the first-class interface.\n\nThe layout is responsive down to phone widths; the per-turn time breakdown and\nthe tok/s trend live on the Profiling page:\n\n

\n \"the\n   \n \"the\n

\n"},"files":{"c/tools/expert_atlas/README.md":"# Expert Atlas — what does each of the 19,456 experts actually do?\n\nProbe harness for #175. Runs a set of topic-tagged prompts, dumps each run's expert-routing\nhistogram, and turns them into a per-expert topic-affinity vector.\n\n```bash\ncd c\nexport COLI_MODEL=/path/to/glm52_i4\n./tools/expert_atlas/sweep.sh # 30 probes (10 topics x 3 prompts)\npython3 tools/expert_atlas/analyze.py --stats atlas_out/stats --out atlas_out/experts.json \\\n --web web/dist/experts.json # optional: feed the web dashboard Atlas\npython3 tools/expert_atlas/validate.py atlas_out/stats 200 # leave-one-prompt-out check\n```\n\n`--web` writes the same atlas in the shape the web dashboard consumes (the Atlas galaxy and the\nBrain hover tooltips): keyed `\"layer:expert\"` with `affinity`/`entropy`/`top`/`label`. It replaces\nthe retired `tools/expert_atlas.py`, whose API-driven probing ran through a live server and was\nexposed to exactly the traps above (server-side `--topp`, speculative drafts, shared `.coli_usage`).\n\n## Read this before you trust any atlas\n\nFour things silently corrupt this measurement. The sweep script controls all of them; if you\nroll your own, don't skip them.\n\n| trap | effect | control |\n|---|---|---|\n| **`--topp`** | prunes experts by cumulative probability — measured: it hides **38% of the distinct experts** (7,587 → 4,687). It is also the *recommended speed setting*. | `TOPP=0` |\n| **speculative drafts** | `eusage` is incremented inside `moe()`, *before* verification, so **rejected** drafts count. Those are experts routed for text the model never emitted. | `MTP=0 DRAFT=0` |\n| **`.coli_usage`** | is loaded at startup and accumulates, so a naive `STATS` dump contains **all prior history**, not this run. | remove per run (script backs it up and restores) |\n| **autocorrelation** | routing inside one run is highly correlated — the same context routes to the same experts token after token. An expert firing 38 times during one prompt is **one** observation, not 38. Chi-square/entropy on raw selections will certify single-prompt flukes as perfect specialists. | `analyze.py` requires the affinity to **replicate across a category's independent prompts** |\n\nThe CUDA expert tier is also not run-to-run deterministic, so the sweep uses `--gpu none`. Tier\nconfig only decides *where weights live*, not what the router picks, so this costs nothing.\n\n## Method\n\n`analyze.py`:\n\n1. `n[e][c]` — selections of expert *e* while running category *c*\n2. `f[e][c] = n[e][c] / N[c]` — normalise by **category size** (prefill routes the prompt too, so a\n verbose category would otherwise look busier)\n3. `p(c|e)` — renormalise into a topic distribution per expert, i.e. base-rate corrected. Ranking\n by raw count instead just rediscovers which experts are popular in general.\n4. `spec(e) = 1 − H(p(c|e)) / log C` — 0 = generalist, 1 = fires for exactly one topic\n5. **replication gate** — an expert is only a candidate specialist for *c* if it fires in ≥2 of *c*'s\n independent prompts\n\n`validate.py` — leave-one-prompt-out. Build each category's top-K specialist set from its *other*\nprompts, then check which set the held-out prompt's routing actually lands in. If specialisation\nwere an artifact of prompt wording, the held-out prompt would not prefer its own category.\n\n## Result on GLM-5.2 744B int4 (Zen5, CPU routing path)\n\n- **Leave-one-prompt-out: 29/30 = 96.7%** (chance 10%). Specialisation is a property of the topic,\n not of prompt wording.\n- The single miss is instructive: `写一首关于秋天的短诗` (\"write a short poem about autumn\") is\n classified **poetry**, not Chinese — routing follows the **task** over the **language**.\n- **Only 7.9% of experts are strong specialists** (spec ≥ 0.5). The \"one expert = one topic\"\n picture is wrong for ~92% of them.\n- Specialisation **rises with depth**: layer 3 ≈ 0.07 (generalist, token/syntax level) → layers\n 18–58 ≈ 0.19–0.27.\n- The replication gate removed **587** experts that looked like flawless specialists on one prompt.\n\n## Extending it\n\n`probes.json` is the whole probe set — add categories and prompts. Use **3+ prompts per category\nwith varied phrasing**, or the replication gate has nothing to check and you are back to measuring\none prompt. Keep prompt lengths in a similar band across categories.\n","c/tools/iq3xxs_grid.json":"[[4, 4, 4, 4], [20, 4, 4, 4], [36, 4, 4, 4], [12, 12, 4, 4], [28, 12, 4, 4], [62, 12, 4, 4], [4, 20, 4, 4], [20, 20, 4, 4], [12, 28, 4, 4], [20, 36, 4, 4], [28, 62, 4, 4], [44, 62, 4, 4], [12, 4, 12, 4], [28, 4, 12, 4], [4, 12, 12, 4], [20, 12, 12, 4], [12, 20, 12, 4], [44, 20, 12, 4], [4, 28, 12, 4], [20, 28, 12, 4], [12, 36, 12, 4], [36, 44, 12, 4], [4, 62, 12, 4], [4, 4, 20, 4], [20, 4, 20, 4], [36, 4, 20, 4], [12, 12, 20, 4], [4, 20, 20, 4], [20, 20, 20, 4], [12, 28, 20, 4], [28, 28, 20, 4], [62, 28, 20, 4], [12, 44, 20, 4], [62, 44, 20, 4], [44, 62, 20, 4], [12, 4, 28, 4], [62, 4, 28, 4], [4, 12, 28, 4], [20, 12, 28, 4], [44, 20, 28, 4], [4, 62, 28, 4], [28, 12, 36, 4], [62, 28, 36, 4], [36, 36, 36, 4], [62, 44, 36, 4], [28, 62, 36, 4], [44, 62, 36, 4], [12, 4, 44, 4], [62, 4, 44, 4], [20, 28, 44, 4], [20, 44, 44, 4], [44, 28, 52, 4], [36, 52, 52, 4], [4, 12, 62, 4], [36, 12, 62, 4], [52, 12, 62, 4], [28, 36, 62, 4], [12, 52, 62, 4], [12, 4, 4, 12], [28, 4, 4, 12], [4, 12, 4, 12], [20, 12, 4, 12], [12, 20, 4, 12], [28, 20, 4, 12], [4, 28, 4, 12], [20, 28, 4, 12], [36, 28, 4, 12], [62, 36, 4, 12], [4, 44, 4, 12], [4, 4, 12, 12], [20, 4, 12, 12], [12, 12, 12, 12], [4, 20, 12, 12], [20, 20, 12, 12], [12, 4, 20, 12], [28, 4, 20, 12], [4, 12, 20, 12], [20, 12, 20, 12], [12, 20, 20, 12], [4, 28, 20, 12], [20, 62, 20, 12], [4, 4, 28, 12], [20, 4, 28, 12], [4, 20, 28, 12], [12, 28, 28, 12], [52, 36, 28, 12], [52, 52, 28, 12], [12, 4, 36, 12], [44, 4, 36, 12], [4, 44, 36, 12], [4, 20, 44, 12], [36, 20, 44, 12], [52, 36, 44, 12], [12, 62, 44, 12], [44, 4, 52, 12], [20, 20, 62, 12], [4, 36, 62, 12], [4, 4, 4, 20], [20, 4, 4, 20], [12, 12, 4, 20], [28, 12, 4, 20], [4, 20, 4, 20], [20, 20, 4, 20], [52, 20, 4, 20], [12, 28, 4, 20], [20, 36, 4, 20], [12, 4, 12, 20], [28, 4, 12, 20], [44, 4, 12, 20], [4, 12, 12, 20], [20, 12, 12, 20], [12, 20, 12, 20], [4, 28, 12, 20], [28, 52, 12, 20], [62, 52, 12, 20], [4, 62, 12, 20], [4, 4, 20, 20], [20, 4, 20, 20], [12, 12, 20, 20], [62, 12, 20, 20], [4, 20, 20, 20], [20, 20, 20, 20], [62, 28, 20, 20], [4, 36, 20, 20], [44, 44, 20, 20], [12, 4, 28, 20], [4, 12, 28, 20], [36, 12, 28, 20], [4, 62, 28, 20], [36, 62, 28, 20], [44, 28, 36, 20], [28, 44, 36, 20], [28, 4, 44, 20], [62, 20, 44, 20], [12, 36, 44, 20], [36, 62, 44, 20], [12, 4, 62, 20], [28, 4, 62, 20], [52, 12, 62, 20], [44, 36, 62, 20], [12, 4, 4, 28], [4, 12, 4, 28], [20, 12, 4, 28], [12, 20, 4, 28], [28, 20, 4, 28], [4, 44, 4, 28], [44, 52, 4, 28], [20, 62, 4, 28], [4, 4, 12, 28], [20, 4, 12, 28], [4, 20, 12, 28], [12, 28, 12, 28], [36, 36, 12, 28], [52, 36, 12, 28], [12, 4, 20, 28], [28, 4, 20, 28], [4, 12, 20, 28], [44, 20, 20, 28], [20, 44, 20, 28], [20, 62, 20, 28], [12, 12, 28, 28], [28, 28, 28, 28], [4, 28, 36, 28], [62, 36, 36, 28], [20, 62, 36, 28], [4, 4, 44, 28], [52, 4, 44, 28], [20, 20, 44, 28], [44, 44, 44, 28], [36, 12, 52, 28], [52, 28, 52, 28], [28, 52, 52, 28], [28, 28, 62, 28], [4, 52, 62, 28], [36, 4, 4, 36], [62, 12, 4, 36], [44, 28, 4, 36], [62, 28, 4, 36], [28, 44, 4, 36], [62, 44, 4, 36], [36, 62, 12, 36], [4, 20, 20, 36], [62, 28, 20, 36], [4, 36, 20, 36], [4, 52, 20, 36], [52, 52, 20, 36], [62, 4, 28, 36], [44, 36, 28, 36], [36, 4, 36, 36], [12, 44, 36, 36], [36, 52, 36, 36], [44, 20, 44, 36], [28, 36, 44, 36], [4, 62, 44, 36], [44, 4, 62, 36], [4, 12, 62, 36], [20, 12, 62, 36], [4, 28, 62, 36], [20, 12, 4, 44], [12, 36, 4, 44], [4, 62, 4, 44], [4, 4, 12, 44], [52, 4, 12, 44], [52, 20, 12, 44], [44, 44, 12, 44], [36, 12, 20, 44], [20, 28, 20, 44], [20, 62, 20, 44], [20, 4, 28, 44], [28, 44, 28, 44], [4, 12, 36, 44], [28, 20, 36, 44], [62, 20, 36, 44], [20, 62, 36, 44], [20, 4, 44, 44], [12, 28, 44, 44], [4, 44, 52, 44], [36, 20, 62, 44], [20, 36, 62, 44], [36, 20, 4, 52], [36, 36, 4, 52], [52, 36, 4, 52], [36, 52, 4, 52], [12, 20, 12, 52], [12, 52, 12, 52], [62, 12, 20, 52], [36, 52, 20, 52], [4, 28, 28, 52], [52, 28, 28, 52], [36, 36, 36, 52], [44, 4, 44, 52], [20, 44, 44, 52], [28, 28, 52, 52], [28, 4, 62, 52], [12, 20, 62, 52], [28, 4, 4, 62], [44, 4, 4, 62], [62, 4, 4, 62], [4, 12, 4, 62], [20, 28, 4, 62], [20, 44, 4, 62], [52, 20, 12, 62], [4, 36, 12, 62], [20, 12, 20, 62], [44, 36, 20, 62], [20, 44, 20, 62], [4, 4, 28, 62], [44, 12, 28, 62], [28, 28, 28, 62], [4, 52, 28, 62], [12, 20, 36, 62], [12, 36, 36, 62], [4, 4, 44, 62], [20, 4, 44, 62], [36, 20, 44, 62], [4, 28, 52, 62]]","c/tools/expert_atlas/probes.json":"{\n \"_comment\": \"Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.\",\n\n \"code_python\": [\n \"Write a Python function that merges two sorted lists into one sorted list.\",\n \"In Python, explain the difference between a generator and a list comprehension.\",\n \"Refactor this Python snippet to avoid a nested loop: for a in xs: for b in ys: if a==b: out.append(a)\"\n ],\n \"code_sql\": [\n \"Write a SQL query that returns the top 5 customers by total order value.\",\n \"Explain what a LEFT JOIN does differently from an INNER JOIN in SQL.\",\n \"Write SQL to add an index on the email column of a users table and explain when it helps.\"\n ],\n \"math_proof\": [\n \"Prove that the square root of 2 is irrational.\",\n \"Show that the sum of the first n odd numbers equals n squared.\",\n \"Explain why the derivative of e^x is e^x.\"\n ],\n \"chinese\": [\n \"请用三句话解释什么是机器学习。\",\n \"写一首关于秋天的短诗。\",\n \"中国的四大发明是什么？请简要说明。\"\n ],\n \"german\": [\n \"Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.\",\n \"Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.\",\n \"Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?\"\n ],\n \"poetry\": [\n \"Write a short poem about the sea at night.\",\n \"Compose four lines of verse about an empty train station.\",\n \"Write a haiku about the first snow of winter.\"\n ],\n \"law\": [\n \"Explain in plain terms what consideration means in contract law.\",\n \"What is the difference between a misdemeanor and a felony?\",\n \"Summarize what the term 'force majeure' covers in a commercial contract.\"\n ],\n \"medicine\": [\n \"Explain the difference between type 1 and type 2 diabetes.\",\n \"What are the common symptoms of iron deficiency anemia?\",\n \"Describe how a vaccine produces immunity.\"\n ],\n \"json_format\": [\n \"Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.\",\n \"Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.\",\n \"Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'.\"\n ],\n \"casual_chat\": [\n \"Hey, what's a good way to spend a rainy Sunday afternoon?\",\n \"I'm feeling pretty tired today. Any tips to get through the afternoon?\",\n \"What's your favourite kind of weather, and why?\"\n ]\n}\n","c/tools/README.md":"# Tools\n\nThese scripts support model preparation and offline engineering work. They are\nnot runtime dependencies of the C engine.\n\n- `convert_fp8_to_int4.py`, `download_glm52.py`: model preparation\n- `qpack_install_policy.py`: source-bound resume decisions and manifest-last\n publication primitives for future qpack download/repack frontends. Hold its\n install session for the full transfer. Qpack v1 has sizes but no content\n hashes, so frontends must also enforce immutable revisions and any available\n transport checksum or ETag. It covers manifest-declared container artifacts,\n not required runtime auxiliaries such as `config.json`. Swiftlet manifests\n vary: the 35B container declares `packed_experts/layout.json` and its\n tokenizer files, the production Qwen3-Next-80B container declares only the\n weights, so `layout.json` is verified when declared and never required\n here. Directory-fsync crash durability is POSIX-only.\n- `qpack_http_install.py`: dependency-free Hugging Face frontend for that\n policy. It resolves a branch or tag to an immutable commit before opening the\n install session, streams manifest-declared files directly into `.part`\n files, resumes only exact HTTP ranges, and publishes `manifest.json` last.\n `config.json` is required, and so is `packed_experts/layout.json`, which\n the reader opens directly: when the manifest does not declare it, it is\n fetched as a required sidecar. Manifest-declared tokenizer/configuration\n files use the same resumable artifact path, while undeclared optional files\n are copied as verified sidecars when present. Strong ETags, documented server\n SHA-256 values, and a locally computed full-file SHA-256 are persisted in\n `.qpack-http.json` and checked on resume. `HF_TOKEN` is sent only to the Hub\n origin and is stripped on cross-origin HTTPS redirects.\n- `qpack_mirror_install.py`: static HTTPS/R2 frontend over the same transfer\n engine. A strict `hashes.json` is required and fetched before the manifest;\n it must cover `manifest.json`, `config.json`, `packed_experts/layout.json`,\n and every manifest artifact.\n The legacy Swiftlet `{\"files\":{\"path\":\"sha256\"}}` schema and the sized\n `qpack.hashes.v1` schema are accepted. The normalized mirror URL plus the\n digest of the exact raw index bytes bind resume state; indexed runtime files receive\n authoritative full-object SHA-256 values, while other valid indexed\n sidecars are ignored. The exact raw index is persisted beside the installed\n container. Mirror installs never inherit\n `HF_TOKEN`; an explicit bearer token can be read from a named environment\n variable and is stripped on cross-origin redirects.\n- `convert_fmt4_to_fmt2.py`: fmt=4 (grouped int4) -> fmt=2 (per-row int4)\n re-quant of a GLM-5.2 container, for Metal-backend compatibility\n (see `docs/METAL-M1ULTRA-FMT2-REPORT.md`)\n- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8) minting a\n standalone-loadable model directory: routed experts plus the full resident\n family (attention projections including `kv_b_proj`, shared-expert and\n dense-MLP weights), norms/router/embed/lm_head copied byte-identically in\n their original BF16/F32, and the non-tensor files the loader opens at\n runtime (`config.json`, `tokenizer.json`; `generation_config.json`\n best-effort). `--mtp` repacks the multi-token-prediction head as a separate\n pass into the same outdir. Caveat: the minted `kv_b_proj` loads today but\n must not serve batched-path decode until the engine's fmt=8 absorb support\n lands (branch `f8/absorb-fmt8`); the failure is a loud crash, not silent\n corruption -- see the module docstring and `docs/FORMATS.md`.\n Synthetic-fixture-tested only, no real-shard runs yet.\n- `make_glm_oracle.py`, `make_glm_bench_model.py`: deterministic fixtures\n- `benchmark_cuda_fixture.py`, `eval_glm.py`, `fetch_benchmarks.py`: benchmarks\n- `gen_unicode.py`: tokenizer table generation\n\nRun them from `c/`, for example:\n\n```sh\npython3 tools/convert_fp8_to_int4.py --selftest\npython3 tools/make_glm_bench_model.py --output /tmp/colibri-bench\npython3 tools/qpack_http_install.py OWNER/REPO \\\n --revision main --output /path/to/model.qpack\npython3 tools/qpack_mirror_install.py \\\n https://models.example/qwen/model.qpack \\\n --output /path/to/model.qpack \\\n --hashes-sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\n```\n\nAn interrupted qpack install can be restarted with the same command. Keep the\nsource repository and revision unchanged. The installer rejects HTTP range or\nvalidator inconsistencies instead of appending uncertain bytes. It requires\n`Content-Length` for large artifacts and never stages the original BF16/FP8\ncheckpoint, but it does not convert a raw checkpoint into qpack.\n\nStatic mirrors must use HTTPS and publish objects first, `manifest.json` next,\nand `hashes.json` last as the remote completion marker. Without\n`--hashes-sha256`, the first valid TLS response is trust on first use: the\nURL-and-index identity prevents a later snapshot from resuming into those\npartials, but it cannot authenticate a mirror that was already compromised on\nthe first request. Prefer a versioned mirror prefix and an out-of-band index\ndigest for production distribution. The installer uses exact byte ranges and\n`If-Match` when a strong ETag is available; final SHA-256 verification remains\nthe authority.\n\n`make_glm_oracle.py` also produces the quantized routed-expert fixtures for the\nfmt=6 (E8/IQ3, rotation-bearing) and fmt=4 (grouped int4, no-rotation control)\nparity gate (#3/#7). Only the routed experts are quantized; shared/dense/attn\nstay f32, and the reference (`ref_glm.json` inside each fixture dir) is computed\nfrom the dequantized weights so the engine reproduces it token-exactly:\n\n```sh\npython3 tools/make_glm_oracle.py --fmt6 # -> glm_tiny_fmt6/\npython3 tools/make_glm_oracle.py --fmt4 # -> glm_tiny_fmt4/\n# verify the engine loads the formats directly (32/32 expected):\nSNAP=./glm_tiny_fmt6 REF=./glm_tiny_fmt6/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16\nSNAP=./glm_tiny_fmt4 REF=./glm_tiny_fmt4/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16\n```\n","docs/api.md":"# OpenAI-compatible API, KV contexts & web UI\n\n## `coli serve`\n\n`coli serve` keeps one model process loaded and exposes a text-only\nOpenAI-compatible HTTP API. The gateway uses only the Python standard library;\ninference still runs in the same dependency-free C engine.\n\n```bash\ncd c\nCOLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \\\n --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri\n\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Authorization: Bearer local-secret' \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"glm-5.2-colibri\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"stream\": true\n }'\n```\n\nImplemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,\n`POST /v1/chat/completions`, legacy `POST /v1/completions` and\n`POST /v1/systemone`, colibri's one decision API ([systemone.md](systemone.md)):\nthe request and reply of TypeSafe's Jev API, scored by a language model through\nits logprob channel or answered natively by a decision model such as\n[Laya](laya.md),\n[GLiNER2.5-Decide](gliner_decide.md) or [Clef](clef.md). Chat and\ncompletion requests support JSON responses, SSE streaming, usage counts,\n`max_tokens`/`max_completion_tokens`, `temperature`, `top_p`, and up to four\ncustom `stop` sequences. `max_tokens` is a ceiling, not a target: when the\nprompt leaves less room than the budget asks for, every engine clamps the\nbudget to what the context holds and the reply ends with `finish_reason:\n\"length\"`; only a prompt that does not fit is refused, with\n`context_length_exceeded` (#260, #1641). Stop sequences are removed from the response and end\ngeneration early in both JSON and streaming modes.\n\nA trailing `assistant` message *continues* that turn instead of starting a new\none: the prompt ends inside it, which is the official template rendered with\n`add_generation_prompt=False`. This is on by default — a message list ending in a\nnon-empty `assistant` turn continues, the same contract as Anthropic's API — and\nthere is no request field for it, because a trailing assistant turn already says\n\"continue me\" and a body extension would only be reachable by hand-written JSON\nrather than from the clients that want it. The server-side switch\n`COLI_CONTINUE_ASSISTANT=0` restores the old behaviour (fold the turn into a\ncompleted one and append a fresh cue). Continuation is refused together with\n`tools`/`tool_calls`, because the tool-call parsers read an assistant turn from\nits start, and the turn must carry text not ending in whitespace: the template\nstrips trailing whitespace, so the model would resume from different bytes than\nthe ones sent. A family whose renderer has no open-turn shape yet falls through\nto the old behaviour rather than erroring — though every shipped family supports\ncontinuation today, Kimi K3 included (its open turn is framed engine-side, in\n`kimi_k3.c`, not derived in the gateway renderer).\n\nA continuation resumes from the exact bytes you send, which makes the split\npoint part of the prompt. Splitting mid-word puts the model at a token boundary\nit would not have produced itself, and the first generated token is conditioned\non that split: measured on GLM-5.3-Flash, `The capital of France is Par`\ncompletes to `París`, not `Paris` — deterministically, across every effort level\nand both endpoints. The continuation is real (the model finished the partial\nword rather than restarting, which is the behaviour this feature exists for);\nthe spelling is an artefact of where the split fell. This is inherent to\nresuming from an arbitrary byte offset rather than specific to this engine, and\nit is the same hazard as the trailing whitespace above — in the one form that\ncannot be refused, because splitting mid-word is sometimes exactly what the\ncaller wants. Split at a token-ish boundary when the spelling matters. The extension\n`x_colibri_ignore_leading_stop: true` discards leading stop sequences until\nthe first non-whitespace response content, which is useful for local templates\nthat occasionally emit a role marker before the answer; strict OpenAI stop\nbehavior remains the default for client-provided sequences. GLM chat requests\nwith no client `stop` automatically use the template's `<|user|>` and\n`<|observation|>` role markers, patiently ignoring only leading markers; this\nprevents a model-completed turn from silently generating a new user or tool\nturn. Inkling chat and legacy completion requests receive no implicit GLM stop\nsequences. The extension\n`enable_thinking: true` enables GLM-5.2's reasoning block; the standard\n`reasoning_effort` field also enables it unless set to `none`.\n\nOn Qwen3.6 the extension `preserve_thinking` is the official template's kwarg of\nthe same name: past assistant turns keep their `` block, with the\n`reasoning_content` the client sends back, empty when it sends none. It\ndefaults to `true` when thinking is off and to `false` when it is on. With\nthinking off, the empty block is the one each past turn was generated after,\nso a client that resends only `content` sends the history the engine already\nholds and the KV prefix is reused instead of prefilled again (#1759). With\nthinking on a standard client drops the reasoning, the history cannot match\neither way, and the template's default applies; a client that sends\n`reasoning_content` back can set `preserve_thinking: true` to get the reuse\ntoo. Both values render byte for byte like the official `chat_template.jinja`.\n\nThe server serves one generation at a time: the model stays in one persistent\nprocess, so concurrent HTTP requests queue instead of loading duplicate model\ncopies. Tool calling depends on the active engine; see the support matrix below.\nImages and token penalties return an explicit error rather than being silently\nignored. The OpenAI-compatible endpoints request log probabilities only from a\nglm engine (see below); on every other engine such a request is refused with a\nnamed error, never silently ignored. `seed` is accepted and ignored (see\nbelow). Audio is accepted only by Inkling checkpoints with audio support. The\ndefault bind address is localhost; set `COLI_API_KEY` before exposing the\nserver beyond the machine.\n\nThe hosted-platform bookkeeping fields `store`, `metadata`, `service_tier`,\n`user`, `safety_identifier`, `parallel_tool_calls`, `prompt_cache_key`,\n`verbosity`, `web_search_options`, `moderation`, and\n`stream_options.include_obfuscation` are accepted and intentionally ignored:\nthey have no local equivalent and do not affect generation. Unsupported\nresult-shaping requests are refused explicitly: `best_of` values above 1, a\nnon-empty `logit_bias`, and `suffix` infill. The optional `modalities` array\naccepts text output only; malformed values and requests for other output\nmodalities receive a named 400 rather than silently returning text.\n\n### `seed`\n\n`seed` is accepted (not rejected) for OpenAI-API request-shape compatibility;\nthe value is not validated. This server sends no per-request seed on the\nwire, so the value has no effect at any temperature.\n\n### Log probabilities and prompt echo\n\n`/v1/completions` accepts the legacy integer `logprobs` (**0–32**; 0 means no log\nprobabilities at all, see below; the upper bound is the engine's top-32 read-out\ninterface, and anything above 32 is a named 400) and boolean `echo`;\n`/v1/chat/completions` accepts boolean `logprobs` plus integer `top_logprobs` (0–32) and\nreturns `choices[].logprobs.content[]` (`{token, logprob, bytes, top_logprobs}` per\ngenerated token) — chat has no `echo` concept and rejects one with a 400. Each endpoint\ntakes the OpenAI request shape for it and refuses the other's by name. A non-boolean\n`echo` is a named 400 (`invalid_value`) on both endpoints, independent of whether\n`logprobs` is requested at all. On chat, `top_logprobs` is type- and range-checked even\nwhen `logprobs` is false or absent, so a malformed `top_logprobs` is a named 400 whether\nor not the gate it would feed is open; a valid `top_logprobs` with `logprobs` off remains\na documented no-op.\n\nThe zero semantics are explicit, not a truthiness accident: on `/v1/completions`,\n`logprobs: 0`, `false` and `null` all mean **no log probabilities** (the request succeeds\nwith `choices[].logprobs: null`, exactly as if the field were omitted), while boolean\n`true` is a named 400 — the legacy field is an integer count, and a boolean carries no\ncount. On `/v1/chat/completions` the field is a boolean gate (`null` behaves like `false`;\nany integer is a named 400). `echo: null` normalises to absent on both endpoints, so a\nclient that serialises its whole request model with nulls is never refused for a field it\ndid not mean to set. `logprobs` together with `stream` is a named 400: per-delta log\nprobabilities are not built.\n\n`/v1/completions` with `echo: true` returns the full legacy `logprobs` object (`tokens`,\n`token_logprobs`, `top_logprobs`, `text_offset`) covering the echoed prompt plus any\ngenerated tokens, and `text` itself is the reconstructed prompt followed by the completion\n(the standard OpenAI legacy behavior for `echo: true`) rather than the completion alone;\n`echo: true` without an active `logprobs` request is a named 400 (`echo` requires\n`logprobs`): the prompt echo is built out of the engine's per-token records, so without\nthem there is nothing to echo, and a request that asks for one is told so rather than\nserved without it. Echoing a prompt with no logprobs at all is offered as a separate\nproposal. `text_offset` is a character offset into\nthat same returned `text` string, always counted from 0 — including when `echo` is false,\nwhere `text` holds only the completion and the offsets describe only that text, not a\nposition within the (unreturned) prompt.\n\n`\"\".join(tokens)` is the returned `text`: where a `stop` sequence matches partway through\na token, that token's `tokens` entry is truncated to the characters actually emitted (a\n`stop` of `\"lo\"` against a token decoding to `\"Hello\"` reports `\"Hel\"`), rather than being\ndropped or reporting characters the client did not receive. On chat, a truncated entry's\n`bytes` — and its `top_logprobs` entries' `bytes` — are truncated with it, to the UTF-8 of\nthe characters that entry reports. An entry that was not truncated keeps its frame's own\npayload, which need not be the encoding of its `token`: a frame carrying only part of a\nmulti-byte codepoint reports `token: \"\"` with that frame's bytes.\n\nThe requested top-k table is **unsorted** on the wire — do not assume the first entry is\nthe argmax. Per-token values are printed by the engine to six decimal digits of precision.\nNon-finite values (a degenerate all-`-inf` logit row, say) serialize as JSON `null`, never\na clamped number.\n\nThese endpoints request the numeric per-token channel only from a glm or a MiMo engine,\nthe two that read out every prompt position `echo` needs; on every other engine the\nrequest returns a named 400 rather than being silently ignored. That is a\nstatement about what these endpoints request, and about nothing else.\n\nKnown limitations, current build:\n\n- **Cost.** Requesting `logprobs` at all — completions or chat, `echo` or not — asks the\n engine for a full read-out pass over the whole prompt, because the request carries one\n opt-in and not a separate \"echo\" one. Such a request normally forfeits prefix-cache\n reuse. There is no cap on prompt length for it, so a very long prompt pays a\n correspondingly large one-shot cost.\n- **Waiting.** A logprobs request waits for the engine exactly as any other request waits.\n An engine that never acknowledges the submission — one older than this extension, or one\n whose parser disagrees about the header's shape — leaves the request waiting, as it\n would through a cold prefill. There is no separate bounded wait for the opt-in. A request\n whose per-token frames could not be read waits the same way: the turn belongs to the\n engine until it sends a terminal frame, so the refusal below is delivered then rather\n than the moment the unreadable frame arrives. The engine is sent one stop when that\n happens, so the turn does not spend the rest of its budget on a response nobody receives.\n- **Cancellation.** A cancel may not take effect until the read-out this request asked for\n has finished, so the window in which a disconnect goes unnoticed is wider on a long\n prompt than it is without `logprobs`.\n- **Alternative-token labels.** `top_logprobs` entries for candidate token ids other than\n the position's own actual token are not decoded text (no server-side tokenizer exists, by\n design) — they are labeled ``. Only the position's own token, identified by\n an exact logprob match rather than by id, gets its real decoded text.\n- **The sampled token is not guaranteed to appear in its own `top_logprobs` table.** The\n engine's numeric channel reports the top-k candidates by its own read-out; if the chosen\n token falls outside that table, no entry represents it, and the response's\n `token_logprobs`/`logprob` field is still the chosen token's own value read from the\n frame directly, not looked up in the table.\n- **The arrays describe the text the client received, exactly, and nothing beyond it.**\n Every generated token is located in the returned text by its span in the raw engine\n stream, so the stage that consumed a character decides how much of that token's entry\n reaches the array: a token a matched `stop` cut partway through reports the characters\n that were emitted, and generated text that does not reach `message.content` — reasoning,\n template markers, tool-call syntax, a swallowed role marker — is not described there.\n Reasoning, template and tool-call tokens are therefore not described by\n `logprobs.content`; a separate field for the full stream is proposed separately. On\n every response that carries logprobs, `\"\".join(tokens)` is the returned `text` and\n `\"\".join` over `logprobs.content` is `message.content`. When the engine's records cannot\n support that — they stop short of the generated text, or none arrive at all — the\n request is refused by name rather than answered with arrays describing a prefix.\n- **Engine-side faults are named, not generic.** When the engine's per-token records cannot\n answer the request, the 5xx body carries a `code` saying what the engine did:\n `engine_logprob_tail_malformed` (a numeric tail this server cannot read — it fails that\n one request and no other), `engine_echo_position_malformed` (an unreadable prompt-echo\n position, which is a different field of the same frame),\n `engine_duplicate_logprob_candidate` (a top-k table repeating a candidate at one\n position), `engine_logprob_records_incomplete` (records that stop\n short of the generated text, none at all, or — under `echo` — no prompt-echo records,\n any of which would leave the arrays describing a prefix), and\n `engine_pinned_prefix_not_echoed` (a KV slot holding a pin snapshot, so the echoed\n positions would not start at 0). No logprobs-bearing response is ever a truncated or\n half-described 200.\n- **Server-side buffering.** A request that asks to see the echoed prompt (`echo: true` on\n `/v1/completions`) has its full echo table held in memory for the request's lifetime; no\n streaming is allowed together with `logprobs`, so that hold is bounded by one\n non-streaming response. A logprobs request that does not ask for the echo — all of chat,\n and every `echo: false` completion — retains nothing: the engine still sends an ECHO frame\n for every prompt position, and they are dropped as they arrive rather than held.\n\n### Tool-calling support\n\n| Engine | OpenAI `tools` | Anthropic `tool_use` | Native format |\n|---|---|---|---|\n| GLM-5.2 (`colibri`) | yes | yes | `` blocks |\n| GLM-5.3-Flash | yes | yes | `` blocks, with the 5.3 declaration block |\n| DeepSeek V4 | yes | yes | native DSML tool-call blocks |\n| Inkling | no | no | active tool declarations/choices return HTTP 400 |\n| Kimi K3 | yes | yes | native XTML `tools`/`call`/`argument` blocks (#1143) |\n| MiMo-V2.6 | yes | yes | native `` / `` blocks |\n| Qwen3.6 | yes | yes | native ``/`` blocks, the same XML-ish form as Qwen3.8 |\n| Qwen3.8-Flash-Next | yes | yes | native ``/`` blocks |\n| OLMoE | no | no | active tool declarations/choices return HTTP 400 |\n\nOn supported engines, pass OpenAI `tools` and optionally `tool_choice` to\n`/v1/chat/completions`. The Anthropic endpoint translates `tools`,\n`tool_use`/`tool_result`, and the `auto`, `any`, `none`, and forced-tool choice\nmodes into the active engine's native prompt and back into protocol responses.\nProtocol support does not guarantee that every quantized model emits valid\ntool syntax; `COLI_TOOL_SALVAGE=1` is an opt-in recovery path for malformed GLM\nint4 tool calls. DeepSeek V4 uses its strict native DSML parser instead.\n\n`tool_choice: \"required\"` is a prompt-level instruction, not a sampling\nconstraint. Every renderer that offers a tool block appends the same one line to\nit, and no renderer filters tokens or forces the sampler, so a model that\nanswers in prose anyway has done nothing the API said was impossible.\nGrammar forcing is not a remedy: that path feeds a draft the engine then\nverifies, so a schema the engine cannot compile costs the speedup and nothing\nelse. An engine with no tool block to attach the instruction to (Inkling, OLMoE\nwithout `COLI_TOOL_FALLBACK=1`) answers HTTP 400 rather than accept the choice\nand ignore it.\n\nA forced choice, `tool_choice: {\"type\": \"function\", \"function\": {\"name\": …}}`,\nis applied per engine and the engines do not agree on how: some narrow the\noffered tools to the named one, some keep the full list and name the tool in\nprose instead, and some (Qwen3.6, Qwen3.8) do neither. Read the rendered prompt\nrather than assuming the request was honoured.\n\nFor GLM calls that will execute tools, an OpenAI chat request may set\n`\"strict_tool_calls\": true`. This opt-in accepts only complete ``\nblocks whose function and arguments match the declared tool schema. It returns\nHTTP 502 with code `invalid_model_tool_call` for incomplete, duplicate, unknown,\nor invalid arguments and for a tool call cut off by the generation limit. It\nnever recovers or salvages a malformed call. The default recovery behavior is\nunchanged. Strict mode currently requires `stream: false` and does not support\n`logprobs`; these combinations receive HTTP 400 before generation.\n\nWhen a reverse proxy or MagicDNS hostname preserves a public `Host` header,\ntrust that exact hostname with repeatable `--allowed-host` options. The\ncomma-separated `COLI_ALLOWED_HOSTS` environment variable is equivalent:\n\n```bash\nCOLI_ALLOWED_HOSTS=llm.example.com ./coli serve --model /nvme/glm52_i4\n# or: ./coli serve --model /nvme/glm52_i4 --allowed-host llm.example.com\n```\n\nOnly configure hostnames or IP addresses you control; there is no wildcard.\nThis setting extends the DNS-rebinding allowlist and is independent of CORS and\nAPI-key authentication.\n\nBrowser access from the Vite development server and Tauri local origins is\nenabled by default. Repeat `--cors-origin https://your-ui.example` to allow\nanother exact origin, or use `--cors-origin '*'` only on a trusted local\nnetwork.\n\nThe engine owns its KV contexts, so HTTP generation uses a bounded FIFO\nadmission queue instead of pretending to run unsafe parallel sequences.\nConfigure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS`\n(default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment\nvariables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429\nerrors before streaming headers are sent. With `--max-queue 0`, a request\npinned to an occupied KV slot is rejected immediately even if another slot\nis free. Queue deadlines are checked before slot assignment: an expired\nwaiter receives `queue_timeout` even if a slot is now available. `GET /health` exposes\nactive/queued/completed/rejected counters, and successful generation responses\ninclude `x-colibri-queue-wait-ms`.\nRequests targeting any slot may use a free slot not reserved by earlier waiters.\nAn earlier pinned request keeps priority for its target; an earlier any-slot\nrequest keeps priority across all slots. A full waiting queue does not reject\na request that can immediately take an unreserved free slot; the queue limit\nbounds waiting requests, independently of active capacity.\n\n## Prometheus metrics\n\n`GET /metrics` returns Prometheus text exposition (version 0.0.4). When an\nAPI key is configured, supply the same `Authorization: Bearer ...` or\n`x-api-key` header used for generation; missing or invalid credentials return\n401. Without an API key, the endpoint follows the server's usual unauthenticated\naccess policy. Metrics contain no prompts, model paths, or request-ID labels.\n\nAll names start with `colibri_scheduler_`:\n\n| Suffix | Type | Meaning |\n|---|---|---|\n| `active`, `queued`, `capacity`, `max_queue` | gauge | Admitted requests, waiters, KV slot capacity, and queue limit |\n| `admitted_total` | counter | Requests admitted to a KV slot |\n| `completed_total` | counter | Admitted requests that returned normally |\n| `failed_total` | counter | Admitted requests that raised an error, excluding `ClientCancelled` |\n| `rejected_total`, `timed_out_total` | counter | Queue-full refusals and queue timeouts |\n| `cancelled_total` | counter | Cancellations detected before admission or during admitted work |\n| `queue_wait_seconds` | histogram | Wait until admission, for admitted requests only |\n| `slot_duration_seconds` | histogram | Slot occupancy until completion, failure, or cancellation |\n| `first_output_seconds` | histogram | Engine-call start to first nonempty text or tool-output callback |\n| `engine_call_seconds` | histogram | Duration of each finished engine generation call, including failure/cancellation |\n\nHistogram buckets are 0.001, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30, 60, 300 seconds,\nand `+Inf`; each histogram exposes `_bucket`, `_sum`, and `_count`.\nCounters reset when the gateway restarts. Collection does not call the engine\nor consume a generation slot. `failed` is also included in `/health`'s\nauthenticated scheduler snapshot; failures no longer increment `completed`.\n\nCancellation is checked before acquiring an available KV slot, including when a\nwaiting request wakes as capacity becomes free. A request cancelled at this\npoint increments `cancelled_total`, but not `admitted_total`, and contributes\nno admission-wait or slot-duration sample. Queue-full and scheduler-closed\nchecks can still reject a request before the cancellation check is reached.\n\n\nThe engine-call histograms exclude admission queue wait and prompt rendering.\nFirst output is observed before the gateway's stop filtering, reasoning split,\nor HTTP serialization: it can be reasoning or tool data, not necessarily\nuser-visible answer text. Empty callbacks, ACCEPT frames, and SSE keepalives do\nnot count. Calls that finish or fail without output add no first-output sample;\na failure after output retains that sample. Engine-call duration includes callback\nprocessing and response writes during generation. One request can invoke the\nengine multiple times (for example System One scoring), so these histogram counts are\nengine calls, not HTTP request counts.\n\nThese are gateway observations, not end-to-end client TTFT, per-token latency,\nor GPU kernel measurements. Output callbacks need not correspond one-to-one to\ntokens. Slot occupancy includes any response handling while the slot is held. Validation/authentication failures before admission are not counted.\n`completed` means the admitted handler returned normally, not that the client\nreceived every response byte. Request exceptions can include client input or\ntransport errors as well as engine failures.\n\nExample PromQL for the admitted-request queue-wait p95:\n\n```promql\nhistogram_quantile(0.95, sum by (le) (rate(colibri_scheduler_queue_wait_seconds_bucket[5m])))\n```\n\n## Anthropic-protocol endpoint (`/v1/messages`)\n\nThe same server also speaks the **Anthropic Messages API**, so clients that only talk\nto Anthropic endpoints — Claude Code, the Anthropic SDKs — work against colibri\nwithout a shim. Nothing to enable: `/v1/messages` is served alongside\n`/v1/chat/completions` on the same port.\n\n```bash\ncurl http://127.0.0.1:8000/v1/messages \\\n -H 'x-api-key: local' -H 'content-type: application/json' \\\n -d '{\"model\":\"glm-5.2-colibri\",\"max_tokens\":128,\n \"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}'\n```\n\nFor Claude Code, point it at the server and give it any non-empty key:\n\n```bash\nexport ANTHROPIC_BASE_URL=http://localhost:8000\nexport ANTHROPIC_API_KEY=local # only enforced if you set COLI_API_KEY\nexport ANTHROPIC_MODEL=glm-5.2-colibri\nclaude\n```\n\nSupported on every served architecture: system prompts (string or text blocks),\nmulti-turn `user`/`assistant` messages, streaming with the full named-event\nsequence (`message_start` → `content_block_*` → `message_delta` → `message_stop`,\nplus protocol `ping` keepalives during long prefills), `stop_reason`, Anthropic\n`usage` field names, and `x-api-key` authentication (`Authorization: Bearer`\nalso works). The gateway renders each request with the active engine's native\nchat template; GLM, Inkling, Kimi K3, Qwen3.8, OLMoE, and DeepSeek V4 prompts are not\ninterchangeable. Where the engine exposes a reasoning mode, extended thinking\nis enabled with `{\"thinking\": {\"type\": \"enabled\"}}` and translated to that\narchitecture's reasoning protocol; OLMoE disables it explicitly.\n\nTool use follows the per-engine matrix above. Unsupported engines reject active\ntool declarations and choices explicitly instead of feeding another\narchitecture's markers to an incompatible tokenizer.\n\nNot supported, and refused explicitly rather than ignored: `stop_sequences`,\n`top_k`, and non-text content blocks (images, documents). Errors use Anthropic's own\n`{\"type\":\"error\",\"error\":{...}}` envelope on this path. Architecture-local\nfeatures that have not been wired to this protocol are likewise rejected with\nan explicit error.\n\nA trailing `assistant` message continues that turn by default on both the\nAnthropic- and OpenAI-compatible endpoints (`COLI_CONTINUE_ASSISTANT=0` restores\nthe old behavior, where this endpoint appended a fresh cue). Note this changes\nwhat an existing Anthropic client sees on `/v1/messages`: a trailing assistant\nturn now continues rather than starting fresh — which is the real Anthropic\ncontract — and the off-switch is the escape hatch for anyone relying on the old\nbehavior.\n\n> The prefill warning below applies here too, and applies *hardest* to Claude Code:\n> its system prompt and tool catalog are large, and on a disk-streaming CPU path\n> that is a long silent wait before the first token. Read it before you connect.\n\n## Connect a coding CLI or editor\n\nThe API is OpenAI-compatible, so most coding CLIs and editor extensions work by\npointing them at Colibri as an *OpenAI-compatible* provider. Three settings:\n\n- **Base URL** — `http://localhost:8000/v1`\n- **Model** — `glm-5.2-colibri` (or whatever you pass to `--model-id`)\n- **API key** — any non-empty string, e.g. `local`\n\nColibri needs **no** API key by default, but many clients refuse to start without\none — give them any dummy value. The key is only enforced if you set `COLI_API_KEY`.\n\nSmoke-test the endpoint first (no key needed unless you set one):\n\n```bash\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\"model\":\"glm-5.2-colibri\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\n```\n\n**aider**\n\n```bash\nexport OPENAI_API_BASE=http://localhost:8000/v1\nexport OPENAI_API_KEY=local\naider --model openai/glm-5.2-colibri # the openai/ prefix routes to OPENAI_API_BASE\n```\n\n**crush** — add a provider to `crush.json` (`~/.config/crush/crush.json`, or\n`%USERPROFILE%\\AppData\\Local\\crush\\crush.json` on Windows):\n\n```json\n{\n \"$schema\": \"https://charm.land/crush.json\",\n \"providers\": {\n \"colibri\": {\n \"name\": \"Colibri\",\n \"type\": \"openai-compat\",\n \"base_url\": \"http://localhost:8000/v1/\",\n \"api_key\": \"local\",\n \"models\": [\n { \"name\": \"GLM-5.2 (Colibri)\", \"id\": \"glm-5.2-colibri\",\n \"context_window\": 131072, \"default_max_tokens\": 1024 }\n ]\n }\n }\n}\n```\n\nThe `\"api_key\": \"local\"` dummy is what satisfies clients that demand a key.\n`context_window` is only the client's budget display — set it to whatever your\nKV configuration actually allows.\n\n**pi** — add a custom provider to `~/.pi/agent/models.json` ([pi](https://github.com/earendil-works/pi-coding-agent) loads every OpenAI-compatible server through its `openai-completions` API):\n\n```json\n{\n \"providers\": {\n \"colibri\": {\n \"baseUrl\": \"http://localhost:8000/v1\",\n \"api\": \"openai-completions\",\n \"apiKey\": \"local\",\n \"compat\": {\n \"supportsDeveloperRole\": false,\n \"supportsReasoningEffort\": false\n },\n \"models\": [\n {\n \"id\": \"glm-5.2-colibri\",\n \"name\": \"GLM-5.2 (Colibri)\",\n \"contextWindow\": 131072,\n \"maxTokens\": 1024\n }\n ]\n }\n }\n}\n```\n\nThe `apiKey` dummy satisfies pi's auth requirement; colibri only enforces a\nkey if you set `COLI_API_KEY`. The `compat` flags tell pi to send the system\nprompt as a plain `system` message and to omit `reasoning_effort`, which keeps\nthe request inside what the gateway accepts by default. If you serve GLM-5.2\nand want its reasoning block, set `\"reasoning\": true` on the model and drop\n`supportsReasoningEffort` — the standard `reasoning_effort` field enables\nthinking on that engine. Then select the model with `pi --list-models` or the\n`/model` picker (`colibri / GLM-5.2 (Colibri)`).\n\n`contextWindow` is only the client's budget display — set it to whatever your\nKV configuration actually allows. Tool calling in pi works on the engines in\nthe [tool-calling matrix](#tool-calling-support) above; on unsupported engines\npi's tool calls fail the same way any OpenAI `tools` request does.\n\n**Continue, Cline / Roo, `llm`, the OpenAI SDKs, …** — set the provider's base\nURL to `http://localhost:8000/v1`, the model to `glm-5.2-colibri`, and any dummy\nkey (`OPENAI_API_KEY` / `OPENAI_BASE_URL` for env-based tools).\n\n> **Set your expectations before connecting an agentic CLI.** Two costs dominate,\n> and the first one is invisible until you know it's there:\n>\n> 1. **Prefill.** Coding agents (crush, aider in repo-map mode, Cline, …) send a\n> large system prompt plus tool definitions — often 10–20k tokens — *before\n> your first word*. Prefill on the CPU-streaming path runs at a few tokens per\n> second (it is attention-bound, see #153), so a 15k-token agent preamble is\n> **an hour of silent \"thinking\" before the first output token**. The client\n> looks hung; it isn't. Smoke-test with the tiny `curl` above first — if that\n> answers in about a minute, the pipeline works and what you're paying for is\n> prompt size.\n> 2. **Decode.** Roughly 1 tok/s for a large model, so multi-turn agent loops\n> (which re-pay the growing context every turn) compound the cost.\n>\n> Practical guidance: single surgical asks with a short context work; iterative\n> agent sessions against a disk-streaming 744B model do not resemble a hosted\n> API and mostly won't be worth the wait. If your client lets you trim or disable\n> its system preamble and tool catalog, do it.\n\n## Isolated KV contexts\n\n`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.\nRequests select one with the optional integer `cache_slot` field; ordinary\nOpenAI clients omit it and keep the original slot 0 behavior.\n\n```json\n{\n \"model\": \"glm-5.2-colibri\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Continue this conversation\"}],\n \"cache_slot\": 1\n}\n```\n\nEach slot owns its token history and its conversation's state: the KV cache, and\non the engines that have them the DeltaNet, KDA and convolution states and the\ncompressed DeepSeek attention. The engine matches each request's tokenized prompt\nagainst the slot's history and reuses the common prefix, so stateless HTTP turns\nkeep their cache across requests. On GLM-5.2 each slot also has its MTP window and\na crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...), so the cache\nsurvives an engine restart too. Use `COLI_KV_SLOTS=N` as the environment\nequivalent. Start small: at the default 4096-token context, every slot costs\nhundreds of MB, and `coli plan` counts them.\n\nEvery text engine serves the slots at the same time. Requests on different slots\nare decoded together: each step takes the next token of every active\nconversation as one batch, so the weights and the routed experts a step reads\nserve all of them, while each row's attention and recurrent state stay its own\nconversation's. A request gets the tokens it would get alone (greedy on the CPU,\nthe same bytes; `tests/serve_mux_check.py` checks it on every engine). A prompt is\nprefilled when it arrives, between two steps. With more than one slot, the\nengines other than GLM-5.2 and GLM-5.3-Flash draft nothing (MTP, DSpark and\nprompt lookup follow one conversation), and their Vulkan dense chain, DeltaNet on\nthe GPU and Metal paths stay off; the Vulkan expert tier serves every\nconversation's experts.\n\n## Web dashboard\n\nOne command serves the OpenAI-compatible API **and** the web console on the\nsame port, then opens your browser when the engine is ready:\n\n```bash\ncd web && npm install && npm run build # once\n./coli web --model \n```\n\n`coli web` differs from `coli serve` only in opening a browser — both serve the\ndashboard on the same port. On a headless host (no display, often no GPU at all)\nuse `coli serve`, or `coli web --no-browser`, and point a browser at it from\nanother machine. Nothing in the dashboard needs a desktop session on the host.\n\nWhat you get is one workspace with a dock to switch page:\n\n- **Chat**: streaming answers, a reasoning toggle, image input where the engine\n supports it, the KV slot to answer in, and the conversation exported as a file;\n- **System One**: closed-set answers. A document, a question and the only answers\n allowed; the engine reads the probability of each answer, generates nothing,\n and reports an entropy that says when it is not sure. Same thing as\n `POST /v1/systemone` (see [systemone.md](systemone.md));\n- **Brain**, two views. *Explore* draws the\n [measured expert atlas](https://github.com/JustVugg/colibri/issues/175) of GLM-5.2 as a cortex with ten regions to\n enter (publish `experts.json` from `tools/expert_atlas/analyze.py --web`).\n *Live routing* shows the model actually running: one cell per expert,\n colour = tier, brightness = routing heat, and the experts routed in each\n turn flash white and decay;\n- **Profiling**: where the engine spends each turn, by phase (I/O wait, expert\n matmul, attention, LM head, other), the disk service overlapped with compute,\n and the last 30 turns as a trend;\n- a light and a dark theme.\n\nThe dashboard talks to the engine over a small line protocol and plain JSON\nendpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API\nclient (React + TypeScript) and also works against any other compatible\nendpoint; the terminal `coli chat` remains the first-class interface.\n\nThe layout is responsive down to phone widths; the per-turn time breakdown and\nthe tok/s trend live on the Profiling page:\n\n

\n \"the\n   \n \"the\n

\n"},"items":[{"name":"api.md","path":"docs/api.md","rawUrl":"https://raw.githubusercontent.com/JustVugg/colibri/HEAD/docs/api.md","title":"docs - API Interface Contract & Specifications","category":"project-spec","format":"markdown","content":"# OpenAI-compatible API, KV contexts & web UI\n\n## `coli serve`\n\n`coli serve` keeps one model process loaded and exposes a text-only\nOpenAI-compatible HTTP API. The gateway uses only the Python standard library;\ninference still runs in the same dependency-free C engine.\n\n```bash\ncd c\nCOLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \\\n --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri\n\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Authorization: Bearer local-secret' \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"glm-5.2-colibri\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"stream\": true\n }'\n```\n\nImplemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,\n`POST /v1/chat/completions`, legacy `POST /v1/completions` and\n`POST /v1/systemone`, colibri's one decision API ([systemone.md](systemone.md)):\nthe request and reply of TypeSafe's Jev API, scored by a language model through\nits logprob channel or answered natively by a decision model such as\n[Laya](laya.md),\n[GLiNER2.5-Decide](gliner_decide.md) or [Clef](clef.md). Chat and\ncompletion requests support JSON responses, SSE streaming, usage counts,\n`max_tokens`/`max_completion_tokens`, `temperature`, `top_p`, and up to four\ncustom `stop` sequences. `max_tokens` is a ceiling, not a target: when the\nprompt leaves less room than the budget asks for, every engine clamps the\nbudget to what the context holds and the reply ends with `finish_reason:\n\"length\"`; only a prompt that does not fit is refused, with\n`context_length_exceeded` (#260, #1641). Stop sequences are removed from the response and end\ngeneration early in both JSON and streaming modes.\n\nA trailing `assistant` message *continues* that turn instead of starting a new\none: the prompt ends inside it, which is the official template rendered with\n`add_generation_prompt=False`. This is on by default — a message list ending in a\nnon-empty `assistant` turn continues, the same contract as Anthropic's API — and\nthere is no request field for it, because a trailing assistant turn already says\n\"continue me\" and a body extension would only be reachable by hand-written JSON\nrather than from the clients that want it. The server-side switch\n`COLI_CONTINUE_ASSISTANT=0` restores the old behaviour (fold the turn into a\ncompleted one and append a fresh cue). Continuation is refused together with\n`tools`/`tool_calls`, because the tool-call parsers read an assistant turn from\nits start, and the turn must carry text not ending in whitespace: the template\nstrips trailing whitespace, so the model would resume from different bytes than\nthe ones sent. A family whose renderer has no open-turn shape yet falls through\nto the old behaviour rather than erroring — though every shipped family supports\ncontinuation today, Kimi K3 included (its open turn is framed engine-side, in\n`kimi_k3.c`, not derived in the gateway renderer).\n\nA continuation resumes from the exact bytes you send, which makes the split\npoint part of the prompt. Splitting mid-word puts the model at a token boundary\nit would not have produced itself, and the first generated token is conditioned\non that split: measured on GLM-5.3-Flash, `The capital of France is Par`\ncompletes to `París`, not `Paris` — deterministically, across every effort level\nand both endpoints. The continuation is real (the model finished the partial\nword rather than restarting, which is the behaviour this feature exists for);\nthe spelling is an artefact of where the split fell. This is inherent to\nresuming from an arbitrary byte offset rather than specific to this engine, and\nit is the same hazard as the trailing whitespace above — in the one form that\ncannot be refused, because splitting mid-word is sometimes exactly what the\ncaller wants. Split at a token-ish boundary when the spelling matters. The extension\n`x_colibri_ignore_leading_stop: true` discards leading stop sequences until\nthe first non-whitespace response content, which is useful for local templates\nthat occasionally emit a role marker before the answer; strict OpenAI stop\nbehavior remains the default for client-provided sequences. GLM chat requests\nwith no client `stop` automatically use the template's `<|user|>` and\n`<|observation|>` role markers, patiently ignoring only leading markers; this\nprevents a model-completed turn from silently generating a new user or tool\nturn. Inkling chat and legacy completion requests receive no implicit GLM stop\nsequences. The extension\n`enable_thinking: true` enables GLM-5.2's reasoning block; the standard\n`reasoning_effort` field also enables it unless set to `none`.\n\nOn Qwen3.6 the extension `preserve_thinking` is the official template's kwarg of\nthe same name: past assistant turns keep their `` block, with the\n`reasoning_content` the client sends back, empty when it sends none. It\ndefaults to `true` when thinking is off and to `false` when it is on. With\nthinking off, the empty block is the one each past turn was generated after,\nso a client that resends only `content` sends the history the engine already\nholds and the KV prefix is reused instead of prefilled again (#1759). With\nthinking on a standard client drops the reasoning, the history cannot match\neither way, and the template's default applies; a client that sends\n`reasoning_content` back can set `preserve_thinking: true` to get the reuse\ntoo. Both values render byte for byte like the official `chat_template.jinja`.\n\nThe server serves one generation at a time: the model stays in one persistent\nprocess, so concurrent HTTP requests queue instead of loading duplicate model\ncopies. Tool calling depends on the active engine; see the support matrix below.\nImages and token penalties return an explicit error rather than being silently\nignored. The OpenAI-compatible endpoints request log probabilities only from a\nglm engine (see below); on every other engine such a request is refused with a\nnamed error, never silently ignored. `seed` is accepted and ignored (see\nbelow). Audio is accepted only by Inkling checkpoints with audio support. The\ndefault bind address is localhost; set `COLI_API_KEY` before exposing the\nserver beyond the machine.\n\nThe hosted-platform bookkeeping fields `store`, `metadata`, `service_tier`,\n`user`, `safety_identifier`, `parallel_tool_calls`, `prompt_cache_key`,\n`verbosity`, `web_search_options`, `moderation`, and\n`stream_options.include_obfuscation` are accepted and intentionally ignored:\nthey have no local equivalent and do not affect generation. Unsupported\nresult-shaping requests are refused explicitly: `best_of` values above 1, a\nnon-empty `logit_bias`, and `suffix` infill. The optional `modalities` array\naccepts text output only; malformed values and requests for other output\nmodalities receive a named 400 rather than silently returning text.\n\n### `seed`\n\n`seed` is accepted (not rejected) for OpenAI-API request-shape compatibility;\nthe value is not validated. This server sends no per-request seed on the\nwire, so the value has no effect at any temperature.\n\n### Log probabilities and prompt echo\n\n`/v1/completions` accepts the legacy integer `logprobs` (**0–32**; 0 means no log\nprobabilities at all, see below; the upper bound is the engine's top-32 read-out\ninterface, and anything above 32 is a named 400) and boolean `echo`;\n`/v1/chat/completions` accepts boolean `logprobs` plus integer `top_logprobs` (0–32) and\nreturns `choices[].logprobs.content[]` (`{token, logprob, bytes, top_logprobs}` per\ngenerated token) — chat has no `echo` concept and rejects one with a 400. Each endpoint\ntakes the OpenAI request shape for it and refuses the other's by name. A non-boolean\n`echo` is a named 400 (`invalid_value`) on both endpoints, independent of whether\n`logprobs` is requested at all. On chat, `top_logprobs` is type- and range-checked even\nwhen `logprobs` is false or absent, so a malformed `top_logprobs` is a named 400 whether\nor not the gate it would feed is open; a valid `top_logprobs` with `logprobs` off remains\na documented no-op.\n\nThe zero semantics are explicit, not a truthiness accident: on `/v1/completions`,\n`logprobs: 0`, `false` and `null` all mean **no log probabilities** (the request succeeds\nwith `choices[].logprobs: null`, exactly as if the field were omitted), while boolean\n`true` is a named 400 — the legacy field is an integer count, and a boolean carries no\ncount. On `/v1/chat/completions` the field is a boolean gate (`null` behaves like `false`;\nany integer is a named 400). `echo: null` normalises to absent on both endpoints, so a\nclient that serialises its whole request model with nulls is never refused for a field it\ndid not mean to set. `logprobs` together with `stream` is a named 400: per-delta log\nprobabilities are not built.\n\n`/v1/completions` with `echo: true` returns the full legacy `logprobs` object (`tokens`,\n`token_logprobs`, `top_logprobs`, `text_offset`) covering the echoed prompt plus any\ngenerated tokens, and `text` itself is the reconstructed prompt followed by the completion\n(the standard OpenAI legacy behavior for `echo: true`) rather than the completion alone;\n`echo: true` without an active `logprobs` request is a named 400 (`echo` requires\n`logprobs`): the prompt echo is built out of the engine's per-token records, so without\nthem there is nothing to echo, and a request that asks for one is told so rather than\nserved without it. Echoing a prompt with no logprobs at all is offered as a separate\nproposal. `text_offset` is a character offset into\nthat same returned `text` string, always counted from 0 — including when `echo` is false,\nwhere `text` holds only the completion and the offsets describe only that text, not a\nposition within the (unreturned) prompt.\n\n`\"\".join(tokens)` is the returned `text`: where a `stop` sequence matches partway through\na token, that token's `tokens` entry is truncated to the characters actually emitted (a\n`stop` of `\"lo\"` against a token decoding to `\"Hello\"` reports `\"Hel\"`), rather than being\ndropped or reporting characters the client did not receive. On chat, a truncated entry's\n`bytes` — and its `top_logprobs` entries' `bytes` — are truncated with it, to the UTF-8 of\nthe characters that entry reports. An entry that was not truncated keeps its frame's own\npayload, which need not be the encoding of its `token`: a frame carrying only part of a\nmulti-byte codepoint reports `token: \"\"` with that frame's bytes.\n\nThe requested top-k table is **unsorted** on the wire — do not assume the first entry is\nthe argmax. Per-token values are printed by the engine to six decimal digits of precision.\nNon-finite values (a degenerate all-`-inf` logit row, say) serialize as JSON `null`, never\na clamped number.\n\nThese endpoints request the numeric per-token channel only from a glm or a MiMo engine,\nthe two that read out every prompt position `echo` needs; on every other engine the\nrequest returns a named 400 rather than being silently ignored. That is a\nstatement about what these endpoints request, and about nothing else.\n\nKnown limitations, current build:\n\n- **Cost.** Requesting `logprobs` at all — completions or chat, `echo` or not — asks the\n engine for a full read-out pass over the whole prompt, because the request carries one\n opt-in and not a separate \"echo\" one. Such a request normally forfeits prefix-cache\n reuse. There is no cap on prompt length for it, so a very long prompt pays a\n correspondingly large one-shot cost.\n- **Waiting.** A logprobs request waits for the engine exactly as any other request waits.\n An engine that never acknowledges the submission — one older than this extension, or one\n whose parser disagrees about the header's shape — leaves the request waiting, as it\n would through a cold prefill. There is no separate bounded wait for the opt-in. A request\n whose per-token frames could not be read waits the same way: the turn belongs to the\n engine until it sends a terminal frame, so the refusal below is delivered then rather\n than the moment the unreadable frame arrives. The engine is sent one stop when that\n happens, so the turn does not spend the rest of its budget on a response nobody receives.\n- **Cancellation.** A cancel may not take effect until the read-out this request asked for\n has finished, so the window in which a disconnect goes unnoticed is wider on a long\n prompt than it is without `logprobs`.\n- **Alternative-token labels.** `top_logprobs` entries for candidate token ids other than\n the position's own actual token are not decoded text (no server-side tokenizer exists, by\n design) — they are labeled ``. Only the position's own token, identified by\n an exact logprob match rather than by id, gets its real decoded text.\n- **The sampled token is not guaranteed to appear in its own `top_logprobs` table.** The\n engine's numeric channel reports the top-k candidates by its own read-out; if the chosen\n token falls outside that table, no entry represents it, and the response's\n `token_logprobs`/`logprob` field is still the chosen token's own value read from the\n frame directly, not looked up in the table.\n- **The arrays describe the text the client received, exactly, and nothing beyond it.**\n Every generated token is located in the returned text by its span in the raw engine\n stream, so the stage that consumed a character decides how much of that token's entry\n reaches the array: a token a matched `stop` cut partway through reports the characters\n that were emitted, and generated text that does not reach `message.content` — reasoning,\n template markers, tool-call syntax, a swallowed role marker — is not described there.\n Reasoning, template and tool-call tokens are therefore not described by\n `logprobs.content`; a separate field for the full stream is proposed separately. On\n every response that carries logprobs, `\"\".join(tokens)` is the returned `text` and\n `\"\".join` over `logprobs.content` is `message.content`. When the engine's records cannot\n support that — they stop short of the generated text, or none arrive at all — the\n request is refused by name rather than answered with arrays describing a prefix.\n- **Engine-side faults are named, not generic.** When the engine's per-token records cannot\n answer the request, the 5xx body carries a `code` saying what the engine did:\n `engine_logprob_tail_malformed` (a numeric tail this server cannot read — it fails that\n one request and no other), `engine_echo_position_malformed` (an unreadable prompt-echo\n position, which is a different field of the same frame),\n `engine_duplicate_logprob_candidate` (a top-k table repeating a candidate at one\n position), `engine_logprob_records_incomplete` (records that stop\n short of the generated text, none at all, or — under `echo` — no prompt-echo records,\n any of which would leave the arrays describing a prefix), and\n `engine_pinned_prefix_not_echoed` (a KV slot holding a pin snapshot, so the echoed\n positions would not start at 0). No logprobs-bearing response is ever a truncated or\n half-described 200.\n- **Server-side buffering.** A request that asks to see the echoed prompt (`echo: true` on\n `/v1/completions`) has its full echo table held in memory for the request's lifetime; no\n streaming is allowed together with `logprobs`, so that hold is bounded by one\n non-streaming response. A logprobs request that does not ask for the echo — all of chat,\n and every `echo: false` completion — retains nothing: the engine still sends an ECHO frame\n for every prompt position, and they are dropped as they arrive rather than held.\n\n### Tool-calling support\n\n| Engine | OpenAI `tools` | Anthropic `tool_use` | Native format |\n|---|---|---|---|\n| GLM-5.2 (`colibri`) | yes | yes | `` blocks |\n| GLM-5.3-Flash | yes | yes | `` blocks, with the 5.3 declaration block |\n| DeepSeek V4 | yes | yes | native DSML tool-call blocks |\n| Inkling | no | no | active tool declarations/choices return HTTP 400 |\n| Kimi K3 | yes | yes | native XTML `tools`/`call`/`argument` blocks (#1143) |\n| MiMo-V2.6 | yes | yes | native `` / `` blocks |\n| Qwen3.6 | yes | yes | native ``/`` blocks, the same XML-ish form as Qwen3.8 |\n| Qwen3.8-Flash-Next | yes | yes | native ``/`` blocks |\n| OLMoE | no | no | active tool declarations/choices return HTTP 400 |\n\nOn supported engines, pass OpenAI `tools` and optionally `tool_choice` to\n`/v1/chat/completions`. The Anthropic endpoint translates `tools`,\n`tool_use`/`tool_result`, and the `auto`, `any`, `none`, and forced-tool choice\nmodes into the active engine's native prompt and back into protocol responses.\nProtocol support does not guarantee that every quantized model emits valid\ntool syntax; `COLI_TOOL_SALVAGE=1` is an opt-in recovery path for malformed GLM\nint4 tool calls. DeepSeek V4 uses its strict native DSML parser instead.\n\n`tool_choice: \"required\"` is a prompt-level instruction, not a sampling\nconstraint. Every renderer that offers a tool block appends the same one line to\nit, and no renderer filters tokens or forces the sampler, so a model that\nanswers in prose anyway has done nothing the API said was impossible.\nGrammar forcing is not a remedy: that path feeds a draft the engine then\nverifies, so a schema the engine cannot compile costs the speedup and nothing\nelse. An engine with no tool block to attach the instruction to (Inkling, OLMoE\nwithout `COLI_TOOL_FALLBACK=1`) answers HTTP 400 rather than accept the choice\nand ignore it.\n\nA forced choice, `tool_choice: {\"type\": \"function\", \"function\": {\"name\": …}}`,\nis applied per engine and the engines do not agree on how: some narrow the\noffered tools to the named one, some keep the full list and name the tool in\nprose instead, and some (Qwen3.6, Qwen3.8) do neither. Read the rendered prompt\nrather than assuming the request was honoured.\n\nFor GLM calls that will execute tools, an OpenAI chat request may set\n`\"strict_tool_calls\": true`. This opt-in accepts only complete ``\nblocks whose function and arguments match the declared tool schema. It returns\nHTTP 502 with code `invalid_model_tool_call` for incomplete, duplicate, unknown,\nor invalid arguments and for a tool call cut off by the generation limit. It\nnever recovers or salvages a malformed call. The default recovery behavior is\nunchanged. Strict mode currently requires `stream: false` and does not support\n`logprobs`; these combinations receive HTTP 400 before generation.\n\nWhen a reverse proxy or MagicDNS hostname preserves a public `Host` header,\ntrust that exact hostname with repeatable `--allowed-host` options. The\ncomma-separated `COLI_ALLOWED_HOSTS` environment variable is equivalent:\n\n```bash\nCOLI_ALLOWED_HOSTS=llm.example.com ./coli serve --model /nvme/glm52_i4\n# or: ./coli serve --model /nvme/glm52_i4 --allowed-host llm.example.com\n```\n\nOnly configure hostnames or IP addresses you control; there is no wildcard.\nThis setting extends the DNS-rebinding allowlist and is independent of CORS and\nAPI-key authentication.\n\nBrowser access from the Vite development server and Tauri local origins is\nenabled by default. Repeat `--cors-origin https://your-ui.example` to allow\nanother exact origin, or use `--cors-origin '*'` only on a trusted local\nnetwork.\n\nThe engine owns its KV contexts, so HTTP generation uses a bounded FIFO\nadmission queue instead of pretending to run unsafe parallel sequences.\nConfigure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS`\n(default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment\nvariables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429\nerrors before streaming headers are sent. With `--max-queue 0`, a request\npinned to an occupied KV slot is rejected immediately even if another slot\nis free. Queue deadlines are checked before slot assignment: an expired\nwaiter receives `queue_timeout` even if a slot is now available. `GET /health` exposes\nactive/queued/completed/rejected counters, and successful generation responses\ninclude `x-colibri-queue-wait-ms`.\nRequests targeting any slot may use a free slot not reserved by earlier waiters.\nAn earlier pinned request keeps priority for its target; an earlier any-slot\nrequest keeps priority across all slots. A full waiting queue does not reject\na request that can immediately take an unreserved free slot; the queue limit\nbounds waiting requests, independently of active capacity.\n\n## Prometheus metrics\n\n`GET /metrics` returns Prometheus text exposition (version 0.0.4). When an\nAPI key is configured, supply the same `Authorization: Bearer ...` or\n`x-api-key` header used for generation; missing or invalid credentials return\n401. Without an API key, the endpoint follows the server's usual unauthenticated\naccess policy. Metrics contain no prompts, model paths, or request-ID labels.\n\nAll names start with `colibri_scheduler_`:\n\n| Suffix | Type | Meaning |\n|---|---|---|\n| `active`, `queued`, `capacity`, `max_queue` | gauge | Admitted requests, waiters, KV slot capacity, and queue limit |\n| `admitted_total` | counter | Requests admitted to a KV slot |\n| `completed_total` | counter | Admitted requests that returned normally |\n| `failed_total` | counter | Admitted requests that raised an error, excluding `ClientCancelled` |\n| `rejected_total`, `timed_out_total` | counter | Queue-full refusals and queue timeouts |\n| `cancelled_total` | counter | Cancellations detected before admission or during admitted work |\n| `queue_wait_seconds` | histogram | Wait until admission, for admitted requests only |\n| `slot_duration_seconds` | histogram | Slot occupancy until completion, failure, or cancellation |\n| `first_output_seconds` | histogram | Engine-call start to first nonempty text or tool-output callback |\n| `engine_call_seconds` | histogram | Duration of each finished engine generation call, including failure/cancellation |\n\nHistogram buckets are 0.001, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30, 60, 300 seconds,\nand `+Inf`; each histogram exposes `_bucket`, `_sum`, and `_count`.\nCounters reset when the gateway restarts. Collection does not call the engine\nor consume a generation slot. `failed` is also included in `/health`'s\nauthenticated scheduler snapshot; failures no longer increment `completed`.\n\nCancellation is checked before acquiring an available KV slot, including when a\nwaiting request wakes as capacity becomes free. A request cancelled at this\npoint increments `cancelled_total`, but not `admitted_total`, and contributes\nno admission-wait or slot-duration sample. Queue-full and scheduler-closed\nchecks can still reject a request before the cancellation check is reached.\n\n\nThe engine-call histograms exclude admission queue wait and prompt rendering.\nFirst output is observed before the gateway's stop filtering, reasoning split,\nor HTTP serialization: it can be reasoning or tool data, not necessarily\nuser-visible answer text. Empty callbacks, ACCEPT frames, and SSE keepalives do\nnot count. Calls that finish or fail without output add no first-output sample;\na failure after output retains that sample. Engine-call duration includes callback\nprocessing and response writes during generation. One request can invoke the\nengine multiple times (for example System One scoring), so these histogram counts are\nengine calls, not HTTP request counts.\n\nThese are gateway observations, not end-to-end client TTFT, per-token latency,\nor GPU kernel measurements. Output callbacks need not correspond one-to-one to\ntokens. Slot occupancy includes any response handling while the slot is held. Validation/authentication failures before admission are not counted.\n`completed` means the admitted handler returned normally, not that the client\nreceived every response byte. Request exceptions can include client input or\ntransport errors as well as engine failures.\n\nExample PromQL for the admitted-request queue-wait p95:\n\n```promql\nhistogram_quantile(0.95, sum by (le) (rate(colibri_scheduler_queue_wait_seconds_bucket[5m])))\n```\n\n## Anthropic-protocol endpoint (`/v1/messages`)\n\nThe same server also speaks the **Anthropic Messages API**, so clients that only talk\nto Anthropic endpoints — Claude Code, the Anthropic SDKs — work against colibri\nwithout a shim. Nothing to enable: `/v1/messages` is served alongside\n`/v1/chat/completions` on the same port.\n\n```bash\ncurl http://127.0.0.1:8000/v1/messages \\\n -H 'x-api-key: local' -H 'content-type: application/json' \\\n -d '{\"model\":\"glm-5.2-colibri\",\"max_tokens\":128,\n \"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}'\n```\n\nFor Claude Code, point it at the server and give it any non-empty key:\n\n```bash\nexport ANTHROPIC_BASE_URL=http://localhost:8000\nexport ANTHROPIC_API_KEY=local # only enforced if you set COLI_API_KEY\nexport ANTHROPIC_MODEL=glm-5.2-colibri\nclaude\n```\n\nSupported on every served architecture: system prompts (string or text blocks),\nmulti-turn `user`/`assistant` messages, streaming with the full named-event\nsequence (`message_start` → `content_block_*` → `message_delta` → `message_stop`,\nplus protocol `ping` keepalives during long prefills), `stop_reason`, Anthropic\n`usage` field names, and `x-api-key` authentication (`Authorization: Bearer`\nalso works). The gateway renders each request with the active engine's native\nchat template; GLM, Inkling, Kimi K3, Qwen3.8, OLMoE, and DeepSeek V4 prompts are not\ninterchangeable. Where the engine exposes a reasoning mode, extended thinking\nis enabled with `{\"thinking\": {\"type\": \"enabled\"}}` and translated to that\narchitecture's reasoning protocol; OLMoE disables it explicitly.\n\nTool use follows the per-engine matrix above. Unsupported engines reject active\ntool declarations and choices explicitly instead of feeding another\narchitecture's markers to an incompatible tokenizer.\n\nNot supported, and refused explicitly rather than ignored: `stop_sequences`,\n`top_k`, and non-text content blocks (images, documents). Errors use Anthropic's own\n`{\"type\":\"error\",\"error\":{...}}` envelope on this path. Architecture-local\nfeatures that have not been wired to this protocol are likewise rejected with\nan explicit error.\n\nA trailing `assistant` message continues that turn by default on both the\nAnthropic- and OpenAI-compatible endpoints (`COLI_CONTINUE_ASSISTANT=0` restores\nthe old behavior, where this endpoint appended a fresh cue). Note this changes\nwhat an existing Anthropic client sees on `/v1/messages`: a trailing assistant\nturn now continues rather than starting fresh — which is the real Anthropic\ncontract — and the off-switch is the escape hatch for anyone relying on the old\nbehavior.\n\n> The prefill warning below applies here too, and applies *hardest* to Claude Code:\n> its system prompt and tool catalog are large, and on a disk-streaming CPU path\n> that is a long silent wait before the first token. Read it before you connect.\n\n## Connect a coding CLI or editor\n\nThe API is OpenAI-compatible, so most coding CLIs and editor extensions work by\npointing them at Colibri as an *OpenAI-compatible* provider. Three settings:\n\n- **Base URL** — `http://localhost:8000/v1`\n- **Model** — `glm-5.2-colibri` (or whatever you pass to `--model-id`)\n- **API key** — any non-empty string, e.g. `local`\n\nColibri needs **no** API key by default, but many clients refuse to start without\none — give them any dummy value. The key is only enforced if you set `COLI_API_KEY`.\n\nSmoke-test the endpoint first (no key needed unless you set one):\n\n```bash\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\"model\":\"glm-5.2-colibri\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\n```\n\n**aider**\n\n```bash\nexport OPENAI_API_BASE=http://localhost:8000/v1\nexport OPENAI_API_KEY=local\naider --model openai/glm-5.2-colibri # the openai/ prefix routes to OPENAI_API_BASE\n```\n\n**crush** — add a provider to `crush.json` (`~/.config/crush/crush.json`, or\n`%USERPROFILE%\\AppData\\Local\\crush\\crush.json` on Windows):\n\n```json\n{\n \"$schema\": \"https://charm.land/crush.json\",\n \"providers\": {\n \"colibri\": {\n \"name\": \"Colibri\",\n \"type\": \"openai-compat\",\n \"base_url\": \"http://localhost:8000/v1/\",\n \"api_key\": \"local\",\n \"models\": [\n { \"name\": \"GLM-5.2 (Colibri)\", \"id\": \"glm-5.2-colibri\",\n \"context_window\": 131072, \"default_max_tokens\": 1024 }\n ]\n }\n }\n}\n```\n\nThe `\"api_key\": \"local\"` dummy is what satisfies clients that demand a key.\n`context_window` is only the client's budget display — set it to whatever your\nKV configuration actually allows.\n\n**pi** — add a custom provider to `~/.pi/agent/models.json` ([pi](https://github.com/earendil-works/pi-coding-agent) loads every OpenAI-compatible server through its `openai-completions` API):\n\n```json\n{\n \"providers\": {\n \"colibri\": {\n \"baseUrl\": \"http://localhost:8000/v1\",\n \"api\": \"openai-completions\",\n \"apiKey\": \"local\",\n \"compat\": {\n \"supportsDeveloperRole\": false,\n \"supportsReasoningEffort\": false\n },\n \"models\": [\n {\n \"id\": \"glm-5.2-colibri\",\n \"name\": \"GLM-5.2 (Colibri)\",\n \"contextWindow\": 131072,\n \"maxTokens\": 1024\n }\n ]\n }\n }\n}\n```\n\nThe `apiKey` dummy satisfies pi's auth requirement; colibri only enforces a\nkey if you set `COLI_API_KEY`. The `compat` flags tell pi to send the system\nprompt as a plain `system` message and to omit `reasoning_effort`, which keeps\nthe request inside what the gateway accepts by default. If you serve GLM-5.2\nand want its reasoning block, set `\"reasoning\": true` on the model and drop\n`supportsReasoningEffort` — the standard `reasoning_effort` field enables\nthinking on that engine. Then select the model with `pi --list-models` or the\n`/model` picker (`colibri / GLM-5.2 (Colibri)`).\n\n`contextWindow` is only the client's budget display — set it to whatever your\nKV configuration actually allows. Tool calling in pi works on the engines in\nthe [tool-calling matrix](#tool-calling-support) above; on unsupported engines\npi's tool calls fail the same way any OpenAI `tools` request does.\n\n**Continue, Cline / Roo, `llm`, the OpenAI SDKs, …** — set the provider's base\nURL to `http://localhost:8000/v1`, the model to `glm-5.2-colibri`, and any dummy\nkey (`OPENAI_API_KEY` / `OPENAI_BASE_URL` for env-based tools).\n\n> **Set your expectations before connecting an agentic CLI.** Two costs dominate,\n> and the first one is invisible until you know it's there:\n>\n> 1. **Prefill.** Coding agents (crush, aider in repo-map mode, Cline, …) send a\n> large system prompt plus tool definitions — often 10–20k tokens — *before\n> your first word*. Prefill on the CPU-streaming path runs at a few tokens per\n> second (it is attention-bound, see #153), so a 15k-token agent preamble is\n> **an hour of silent \"thinking\" before the first output token**. The client\n> looks hung; it isn't. Smoke-test with the tiny `curl` above first — if that\n> answers in about a minute, the pipeline works and what you're paying for is\n> prompt size.\n> 2. **Decode.** Roughly 1 tok/s for a large model, so multi-turn agent loops\n> (which re-pay the growing context every turn) compound the cost.\n>\n> Practical guidance: single surgical asks with a short context work; iterative\n> agent sessions against a disk-streaming 744B model do not resemble a hosted\n> API and mostly won't be worth the wait. If your client lets you trim or disable\n> its system preamble and tool catalog, do it.\n\n## Isolated KV contexts\n\n`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.\nRequests select one with the optional integer `cache_slot` field; ordinary\nOpenAI clients omit it and keep the original slot 0 behavior.\n\n```json\n{\n \"model\": \"glm-5.2-colibri\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Continue this conversation\"}],\n \"cache_slot\": 1\n}\n```\n\nEach slot owns its token history and its conversation's state: the KV cache, and\non the engines that have them the DeltaNet, KDA and convolution states and the\ncompressed DeepSeek attention. The engine matches each request's tokenized prompt\nagainst the slot's history and reuses the common prefix, so stateless HTTP turns\nkeep their cache across requests. On GLM-5.2 each slot also has its MTP window and\na crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...), so the cache\nsurvives an engine restart too. Use `COLI_KV_SLOTS=N` as the environment\nequivalent. Start small: at the default 4096-token context, every slot costs\nhundreds of MB, and `coli plan` counts them.\n\nEvery text engine serves the slots at the same time. Requests on different slots\nare decoded together: each step takes the next token of every active\nconversation as one batch, so the weights and the routed experts a step reads\nserve all of them, while each row's attention and recurrent state stay its own\nconversation's. A request gets the tokens it would get alone (greedy on the CPU,\nthe same bytes; `tests/serve_mux_check.py` checks it on every engine). A prompt is\nprefilled when it arrives, between two steps. With more than one slot, the\nengines other than GLM-5.2 and GLM-5.3-Flash draft nothing (MTP, DSpark and\nprompt lookup follow one conversation), and their Vulkan dense chain, DeltaNet on\nthe GPU and Metal paths stay off; the Vulkan expert tier serves every\nconversation's experts.\n\n## Web dashboard\n\nOne command serves the OpenAI-compatible API **and** the web console on the\nsame port, then opens your browser when the engine is ready:\n\n```bash\ncd web && npm install && npm run build # once\n./coli web --model \n```\n\n`coli web` differs from `coli serve` only in opening a browser — both serve the\ndashboard on the same port. On a headless host (no display, often no GPU at all)\nuse `coli serve`, or `coli web --no-browser`, and point a browser at it from\nanother machine. Nothing in the dashboard needs a desktop session on the host.\n\nWhat you get is one workspace with a dock to switch page:\n\n- **Chat**: streaming answers, a reasoning toggle, image input where the engine\n supports it, the KV slot to answer in, and the conversation exported as a file;\n- **System One**: closed-set answers. A document, a question and the only answers\n allowed; the engine reads the probability of each answer, generates nothing,\n and reports an entropy that says when it is not sure. Same thing as\n `POST /v1/systemone` (see [systemone.md](systemone.md));\n- **Brain**, two views. *Explore* draws the\n [measured expert atlas](https://github.com/JustVugg/colibri/issues/175) of GLM-5.2 as a cortex with ten regions to\n enter (publish `experts.json` from `tools/expert_atlas/analyze.py --web`).\n *Live routing* shows the model actually running: one cell per expert,\n colour = tier, brightness = routing heat, and the experts routed in each\n turn flash white and decay;\n- **Profiling**: where the engine spends each turn, by phase (I/O wait, expert\n matmul, attention, LM head, other), the disk service overlapped with compute,\n and the last 30 turns as a trend;\n- a light and a dark theme.\n\nThe dashboard talks to the engine over a small line protocol and plain JSON\nendpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API\nclient (React + TypeScript) and also works against any other compatible\nendpoint; the terminal `coli chat` remains the first-class interface.\n\nThe layout is responsive down to phone widths; the per-turn time breakdown and\nthe tok/s trend live on the Profiling page:\n\n

\n \"the\n   \n \"the\n

\n","isInternal":false,"tokens":8720,"sizeBytes":35840},{"name":"probes.json","path":"c/tools/expert_atlas/probes.json","rawUrl":"https://raw.githubusercontent.com/JustVugg/colibri/HEAD/c/tools/expert_atlas/probes.json","title":"Expert_atlas Skill","category":"anthropic-skill","format":"json","content":"{\n \"_comment\": \"Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.\",\n\n \"code_python\": [\n \"Write a Python function that merges two sorted lists into one sorted list.\",\n \"In Python, explain the difference between a generator and a list comprehension.\",\n \"Refactor this Python snippet to avoid a nested loop: for a in xs: for b in ys: if a==b: out.append(a)\"\n ],\n \"code_sql\": [\n \"Write a SQL query that returns the top 5 customers by total order value.\",\n \"Explain what a LEFT JOIN does differently from an INNER JOIN in SQL.\",\n \"Write SQL to add an index on the email column of a users table and explain when it helps.\"\n ],\n \"math_proof\": [\n \"Prove that the square root of 2 is irrational.\",\n \"Show that the sum of the first n odd numbers equals n squared.\",\n \"Explain why the derivative of e^x is e^x.\"\n ],\n \"chinese\": [\n \"请用三句话解释什么是机器学习。\",\n \"写一首关于秋天的短诗。\",\n \"中国的四大发明是什么？请简要说明。\"\n ],\n \"german\": [\n \"Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.\",\n \"Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.\",\n \"Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?\"\n ],\n \"poetry\": [\n \"Write a short poem about the sea at night.\",\n \"Compose four lines of verse about an empty train station.\",\n \"Write a haiku about the first snow of winter.\"\n ],\n \"law\": [\n \"Explain in plain terms what consideration means in contract law.\",\n \"What is the difference between a misdemeanor and a felony?\",\n \"Summarize what the term 'force majeure' covers in a commercial contract.\"\n ],\n \"medicine\": [\n \"Explain the difference between type 1 and type 2 diabetes.\",\n \"What are the common symptoms of iron deficiency anemia?\",\n \"Describe how a vaccine produces immunity.\"\n ],\n \"json_format\": [\n \"Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.\",\n \"Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.\",\n \"Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'.\"\n ],\n \"casual_chat\": [\n \"Hey, what's a good way to spend a rainy Sunday afternoon?\",\n \"I'm feeling pretty tired today. Any tips to get through the afternoon?\",\n \"What's your favourite kind of weather, and why?\"\n ]\n}\n","frontmatter":{"_comment":"Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.","code_python":["Write a Python function that merges two sorted lists into one sorted list.","In Python, explain the difference between a generator and a list comprehension.","Refactor this Python snippet to avoid a nested loop: for a in xs: for b in ys: if a==b: out.append(a)"],"code_sql":["Write a SQL query that returns the top 5 customers by total order value.","Explain what a LEFT JOIN does differently from an INNER JOIN in SQL.","Write SQL to add an index on the email column of a users table and explain when it helps."],"math_proof":["Prove that the square root of 2 is irrational.","Show that the sum of the first n odd numbers equals n squared.","Explain why the derivative of e^x is e^x."],"chinese":["请用三句话解释什么是机器学习。","写一首关于秋天的短诗。","中国的四大发明是什么？请简要说明。"],"german":["Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.","Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.","Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?"],"poetry":["Write a short poem about the sea at night.","Compose four lines of verse about an empty train station.","Write a haiku about the first snow of winter."],"law":["Explain in plain terms what consideration means in contract law.","What is the difference between a misdemeanor and a felony?","Summarize what the term 'force majeure' covers in a commercial contract."],"medicine":["Explain the difference between type 1 and type 2 diabetes.","What are the common symptoms of iron deficiency anemia?","Describe how a vaccine produces immunity."],"json_format":["Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.","Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.","Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'."],"casual_chat":["Hey, what's a good way to spend a rainy Sunday afternoon?","I'm feeling pretty tired today. Any tips to get through the afternoon?","What's your favourite kind of weather, and why?"]},"isInternal":false,"tokens":691,"sizeBytes":2731},{"name":"iq3xxs_grid.json","path":"c/tools/iq3xxs_grid.json","rawUrl":"https://raw.githubusercontent.com/JustVugg/colibri/HEAD/c/tools/iq3xxs_grid.json","title":"Tools Skill","category":"anthropic-skill","format":"json","content":"[[4, 4, 4, 4], [20, 4, 4, 4], [36, 4, 4, 4], [12, 12, 4, 4], [28, 12, 4, 4], [62, 12, 4, 4], [4, 20, 4, 4], [20, 20, 4, 4], [12, 28, 4, 4], [20, 36, 4, 4], [28, 62, 4, 4], [44, 62, 4, 4], [12, 4, 12, 4], [28, 4, 12, 4], [4, 12, 12, 4], [20, 12, 12, 4], [12, 20, 12, 4], [44, 20, 12, 4], [4, 28, 12, 4], [20, 28, 12, 4], [12, 36, 12, 4], [36, 44, 12, 4], [4, 62, 12, 4], [4, 4, 20, 4], [20, 4, 20, 4], [36, 4, 20, 4], [12, 12, 20, 4], [4, 20, 20, 4], [20, 20, 20, 4], [12, 28, 20, 4], [28, 28, 20, 4], [62, 28, 20, 4], [12, 44, 20, 4], [62, 44, 20, 4], [44, 62, 20, 4], [12, 4, 28, 4], [62, 4, 28, 4], [4, 12, 28, 4], [20, 12, 28, 4], [44, 20, 28, 4], [4, 62, 28, 4], [28, 12, 36, 4], [62, 28, 36, 4], [36, 36, 36, 4], [62, 44, 36, 4], [28, 62, 36, 4], [44, 62, 36, 4], [12, 4, 44, 4], [62, 4, 44, 4], [20, 28, 44, 4], [20, 44, 44, 4], [44, 28, 52, 4], [36, 52, 52, 4], [4, 12, 62, 4], [36, 12, 62, 4], [52, 12, 62, 4], [28, 36, 62, 4], [12, 52, 62, 4], [12, 4, 4, 12], [28, 4, 4, 12], [4, 12, 4, 12], [20, 12, 4, 12], [12, 20, 4, 12], [28, 20, 4, 12], [4, 28, 4, 12], [20, 28, 4, 12], [36, 28, 4, 12], [62, 36, 4, 12], [4, 44, 4, 12], [4, 4, 12, 12], [20, 4, 12, 12], [12, 12, 12, 12], [4, 20, 12, 12], [20, 20, 12, 12], [12, 4, 20, 12], [28, 4, 20, 12], [4, 12, 20, 12], [20, 12, 20, 12], [12, 20, 20, 12], [4, 28, 20, 12], [20, 62, 20, 12], [4, 4, 28, 12], [20, 4, 28, 12], [4, 20, 28, 12], [12, 28, 28, 12], [52, 36, 28, 12], [52, 52, 28, 12], [12, 4, 36, 12], [44, 4, 36, 12], [4, 44, 36, 12], [4, 20, 44, 12], [36, 20, 44, 12], [52, 36, 44, 12], [12, 62, 44, 12], [44, 4, 52, 12], [20, 20, 62, 12], [4, 36, 62, 12], [4, 4, 4, 20], [20, 4, 4, 20], [12, 12, 4, 20], [28, 12, 4, 20], [4, 20, 4, 20], [20, 20, 4, 20], [52, 20, 4, 20], [12, 28, 4, 20], [20, 36, 4, 20], [12, 4, 12, 20], [28, 4, 12, 20], [44, 4, 12, 20], [4, 12, 12, 20], [20, 12, 12, 20], [12, 20, 12, 20], [4, 28, 12, 20], [28, 52, 12, 20], [62, 52, 12, 20], [4, 62, 12, 20], [4, 4, 20, 20], [20, 4, 20, 20], [12, 12, 20, 20], [62, 12, 20, 20], [4, 20, 20, 20], [20, 20, 20, 20], [62, 28, 20, 20], [4, 36, 20, 20], [44, 44, 20, 20], [12, 4, 28, 20], [4, 12, 28, 20], [36, 12, 28, 20], [4, 62, 28, 20], [36, 62, 28, 20], [44, 28, 36, 20], [28, 44, 36, 20], [28, 4, 44, 20], [62, 20, 44, 20], [12, 36, 44, 20], [36, 62, 44, 20], [12, 4, 62, 20], [28, 4, 62, 20], [52, 12, 62, 20], [44, 36, 62, 20], [12, 4, 4, 28], [4, 12, 4, 28], [20, 12, 4, 28], [12, 20, 4, 28], [28, 20, 4, 28], [4, 44, 4, 28], [44, 52, 4, 28], [20, 62, 4, 28], [4, 4, 12, 28], [20, 4, 12, 28], [4, 20, 12, 28], [12, 28, 12, 28], [36, 36, 12, 28], [52, 36, 12, 28], [12, 4, 20, 28], [28, 4, 20, 28], [4, 12, 20, 28], [44, 20, 20, 28], [20, 44, 20, 28], [20, 62, 20, 28], [12, 12, 28, 28], [28, 28, 28, 28], [4, 28, 36, 28], [62, 36, 36, 28], [20, 62, 36, 28], [4, 4, 44, 28], [52, 4, 44, 28], [20, 20, 44, 28], [44, 44, 44, 28], [36, 12, 52, 28], [52, 28, 52, 28], [28, 52, 52, 28], [28, 28, 62, 28], [4, 52, 62, 28], [36, 4, 4, 36], [62, 12, 4, 36], [44, 28, 4, 36], [62, 28, 4, 36], [28, 44, 4, 36], [62, 44, 4, 36], [36, 62, 12, 36], [4, 20, 20, 36], [62, 28, 20, 36], [4, 36, 20, 36], [4, 52, 20, 36], [52, 52, 20, 36], [62, 4, 28, 36], [44, 36, 28, 36], [36, 4, 36, 36], [12, 44, 36, 36], [36, 52, 36, 36], [44, 20, 44, 36], [28, 36, 44, 36], [4, 62, 44, 36], [44, 4, 62, 36], [4, 12, 62, 36], [20, 12, 62, 36], [4, 28, 62, 36], [20, 12, 4, 44], [12, 36, 4, 44], [4, 62, 4, 44], [4, 4, 12, 44], [52, 4, 12, 44], [52, 20, 12, 44], [44, 44, 12, 44], [36, 12, 20, 44], [20, 28, 20, 44], [20, 62, 20, 44], [20, 4, 28, 44], [28, 44, 28, 44], [4, 12, 36, 44], [28, 20, 36, 44], [62, 20, 36, 44], [20, 62, 36, 44], [20, 4, 44, 44], [12, 28, 44, 44], [4, 44, 52, 44], [36, 20, 62, 44], [20, 36, 62, 44], [36, 20, 4, 52], [36, 36, 4, 52], [52, 36, 4, 52], [36, 52, 4, 52], [12, 20, 12, 52], [12, 52, 12, 52], [62, 12, 20, 52], [36, 52, 20, 52], [4, 28, 28, 52], [52, 28, 28, 52], [36, 36, 36, 52], [44, 4, 44, 52], [20, 44, 44, 52], [28, 28, 52, 52], [28, 4, 62, 52], [12, 20, 62, 52], [28, 4, 4, 62], [44, 4, 4, 62], [62, 4, 4, 62], [4, 12, 4, 62], [20, 28, 4, 62], [20, 44, 4, 62], [52, 20, 12, 62], [4, 36, 12, 62], [20, 12, 20, 62], [44, 36, 20, 62], [20, 44, 20, 62], [4, 4, 28, 62], [44, 12, 28, 62], [28, 28, 28, 62], [4, 52, 28, 62], [12, 20, 36, 62], [12, 36, 36, 62], [4, 4, 44, 62], [20, 4, 44, 62], [36, 20, 44, 62], [4, 28, 52, 62]]","isInternal":false,"tokens":3072,"sizeBytes":4377},{"name":"README.md","path":"c/tools/expert_atlas/README.md","rawUrl":"https://raw.githubusercontent.com/JustVugg/colibri/HEAD/c/tools/expert_atlas/README.md","title":"expert_atlas Documentation","category":"plugin-manifest","format":"markdown","content":"# Expert Atlas — what does each of the 19,456 experts actually do?\n\nProbe harness for #175. Runs a set of topic-tagged prompts, dumps each run's expert-routing\nhistogram, and turns them into a per-expert topic-affinity vector.\n\n```bash\ncd c\nexport COLI_MODEL=/path/to/glm52_i4\n./tools/expert_atlas/sweep.sh # 30 probes (10 topics x 3 prompts)\npython3 tools/expert_atlas/analyze.py --stats atlas_out/stats --out atlas_out/experts.json \\\n --web web/dist/experts.json # optional: feed the web dashboard Atlas\npython3 tools/expert_atlas/validate.py atlas_out/stats 200 # leave-one-prompt-out check\n```\n\n`--web` writes the same atlas in the shape the web dashboard consumes (the Atlas galaxy and the\nBrain hover tooltips): keyed `\"layer:expert\"` with `affinity`/`entropy`/`top`/`label`. It replaces\nthe retired `tools/expert_atlas.py`, whose API-driven probing ran through a live server and was\nexposed to exactly the traps above (server-side `--topp`, speculative drafts, shared `.coli_usage`).\n\n## Read this before you trust any atlas\n\nFour things silently corrupt this measurement. The sweep script controls all of them; if you\nroll your own, don't skip them.\n\n| trap | effect | control |\n|---|---|---|\n| **`--topp`** | prunes experts by cumulative probability — measured: it hides **38% of the distinct experts** (7,587 → 4,687). It is also the *recommended speed setting*. | `TOPP=0` |\n| **speculative drafts** | `eusage` is incremented inside `moe()`, *before* verification, so **rejected** drafts count. Those are experts routed for text the model never emitted. | `MTP=0 DRAFT=0` |\n| **`.coli_usage`** | is loaded at startup and accumulates, so a naive `STATS` dump contains **all prior history**, not this run. | remove per run (script backs it up and restores) |\n| **autocorrelation** | routing inside one run is highly correlated — the same context routes to the same experts token after token. An expert firing 38 times during one prompt is **one** observation, not 38. Chi-square/entropy on raw selections will certify single-prompt flukes as perfect specialists. | `analyze.py` requires the affinity to **replicate across a category's independent prompts** |\n\nThe CUDA expert tier is also not run-to-run deterministic, so the sweep uses `--gpu none`. Tier\nconfig only decides *where weights live*, not what the router picks, so this costs nothing.\n\n## Method\n\n`analyze.py`:\n\n1. `n[e][c]` — selections of expert *e* while running category *c*\n2. `f[e][c] = n[e][c] / N[c]` — normalise by **category size** (prefill routes the prompt too, so a\n verbose category would otherwise look busier)\n3. `p(c|e)` — renormalise into a topic distribution per expert, i.e. base-rate corrected. Ranking\n by raw count instead just rediscovers which experts are popular in general.\n4. `spec(e) = 1 − H(p(c|e)) / log C` — 0 = generalist, 1 = fires for exactly one topic\n5. **replication gate** — an expert is only a candidate specialist for *c* if it fires in ≥2 of *c*'s\n independent prompts\n\n`validate.py` — leave-one-prompt-out. Build each category's top-K specialist set from its *other*\nprompts, then check which set the held-out prompt's routing actually lands in. If specialisation\nwere an artifact of prompt wording, the held-out prompt would not prefer its own category.\n\n## Result on GLM-5.2 744B int4 (Zen5, CPU routing path)\n\n- **Leave-one-prompt-out: 29/30 = 96.7%** (chance 10%). Specialisation is a property of the topic,\n not of prompt wording.\n- The single miss is instructive: `写一首关于秋天的短诗` (\"write a short poem about autumn\") is\n classified **poetry**, not Chinese — routing follows the **task** over the **language**.\n- **Only 7.9% of experts are strong specialists** (spec ≥ 0.5). The \"one expert = one topic\"\n picture is wrong for ~92% of them.\n- Specialisation **rises with depth**: layer 3 ≈ 0.07 (generalist, token/syntax level) → layers\n 18–58 ≈ 0.19–0.27.\n- The replication gate removed **587** experts that looked like flawless specialists on one prompt.\n\n## Extending it\n\n`probes.json` is the whole probe set — add categories and prompts. Use **3+ prompts per category\nwith varied phrasing**, or the replication gate has nothing to check and you are back to measuring\none prompt. Keep prompt lengths in a similar band across categories.\n","isInternal":false,"tokens":1133,"sizeBytes":4383},{"name":"README.md","path":"c/tools/README.md","rawUrl":"https://raw.githubusercontent.com/JustVugg/colibri/HEAD/c/tools/README.md","title":"tools Documentation","category":"plugin-manifest","format":"markdown","content":"# Tools\n\nThese scripts support model preparation and offline engineering work. They are\nnot runtime dependencies of the C engine.\n\n- `convert_fp8_to_int4.py`, `download_glm52.py`: model preparation\n- `qpack_install_policy.py`: source-bound resume decisions and manifest-last\n publication primitives for future qpack download/repack frontends. Hold its\n install session for the full transfer. Qpack v1 has sizes but no content\n hashes, so frontends must also enforce immutable revisions and any available\n transport checksum or ETag. It covers manifest-declared container artifacts,\n not required runtime auxiliaries such as `config.json`. Swiftlet manifests\n vary: the 35B container declares `packed_experts/layout.json` and its\n tokenizer files, the production Qwen3-Next-80B container declares only the\n weights, so `layout.json` is verified when declared and never required\n here. Directory-fsync crash durability is POSIX-only.\n- `qpack_http_install.py`: dependency-free Hugging Face frontend for that\n policy. It resolves a branch or tag to an immutable commit before opening the\n install session, streams manifest-declared files directly into `.part`\n files, resumes only exact HTTP ranges, and publishes `manifest.json` last.\n `config.json` is required, and so is `packed_experts/layout.json`, which\n the reader opens directly: when the manifest does not declare it, it is\n fetched as a required sidecar. Manifest-declared tokenizer/configuration\n files use the same resumable artifact path, while undeclared optional files\n are copied as verified sidecars when present. Strong ETags, documented server\n SHA-256 values, and a locally computed full-file SHA-256 are persisted in\n `.qpack-http.json` and checked on resume. `HF_TOKEN` is sent only to the Hub\n origin and is stripped on cross-origin HTTPS redirects.\n- `qpack_mirror_install.py`: static HTTPS/R2 frontend over the same transfer\n engine. A strict `hashes.json` is required and fetched before the manifest;\n it must cover `manifest.json`, `config.json`, `packed_experts/layout.json`,\n and every manifest artifact.\n The legacy Swiftlet `{\"files\":{\"path\":\"sha256\"}}` schema and the sized\n `qpack.hashes.v1` schema are accepted. The normalized mirror URL plus the\n digest of the exact raw index bytes bind resume state; indexed runtime files receive\n authoritative full-object SHA-256 values, while other valid indexed\n sidecars are ignored. The exact raw index is persisted beside the installed\n container. Mirror installs never inherit\n `HF_TOKEN`; an explicit bearer token can be read from a named environment\n variable and is stripped on cross-origin redirects.\n- `convert_fmt4_to_fmt2.py`: fmt=4 (grouped int4) -> fmt=2 (per-row int4)\n re-quant of a GLM-5.2 container, for Metal-backend compatibility\n (see `docs/METAL-M1ULTRA-FMT2-REPORT.md`)\n- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8) minting a\n standalone-loadable model directory: routed experts plus the full resident\n family (attention projections including `kv_b_proj`, shared-expert and\n dense-MLP weights), norms/router/embed/lm_head copied byte-identically in\n their original BF16/F32, and the non-tensor files the loader opens at\n runtime (`config.json`, `tokenizer.json`; `generation_config.json`\n best-effort). `--mtp` repacks the multi-token-prediction head as a separate\n pass into the same outdir. Caveat: the minted `kv_b_proj` loads today but\n must not serve batched-path decode until the engine's fmt=8 absorb support\n lands (branch `f8/absorb-fmt8`); the failure is a loud crash, not silent\n corruption -- see the module docstring and `docs/FORMATS.md`.\n Synthetic-fixture-tested only, no real-shard runs yet.\n- `make_glm_oracle.py`, `make_glm_bench_model.py`: deterministic fixtures\n- `benchmark_cuda_fixture.py`, `eval_glm.py`, `fetch_benchmarks.py`: benchmarks\n- `gen_unicode.py`: tokenizer table generation\n\nRun them from `c/`, for example:\n\n```sh\npython3 tools/convert_fp8_to_int4.py --selftest\npython3 tools/make_glm_bench_model.py --output /tmp/colibri-bench\npython3 tools/qpack_http_install.py OWNER/REPO \\\n --revision main --output /path/to/model.qpack\npython3 tools/qpack_mirror_install.py \\\n https://models.example/qwen/model.qpack \\\n --output /path/to/model.qpack \\\n --hashes-sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\n```\n\nAn interrupted qpack install can be restarted with the same command. Keep the\nsource repository and revision unchanged. The installer rejects HTTP range or\nvalidator inconsistencies instead of appending uncertain bytes. It requires\n`Content-Length` for large artifacts and never stages the original BF16/FP8\ncheckpoint, but it does not convert a raw checkpoint into qpack.\n\nStatic mirrors must use HTTPS and publish objects first, `manifest.json` next,\nand `hashes.json` last as the remote completion marker. Without\n`--hashes-sha256`, the first valid TLS response is trust on first use: the\nURL-and-index identity prevents a later snapshot from resuming into those\npartials, but it cannot authenticate a mirror that was already compromised on\nthe first request. Prefer a versioned mirror prefix and an out-of-band index\ndigest for production distribution. The installer uses exact byte ranges and\n`If-Match` when a strong ETag is available; final SHA-256 verification remains\nthe authority.\n\n`make_glm_oracle.py` also produces the quantized routed-expert fixtures for the\nfmt=6 (E8/IQ3, rotation-bearing) and fmt=4 (grouped int4, no-rotation control)\nparity gate (#3/#7). Only the routed experts are quantized; shared/dense/attn\nstay f32, and the reference (`ref_glm.json` inside each fixture dir) is computed\nfrom the dequantized weights so the engine reproduces it token-exactly:\n\n```sh\npython3 tools/make_glm_oracle.py --fmt6 # -> glm_tiny_fmt6/\npython3 tools/make_glm_oracle.py --fmt4 # -> glm_tiny_fmt4/\n# verify the engine loads the formats directly (32/32 expected):\nSNAP=./glm_tiny_fmt6 REF=./glm_tiny_fmt6/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16\nSNAP=./glm_tiny_fmt4 REF=./glm_tiny_fmt4/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16\n```\n","isInternal":false,"tokens":1526,"sizeBytes":6137}],"systemPromptSnippet":"\n\n# OpenAI-compatible API, KV contexts & web UI\n\n## `coli serve`\n\n`coli serve` keeps one model process loaded and exposes a text-only\nOpenAI-compatible HTTP API. The gateway uses only the Python standard library;\ninference still runs in the same dependency-free C engine.\n\n```bash\ncd c\nCOLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \\\n --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri\n\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Authorization: Bearer local-secret' \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"glm-5.2-colibri\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"stream\": true\n }'\n```\n\nImplemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,\n`POST /v1/chat/completions`, legacy `POST /v1/completions` and\n`POST /v1/systemone`, colibri's one decision API ([systemone.md](systemone.md)):\nthe request and reply of TypeSafe's Jev API, scored by a language model through\nits logprob channel or answered natively by a decision model such as\n[Laya](laya.md),\n[GLiNER2.5-Decide](gliner_decide.md) or [Clef](clef.md). Chat and\ncompletion requests support JSON responses, SSE streaming, usage counts,\n`max_tokens`/`max_completion_tokens`, `temperature`, `top_p`, and up to four\ncustom `stop` sequences. `max_tokens` is a ceiling, not a target: when the\nprompt leaves less room than the budget asks for, every engine clamps the\nbudget to what the context holds and the reply ends with `finish_reason:\n\"length\"`; only a prompt that does not fit is refused, with\n`context_length_exceeded` (#260, #1641). Stop sequences are removed from the response and end\ngeneration early in both JSON and streaming modes.\n\nA trailing `assistant` message *continues* that turn instead of starting a new\none: the prompt ends inside it, which is the official template rendered with\n`add_generation_prompt=False`. This is on by default — a message list ending in a\nnon-empty `assistant` turn continues, the same contract as Anthropic's API — and\nthere is no request field for it, because a trailing assistant turn already says\n\"continue me\" and a body extension would only be reachable by hand-written JSON\nrather than from the clients that want it. The server-side switch\n`COLI_CONTINUE_ASSISTANT=0` restores the old behaviour (fold the turn into a\ncompleted one and append a fresh cue). Continuation is refused together with\n`tools`/`tool_calls`, because the tool-call parsers read an assistant turn from\nits start, and the turn must carry text not ending in whitespace: the template\nstrips trailing whitespace, so the model would resume from different bytes than\nthe ones sent. A family whose renderer has no open-turn shape yet falls through\nto the old behaviour rather than erroring — though every shipped family supports\ncontinuation today, Kimi K3 included (its open turn is framed engine-side, in\n`kimi_k3.c`, not derived in the gateway renderer).\n\nA continuation resumes from the exact bytes you send, which makes the split\npoint part of the prompt. Splitting mid-word puts the model at a token boundary\nit would not have produced itself, and the first generated token is conditioned\non that split: measured on GLM-5.3-Flash, `The capital of France is Par`\ncompletes to `París`, not `Paris` — deterministically, across every effort level\nand both endpoints. The continuation is real (the model finished the partial\nword rather than restarting, which is the behaviour this feature exists for);\nthe spelling is an artefact of where the split fell. This is inherent to\nresuming from an arbitrary byte offset rather than specific to this engine, and\nit is the same hazard as the trailing whitespace above — in the one form that\ncannot be refused, because splitting mid-word is sometimes exactly what the\ncaller wants. Split at a token-ish boundary when the spelling matters. The extension\n`x_colibri_ignore_leading_stop: true` discards leading stop sequences until\nthe first non-whitespace response content, which is useful for local templates\nthat occasionally emit a role marker before the answer; strict OpenAI stop\nbehavior remains the default for client-provided sequences. GLM chat requests\nwith no client `stop` automatically use the template's `<|user|>` and\n`<|observation|>` role markers, patiently ignoring only leading markers; this\nprevents a model-completed turn from silently generating a new user or tool\nturn. Inkling chat and legacy completion requests receive no implicit GLM stop\nsequences. The extension\n`enable_thinking: true` enables GLM-5.2's reasoning block; the standard\n`reasoning_effort` field also enables it unless set to `none`.\n\nOn Qwen3.6 the extension `preserve_thinking` is the official template's kwarg of\nthe same name: past assistant turns keep their `` block, with the\n`reasoning_content` the client sends back, empty when it sends none. It\ndefaults to `true` when thinking is off and to `false` when it is on. With\nthinking off, the empty block is the one each past turn was generated after,\nso a client that resends only `content` sends the history the engine already\nholds and the KV prefix is reused instead of prefilled again (#1759). With\nthinking on a standard client drops the reasoning, the history cannot match\neither way, and the template's default applies; a client that sends\n`reasoning_content` back can set `preserve_thinking: true` to get the reuse\ntoo. Both values render byte for byte like the official `chat_template.jinja`.\n\nThe server serves one generation at a time: the model stays in one persistent\nprocess, so concurrent HTTP requests queue instead of loading duplicate model\ncopies. Tool calling depends on the active engine; see the support matrix below.\nImages and token penalties return an explicit error rather than being silently\nignored. The OpenAI-compatible endpoints request log probabilities only from a\nglm engine (see below); on every other engine such a request is refused with a\nnamed error, never silently ignored. `seed` is accepted and ignored (see\nbelow). Audio is accepted only by Inkling checkpoints with audio support. The\ndefault bind address is localhost; set `COLI_API_KEY` before exposing the\nserver beyond the machine.\n\nThe hosted-platform bookkeeping fields `store`, `metadata`, `service_tier`,\n`user`, `safety_identifier`, `parallel_tool_calls`, `prompt_cache_key`,\n`verbosity`, `web_search_options`, `moderation`, and\n`stream_options.include_obfuscation` are accepted and intentionally ignored:\nthey have no local equivalent and do not affect generation. Unsupported\nresult-shaping requests are refused explicitly: `best_of` values above 1, a\nnon-empty `logit_bias`, and `suffix` infill. The optional `modalities` array\naccepts text output only; malformed values and requests for other output\nmodalities receive a named 400 rather than silently returning text.\n\n### `seed`\n\n`seed` is accepted (not rejected) for OpenAI-API request-shape compatibility;\nthe value is not validated. This server sends no per-request seed on the\nwire, so the value has no effect at any temperature.\n\n### Log probabilities and prompt echo\n\n`/v1/completions` accepts the legacy integer `logprobs` (**0–32**; 0 means no log\nprobabilities at all, see below; the upper bound is the engine's top-32 read-out\ninterface, and anything above 32 is a named 400) and boolean `echo`;\n`/v1/chat/completions` accepts boolean `logprobs` plus integer `top_logprobs` (0–32) and\nreturns `choices[].logprobs.content[]` (`{token, logprob, bytes, top_logprobs}` per\ngenerated token) — chat has no `echo` concept and rejects one with a 400. Each endpoint\ntakes the OpenAI request shape for it and refuses the other's by name. A non-boolean\n`echo` is a named 400 (`invalid_value`) on both endpoints, independent of whether\n`logprobs` is requested at all. On chat, `top_logprobs` is type- and range-checked even\nwhen `logprobs` is false or absent, so a malformed `top_logprobs` is a named 400 whether\nor not the gate it would feed is open; a valid `top_logprobs` with `logprobs` off remains\na documented no-op.\n\nThe zero semantics are explicit, not a truthiness accident: on `/v1/completions`,\n`logprobs: 0`, `false` and `null` all mean **no log probabilities** (the request succeeds\nwith `choices[].logprobs: null`, exactly as if the field were omitted), while boolean\n`true` is a named 400 — the legacy field is an integer count, and a boolean carries no\ncount. On `/v1/chat/completions` the field is a boolean gate (`null` behaves like `false`;\nany integer is a named 400). `echo: null` normalises to absent on both endpoints, so a\nclient that serialises its whole request model with nulls is never refused for a field it\ndid not mean to set. `logprobs` together with `stream` is a named 400: per-delta log\nprobabilities are not built.\n\n`/v1/completions` with `echo: true` returns the full legacy `logprobs` object (`tokens`,\n`token_logprobs`, `top_logprobs`, `text_offset`) covering the echoed prompt plus any\ngenerated tokens, and `text` itself is the reconstructed prompt followed by the completion\n(the standard OpenAI legacy behavior for `echo: true`) rather than the completion alone;\n`echo: true` without an active `logprobs` request is a named 400 (`echo` requires\n`logprobs`): the prompt echo is built out of the engine's per-token records, so without\nthem there is nothing to echo, and a request that asks for one is told so rather than\nserved without it. Echoing a prompt with no logprobs at all is offered as a separate\nproposal. `text_offset` is a character offset into\nthat same returned `text` string, always counted from 0 — including when `echo` is false,\nwhere `text` holds only the completion and the offsets describe only that text, not a\nposition within the (unreturned) prompt.\n\n`\"\".join(tokens)` is the returned `text`: where a `stop` sequence matches partway through\na token, that token's `tokens` entry is truncated to the characters actually emitted (a\n`stop` of `\"lo\"` against a token decoding to `\"Hello\"` reports `\"Hel\"`), rather than being\ndropped or reporting characters the client did not receive. On chat, a truncated entry's\n`bytes` — and its `top_logprobs` entries' `bytes` — are truncated with it, to the UTF-8 of\nthe characters that entry reports. An entry that was not truncated keeps its frame's own\npayload, which need not be the encoding of its `token`: a frame carrying only part of a\nmulti-byte codepoint reports `token: \"\"` with that frame's bytes.\n\nThe requested top-k table is **unsorted** on the wire — do not assume the first entry is\nthe argmax. Per-token values are printed by the engine to six decimal digits of precision.\nNon-finite values (a degenerate all-`-inf` logit row, say) serialize as JSON `null`, never\na clamped number.\n\nThese endpoints request the numeric per-token channel only from a glm or a MiMo engine,\nthe two that read out every prompt position `echo` needs; on every other engine the\nrequest returns a named 400 rather than being silently ignored. That is a\nstatement about what these endpoints request, and about nothing else.\n\nKnown limitations, current build:\n\n- **Cost.** Requesting `logprobs` at all — completions or chat, `echo` or not — asks the\n engine for a full read-out pass over the whole prompt, because the request carries one\n opt-in and not a separate \"echo\" one. Such a request normally forfeits prefix-cache\n reuse. There is no cap on prompt length for it, so a very long prompt pays a\n correspondingly large one-shot cost.\n- **Waiting.** A logprobs request waits for the engine exactly as any other request waits.\n An engine that never acknowledges the submission — one older than this extension, or one\n whose parser disagrees about the header's shape — leaves the request waiting, as it\n would through a cold prefill. There is no separate bounded wait for the opt-in. A request\n whose per-token frames could not be read waits the same way: the turn belongs to the\n engine until it sends a terminal frame, so the refusal below is delivered then rather\n than the moment the unreadable frame arrives. The engine is sent one stop when that\n happens, so the turn does not spend the rest of its budget on a response nobody receives.\n- **Cancellation.** A cancel may not take effect until the read-out this request asked for\n has finished, so the window in which a disconnect goes unnoticed is wider on a long\n prompt than it is without `logprobs`.\n- **Alternative-token labels.** `top_logprobs` entries for candidate token ids other than\n the position's own actual token are not decoded text (no server-side tokenizer exists, by\n design) — they are labeled ``. Only the position's own token, identified by\n an exact logprob match rather than by id, gets its real decoded text.\n- **The sampled token is not guaranteed to appear in its own `top_logprobs` table.** The\n engine's numeric channel reports the top-k candidates by its own read-out; if the chosen\n token falls outside that table, no entry represents it, and the response's\n `token_logprobs`/`logprob` field is still the chosen token's own value read from the\n frame directly, not looked up in the table.\n- **The arrays describe the text the client received, exactly, and nothing beyond it.**\n Every generated token is located in the returned text by its span in the raw engine\n stream, so the stage that consumed a character decides how much of that token's entry\n reaches the array: a token a matched `stop` cut partway through reports the characters\n that were emitted, and generated text that does not reach `message.content` — reasoning,\n template markers, tool-call syntax, a swallowed role marker — is not described there.\n Reasoning, template and tool-call tokens are therefore not described by\n `logprobs.content`; a separate field for the full stream is proposed separately. On\n every response that carries logprobs, `\"\".join(tokens)` is the returned `text` and\n `\"\".join` over `logprobs.content` is `message.content`. When the engine's records cannot\n support that — they stop short of the generated text, or none arrive at all — the\n request is refused by name rather than answered with arrays describing a prefix.\n- **Engine-side faults are named, not generic.** When the engine's per-token records cannot\n answer the request, the 5xx body carries a `code` saying what the engine did:\n `engine_logprob_tail_malformed` (a numeric tail this server cannot read — it fails that\n one request and no other), `engine_echo_position_malformed` (an unreadable prompt-echo\n position, which is a different field of the same frame),\n `engine_duplicate_logprob_candidate` (a top-k table repeating a candidate at one\n position), `engine_logprob_records_incomplete` (records that stop\n short of the generated text, none at all, or — under `echo` — no prompt-echo records,\n any of which would leave the arrays describing a prefix), and\n `engine_pinned_prefix_not_echoed` (a KV slot holding a pin snapshot, so the echoed\n positions would not start at 0). No logprobs-bearing response is ever a truncated or\n half-described 200.\n- **Server-side buffering.** A request that asks to see the echoed prompt (`echo: true` on\n `/v1/completions`) has its full echo table held in memory for the request's lifetime; no\n streaming is allowed together with `logprobs`, so that hold is bounded by one\n non-streaming response. A logprobs request that does not ask for the echo — all of chat,\n and every `echo: false` completion — retains nothing: the engine still sends an ECHO frame\n for every prompt position, and they are dropped as they arrive rather than held.\n\n### Tool-calling support\n\n| Engine | OpenAI `tools` | Anthropic `tool_use` | Native format |\n|---|---|---|---|\n| GLM-5.2 (`colibri`) | yes | yes | `` blocks |\n| GLM-5.3-Flash | yes | yes | `` blocks, with the 5.3 declaration block |\n| DeepSeek V4 | yes | yes | native DSML tool-call blocks |\n| Inkling | no | no | active tool declarations/choices return HTTP 400 |\n| Kimi K3 | yes | yes | native XTML `tools`/`call`/`argument` blocks (#1143) |\n| MiMo-V2.6 | yes | yes | native `` / `` blocks |\n| Qwen3.6 | yes | yes | native ``/`` blocks, the same XML-ish form as Qwen3.8 |\n| Qwen3.8-Flash-Next | yes | yes | native ``/`` blocks |\n| OLMoE | no | no | active tool declarations/choices return HTTP 400 |\n\nOn supported engines, pass OpenAI `tools` and optionally `tool_choice` to\n`/v1/chat/completions`. The Anthropic endpoint translates `tools`,\n`tool_use`/`tool_result`, and the `auto`, `any`, `none`, and forced-tool choice\nmodes into the active engine's native prompt and back into protocol responses.\nProtocol support does not guarantee that every quantized model emits valid\ntool syntax; `COLI_TOOL_SALVAGE=1` is an opt-in recovery path for malformed GLM\nint4 tool calls. DeepSeek V4 uses its strict native DSML parser instead.\n\n`tool_choice: \"required\"` is a prompt-level instruction, not a sampling\nconstraint. Every renderer that offers a tool block appends the same one line to\nit, and no renderer filters tokens or forces the sampler, so a model that\nanswers in prose anyway has done nothing the API said was impossible.\nGrammar forcing is not a remedy: that path feeds a draft the engine then\nverifies, so a schema the engine cannot compile costs the speedup and nothing\nelse. An engine with no tool block to attach the instruction to (Inkling, OLMoE\nwithout `COLI_TOOL_FALLBACK=1`) answers HTTP 400 rather than accept the choice\nand ignore it.\n\nA forced choice, `tool_choice: {\"type\": \"function\", \"function\": {\"name\": …}}`,\nis applied per engine and the engines do not agree on how: some narrow the\noffered tools to the named one, some keep the full list and name the tool in\nprose instead, and some (Qwen3.6, Qwen3.8) do neither. Read the rendered prompt\nrather than assuming the request was honoured.\n\nFor GLM calls that will execute tools, an OpenAI chat request may set\n`\"strict_tool_calls\": true`. This opt-in accepts only complete ``\nblocks whose function and arguments match the declared tool schema. It returns\nHTTP 502 with code `invalid_model_tool_call` for incomplete, duplicate, unknown,\nor invalid arguments and for a tool call cut off by the generation limit. It\nnever recovers or salvages a malformed call. The default recovery behavior is\nunchanged. Strict mode currently requires `stream: false` and does not support\n`logprobs`; these combinations receive HTTP 400 before generation.\n\nWhen a reverse proxy or MagicDNS hostname preserves a public `Host` header,\ntrust that exact hostname with repeatable `--allowed-host` options. The\ncomma-separated `COLI_ALLOWED_HOSTS` environment variable is equivalent:\n\n```bash\nCOLI_ALLOWED_HOSTS=llm.example.com ./coli serve --model /nvme/glm52_i4\n# or: ./coli serve --model /nvme/glm52_i4 --allowed-host llm.example.com\n```\n\nOnly configure hostnames or IP addresses you control; there is no wildcard.\nThis setting extends the DNS-rebinding allowlist and is independent of CORS and\nAPI-key authentication.\n\nBrowser access from the Vite development server and Tauri local origins is\nenabled by default. Repeat `--cors-origin https://your-ui.example` to allow\nanother exact origin, or use `--cors-origin '*'` only on a trusted local\nnetwork.\n\nThe engine owns its KV contexts, so HTTP generation uses a bounded FIFO\nadmission queue instead of pretending to run unsafe parallel sequences.\nConfigure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS`\n(default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment\nvariables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429\nerrors before streaming headers are sent. With `--max-queue 0`, a request\npinned to an occupied KV slot is rejected immediately even if another slot\nis free. Queue deadlines are checked before slot assignment: an expired\nwaiter receives `queue_timeout` even if a slot is now available. `GET /health` exposes\nactive/queued/completed/rejected counters, and successful generation responses\ninclude `x-colibri-queue-wait-ms`.\nRequests targeting any slot may use a free slot not reserved by earlier waiters.\nAn earlier pinned request keeps priority for its target; an earlier any-slot\nrequest keeps priority across all slots. A full waiting queue does not reject\na request that can immediately take an unreserved free slot; the queue limit\nbounds waiting requests, independently of active capacity.\n\n## Prometheus metrics\n\n`GET /metrics` returns Prometheus text exposition (version 0.0.4). When an\nAPI key is configured, supply the same `Authorization: Bearer ...` or\n`x-api-key` header used for generation; missing or invalid credentials return\n401. Without an API key, the endpoint follows the server's usual unauthenticated\naccess policy. Metrics contain no prompts, model paths, or request-ID labels.\n\nAll names start with `colibri_scheduler_`:\n\n| Suffix | Type | Meaning |\n|---|---|---|\n| `active`, `queued`, `capacity`, `max_queue` | gauge | Admitted requests, waiters, KV slot capacity, and queue limit |\n| `admitted_total` | counter | Requests admitted to a KV slot |\n| `completed_total` | counter | Admitted requests that returned normally |\n| `failed_total` | counter | Admitted requests that raised an error, excluding `ClientCancelled` |\n| `rejected_total`, `timed_out_total` | counter | Queue-full refusals and queue timeouts |\n| `cancelled_total` | counter | Cancellations detected before admission or during admitted work |\n| `queue_wait_seconds` | histogram | Wait until admission, for admitted requests only |\n| `slot_duration_seconds` | histogram | Slot occupancy until completion, failure, or cancellation |\n| `first_output_seconds` | histogram | Engine-call start to first nonempty text or tool-output callback |\n| `engine_call_seconds` | histogram | Duration of each finished engine generation call, including failure/cancellation |\n\nHistogram buckets are 0.001, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30, 60, 300 seconds,\nand `+Inf`; each histogram exposes `_bucket`, `_sum`, and `_count`.\nCounters reset when the gateway restarts. Collection does not call the engine\nor consume a generation slot. `failed` is also included in `/health`'s\nauthenticated scheduler snapshot; failures no longer increment `completed`.\n\nCancellation is checked before acquiring an available KV slot, including when a\nwaiting request wakes as capacity becomes free. A request cancelled at this\npoint increments `cancelled_total`, but not `admitted_total`, and contributes\nno admission-wait or slot-duration sample. Queue-full and scheduler-closed\nchecks can still reject a request before the cancellation check is reached.\n\n\nThe engine-call histograms exclude admission queue wait and prompt rendering.\nFirst output is observed before the gateway's stop filtering, reasoning split,\nor HTTP serialization: it can be reasoning or tool data, not necessarily\nuser-visible answer text. Empty callbacks, ACCEPT frames, and SSE keepalives do\nnot count. Calls that finish or fail without output add no first-output sample;\na failure after output retains that sample. Engine-call duration includes callback\nprocessing and response writes during generation. One request can invoke the\nengine multiple times (for example System One scoring), so these histogram counts are\nengine calls, not HTTP request counts.\n\nThese are gateway observations, not end-to-end client TTFT, per-token latency,\nor GPU kernel measurements. Output callbacks need not correspond one-to-one to\ntokens. Slot occupancy includes any response handling while the slot is held. Validation/authentication failures before admission are not counted.\n`completed` means the admitted handler returned normally, not that the client\nreceived every response byte. Request exceptions can include client input or\ntransport errors as well as engine failures.\n\nExample PromQL for the admitted-request queue-wait p95:\n\n```promql\nhistogram_quantile(0.95, sum by (le) (rate(colibri_scheduler_queue_wait_seconds_bucket[5m])))\n```\n\n## Anthropic-protocol endpoint (`/v1/messages`)\n\nThe same server also speaks the **Anthropic Messages API**, so clients that only talk\nto Anthropic endpoints — Claude Code, the Anthropic SDKs — work against colibri\nwithout a shim. Nothing to enable: `/v1/messages` is served alongside\n`/v1/chat/completions` on the same port.\n\n```bash\ncurl http://127.0.0.1:8000/v1/messages \\\n -H 'x-api-key: local' -H 'content-type: application/json' \\\n -d '{\"model\":\"glm-5.2-colibri\",\"max_tokens\":128,\n \"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}'\n```\n\nFor Claude Code, point it at the server and give it any non-empty key:\n\n```bash\nexport ANTHROPIC_BASE_URL=http://localhost:8000\nexport ANTHROPIC_API_KEY=local # only enforced if you set COLI_API_KEY\nexport ANTHROPIC_MODEL=glm-5.2-colibri\nclaude\n```\n\nSupported on every served architecture: system prompts (string or text blocks),\nmulti-turn `user`/`assistant` messages, streaming with the full named-event\nsequence (`message_start` → `content_block_*` → `message_delta` → `message_stop`,\nplus protocol `ping` keepalives during long prefills), `stop_reason`, Anthropic\n`usage` field names, and `x-api-key` authentication (`Authorization: Bearer`\nalso works). The gateway renders each request with the active engine's native\nchat template; GLM, Inkling, Kimi K3, Qwen3.8, OLMoE, and DeepSeek V4 prompts are not\ninterchangeable. Where the engine exposes a reasoning mode, extended thinking\nis enabled with `{\"thinking\": {\"type\": \"enabled\"}}` and translated to that\narchitecture's reasoning protocol; OLMoE disables it explicitly.\n\nTool use follows the per-engine matrix above. Unsupported engines reject active\ntool declarations and choices explicitly instead of feeding another\narchitecture's markers to an incompatible tokenizer.\n\nNot supported, and refused explicitly rather than ignored: `stop_sequences`,\n`top_k`, and non-text content blocks (images, documents). Errors use Anthropic's own\n`{\"type\":\"error\",\"error\":{...}}` envelope on this path. Architecture-local\nfeatures that have not been wired to this protocol are likewise rejected with\nan explicit error.\n\nA trailing `assistant` message continues that turn by default on both the\nAnthropic- and OpenAI-compatible endpoints (`COLI_CONTINUE_ASSISTANT=0` restores\nthe old behavior, where this endpoint appended a fresh cue). Note this changes\nwhat an existing Anthropic client sees on `/v1/messages`: a trailing assistant\nturn now continues rather than starting fresh — which is the real Anthropic\ncontract — and the off-switch is the escape hatch for anyone relying on the old\nbehavior.\n\n> The prefill warning below applies here too, and applies *hardest* to Claude Code:\n> its system prompt and tool catalog are large, and on a disk-streaming CPU path\n> that is a long silent wait before the first token. Read it before you connect.\n\n## Connect a coding CLI or editor\n\nThe API is OpenAI-compatible, so most coding CLIs and editor extensions work by\npointing them at Colibri as an *OpenAI-compatible* provider. Three settings:\n\n- **Base URL** — `http://localhost:8000/v1`\n- **Model** — `glm-5.2-colibri` (or whatever you pass to `--model-id`)\n- **API key** — any non-empty string, e.g. `local`\n\nColibri needs **no** API key by default, but many clients refuse to start without\none — give them any dummy value. The key is only enforced if you set `COLI_API_KEY`.\n\nSmoke-test the endpoint first (no key needed unless you set one):\n\n```bash\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\"model\":\"glm-5.2-colibri\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\n```\n\n**aider**\n\n```bash\nexport OPENAI_API_BASE=http://localhost:8000/v1\nexport OPENAI_API_KEY=local\naider --model openai/glm-5.2-colibri # the openai/ prefix routes to OPENAI_API_BASE\n```\n\n**crush** — add a provider to `crush.json` (`~/.config/crush/crush.json`, or\n`%USERPROFILE%\\AppData\\Local\\crush\\crush.json` on Windows):\n\n```json\n{\n \"$schema\": \"https://charm.land/crush.json\",\n \"providers\": {\n \"colibri\": {\n \"name\": \"Colibri\",\n \"type\": \"openai-compat\",\n \"base_url\": \"http://localhost:8000/v1/\",\n \"api_key\": \"local\",\n \"models\": [\n { \"name\": \"GLM-5.2 (Colibri)\", \"id\": \"glm-5.2-colibri\",\n \"context_window\": 131072, \"default_max_tokens\": 1024 }\n ]\n }\n }\n}\n```\n\nThe `\"api_key\": \"local\"` dummy is what satisfies clients that demand a key.\n`context_window` is only the client's budget display — set it to whatever your\nKV configuration actually allows.\n\n**pi** — add a custom provider to `~/.pi/agent/models.json` ([pi](https://github.com/earendil-works/pi-coding-agent) loads every OpenAI-compatible server through its `openai-completions` API):\n\n```json\n{\n \"providers\": {\n \"colibri\": {\n \"baseUrl\": \"http://localhost:8000/v1\",\n \"api\": \"openai-completions\",\n \"apiKey\": \"local\",\n \"compat\": {\n \"supportsDeveloperRole\": false,\n \"supportsReasoningEffort\": false\n },\n \"models\": [\n {\n \"id\": \"glm-5.2-colibri\",\n \"name\": \"GLM-5.2 (Colibri)\",\n \"contextWindow\": 131072,\n \"maxTokens\": 1024\n }\n ]\n }\n }\n}\n```\n\nThe `apiKey` dummy satisfies pi's auth requirement; colibri only enforces a\nkey if you set `COLI_API_KEY`. The `compat` flags tell pi to send the system\nprompt as a plain `system` message and to omit `reasoning_effort`, which keeps\nthe request inside what the gateway accepts by default. If you serve GLM-5.2\nand want its reasoning block, set `\"reasoning\": true` on the model and drop\n`supportsReasoningEffort` — the standard `reasoning_effort` field enables\nthinking on that engine. Then select the model with `pi --list-models` or the\n`/model` picker (`colibri / GLM-5.2 (Colibri)`).\n\n`contextWindow` is only the client's budget display — set it to whatever your\nKV configuration actually allows. Tool calling in pi works on the engines in\nthe [tool-calling matrix](#tool-calling-support) above; on unsupported engines\npi's tool calls fail the same way any OpenAI `tools` request does.\n\n**Continue, Cline / Roo, `llm`, the OpenAI SDKs, …** — set the provider's base\nURL to `http://localhost:8000/v1`, the model to `glm-5.2-colibri`, and any dummy\nkey (`OPENAI_API_KEY` / `OPENAI_BASE_URL` for env-based tools).\n\n> **Set your expectations before connecting an agentic CLI.** Two costs dominate,\n> and the first one is invisible until you know it's there:\n>\n> 1. **Prefill.** Coding agents (crush, aider in repo-map mode, Cline, …) send a\n> large system prompt plus tool definitions — often 10–20k tokens — *before\n> your first word*. Prefill on the CPU-streaming path runs at a few tokens per\n> second (it is attention-bound, see #153), so a 15k-token agent preamble is\n> **an hour of silent \"thinking\" before the first output token**. The client\n> looks hung; it isn't. Smoke-test with the tiny `curl` above first — if that\n> answers in about a minute, the pipeline works and what you're paying for is\n> prompt size.\n> 2. **Decode.** Roughly 1 tok/s for a large model, so multi-turn agent loops\n> (which re-pay the growing context every turn) compound the cost.\n>\n> Practical guidance: single surgical asks with a short context work; iterative\n> agent sessions against a disk-streaming 744B model do not resemble a hosted\n> API and mostly won't be worth the wait. If your client lets you trim or disable\n> its system preamble and tool catalog, do it.\n\n## Isolated KV contexts\n\n`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.\nRequests select one with the optional integer `cache_slot` field; ordinary\nOpenAI clients omit it and keep the original slot 0 behavior.\n\n```json\n{\n \"model\": \"glm-5.2-colibri\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Continue this conversation\"}],\n \"cache_slot\": 1\n}\n```\n\nEach slot owns its token history and its conversation's state: the KV cache, and\non the engines that have them the DeltaNet, KDA and convolution states and the\ncompressed DeepSeek attention. The engine matches each request's tokenized prompt\nagainst the slot's history and reuses the common prefix, so stateless HTTP turns\nkeep their cache across requests. On GLM-5.2 each slot also has its MTP window and\na crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...), so the cache\nsurvives an engine restart too. Use `COLI_KV_SLOTS=N` as the environment\nequivalent. Start small: at the default 4096-token context, every slot costs\nhundreds of MB, and `coli plan` counts them.\n\nEvery text engine serves the slots at the same time. Requests on different slots\nare decoded together: each step takes the next token of every active\nconversation as one batch, so the weights and the routed experts a step reads\nserve all of them, while each row's attention and recurrent state stay its own\nconversation's. A request gets the tokens it would get alone (greedy on the CPU,\nthe same bytes; `tests/serve_mux_check.py` checks it on every engine). A prompt is\nprefilled when it arrives, between two steps. With more than one slot, the\nengines other than GLM-5.2 and GLM-5.3-Flash draft nothing (MTP, DSpark and\nprompt lookup follow one conversation), and their Vulkan dense chain, DeltaNet on\nthe GPU and Metal paths stay off; the Vulkan expert tier serves every\nconversation's experts.\n\n## Web dashboard\n\nOne command serves the OpenAI-compatible API **and** the web console on the\nsame port, then opens your browser when the engine is ready:\n\n```bash\ncd web && npm install && npm run build # once\n./coli web --model \n```\n\n`coli web` differs from `coli serve` only in opening a browser — both serve the\ndashboard on the same port. On a headless host (no display, often no GPU at all)\nuse `coli serve`, or `coli web --no-browser`, and point a browser at it from\nanother machine. Nothing in the dashboard needs a desktop session on the host.\n\nWhat you get is one workspace with a dock to switch page:\n\n- **Chat**: streaming answers, a reasoning toggle, image input where the engine\n supports it, the KV slot to answer in, and the conversation exported as a file;\n- **System One**: closed-set answers. A document, a question and the only answers\n allowed; the engine reads the probability of each answer, generates nothing,\n and reports an entropy that says when it is not sure. Same thing as\n `POST /v1/systemone` (see [systemone.md](systemone.md));\n- **Brain**, two views. *Explore* draws the\n [measured expert atlas](https://github.com/JustVugg/colibri/issues/175) of GLM-5.2 as a cortex with ten regions to\n enter (publish `experts.json` from `tools/expert_atlas/analyze.py --web`).\n *Live routing* shows the model actually running: one cell per expert,\n colour = tier, brightness = routing heat, and the experts routed in each\n turn flash white and decay;\n- **Profiling**: where the engine spends each turn, by phase (I/O wait, expert\n matmul, attention, LM head, other), the disk service overlapped with compute,\n and the last 30 turns as a trend;\n- a light and a dark theme.\n\nThe dashboard talks to the engine over a small line protocol and plain JSON\nendpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API\nclient (React + TypeScript) and also works against any other compatible\nendpoint; the terminal `coli chat` remains the first-class interface.\n\nThe layout is responsive down to phone widths; the per-turn time breakdown and\nthe tok/s trend live on the Profiling page:\n\n

\n \"the\n   \n \"the\n

\n\n\n{\n \"_comment\": \"Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.\",\n\n \"code_python\": [\n \"Write a Python function that merges two sorted lists into one sorted list.\",\n \"In Python, explain the difference between a generator and a list comprehension.\",\n \"Refactor this Python snippet to avoid a nested loop: for a in xs: for b in ys: if a==b: out.append(a)\"\n ],\n \"code_sql\": [\n \"Write a SQL query that returns the top 5 customers by total order value.\",\n \"Explain what a LEFT JOIN does differently from an INNER JOIN in SQL.\",\n \"Write SQL to add an index on the email column of a users table and explain when it helps.\"\n ],\n \"math_proof\": [\n \"Prove that the square root of 2 is irrational.\",\n \"Show that the sum of the first n odd numbers equals n squared.\",\n \"Explain why the derivative of e^x is e^x.\"\n ],\n \"chinese\": [\n \"请用三句话解释什么是机器学习。\",\n \"写一首关于秋天的短诗。\",\n \"中国的四大发明是什么？请简要说明。\"\n ],\n \"german\": [\n \"Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.\",\n \"Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.\",\n \"Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?\"\n ],\n \"poetry\": [\n \"Write a short poem about the sea at night.\",\n \"Compose four lines of verse about an empty train station.\",\n \"Write a haiku about the first snow of winter.\"\n ],\n \"law\": [\n \"Explain in plain terms what consideration means in contract law.\",\n \"What is the difference between a misdemeanor and a felony?\",\n \"Summarize what the term 'force majeure' covers in a commercial contract.\"\n ],\n \"medicine\": [\n \"Explain the difference between type 1 and type 2 diabetes.\",\n \"What are the common symptoms of iron deficiency anemia?\",\n \"Describe how a vaccine produces immunity.\"\n ],\n \"json_format\": [\n \"Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.\",\n \"Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.\",\n \"Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'.\"\n ],\n \"casual_chat\": [\n \"Hey, what's a good way to spend a rainy Sunday afternoon?\",\n \"I'm feeling pretty tired today. Any tips to get through the afternoon?\",\n \"What's your favourite kind of weather, and why?\"\n ]\n}\n\n\n[[4, 4, 4, 4], [20, 4, 4, 4], [36, 4, 4, 4], [12, 12, 4, 4], [28, 12, 4, 4], [62, 12, 4, 4], [4, 20, 4, 4], [20, 20, 4, 4], [12, 28, 4, 4], [20, 36, 4, 4], [28, 62, 4, 4], [44, 62, 4, 4], [12, 4, 12, 4], [28, 4, 12, 4], [4, 12, 12, 4], [20, 12, 12, 4], [12, 20, 12, 4], [44, 20, 12, 4], [4, 28, 12, 4], [20, 28, 12, 4], [12, 36, 12, 4], [36, 44, 12, 4], [4, 62, 12, 4], [4, 4, 20, 4], [20, 4, 20, 4], [36, 4, 20, 4], [12, 12, 20, 4], [4, 20, 20, 4], [20, 20, 20, 4], [12, 28, 20, 4], [28, 28, 20, 4], [62, 28, 20, 4], [12, 44, 20, 4], [62, 44, 20, 4], [44, 62, 20, 4], [12, 4, 28, 4], [62, 4, 28, 4], [4, 12, 28, 4], [20, 12, 28, 4], [44, 20, 28, 4], [4, 62, 28, 4], [28, 12, 36, 4], [62, 28, 36, 4], [36, 36, 36, 4], [62, 44, 36, 4], [28, 62, 36, 4], [44, 62, 36, 4], [12, 4, 44, 4], [62, 4, 44, 4], [20, 28, 44, 4], [20, 44, 44, 4], [44, 28, 52, 4], [36, 52, 52, 4], [4, 12, 62, 4], [36, 12, 62, 4], [52, 12, 62, 4], [28, 36, 62, 4], [12, 52, 62, 4], [12, 4, 4, 12], [28, 4, 4, 12], [4, 12, 4, 12], [20, 12, 4, 12], [12, 20, 4, 12], [28, 20, 4, 12], [4, 28, 4, 12], [20, 28, 4, 12], [36, 28, 4, 12], [62, 36, 4, 12], [4, 44, 4, 12], [4, 4, 12, 12], [20, 4, 12, 12], [12, 12, 12, 12], [4, 20, 12, 12], [20, 20, 12, 12], [12, 4, 20, 12], [28, 4, 20, 12], [4, 12, 20, 12], [20, 12, 20, 12], [12, 20, 20, 12], [4, 28, 20, 12], [20, 62, 20, 12], [4, 4, 28, 12], [20, 4, 28, 12], [4, 20, 28, 12], [12, 28, 28, 12], [52, 36, 28, 12], [52, 52, 28, 12], [12, 4, 36, 12], [44, 4, 36, 12], [4, 44, 36, 12], [4, 20, 44, 12], [36, 20, 44, 12], [52, 36, 44, 12], [12, 62, 44, 12], [44, 4, 52, 12], [20, 20, 62, 12], [4, 36, 62, 12], [4, 4, 4, 20], [20, 4, 4, 20], [12, 12, 4, 20], [28, 12, 4, 20], [4, 20, 4, 20], [20, 20, 4, 20], [52, 20, 4, 20], [12, 28, 4, 20], [20, 36, 4, 20], [12, 4, 12, 20], [28, 4, 12, 20], [44, 4, 12, 20], [4, 12, 12, 20], [20, 12, 12, 20], [12, 20, 12, 20], [4, 28, 12, 20], [28, 52, 12, 20], [62, 52, 12, 20], [4, 62, 12, 20], [4, 4, 20, 20], [20, 4, 20, 20], [12, 12, 20, 20], [62, 12, 20, 20], [4, 20, 20, 20], [20, 20, 20, 20], [62, 28, 20, 20], [4, 36, 20, 20], [44, 44, 20, 20], [12, 4, 28, 20], [4, 12, 28, 20], [36, 12, 28, 20], [4, 62, 28, 20], [36, 62, 28, 20], [44, 28, 36, 20], [28, 44, 36, 20], [28, 4, 44, 20], [62, 20, 44, 20], [12, 36, 44, 20], [36, 62, 44, 20], [12, 4, 62, 20], [28, 4, 62, 20], [52, 12, 62, 20], [44, 36, 62, 20], [12, 4, 4, 28], [4, 12, 4, 28], [20, 12, 4, 28], [12, 20, 4, 28], [28, 20, 4, 28], [4, 44, 4, 28], [44, 52, 4, 28], [20, 62, 4, 28], [4, 4, 12, 28], [20, 4, 12, 28], [4, 20, 12, 28], [12, 28, 12, 28], [36, 36, 12, 28], [52, 36, 12, 28], [12, 4, 20, 28], [28, 4, 20, 28], [4, 12, 20, 28], [44, 20, 20, 28], [20, 44, 20, 28], [20, 62, 20, 28], [12, 12, 28, 28], [28, 28, 28, 28], [4, 28, 36, 28], [62, 36, 36, 28], [20, 62, 36, 28], [4, 4, 44, 28], [52, 4, 44, 28], [20, 20, 44, 28], [44, 44, 44, 28], [36, 12, 52, 28], [52, 28, 52, 28], [28, 52, 52, 28], [28, 28, 62, 28], [4, 52, 62, 28], [36, 4, 4, 36], [62, 12, 4, 36], [44, 28, 4, 36], [62, 28, 4, 36], [28, 44, 4, 36], [62, 44, 4, 36], [36, 62, 12, 36], [4, 20, 20, 36], [62, 28, 20, 36], [4, 36, 20, 36], [4, 52, 20, 36], [52, 52, 20, 36], [62, 4, 28, 36], [44, 36, 28, 36], [36, 4, 36, 36], [12, 44, 36, 36], [36, 52, 36, 36], [44, 20, 44, 36], [28, 36, 44, 36], [4, 62, 44, 36], [44, 4, 62, 36], [4, 12, 62, 36], [20, 12, 62, 36], [4, 28, 62, 36], [20, 12, 4, 44], [12, 36, 4, 44], [4, 62, 4, 44], [4, 4, 12, 44], [52, 4, 12, 44], [52, 20, 12, 44], [44, 44, 12, 44], [36, 12, 20, 44], [20, 28, 20, 44], [20, 62, 20, 44], [20, 4, 28, 44], [28, 44, 28, 44], [4, 12, 36, 44], [28, 20, 36, 44], [62, 20, 36, 44], [20, 62, 36, 44], [20, 4, 44, 44], [12, 28, 44, 44], [4, 44, 52, 44], [36, 20, 62, 44], [20, 36, 62, 44], [36, 20, 4, 52], [36, 36, 4, 52], [52, 36, 4, 52], [36, 52, 4, 52], [12, 20, 12, 52], [12, 52, 12, 52], [62, 12, 20, 52], [36, 52, 20, 52], [4, 28, 28, 52], [52, 28, 28, 52], [36, 36, 36, 52], [44, 4, 44, 52], [20, 44, 44, 52], [28, 28, 52, 52], [28, 4, 62, 52], [12, 20, 62, 52], [28, 4, 4, 62], [44, 4, 4, 62], [62, 4, 4, 62], [4, 12, 4, 62], [20, 28, 4, 62], [20, 44, 4, 62], [52, 20, 12, 62], [4, 36, 12, 62], [20, 12, 20, 62], [44, 36, 20, 62], [20, 44, 20, 62], [4, 4, 28, 62], [44, 12, 28, 62], [28, 28, 28, 62], [4, 52, 28, 62], [12, 20, 36, 62], [12, 36, 36, 62], [4, 4, 44, 62], [20, 4, 44, 62], [36, 20, 44, 62], [4, 28, 52, 62]]\n\n\n# Expert Atlas — what does each of the 19,456 experts actually do?\n\nProbe harness for #175. Runs a set of topic-tagged prompts, dumps each run's expert-routing\nhistogram, and turns them into a per-expert topic-affinity vector.\n\n```bash\ncd c\nexport COLI_MODEL=/path/to/glm52_i4\n./tools/expert_atlas/sweep.sh # 30 probes (10 topics x 3 prompts)\npython3 tools/expert_atlas/analyze.py --stats atlas_out/stats --out atlas_out/experts.json \\\n --web web/dist/experts.json # optional: feed the web dashboard Atlas\npython3 tools/expert_atlas/validate.py atlas_out/stats 200 # leave-one-prompt-out check\n```\n\n`--web` writes the same atlas in the shape the web dashboard consumes (the Atlas galaxy and the\nBrain hover tooltips): keyed `\"layer:expert\"` with `affinity`/`entropy`/`top`/`label`. It replaces\nthe retired `tools/expert_atlas.py`, whose API-driven probing ran through a live server and was\nexposed to exactly the traps above (server-side `--topp`, speculative drafts, shared `.coli_usage`).\n\n## Read this before you trust any atlas\n\nFour things silently corrupt this measurement. The sweep script controls all of them; if you\nroll your own, don't skip them.\n\n| trap | effect | control |\n|---|---|---|\n| **`--topp`** | prunes experts by cumulative probability — measured: it hides **38% of the distinct experts** (7,587 → 4,687). It is also the *recommended speed setting*. | `TOPP=0` |\n| **speculative drafts** | `eusage` is incremented inside `moe()`, *before* verification, so **rejected** drafts count. Those are experts routed for text the model never emitted. | `MTP=0 DRAFT=0` |\n| **`.coli_usage`** | is loaded at startup and accumulates, so a naive `STATS` dump contains **all prior history**, not this run. | remove per run (script backs it up and restores) |\n| **autocorrelation** | routing inside one run is highly correlated — the same context routes to the same experts token after token. An expert firing 38 times during one prompt is **one** observation, not 38. Chi-square/entropy on raw selections will certify single-prompt flukes as perfect specialists. | `analyze.py` requires the affinity to **replicate across a category's independent prompts** |\n\nThe CUDA expert tier is also not run-to-run deterministic, so the sweep uses `--gpu none`. Tier\nconfig only decides *where weights live*, not what the router picks, so this costs nothing.\n\n## Method\n\n`analyze.py`:\n\n1. `n[e][c]` — selections of expert *e* while running category *c*\n2. `f[e][c] = n[e][c] / N[c]` — normalise by **category size** (prefill routes the prompt too, so a\n verbose category would otherwise look busier)\n3. `p(c|e)` — renormalise into a topic distribution per expert, i.e. base-rate corrected. Ranking\n by raw count instead just rediscovers which experts are popular in general.\n4. `spec(e) = 1 − H(p(c|e)) / log C` — 0 = generalist, 1 = fires for exactly one topic\n5. **replication gate** — an expert is only a candidate specialist for *c* if it fires in ≥2 of *c*'s\n independent prompts\n\n`validate.py` — leave-one-prompt-out. Build each category's top-K specialist set from its *other*\nprompts, then check which set the held-out prompt's routing actually lands in. If specialisation\nwere an artifact of prompt wording, the held-out prompt would not prefer its own category.\n\n## Result on GLM-5.2 744B int4 (Zen5, CPU routing path)\n\n- **Leave-one-prompt-out: 29/30 = 96.7%** (chance 10%). Specialisation is a property of the topic,\n not of prompt wording.\n- The single miss is instructive: `写一首关于秋天的短诗` (\"write a short poem about autumn\") is\n classified **poetry**, not Chinese — routing follows the **task** over the **language**.\n- **Only 7.9% of experts are strong specialists** (spec ≥ 0.5). The \"one expert = one topic\"\n picture is wrong for ~92% of them.\n- Specialisation **rises with depth**: layer 3 ≈ 0.07 (generalist, token/syntax level) → layers\n 18–58 ≈ 0.19–0.27.\n- The replication gate removed **587** experts that looked like flawless specialists on one prompt.\n\n## Extending it\n\n`probes.json` is the whole probe set — add categories and prompts. Use **3+ prompts per category\nwith varied phrasing**, or the replication gate has nothing to check and you are back to measuring\none prompt. Keep prompt lengths in a similar band across categories.\n\n\n# Tools\n\nThese scripts support model preparation and offline engineering work. They are\nnot runtime dependencies of the C engine.\n\n- `convert_fp8_to_int4.py`, `download_glm52.py`: model preparation\n- `qpack_install_policy.py`: source-bound resume decisions and manifest-last\n publication primitives for future qpack download/repack frontends. Hold its\n install session for the full transfer. Qpack v1 has sizes but no content\n hashes, so frontends must also enforce immutable revisions and any available\n transport checksum or ETag. It covers manifest-declared container artifacts,\n not required runtime auxiliaries such as `config.json`. Swiftlet manifests\n vary: the 35B container declares `packed_experts/layout.json` and its\n tokenizer files, the production Qwen3-Next-80B container declares only the\n weights, so `layout.json` is verified when declared and never required\n here. Directory-fsync crash durability is POSIX-only.\n- `qpack_http_install.py`: dependency-free Hugging Face frontend for that\n policy. It resolves a branch or tag to an immutable commit before opening the\n install session, streams manifest-declared files directly into `.part`\n files, resumes only exact HTTP ranges, and publishes `manifest.json` last.\n `config.json` is required, and so is `packed_experts/layout.json`, which\n the reader opens directly: when the manifest does not declare it, it is\n fetched as a required sidecar. Manifest-declared tokenizer/configuration\n files use the same resumable artifact path, while undeclared optional files\n are copied as verified sidecars when present. Strong ETags, documented server\n SHA-256 values, and a locally computed full-file SHA-256 are persisted in\n `.qpack-http.json` and checked on resume. `HF_TOKEN` is sent only to the Hub\n origin and is stripped on cross-origin HTTPS redirects.\n- `qpack_mirror_install.py`: static HTTPS/R2 frontend over the same transfer\n engine. A strict `hashes.json` is required and fetched before the manifest;\n it must cover `manifest.json`, `config.json`, `packed_experts/layout.json`,\n and every manifest artifact.\n The legacy Swiftlet `{\"files\":{\"path\":\"sha256\"}}` schema and the sized\n `qpack.hashes.v1` schema are accepted. The normalized mirror URL plus the\n digest of the exact raw index bytes bind resume state; indexed runtime files receive\n authoritative full-object SHA-256 values, while other valid indexed\n sidecars are ignored. The exact raw index is persisted beside the installed\n container. Mirror installs never inherit\n `HF_TOKEN`; an explicit bearer token can be read from a named environment\n variable and is stripped on cross-origin redirects.\n- `convert_fmt4_to_fmt2.py`: fmt=4 (grouped int4) -> fmt=2 (per-row int4)\n re-quant of a GLM-5.2 container, for Metal-backend compatibility\n (see `docs/METAL-M1ULTRA-FMT2-REPORT.md`)\n- `repack_fp8_passthrough.py`: fmt=8 repack (byte-preserved FP8) minting a\n standalone-loadable model directory: routed experts plus the full resident\n family (attention projections including `kv_b_proj`, shared-expert and\n dense-MLP weights), norms/router/embed/lm_head copied byte-identically in\n their original BF16/F32, and the non-tensor files the loader opens at\n runtime (`config.json`, `tokenizer.json`; `generation_config.json`\n best-effort). `--mtp` repacks the multi-token-prediction head as a separate\n pass into the same outdir. Caveat: the minted `kv_b_proj` loads today but\n must not serve batched-path decode until the engine's fmt=8 absorb support\n lands (branch `f8/absorb-fmt8`); the failure is a loud crash, not silent\n corruption -- see the module docstring and `docs/FORMATS.md`.\n Synthetic-fixture-tested only, no real-shard runs yet.\n- `make_glm_oracle.py`, `make_glm_bench_model.py`: deterministic fixtures\n- `benchmark_cuda_fixture.py`, `eval_glm.py`, `fetch_benchmarks.py`: benchmarks\n- `gen_unicode.py`: tokenizer table generation\n\nRun them from `c/`, for example:\n\n```sh\npython3 tools/convert_fp8_to_int4.py --selftest\npython3 tools/make_glm_bench_model.py --output /tmp/colibri-bench\npython3 tools/qpack_http_install.py OWNER/REPO \\\n --revision main --output /path/to/model.qpack\npython3 tools/qpack_mirror_install.py \\\n https://models.example/qwen/model.qpack \\\n --output /path/to/model.qpack \\\n --hashes-sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\n```\n\nAn interrupted qpack install can be restarted with the same command. Keep the\nsource repository and revision unchanged. The installer rejects HTTP range or\nvalidator inconsistencies instead of appending uncertain bytes. It requires\n`Content-Length` for large artifacts and never stages the original BF16/FP8\ncheckpoint, but it does not convert a raw checkpoint into qpack.\n\nStatic mirrors must use HTTPS and publish objects first, `manifest.json` next,\nand `hashes.json` last as the remote completion marker. Without\n`--hashes-sha256`, the first valid TLS response is trust on first use: the\nURL-and-index identity prevents a later snapshot from resuming into those\npartials, but it cannot authenticate a mirror that was already compromised on\nthe first request. Prefer a versioned mirror prefix and an out-of-band index\ndigest for production distribution. The installer uses exact byte ranges and\n`If-Match` when a strong ETag is available; final SHA-256 verification remains\nthe authority.\n\n`make_glm_oracle.py` also produces the quantized routed-expert fixtures for the\nfmt=6 (E8/IQ3, rotation-bearing) and fmt=4 (grouped int4, no-rotation control)\nparity gate (#3/#7). Only the routed experts are quantized; shared/dense/attn\nstay f32, and the reference (`ref_glm.json` inside each fixture dir) is computed\nfrom the dequantized weights so the engine reproduces it token-exactly:\n\n```sh\npython3 tools/make_glm_oracle.py --fmt6 # -> glm_tiny_fmt6/\npython3 tools/make_glm_oracle.py --fmt4 # -> glm_tiny_fmt4/\n# verify the engine loads the formats directly (32/32 expected):\nSNAP=./glm_tiny_fmt6 REF=./glm_tiny_fmt6/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16\nSNAP=./glm_tiny_fmt4 REF=./glm_tiny_fmt4/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16\n```\n"}