{"owner":"Kiln-AI","repo":"Kiln","hasSkills":true,"totalSkillsCount":7,"totalTokensCount":30234,"categories":["anthropic-skill","root-instruction","mcp-config"],"hasMcp":true,"mcpConfig":{"mcpServers":{"HooksMCP":{"command":"uvx","args":["--from","hooks-mcp>=0.2.5","hooks-mcp",".config/hooks_mcp.yaml","--working-directory","."]}}},"found":[".agents/skills/kiln-check-finetune-deprecation/SKILL.md",".agents/skills/kiln-prerelease-check/SKILL.md",".agents/skills/claude-maintain-models/SKILL.md",".agents/skills/release-digest/SKILL.md",".agents/skills/kiln-check-deprecation/SKILL.md","AGENTS.md",".agents/mcp.json"],"skills":{".agents/skills/kiln-check-finetune-deprecation/SKILL.md":"---\nname: kiln-check-finetune-deprecation\ndescription: Check Kiln's fine-tunable model list for deprecated or unsupported base models. Use when the user wants to audit fine-tuning support, check if fine-tune base models are still valid, or mentions fine-tune model deprecation.\n---\n\n# Check Fine-Tune Model Deprecation in Kiln\n\nAudit the fine-tunable models listed in Kiln to find base models that are no longer supported for fine-tuning by their providers. This is separate from general model deprecation — a model can still be available for inference but no longer supported as a fine-tuning base.\n\n---\n\n## Global Rules\n\n- **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: [\"all\"]`. The sandbox blocks network access.\n- **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)`\n- **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate.\n- **Non-destructive:** This skill only reports findings. It does not automatically remove or deprecate models.\n\n---\n\n## Supported Providers\n\n| Provider | Env Var | Auth |\n|----------|---------|------|\n| OpenAI | `OPENAI_API_KEY` | Bearer |\n| Together AI | `TOGETHER_API_KEY` | Bearer |\n| Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) |\n| Fireworks AI | `FIREWORKS_API_KEY` | Bearer |\n\n---\n\n## Background\n\nKiln has two types of fine-tunable model entries:\n\n1. **Static entries** — Models in `libs/core/kiln_ai/adapters/ml_model_list.py` with `provider_finetune_id` set. Currently used by Together AI and Vertex AI.\n2. **Dynamic entries (Fireworks)** — Fetched at runtime from `api.fireworks.ai/v1/accounts/fireworks/models` filtering by `tunable=True`. This list is managed by Fireworks and may include stale/unsupported models.\n\nThe API endpoint that serves the fine-tune dropdown is `GET /api/finetune_providers` in `app/desktop/studio_server/finetune_api.py`.\n\n---\n\n## Phase 1 – Check Static Fine-Tune IDs\n\nRun the check script from the repo root:\n\n```bash\nuv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py static > /tmp/kiln_finetune_static.json\n```\n\nThis extracts all `provider_finetune_id` entries from `ml_model_list.py` and checks each provider's docs/API to see if the model is still available for fine-tuning:\n\n- **Together AI:** Scrapes the Together AI docs page for the list of supported fine-tune models. Together uses full HuggingFace-style IDs (e.g. `meta-llama/Meta-Llama-3.1-8B-Instruct-Reference`).\n- **Vertex AI:** Checks the Vertex AI publisher models API. Vertex fine-tuning uses specific versioned model IDs (e.g. `gemini-2.0-flash-001`).\n\n**Output:** JSON to stdout with per-provider results, human summary to stderr.\n\n---\n\n## Phase 2 – Check Fireworks Dynamic Models\n\nFireworks fine-tunable models are fetched dynamically from their API, not stored in `ml_model_list.py`. The script checks the API's `supervisedLoraTunable` and `supervisedFullParameterTunable` fields (not the old `tunable` field, which is stale) and cross-references against our allowlist.\n\n```bash\nuv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py fireworks > /tmp/kiln_finetune_fireworks.json\n```\n\nThis script:\n1. Fetches all models with `supervisedLoraTunable=True` or `supervisedFullParameterTunable=True` from `api.fireworks.ai/v1/accounts/fireworks/models`\n2. Cross-references against `FIREWORKS_SUPPORTED_FINETUNE_MODELS` from `libs/core/kiln_ai/adapters/fine_tune/fireworks_finetune.py` — the same allowlist that filters the runtime fine-tune dropdown\n3. Checks both directions:\n - **Stale allowlist entries**: models in our allowlist that the API no longer marks as supervised-tunable — these may need to be removed\n - **Missing from allowlist**: models the API marks as supervised-tunable but aren't in our allowlist — these are candidates to add\n\nThe canonical allowlist lives in `fireworks_finetune.py` and is shared between the runtime dropdown filter and this audit skill. There is a single place to update when Fireworks changes their supported models.\n\n**Interpreting results:** The supervised API fields return ~43 models, but our allowlist intentionally filters to ~20 that match Fireworks' documented supported models. Seeing ~23 \"models in API but NOT in allowlist\" is normal — these are models Fireworks marks as technically tunable but doesn't officially document or support. Only flag these if a model appears in the Fireworks docs but is missing from our allowlist. The important direction is **stale allowlist entries** — models we ship that the API no longer marks as supervised-tunable.\n\n**Output:** JSON to stdout, human summary to stderr.\n\n---\n\n## Phase 3 – Report Findings\n\nRead the JSON outputs and present findings in a clear table:\n\n```text\nFine-Tune Deprecation Audit Results\n====================================\n\nSTATIC PROVIDERS (provider_finetune_id in ml_model_list.py):\n\n ✅ OpenAI: 5/5 found\n gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14,\n gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18\n\n ✅ Vertex AI: 2/2 found\n gemini-2.0-flash-001, gemini-2.0-flash-lite-001\n\n ❌ Together AI: 2/4 missing\n ✅ Qwen/Qwen2.5-72B-Instruct — found\n ✅ Qwen/Qwen2.5-14B-Instruct — found\n ❌ meta-llama/Meta-Llama-3.1-8B-Instruct-Reference — NOT FOUND\n ❌ meta-llama/Meta-Llama-3.1-70B-Instruct-Reference — NOT FOUND\n\n ⏭️ SKIPPED (no credentials):\n - vertex (VERTEX_PROJECT_ID not set)\n\nDYNAMIC PROVIDER (Fireworks):\n\n ⚠️ 167 models marked tunable in API\n ⚠️ 15 models in known-good docs list\n ✅ 11 models in both API and docs\n ❌ 156 models in API but NOT in docs (likely stale)\n - accounts/fireworks/models/qwen2-72b-instruct\n - accounts/fireworks/models/code-llama-7b\n - ...\n ⚠️ 4 models in docs but NOT in API\n - accounts/fireworks/models/gemma-2-9b-it\n - ...\n```\n\nNote: The example above is illustrative. Actual model counts and entries will vary.\n\n---\n\n## Phase 4 – Remediation\n\nBased on findings, recommend specific actions:\n\n### For static entries (`provider_finetune_id`):\n- If a model's fine-tune ID is no longer valid, either:\n - **Update** the `provider_finetune_id` to a newer version if one exists\n - **Remove** the `provider_finetune_id` field to stop offering fine-tuning for that model on that provider\n - **Set `deprecated=True`** on the provider entry if the model is fully gone\n\n### For Fireworks dynamic entries:\n- The stale models come from Fireworks' API, not our code. Options:\n - **Filter in our code** — update `fetch_fireworks_finetune_models()` in `finetune_api.py` to filter against a known-good list\n - **Report to Fireworks** — flag the stale models to Fireworks support\n - **Both** — filter now, report later\n\n**Always ask the user to confirm** before making any code changes.\n\n---\n\n## Phase 5 – Verify\n\nAfter any changes, run the relevant tests:\n\n```bash\nuv run python3 -m pytest app/desktop/studio_server/test_finetune_api.py -q\n```\n\n---\n\n## Checklist\n\n- [ ] Env vars sourced from `.env`\n- [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex)\n- [ ] Static fine-tune IDs checked against provider APIs (Together, Vertex)\n- [ ] Fireworks dynamic models cross-referenced against known-good list\n- [ ] Findings reported to user with clear table\n- [ ] User confirmed remediation approach\n- [ ] Code changes made (if any)\n- [ ] Tests pass after changes\n- [ ] FIREWORKS_SUPPORTED_FINETUNE_MODELS in fireworks_finetune.py updated if docs have changed\n",".agents/skills/kiln-prerelease-check/SKILL.md":"---\nname: kiln-prerelease-check\ndescription: Run the Kiln pre-release smoke test suite plus the standard CI checks (checks.sh), diagnose every prerelease test that broke (and why), and write a clean readable report with recommended actions. Read-only — it never edits code. Use when the user wants to validate a release candidate, run prerelease tests, or asks for a \"prerelease check / smoke / audit\".\n---\n\n# Kiln Pre-release Check\n\nEnd-to-end pre-release verification and **diagnosis**. This skill:\n\n1. Runs the standard CI checks (`checks.sh` — lint / format / typecheck / unit tests).\n2. Runs the curated `@pytest.mark.prerelease` smoke set (real, paid API calls).\n3. For every test that broke, works out **what** broke and **why** — including whether the cause is a model that's gone deprecated (per `ml_model_list.py`) or that the provider no longer recognizes the slug.\n4. Sweeps the prerelease model pins for staleness and newer model versions.\n5. Writes a clean, scan-readable report with a clear **recommended action** for each finding.\n\n**This skill is read-only. It does not change any code, tests, whitelists, or prod source — ever.** Its entire output is a report at `.prerelease//REPORT.md` (this directory is gitignored). When the right action is to bump a model pin or update the whitelist, the report *recommends* it; the user makes the change.\n\n---\n\n## What \"prerelease\" means here\n\nA prerelease test is a paid test (it makes real API calls) that we **cannot** meaningfully replace with a mock — its whole point is to verify a live third-party API still behaves how we depend on it. Examples: `test_connect_vertex_live`, basic OpenAI / Claude / Gemini calls, embedding API across providers, reranker integration, Fireworks fine-tune deployment listing, etc.\n\nThe set is a **curated subset of `@pytest.mark.paid`**. Tests carry both markers — `@pytest.mark.paid` for the cost gate, `@pytest.mark.prerelease` for the curated selection.\n\nConcretely:\n\n- `--runpaid` runs the full paid suite (slow, expensive, intended for one-off use).\n- `--runprerelease` runs **only** the prerelease subset (still paid, but a small, focused, releaseable list). It implies `--runpaid` for those tests.\n\nMarker registration lives in `pyproject.toml` (`[tool.pytest.ini_options].markers`); the flag is wired up in the top-level `conftest.py` via `pytest_addoption` + `pytest_collection_modifyitems`.\n\n### Whitelist for litellm-adapter tests\n\nSeveral paid tests (embeddings, document extraction, thinking-level reasoning) fan out across **every** `(model, provider)` pair in `ml_model_list.py` / `ml_embedding_model_list.py`. Running those under `--runprerelease` would be slow and expensive. Instead, those tests have dedicated `*_prerelease_smoke` sibling functions that iterate only over a small curated whitelist living in:\n\n```\nlibs/core/kiln_ai/adapters/pytest_prerelease_whitelist.py\n```\n\nWhitelisted lists:\n\n- `PRERELEASE_CHAT_MODELS` — handful of chat models, one per major provider (OpenAI + GPT-5, Anthropic + Claude Sonnet/Haiku, Gemini, OpenRouter, Groq, Fireworks, Together, Cloudflare).\n- `PRERELEASE_EMBEDDING_MODELS` — one embedding model per embedding-supporting provider.\n- `PRERELEASE_EXTRACTION_MODELS` — one multimodal model per major vendor (OpenAI, Anthropic, Gemini).\n- `PRERELEASE_EXTRACTION_MIME_PROBES` — three mime probes (PDF, PNG, MP3) for the extraction smoke; the full paid test sweeps all 13 mime types per model.\n- `PRERELEASE_THINKING_MODELS` — five (provider, model, thinking_level) triples covering reasoning content + a \"none\" negation case.\n- `PRERELEASE_JEV_MODELS` — TypeSafe AI's Jev models, used by the Jev paid smoke tests. Separate from `PRERELEASE_CHAT_MODELS` because Jev does not go through LiteLLM.\n\nThis whitelist is the thing most likely to go stale, so it's the main target of the pin sweep in Phase 4. The fan-out tests (e.g. `test_extract_document_success` over every model × mime type, `test_paid_generate_embeddings_basic` over every embedding) are **only `@pytest.mark.paid`** — not `@pytest.mark.prerelease` — and are out of scope for this skill.\n\n---\n\n## Global rules\n\n- **Read-only. Make no edits.** Do not modify test files, the whitelist, prod source, config, or anything else. Every actionable finding is a *recommendation* in the report. If you catch yourself about to call Edit/Write on anything other than files under `.prerelease//`, stop.\n- **Network access is required.** Every test phase makes real API calls or hits remote model-list endpoints. If you are running this skill inside a sandboxed Bash session, request `required_permissions: [\"all\"]` for the test commands.\n- **Env vars:** Source `.env` before running anything that needs API keys. Use a real sourcing pass (not `export $(... | xargs)`, which mangles any value containing spaces or quotes):\n ```bash\n set -a; . ./.env; set +a\n ```\n Provider-specific env vars the prerelease set may touch: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `GROQ_API_KEY`, `OPENROUTER_API_KEY`, `FIREWORKS_API_KEY`, `TOGETHERAI_API_KEY`, `SILICONFLOW_CN_API_KEY`, `COHERE_API_KEY`, `TYPESAFE_API_KEY`, `CLOUDFLARE_API_KEY` + `CLOUDFLARE_ACCOUNT_ID` (used by `test_cloudflare` in `test_prompt_adaptors.py`, which runs GLM 4.7 Flash; a free-plan Cloudflare account is enough), `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`, plus `KILN_TEST_VERTEX_PROJECT_ID` (+ optional `KILN_TEST_VERTEX_LOCATION`) for the Vertex live check, which also requires `gcloud auth application-default login` against a project with `aiplatform.googleapis.com` enabled.\n- **A missing key is a coverage gap, not a failure.** Each missing key makes the relevant test `pytest.skip(...)`. That isn't a prerelease failure — surface it in the report so the user can decide whether to provide the key and re-run.\n- **`ml_model_list.py` never deletes entries — it only marks them `deprecated=True`.** So a slug missing from our list is not a signal you'll normally encounter. The signals that matter when diagnosing a break or staleness are:\n 1. The provider entry has `deprecated=True` in our list.\n 2. The provider itself rejects the slug at runtime (4xx / \"model not found\") — note our list trails provider deprecations, so this can happen *before* we've marked it deprecated. This is how you'd discover a model \"no longer exists\" even when our list still lists it.\n 3. A newer non-deprecated sibling in the same family now exists in our list, regardless of whether the current pin still works.\n- **Single source of truth for the model list** is `libs/core/kiln_ai/adapters/ml_model_list.py` (chat) and `ml_embedding_model_list.py` (embeddings). Cross-reference there when diagnosing.\n- **The pin sweep is mandatory, not \"only when something fails.\"** Even on a fully green run, every hardcoded slug in `pytest_prerelease_whitelist.py` and in prerelease test parametrize lists must be cross-checked against the model list and reported (kept-and-current vs. has-newer-sibling vs. deprecated). Zero silent \"looks fine, moving on\" calls.\n\n---\n\n## Phase 1 — Set up the output directory\n\n```bash\nTS=$(date -u +%Y-%m-%dT%H-%M-%SZ)\nOUT=\".prerelease/${TS}\"\nSCRATCH=\"${OUT}/_scratch\"\nmkdir -p \"${SCRATCH}\"\necho \"Writing prerelease results to ${OUT}\"\n```\n\n`.prerelease/` is gitignored. This timestamped folder is the **only** place this skill writes. Keep it tidy — the top level should hold just the human-readable artifacts:\n\n- `REPORT.md` — the deliverable.\n- `checks.log`, `prerelease.log`, `retry.log` — raw run logs (kept for drill-down).\n\nEverything else — the grep/collect/diff intermediates used while diagnosing (`current_model_ids.txt`, `prerelease_test_files.txt`, `paid_tests.txt`, `prod_probes.txt`, etc.) — goes in `${SCRATCH}` (`${OUT}/_scratch`), **not** the top level. Don't scatter raw filename dumps where the user has to wade through them; the report is what they read.\n\n---\n\n## Phase 2 — Run `checks.sh`\n\nThese are not paid but are mandatory for any release. If they fail, the prerelease check fails before we even spend money on API calls.\n\n```bash\nuv run ./checks.sh --agent-mode 2>&1 | tee \"${OUT}/checks.log\"\n```\n\nCapture the exit code. If it's non-zero, **don't bail** — keep going with the paid tests so the report covers everything in one pass, but mark the overall run as failed.\n\n---\n\n## Phase 3 — Run the prerelease pytest set\n\n```bash\nuv run python3 -m pytest --runprerelease -v --tb=short \\\n -o \"addopts=\" \\\n 2>&1 | tee \"${OUT}/prerelease.log\"\n```\n\nNotes:\n\n- `--runprerelease` is registered in the top-level `conftest.py`. It runs only `@pytest.mark.prerelease` tests and implies `--runpaid` for them.\n- `-o \"addopts=\"` overrides the project default of `-n auto` (xdist). xdist is incompatible with some of the live-API tests (especially anything that touches global litellm state), so run serially.\n- This will be slow. Many of the parametrized tests fan out to dozens of cases.\n- Tests with missing API keys will `skip`, not fail. Capture skip counts.\n\nIf the user wants a faster narrowed run (e.g. only OpenAI + Anthropic), use `-k` to filter:\n\n```bash\nuv run python3 -m pytest --runprerelease -v --tb=short -o \"addopts=\" \\\n -k \"openai or anthropic or vertex\" \\\n 2>&1 | tee \"${OUT}/prerelease.log\"\n```\n\n---\n\n## Phase 4 — Diagnose\n\nThis is where the skill earns its keep: for every failure, explain the cause; for every pin, report staleness. No edits — only findings and recommendations.\n\n### 4a. Diagnose each failure (what broke, why, recommended action)\n\nFor every test that did **not** pass in Phase 3, build one finding. Walk the failure through this decision tree to land on a cause and a recommended action:\n\n1. **Missing credentials → skip, not failure.** (`pytest.skip` for absent key.) Don't list under failures; list under \"Skipped tests\". Recommended action: add the key and re-run if that coverage matters.\n2. **Provider rejected the model slug** — error mentions `model not found`, `does not exist`, `404`, `model_not_found`, `invalid model`, decommissioned, etc. Cross-reference the slug against `ml_model_list.py`:\n - If our list marks that provider entry `deprecated=True` → cause: **deprecated model, provider has dropped it**. Recommended action: bump the pin/whitelist to the newest non-deprecated sibling in the same family (name the candidate).\n - If our list still lists it as non-deprecated → cause: **model no longer exists at the provider; our list trails reality**. Recommended action: (a) bump the test pin/whitelist to a current sibling, and (b) flag that `ml_model_list.py` should mark this entry `deprecated=True`. Name both.\n3. **Auth / quota / rate-limit / transient network** — 401/403/429/5xx, timeouts. Cause: environment or provider-side, not our code. Recommended action: check the key/quota and re-run; only escalate if it reproduces.\n4. **Behavioral assertion mismatch** — the call succeeded but the response shape, content, token/cache fields, streaming chunks, or reasoning output changed. Cause: **provider changed behavior we depend on (real regression)**. Recommended action: investigate the prod path that relies on this; do not assume it's a stale pin. This is the highest-signal failure class — call it out prominently.\n5. **Our-code error** — traceback originates in `libs/core` / `app/` rather than the provider response. Cause: **regression in Kiln**. Recommended action: investigate the implicated code; name the file/function from the traceback.\n\nIf you can't confidently distinguish a stale-pin cause (2) from a real regression (4 or 5), **report it as a possible regression** and say what additional check would disambiguate (e.g. \"try the same call with a current sibling slug to see if it's the model or the code\"). Never recommend a silent pin bump for an ambiguous failure.\n\nRecord, per failure: test ID · provider · cause class · the model slug involved (if any) and its `ml_model_list.py` status · a short excerpt of the actual error · recommended action.\n\n**Retry likely-transient failures once before finalizing.** Live-API tests are flaky: rate limits, 5xx, timeouts, dropped streams, and occasional empty/under-populated responses (e.g. a thinking model returning no reasoning content on one call but not the next) routinely fail a single run without indicating any real breakage. So before you write up the failures, re-run the failing tests in isolation **once** to separate flakes from real defects:\n\n- **Retry these cause classes:** auth/quota/rate-limit/transient (class 3) and ambiguous behavioral/streaming mismatches (class 4) — anything that could plausibly be a one-off. **Do not retry** failures with a deterministic, self-explanatory cause: a deprecated/removed model slug (class 2), a Kiln-code traceback (class 5), or a missing-credential skip. Retrying those just burns money and time.\n- **How:** re-run only the failing node IDs, serially, into a separate log so the original run is preserved:\n ```bash\n uv run python3 -m pytest --runprerelease -v --tb=short -o \"addopts=\" \\\n \"\" \"\" ... \\\n 2>&1 | tee \"${OUT}/retry.log\"\n ```\n One retry pass is enough — this is a flake filter, not a stability loop. Don't retry a test more than once.\n- **Reclassify by the retry outcome:** a test that **passes on retry** is **transient/flaky**, not a release blocker — move it out of the hard-failure set. A test that **fails again** is **reproducible** — diagnose it as a real finding per the tree above.\n- **Always disclose the retry in the report.** Never silently swap a first-run FAIL for a PASS. Every retried test must show its first-run result, its retry result, and the verdict (transient vs. reproducible) — see the report template's \"Failures\" section and the retry note in the Verdict. A green retry is evidence the suite is flaky here, which the user needs to know before tagging.\n\n### 4b. Model-pin staleness sweep (mandatory every run, even when green)\n\nIndependent of failures, every hardcoded model slug the prerelease set depends on must be checked against the model list and reported. A pin can be perfectly green today and still be worth bumping because a newer sibling shipped — prerelease only catches what real users hit if it tracks what real users actually use.\n\n**In scope (check these every run):**\n- Every entry in `libs/core/kiln_ai/adapters/pytest_prerelease_whitelist.py`.\n- Hardcoded model slugs inside the body or parametrize list of any `@pytest.mark.prerelease` test (notably `libs/core/kiln_ai/adapters/test_prompt_adaptors.py::test_openrouter`).\n\n**Out of scope (don't report as stale):**\n- Hardcoded slugs in mock-only unit tests (no `@pytest.mark.paid` / `@pytest.mark.prerelease`) — the slug is a label, no real call happens.\n- Slugs in fixtures, recorded VCR cassettes, snapshot files.\n- Slugs in `@pytest.mark.paid` tests that aren't also `@pytest.mark.prerelease`.\n\n**Step 1 — status of each whitelist entry.** Run:\n\n```bash\nuv run python3 - <<'PYEOF'\nfrom kiln_ai.adapters.ml_model_list import built_in_models\nfrom kiln_ai.adapters.ml_embedding_model_list import built_in_embedding_models\nfrom kiln_ai.adapters.pytest_prerelease_whitelist import (\n PRERELEASE_CHAT_MODELS, PRERELEASE_EMBEDDING_MODELS,\n PRERELEASE_EXTRACTION_MODELS, PRERELEASE_THINKING_MODELS,\n PRERELEASE_JEV_MODELS,\n)\n\ndef chat_status(name, provider):\n for m in built_in_models:\n if m.name == name:\n p = next((p for p in m.providers if p.name.value == provider), None)\n return 'NO_PROVIDER' if p is None else ('DEPRECATED' if p.deprecated else 'OK')\n return 'MODEL_REMOVED_OR_RENAMED'\n\ndef emb_status(name, provider):\n for m in built_in_embedding_models:\n if m.name == name:\n p = next((p for p in m.providers if p.name.value == provider), None)\n return 'NO_PROVIDER' if p is None else 'OK'\n return 'MODEL_REMOVED_OR_RENAMED'\n\nprint('== chat / extraction ==')\nfor n,p in PRERELEASE_CHAT_MODELS + PRERELEASE_EXTRACTION_MODELS + PRERELEASE_JEV_MODELS:\n print(f'{chat_status(n,p):>22} {n} ({p})')\nprint('== embedding ==')\nfor n,p in PRERELEASE_EMBEDDING_MODELS:\n print(f'{emb_status(n,p):>22} {n} ({p})')\nprint('== thinking ==')\nfor prov,name,lvl in PRERELEASE_THINKING_MODELS:\n print(f'{chat_status(name,prov):>22} {name} ({prov}, level={lvl})')\nPYEOF\n```\n\nAnything not `OK` is stale and goes in the sweep table with a recommended replacement (newest non-deprecated sibling in the same family).\n\n**Step 2 — newer-sibling pass for the `OK` entries.** For each entry that's `OK`, scan `ml_model_list.py` for a newer member of the same family. List the canonical model ids to compare against:\n\n```bash\ngrep -oE '^[[:space:]]+[a-z][a-z0-9_]+ = \"[a-z0-9_]+\"' \\\n libs/core/kiln_ai/adapters/ml_model_list.py | \\\n awk -F'\"' '{print $2}' | sort -u > \"${SCRATCH}/current_model_ids.txt\"\n```\n\nExamples of the judgment call:\n- `PRERELEASE_CHAT_MODELS` on `gpt_4o_mini` while `gpt_5_x_mini` exists → recommend bump.\n- `PRERELEASE_CHAT_MODELS` on `claude_sonnet_4_5` while `claude_sonnet_4_6`/`4_7` exists → recommend bump.\n- `gemini_1_5_*` → `gemini_2_5_*` / `gemini_3_*`; `claude_3_5_*` → `claude_sonnet_4_*` / `claude_4_5_haiku`; `llama_3_x` → `llama_3_3_70b` / `llama_4_*`.\n\nRule of thumb: if `ml_model_list.py` ordering puts a newer sibling above the current pin, recommend the bump. If it's genuinely ambiguous which sibling is \"current\", say so and let the user pick — don't assert a churny recommendation.\n\n**Step 3 — per-test hardcoded slugs.** Enumerate the prerelease test files and scan them for slugs not in `current_model_ids.txt`:\n\n```bash\nuv run python3 -m pytest --runprerelease --collect-only -q -o \"addopts=\" \\\n 2>&1 | grep '::' | awk -F'::' '{print $1}' | sort -u > \"${SCRATCH}/prerelease_test_files.txt\"\n```\n\nCross-reference each pinned slug against `current_model_ids.txt` and the deprecation status. Report any obsolete pins as recommendations.\n\nEvery in-scope pin lands in the sweep table — including the ones that are current and need no action. Silent omission is not allowed.\n\n### 4c. Coverage gaps\n\nThe prerelease set should cover, at minimum:\n\n| Area | Test(s) |\n|---|---|\n| Vertex live connect | `app/desktop/studio_server/test_provider_api.py::test_connect_vertex_live` |\n| OpenAI basic call | `libs/core/kiln_ai/adapters/test_prompt_adaptors.py::test_openai` |\n| Anthropic via OpenRouter / Anthropic API | `test_structured_output_anthropic_*`, `test_structured_output_openrouter_function_calling_nested_object` |\n| Gemini API | covered via embeddings + prompt caching + thinking levels |\n| Groq | `test_groq` |\n| OpenRouter multi-model | `test_openrouter[...]` (parametrized) |\n| Amazon Bedrock | `test_amazon_bedrock` |\n| Embedding API (whitelist) | `test_paid_generate_embeddings_basic_prerelease_smoke`, `test_generate_embedding_prerelease_smoke` |\n| Custom-dimension embeddings (whitelist) | `test_paid_generate_embeddings_custom_dimensions_prerelease_smoke`, `test_generate_embedding_with_user_supplied_dimensions_prerelease_smoke` |\n| Structured output (json_schema, function_calling, flat/nested/array) | `test_structured_output_*` |\n| Streaming (OpenAI protocol + AI SDK protocol) | `test_invoke_openai_stream_chunks`, `test_invoke_ai_sdk_stream` |\n| Tool calling | `test_tools_gpt_4_1_mini` |\n| Thinking-level reasoning (whitelist) | `test_thinking_level_reasoning_content_prerelease_smoke` |\n| Reranker | `test_reranker_integration_success` |\n| Document extraction (whitelist of models × probe mime types) | `test_extract_document_success_prerelease_smoke`, `test_provider_bad_request_prerelease_smoke` |\n| Semantic chunker (real embedding integration) | `test_semantic_chunker_real_integration` |\n| Fireworks fine-tune | `test_fetch_all_deployments` |\n| Prompt caching (Anthropic + OpenAI + Gemini + Fireworks + Together) | `test_prompt_caching_cache_hit` |\n\nIf a new provider/family in `ml_model_list.py` isn't represented in the prerelease set, note the gap as a recommendation.\n\nAlso list recently added `@pytest.mark.paid` tests that are *not* `@pytest.mark.prerelease`-tagged, so the user can decide if any should be promoted:\n\n```bash\n# Compare collected test NODE IDs, not marker line numbers: a test carrying both\n# markers on different lines would be misclassified by a file:line diff. Marker\n# expressions select by marker regardless of the skip gates, so collect-only works.\n# `sed 's/\\[.*//'` collapses parametrized cases to one row per test function\n# (the paid fan-out tests expand to thousands of cases otherwise).\nuv run python3 -m pytest --collect-only -q -m \"paid and not prerelease\" -o \"addopts=\" \\\n 2>/dev/null | grep '::' | sed 's/\\[.*//' | sort -u > \"${SCRATCH}/paid_only_nodes.txt\"\n# Recently added files among those paid-only tests (pipe to xargs to avoid\n# \"argument list too long\" on a large list):\nawk -F'::' '{print $1}' \"${SCRATCH}/paid_only_nodes.txt\" | sort -u | \\\n xargs git log --since=\"3 months ago\" --diff-filter=AM --name-only --pretty=format: -- \\\n 2>/dev/null | sort -u | head -50 > \"${SCRATCH}/recent_paid_only_files.txt\"\n```\n\n### 4d. Prod-code model probes (flag only)\n\nSome prod-code paths hardcode a model slug as a **probe** — a narrow connectivity/auth check where the model is just a vehicle to hit the API, not a feature the user picked. The canonical example is `app/desktop/studio_server/provider_api.py::connect_vertex`, which calls `litellm.acompletion(model=\"vertex_ai/gemini-X.Y-flash\", …)` to verify Vertex credentials when the user clicks \"Connect\".\n\nThese should stay on the latest available model in their family, otherwise the Connect button drifts toward a deprecated/removed model. **This skill only flags them** — it does not edit prod code, and the recommendation in the report is the artifact the user acts on.\n\nSweep for hardcoded inference-probe slugs in non-test prod files:\n\n```bash\ngrep -rnE 'model=[\"'\\''][^\"'\\'']*((gemini|gpt|claude|llama)[^\"'\\'']*)[\"'\\''']' \\\n app/desktop libs/core/kiln_ai libs/server/kiln_server \\\n --include=\"*.py\" 2>/dev/null \\\n | grep -v \"test_\" | grep -v \"/build/\" \\\n | tee \"${SCRATCH}/prod_probes.txt\"\n```\n\nFor each hit:\n1. Decide whether it's a probe (connectivity/auth check, model incidental) or real prod behavior (a default the user sees, fine-tune flow, eval scoring). If unsure, treat it as real prod behavior.\n2. For probes, check `ml_model_list.py` for the newest non-deprecated sibling at that provider.\n3. Record in the probe table: file:line · current slug · suggested newer slug (or \"none — already latest\") · the verifying live test and its result.\n\nEvery grep hit goes in the table even when no newer sibling exists. If the verifying live test failed outright, also list it under Failures (it's a real-failure signal).\n\n---\n\n## Phase 5 — Write the report\n\nCreate `.prerelease//REPORT.md`. Lead with the verdict; make every finding skim-readable with an explicit recommended action. This is the only artifact the skill produces.\n\n```markdown\n# Kiln Prerelease Check — \n\n## Verdict\n- Overall: PASS / FAIL / PASS WITH RECOMMENDATIONS\n- checks.sh: \n- prerelease pytest: \n- Retries: \n- Headline: \n\n## Test results\nOne scannable table covering **every prerelease test that ran**, grouped by coverage area, with an emoji outcome and a short note where it helps — this is the at-a-glance summary the user reads first. Aggregate parametrized cases into one row with per-case emoji counts (e.g. `✅×8 ❌×1`); don't spell out long node-id / filename lists here (raw lists live in the logs and `_scratch/`). Outcome legend: ✅ pass · ❌ fail · ⏭️ skip (missing key / N/A) · 🔁 transient (failed then passed on retry). Leave the Notes cell blank when the row is a clean all-pass; use it to name the failing case, why it skipped, or the retry result.\n\n| Area | Test(s) | Result | Notes |\n|---|---|---|---|\n| OpenAI basic call | `test_openai` | ✅ | |\n| Groq | `test_groq` | ✅ | |\n| OpenRouter multi-model | `test_openrouter` (9) | ✅×9 | |\n| Amazon Bedrock | `test_amazon_bedrock` | ✅ | |\n| Structured output | `test_structured_output_*` (6) | ✅×6 | |\n| Streaming | `test_invoke_openai_stream_chunks` / `test_invoke_ai_sdk_stream` (N) | ✅×N | |\n| Tool calling | `test_tools_gpt_4_1_mini` | ✅ | |\n| Thinking-level reasoning | `…reasoning_content_prerelease_smoke` (N) | ✅×N | |\n| Embeddings | `…embeddings_basic_prerelease_smoke` (6) | ✅×4 ⏭️×2 | |\n| Reranker | `test_reranker_integration_success` (N) | ✅×1 ❌×2 | |\n| Prompt caching | `test_prompt_caching_cache_hit` (7) | ✅×7 | |\n| Document extraction | `…extract_document_*_prerelease_smoke` (N) | ✅×N | |\n| Semantic chunker | `test_semantic_chunker_real_integration` | ✅ | |\n| Vertex live connect | `test_connect_vertex_live` | ❌ | |\n| Fireworks fine-tune | `test_fetch_all_deployments` | ✅ | |\n\nUse the real areas/tests/counts from this run (mirror the coverage table in Phase 4c). Put 🔁 on anything that needed a retry and add the retry detail in the Notes cell + under Failures.\n\n## Failures — what broke, why, and what to do\nOne block per failing test. Sorted most-actionable first (real regressions before stale pins).\nIf any failures were retried (Phase 4a), open this section with a small first-run-vs-retry table so transient flakes are visible at a glance, then write a block only for the tests that **reproduced** (note retried-and-passed ones as transient — don't dignify them with a full diagnosis block).\n\n### \n- Provider: \n- Cause: \n- Model slug: — `ml_model_list.py` status: \n- Retry: \n- Evidence: \n- Recommended action: \n\n## Skipped tests (missing credentials)\nList by provider, with which env var is missing, so the user can decide what to add and re-run.\n\n## Model-pin staleness sweep (mandatory, every run)\nEvery in-scope hardcoded slug — including current ones needing no action.\n\n| Location (file:line) | Current slug | `ml_model_list.py` status | Newer sibling? | Recommendation |\n|---|---|---|---|---|\n| `…/pytest_prerelease_whitelist.py:NN` | `claude_sonnet_4_5` | OK | yes — `claude_sonnet_4_6` | bump to `claude_sonnet_4_6` |\n| `…/pytest_prerelease_whitelist.py:NN` | `gpt_4o_mini` | OK | no | keep (current) |\n| `…/test_prompt_adaptors.py:NN` | `gemini_1_5_flash` (openrouter) | deprecated | n/a | bump to `gemini_2_5_flash` |\n\n`status` ∈ {OK, deprecated, no-provider-entry, removed/renamed}. `Recommendation` ∈ {keep (current), bump to , investigate}.\n\n## Prod-code probe sweep (flag only — no edits)\nEvery hardcoded inference-probe slug found in prod code (Phase 4d).\n\n| File:line | Current slug | Suggested newer slug | Verifying test | Test result | Notes |\n|---|---|---|---|---|---|\n| `app/desktop/studio_server/provider_api.py:NNNN` | `vertex_ai/gemini-2.5-flash` | `vertex_ai/gemini-3-flash` | `test_connect_vertex_live` | PASS | probe still works; newer family member exists |\n| `path/to/file.py:NN` | `…` | none — already latest | `test_…` | PASS | no action |\n\n## Coverage gaps\n- Providers/families in `ml_model_list.py` with no prerelease test.\n- Recently added paid tests not in the prerelease set (one line each on why it MIGHT be prerelease-worthy).\n\n## Flagged for user review\nAnything you noticed but couldn't safely classify: ambiguous failures (model-vs-code unclear), prod-path code that isn't a probe, suspicious behavior. Each row: file:line / test · what's suspicious · what would disambiguate.\n\n## Suggested next steps\nOrdered checklist of what the user should do before tagging the release (apply recommended pin bumps, investigate regression X, add key for skipped provider Y, etc.).\n```\n\nKeep it scan-readable. Lead with the verdict. Don't bury real regressions under pin-housekeeping.\n\n---\n\n## Phase 6 — Hand back to the user\n\nPrint the report path and a one-line verdict:\n\n```bash\necho \"Prerelease check complete. Report: ${OUT}/REPORT.md\"\n```\n\nThen summarize in chat (one or two sentences):\n\n- All green, nothing stale: `Prerelease check: PASS. passed, skipped (missing keys: ). No stale pins. Report at .`\n- Green but recommendations exist: `Prerelease check: PASS WITH RECOMMENDATIONS. passed; stale/older pins to consider bumping (no changes made). Report at .`\n- Something failed: `Prerelease check: FAIL. failed across — . See /REPORT.md before tagging.`\n\nRemember: the skill never made any edits. If the report recommends pin bumps or a prod-probe change, those are for the user to apply.\n\n---\n\n## Additional release checklists\n\n- **Code Tools / Code Evals cross-OS spawn sanity**: see `specs/projects/code_tools/cross_os_spawn_checklist.md` for platform-specific verification steps (macOS/Windows/Linux + PyInstaller frozen builds). Must be completed before any release that includes code tools or code evals.\n\n## Checklist\n\n- [ ] `.prerelease//` directory created (the only place anything was written)\n- [ ] `checks.sh --agent-mode` run, log captured\n- [ ] `--runprerelease` pytest run, log captured\n- [ ] Likely-transient failures (auth/quota/transient + ambiguous behavioral) retried once in isolation; deterministic failures (deprecated/removed slug, Kiln-code traceback) not retried\n- [ ] Every retried test disclosed in the report (first-run result, retry result, transient-vs-reproducible verdict) — no silent FAIL→PASS swaps\n- [ ] Every failure diagnosed with a cause class and a concrete recommended action (regression vs. deprecated-model vs. model-gone vs. auth/transient)\n- [ ] Model-pin staleness sweep table filled for **every** in-scope slug, including current ones needing no action\n- [ ] Newer-sibling recommendations included even for green pins\n- [ ] Prod-probe sweep done; every grep hit in the probe table with a suggested slug (or \"none — already latest\")\n- [ ] Coverage-gap audit done\n- [ ] Skipped tests listed by provider + missing env var\n- [ ] **No files edited outside `.prerelease//`** — all fixes are recommendations\n- [ ] `REPORT.md` written, verdict first, recommended actions explicit\n- [ ] Verdict surfaced to the user with the report path\n",".agents/skills/claude-maintain-models/SKILL.md":"---\nname: claude-maintain-models\ndescription: Add new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, or wants to add a net-new AI provider to Kiln.\nallowed-tools: Read Edit Write Bash Grep Glob Agent WebSearch WebFetch\n---\n\n# Add a New AI Model to Kiln\n\n**Branch check first:** if the request involves a provider Kiln does not\nsupport yet, start at [Adding a Net-New Provider](#adding-a-net-new-provider).\nThat workflow spans core, server, UI, tests, and tooling, and is gated on a\nclient release — the model-entry steps below are only its final, gated piece.\n\nFor a new model on an already-supported provider, integrating it into\n`libs/core/kiln_ai/adapters/ml_model_list.py` requires:\n\n1. **`ModelName` enum** – add an enum member\n2. **`built_in_models` list** – add a `KilnModel(...)` entry with providers\n3. **`ModelFamily` enum** – only if the vendor is brand-new\n\nAfter code changes, run paid integration tests, then draft a Discord post.\n\n---\n\n## Global Rules\n\nThese apply throughout the entire workflow.\n\n- **Slug verification:** NEVER guess or infer model slugs from naming patterns. Every `model_id` must come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it.\n- **Date awareness:** These models are often released very recently. Web search for current info before assuming you know the details.\n\n---\n\n## Phase 1 – Model Discovery (only when asked to find new/missing models)\n\nIf the user asks you to find new models, **do NOT just web search \"new AI models this week\"** — that only surfaces major releases. Instead, systematically check each family against **both** the LiteLLM catalog **and** models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.\n\n1. **Read the `ModelFamily` and `ModelName` enums** to know what we already have.\n\n2. **Query both catalogs for each family** (run in parallel where possible):\n\n **LiteLLM catalog** — filters out mirror providers to avoid duplicates:\n ```bash\n curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != \"openrouter\" and .provider != \"bedrock\" and .provider != \"bedrock_converse\" and .provider != \"vertex_ai-anthropic_models\" and .provider != \"azure\") | .id] | unique | .[]'\n ```\n\n **models.dev** — search all model IDs across all providers:\n ```bash\n curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i \"SEARCH_TERM\"\n ```\n For details on a specific provider+model: `curl -s https://models.dev/api.json | jq '.[\"PROVIDER\"].models[\"MODEL_ID\"]'`\n\n3. **Search terms** (one query per term):\n `claude`, `gpt`, `o1`, `o3`, `o4` (OpenAI reasoning), `gemini`, `llama`, `deepseek`, `qwen`, `qwq`, `mistral`, `grok`, `kimi`, `glm`, `minimax`, `hunyuan`, `ernie`, `phi`, `gemma`, `seed`, `step`, `pangu`\n\n4. **Union and cross-reference** results from both catalogs against `ModelName`. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). **Skip pure coding models** (e.g. `codestral`, `deepseek-coder`, `qwen-coder`).\n\n5. **Run targeted web searches** per family to catch very fresh releases not yet in either catalog:\n - `\"[family] new model [current year]\"`\n - `\"[family] release [current month] [current year]\"`\n\n6. **Present findings** as a summary. Let the user decide which to add.\n\n---\n\n## Phase 1B – Lagging-Provider Backfill Check (every run)\n\nSome providers — **Fireworks AI**, **Together AI**, **SiliconFlow**, **Cloudflare** — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM (or, for Cloudflare, never reach LiteLLM at all). Relying only on those two catalogs will both under-populate the provider list for the model you're adding now **and** miss the window to backfill recently-added models whose provider support has since grown.\n\nRun this check on **every invocation** of the skill, regardless of whether you're in discovery mode or adding a specific model.\n\n1. **Pull the 10 most recently added models from git history.** List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:\n ```bash\n git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E \"^\\+\\s+name=ModelName\\.\" | head -20\n ```\n\n2. **For the model you're adding (if any) AND each of those 10 models**, cross-check Fireworks, Together, SiliconFlow and Cloudflare directly using the endpoints in the [Lagging Providers Reference](#lagging-providers). Do NOT trust `models.dev` / LiteLLM as the final word for these providers.\n\n3. **If a lagging provider now supports a recently-added model that isn't yet in its `KilnModel` entry**, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.\n\n4. **Run the Cloudflare thinking-level checker** (see [Cloudflare thinking levels](#cloudflare-thinking-levels)). It checks every Cloudflare entry, not only the recent ones, because Cloudflare edits the effort lists of models Kiln already ships. Treat each ❌ finding like a backfill: fix the entry in the current change or propose a separate PR.\n\n---\n\n## Phase 2 – Gather Context\n\n1. **Read the predecessor model** in `ml_model_list.py` (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.\n\n2. **Query the LiteLLM catalog** for the new model. This is the primary slug source since Kiln uses LiteLLM. See the [Slug Lookup Reference](#slug-lookup-reference) for query syntax and all verified sources.\n\n3. **Get the OpenRouter slug** via:\n - `curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i \"SEARCH_TERM\"`\n - Fallback: WebSearch for `openrouter [model name] model id`\n\n4. **Get the direct-provider slug** (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the [Slug Lookup Reference](#slug-lookup-reference) for provider-specific URLs.\n\n5. **Identify quirks** — check the [Provider Quirks Reference](#provider-quirks-reference) for the relevant provider, and web search for any new quirks:\n - Structured output mode (JSON schema vs function calling)?\n - Reasoning model (needs `reasoning_capable`, parsers, OpenRouter options)?\n - Vision/multimodal support? Which MIME types?\n - Provider-specific flags (`temp_top_p_exclusive`, etc.)?\n - Rate limit concerns (`max_parallel_requests`)?\n\n6. **Determine thinking levels** — does the model support configurable reasoning effort? See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain. Key quick checks:\n - Check the **vendor model page** (e.g. OpenAI model pages say \"Reasoning.effort supports: X, Y, Z\")\n - Check **OpenRouter** `supported_parameters` — if `reasoning` is absent, skip thinking levels\n - R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts\n\n---\n\n## Phase 3 – Code Changes\n\nAll changes go in `libs/core/kiln_ai/adapters/ml_model_list.py`.\n\n### 3a. `ModelName` enum\n\n- snake_case: `claude_opus_4_6 = \"claude_opus_4_6\"`\n- Place **before** predecessor (newer first within group). If the vendor is\n brand-new there is no predecessor — start a new group at the end of the enum\n- Follow existing grouping (all claude together, all gpt together, etc.)\n\n### 3b. `KilnModel` entry in `built_in_models`\n\n- Place per the **ordering rules in 3c** — this placement is user-visible, get it right\n- Copy predecessor's structure and modify: `name`, `friendly_name`, `model_id` per provider, flags\n- **`friendly_name` must follow the existing naming pattern** of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use `\"Claude {version} Sonnet\"` (e.g. \"Claude 4.5 Sonnet\"), not `\"Claude Sonnet {version}\"`. Do NOT use the vendor's marketing name if it differs from Kiln's established convention.\n\n**Provider `model_id` formats:**\n\n| Provider | Format | Notes |\n|----------|--------|-------|\n| `openrouter` | `vendor/model-name` | Always verify via API |\n| `openai` | Bare model name | Verify via OpenAI docs |\n| `anthropic` | Variable — older models have date stamps, newer may not | Always verify via Anthropic docs |\n| `gemini_api` | Bare name | Verify via Google AI Studio docs |\n| `fireworks_ai` | `accounts/fireworks/models/...` | Verify via Fireworks docs |\n| `together_ai` | Vendor path format | Verify via Together docs |\n| `vertex` | Usually same as gemini_api | Verify via Vertex docs |\n| `siliconflow_cn` | Vendor/model format | Verify via SiliconFlow docs |\n| `featherless_ai` | HuggingFace repo id, case-sensitive (`zai-org/GLM-5.2`) | Verify via their `/v1/models` — see [Featherless](#featherless-ai) |\n| `typesafe` | Dotted System One version (`jev-1.13.0`) | Their `/v1/models` lists aliases only — verify by `POST /v1/systemone`, see [TypeSafe AI](#typesafe-ai) |\n| `cloudflare` | Cloudflare's own `@cf//` id, verbatim (`@cf/zai-org/glm-5.3`) — no `cloudflare/` or `openai/` prefix | Verify via Cloudflare's model JSON — see [Cloudflare](#cloudflare-workers-ai) |\n\n**Every single `model_id` must be verified from an authoritative source. No exceptions.**\n\n**Setting flags — use catalog data + predecessor as dual signals:**\n\nThe LiteLLM catalog and models.dev responses include capability flags (`supports_vision`, `supports_function_calling`, `supports_reasoning`, etc.). Use these as the **primary signal** for what to enable on the new model:\n\n- If the catalog says `supports_vision: true` → enable `supports_vision`, `multimodal_capable`, and vision MIME types (see 2c)\n- If the catalog says `supports_function_calling: true` → use `StructuredOutputMode.json_schema` (or `function_calling` depending on provider norms — check predecessor)\n- If the catalog says `supports_reasoning: true` → the model can reason, but **do NOT reflexively set `reasoning_capable=True`** — default to `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)). Still add `available_thinking_levels` if it supports effort levels, and check parser/formatter flags.\n\nThen **cross-check against the predecessor**. The predecessor tells you *how* Kiln configures a similar model (which `structured_output_mode`, which provider-specific flags, etc.). The catalog tells you *what* the model can do. Use both:\n- Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.\n- Predecessor has `temp_top_p_exclusive` but nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track.\n- Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.\n\n**Common flags:**\n- `structured_output_mode` – how the model handles JSON output\n- `suggested_for_evals` / `suggested_for_data_gen` – see **zero-sum rule** below. If the entry also sets `available_thinking_levels`, its `default_thinking_level` must be a reasoning level — see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)\n- `multimodal_capable` / `supports_vision` / `supports_doc_extraction` – see **multimodal rules** below\n- `reasoning_capable` – for thinking/reasoning models. **Default new models to `reasoning_capable=False`** unless the model *always* emits its reasoning (see [Reasoning Capable Default](#reasoning-capable-default))\n- `temp_top_p_exclusive` – Anthropic models that can't have both temp and top_p\n- `parser` / `formatter` – for models needing special parsing (e.g. R1-style thinking)\n\n#### 2c. Multimodal capabilities\n\nIf the model supports non-text inputs, configure:\n\n- `multimodal_capable=True` and `supports_doc_extraction=True` if it supports any MIME types\n- `supports_vision=True` if it supports images\n- `multimodal_requires_pdf_as_image=True` if vision-capable but no native PDF support (also add `KilnMimeType.PDF` to MIME list). **Always set this on OpenRouter providers** — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.\n- Always include `KilnMimeType.TXT` and `KilnMimeType.MD` on any `multimodal_capable` model\n\n**Strategy: start broad, narrow based on test failures.** Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.\n\nFull MIME superset (Gemini uses all):\n```python\n# documents\nKilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD\n# images\nKilnMimeType.JPG, KilnMimeType.PNG\n# audio\nKilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG\n# video\nKilnMimeType.MP4, KilnMimeType.MOV\n```\n\n### 3c. Ordering — the list IS the UI\n\nThe order of `built_in_models` is exactly the order users see: model dropdowns\ngroup by provider and list each provider's models in list order, and the model\nlibrary page lists models in list order. A misplaced entry ships a scrambled\ndropdown to every client via the remote config. Rules:\n\n1. **Families are contiguous.** Every model of a family sits in one block.\n Never append a new family member at the bottom of the file or after another\n family — that strands it (past bugs: Mistral Small 4/3 ended up inside the\n Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).\n2. **Within a family, versions run newest → oldest, top to bottom.**\n 4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the\n family block. A new model of an *existing* version goes inside that\n version's group — NOT at the top of the family and NOT below older\n versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7\n landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).\n3. **Within a version, big → small. Always.**\n - Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite;\n Large > Medium > Small.\n - Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking\n variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and\n VL Thinking blocks) stay intact, ordered descending internally.\n - Tier-grouped families follow the same rule for their tier blocks: Claude\n runs Fable, then Opus > Sonnet > Haiku, versions descending inside each\n tier. (It once ran Haiku-first, and several families ran sizes\n ascending — those were bugs, not conventions. Do not preserve an\n ascending run because it's \"what the family already does.\")\n4. **A net-new family's block goes at the END of `built_in_models`** (this is\n the existing convention — the newest niche vendors sit at the bottom of the\n list). The \"place before predecessor\" rule only applies within an existing\n family; a new family has no predecessor.\n5. **Verify after editing** — print the order you just shipped and eyeball the\n affected family:\n\n ```bash\n uv run python -c \"\n from kiln_ai.adapters.ml_model_list import built_in_models\n for m in built_in_models: print(m.family, '|', m.friendly_name)\"\n ```\n\n### 3d. `suggested_for_evals` / `suggested_for_data_gen`\n\n**Only set these if** the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).\n\n**Zero-sum rule:** When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. **Ask the user to confirm** the swap before making changes.\n\n### 3e. `ModelFamily` enum (only if needed)\n\nOnly add a new family if the vendor is completely new.\n\n### 3f. Thinking Levels (`available_thinking_levels` / `default_thinking_level`)\n\nIf the model supports configurable reasoning effort (not just on/off), add `available_thinking_levels` and `default_thinking_level` to each provider entry. See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain and existing constants.\n\n**Quick rules:**\n- Reuse an existing `_THINKING_LEVELS` constant if the levels match exactly\n- Create a new constant only if levels differ; name it `{MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS`\n- `default_thinking_level` must be one of the values in `available_thinking_levels`\n- If the entry sets `suggested_for_evals=True`, `default_thinking_level` must be `\"medium\"` (or `\"low\"` if the model has no medium level) — never `\"none\"`. See [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)\n\n---\n\n## Phase 4 – Run Tests\n\nTests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.\n\n**Vertex AI authentication:** Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run `gcloud auth application-default login` before trying. These failures are auth issues, not model config problems.\n\n**`-k` filter syntax:** Always use bracket notation for model+provider filtering, never `and`:\n- Good: `-k \"test_name[glm_5-fireworks_ai]\"` or `-k \"glm_5\"`\n- Bad: `-k \"glm_5 and fireworks\"` — `and` is a pytest keyword expression that can match wrong tests\n\n### 4.0 — If the test env can't build (blocked git dependency)\n\nKiln's core lib pins `together` to a git fork (`libs/core/pyproject.toml`: `together = { git = \"https://github.com/scosman/together-python\" }`). In a sandboxed environment whose GitHub access is scoped to `kiln-ai/kiln` only (e.g. Claude Code Web), `uv sync` fails to fetch that fork with a **403** from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only `kiln-ai/kiln` resolves.\n\nWorkaround to build the venv for testing (the `together` fork isn't exercised by model-integration tests, which route through LiteLLM):\n\n1. **Do NOT use `uv sync --no-sources`** — it strips the workspace `workspace = true` sources too and makes `kiln-root → kiln-ai` unsatisfiable.\n2. Instead, temporarily comment out ONLY the `together = { git = ... }` line in `libs/core/pyproject.toml`, then run `uv sync` (this resolves `together` from PyPI).\n3. Run the tests.\n4. **Revert** the workaround with `git checkout -- libs/core/pyproject.toml uv.lock` — this restores the commented-out `together` git-fork pin in `libs/core/pyproject.toml` (NOT the repo-root `pyproject.toml`, which the workaround never touches) and reverts `uv.lock` if it changed. Only `ml_model_list.py` (and any intended test-file edits) should remain modified.\n\nNote: the PyPI `together` may pull slightly different transitive deps (e.g. a newer `starlette`), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your `-k` filters to the model files and ignore those.\n\n### 4a. Parallel testing + API keys\n\n**Parallel testing is already on.** There is no `pytest.ini` — pytest config lives in `[tool.pytest.ini_options]` in the root `pyproject.toml`, and `addopts = \"-n auto\"` is active. No edit and no revert are needed. (Only override to `-n 8` if a provider rate-limits you.)\n\n**Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings.** `conftest.py` has an autouse `use_temp_settings_dir` fixture that points `Config.settings_path` at a temp dir, so `~/.kiln_ai/settings.yaml` is deliberately ignored during tests. A key the user added through the app's provider page **will not be seen** — the test fails with \"Attempted to use X without an API key set\", which looks like a config bug but isn't.\n\nBridge the key from the user's settings into the test environment without ever printing it:\n\n```bash\nexport FEATHERLESS_AI_API_KEY=\"$(uv run python -c \\\n \"from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)\" 2>/dev/null | tail -1)\"\n```\n\nUse the provider's `env_var` name from `libs/core/kiln_ai/utils/config.py`. Some providers need more than one value: Cloudflare needs both `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` (and optionally `CLOUDFLARE_AI_GATEWAY_ID`). To check which keys are available before running, print booleans only — never the values:\n\n```bash\nuv run python -c \"from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))\"\n```\n\n**Many paid tests also carry the `ollama` marker**, so `--runpaid` alone silently skips them. Always pass `--runpaid --ollama` together, and use `-rs` to see skip reasons when a test you expected to run reports as skipped.\n\n### 4b. Smoke test — verify slug works\n\nRun a single test+provider combo first:\n\n```bash\nuv run pytest --runpaid --ollama -k \"test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]\"\n```\n\nIf it fails, fix the slug/config before proceeding. Use `--collect-only` to find exact parameter IDs if unsure.\n\n### 4c. Full test suite\n\n```bash\nuv run pytest --runpaid --ollama -k \"MODEL_ENUM\" -v 2>&1 | grep -E \"PASSED|FAILED|ERROR|short test|=====|collected\"\n```\n\n**If tests fail — debug one at a time:**\n1. Pick ONE failing test, run it with `-v` for full output\n2. Fix the config\n3. Re-run that single test to verify\n4. Only re-run the full suite once the single test passes\n\n**Anthropic API key gotcha:** if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as `KILN_ANTHROPIC_API_KEY` instead of `ANTHROPIC_API_KEY` (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't `export` it globally:\n\n```bash\nANTHROPIC_API_KEY=\"$KILN_ANTHROPIC_API_KEY\" uv run pytest --runpaid ...\n```\n\n### 4d. Extraction tests (if `supports_doc_extraction=True`)\n\nTests are in `libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py`.\n\n```bash\n# See what will run:\nuv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM\n\n# Run them:\nuv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k \"MODEL_ENUM\"\n```\n\nIf a provider rejects a data type (400 error), remove that `KilnMimeType` and re-run.\n\n### 4e. Confirm failures are actually yours\n\nBefore treating a failure as a problem with your change, check whether the same test already fails for an **existing** provider of that model. Several assertions are provider-independent and fail regardless.\n\nKnown example: `test_structured_input_cot_prompt_builder` asserts `len(trace) == 5` unconditionally, which is incompatible with any provider setting `reasoning_capable=True` (that selects the single-call strategy, producing 3 messages). It fails for `gpt_oss_120b` on `fireworks_ai` on a clean tree.\n\n```bash\nuv run pytest --runpaid --ollama -q \"path::test_name[MODEL-OTHER_PROVIDER]\"\n```\n\nIf it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.\n\n### 4f. Test output format\n\nCollect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:\n- ✅ for passed tests\n- ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason\n- ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason\n- List every test using the full pytest parametrize ID, grouped by provider\n- Include extraction tests (Phase 4d) if they were run\n\n---\n\n## Phase 5 – Create Pull Request\n\n### 5.0 — Important context about Claude Code Web's stop hook\n\nThis skill is often run via Claude Code Web (Slack connector). That environment has a **non-user-configurable stop hook** which, at end of session, will:\n- Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits\n- Instruct the agent to commit and push any local work before stopping\n- Explicitly tell the agent NOT to create a PR unless the user asked for one\n\n**The problems this causes:**\n1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned `add-model/*` branches on the remote.\n2. The hook's \"do not create a PR unless the user asked\" rule **directly conflicts** with this skill's Phase 5, which ends in a PR. Running this skill *is* the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.\n\n**The user's desires, in priority order:**\n1. **Ask before you push.** If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code \"just to satisfy the stop hook.\"\n2. **No abandoned branches.** Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.\n3. **If the user says to abandon:** revert your local changes (`git restore` / `git clean` the specific files you touched) and delete any branch you created (`git checkout main && git branch -D add-model/MODEL_NAME`) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.\n4. **On a successful run, push and open the PR as described in 5a/5b.** Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says \"don't create a PR.\" Only re-ask if tests failed or the user hasn't confirmed the results.\n\n### 5.1 — Gate before pushing\n\nDo NOT commit, push, or create a branch if any of the following are true:\n- Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)\n- The smoke test (4b) failed and wasn't resolved\n- Any step in Phases 2–4 was skipped or incomplete\n- You are unsure whether a ⚠️ flake is actually a real failure\n\nIf any of the above apply, **stop and ask the user** what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.\n\nAfter all tests pass, commit the changes and open a PR against `main`.\n\n### 5a. Commit and push\n\n1. Create a new branch named `add-model/MODEL_NAME` (e.g. `add-model/glm-5-1`)\n2. Stage only the changed files (typically just `ml_model_list.py`)\n3. Commit with a concise message (e.g. \"Add GLM 5.1 to model list (together_ai, siliconflow_cn)\")\n4. Push the branch\n\n### 5b. Create the PR\n\nUse `gh pr create` against `main`. Follow Rule 0, Rule 1 and Step 3 of the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) for the human header, the CLA and the title:\n\n- The title is `WIP: chore: ` followed by the model-list subject, for example `WIP: chore: add GLM 5.1 to model list`. Use `chore`, because adding a model is maintenance work, not a new feature. Only a human removes the `WIP: ` prefix.\n- Copy everything above `# Agentic PR Summary` from `.github/pull_request_template.md` with no change. Never fill in a placeholder or tick a box there, and never sign the CLA. Remove the `## Contributor License Agreement` section only when the PR author's GitHub username is `scosman`, `sfierro`, `leonardmq`, `tawnymanticore`, or `chiang-daniel`. Keep it, unsigned, for any other author.\n\nReplace the `` `Insert AI summary of PR using .agents/skills/open-pr/SKILL.md` `` placeholder under `# Agentic PR Summary` with the test results, in this exact format:\n\n```\n## What does this PR do?\n\n Test Results\n\n[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]\n\n[Model Name] ([provider]):\n- [N] passed, [N] skipped[, [N] failed]\n- [Any notable failures or flakes]\n\n[Repeat for each model+provider combo]\n\n---\n[Model Name] ([provider]):\n✅ test_data_gen_all_models_providers[model_enum-provider]\n✅ test_data_gen_sample_all_models_providers[model_enum-provider]\n✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]\n✅ test_all_built_in_models_llm_as_judge[model_enum-provider]\n✅ test_all_built_in_models_structured_output[model_enum-provider]\n✅ test_all_built_in_models_structured_input[model_enum-provider]\n✅ test_structured_output_cot_prompt_builder[model_enum-provider]\n✅ test_all_models_providers_plaintext[model_enum-provider]\n✅ test_cot_prompt_builder[model_enum-provider]\n⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason\n❌ test_name[model_enum-provider] — brief reason\n\n[Repeat for each model+provider combo]\n```\n\n**Rules for the PR body:**\n- Every test that ran must appear in the per-test dump, using the full pytest parametrize ID\n- Group tests by `[Model Name] ([provider]):` headers\n- The summary section at the top gives a quick pass/skip/fail count per model+provider\n- The detailed section below the `---` inside the summary (not the `----` above `# Agentic PR Summary`) lists every individual test result\n- Use ⚠️ for content quality flakes (not real failures), ❌ for real errors\n\n---\n\n## Checklist\n\n- [ ] `ModelName` enum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor)\n- [ ] `KilnModel` entry added to `built_in_models` per the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list)\n- [ ] Printed the resulting list order and eyeballed the affected family (3c verification command)\n- [ ] `friendly_name` matches the naming pattern of sibling models in the same family\n- [ ] If the provider is net-new: followed [Adding a Net-New Provider](#adding-a-net-new-provider), including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)\n- [ ] `ModelFamily` enum updated (only if new family)\n- [ ] All provider slugs verified from authoritative sources\n- [ ] Flags inherited from predecessor and adjusted for quirks\n- [ ] `reasoning_capable` defaulted to `False` for adaptive-reasoning models (only `True` for always-emits-reasoning models — see [Reasoning Capable Default](#reasoning-capable-default))\n- [ ] Thinking levels configured if model supports reasoning effort (see [Thinking Levels Reference](#thinking-levels-reference))\n- [ ] Cloudflare and OpenRouter: `check_thinking_levels.py` reports no ❌ finding (see [Cloudflare thinking levels](#cloudflare-thinking-levels) and [OpenRouter (general)](#openrouter-general))\n- [ ] Every `suggested_for_evals` entry with thinking levels defaults to `\"medium\"` (or `\"low\"`), never `\"none\"` (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default))\n- [ ] Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)\n- [ ] Zero-sum applied if model is suggested for evals/data gen\n- [ ] RAG config templates updated if the new model replaces one used in `app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts`\n- [ ] API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)\n- [ ] Smoke test passed\n- [ ] Full test suite passed\n- [ ] Failures cross-checked against an existing provider before being called regressions (see 4e)\n- [ ] PR created against `main` with a `WIP: ` title, the untouched human header, and test results in the Agentic PR Summary\n\n---\n\n## Reasoning Capable Default\n\n**Default newly-added models to `reasoning_capable=False`, even when the catalog reports `supports_reasoning: true`.**\n\nMost recent models use *adaptive* reasoning and sometimes return no reasoning at all. Kiln raises `RuntimeError(\"Reasoning is required for this model, but no reasoning was returned.\")` whenever a provider has `reasoning_capable=True` and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.\n\n`reasoning_capable` is **orthogonal to `available_thinking_levels`** — you keep the important behavior with it set to `False`:\n- Thinking-level / effort selection still works (every GPT-5.x entry sets `available_thinking_levels` with `reasoning_capable` unset).\n- Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).\n- The `test_thinking_level_reasoning_content` paid test still runs and still asserts reasoning is present — it is gated on `available_thinking_levels`, not `reasoning_capable`. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels.\n\nWhat you give up with `reasoning_capable=False`:\n- The conditional \"reasoning is present in `intermediate_outputs`\" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped).\n- When a user attaches a chain-of-thought prompt, the model uses the two-call `two_message_cot` strategy instead of the single-call native `single_turn_r1_thinking`.\n\n**Keep `reasoning_capable=True` only for models that *always* emit reasoning** in a native `` format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.\n\n**Narrower alternative:** if a model reliably reasons but you only hit the error on structured output, set `reasoning_optional_for_structured_output=True` (requires `reasoning_capable=True`) instead of disabling reasoning entirely.\n\n---\n\n## Adding a Net-New Provider\n\nAdding a provider Kiln has never supported is a bigger job than adding a model,\nand it has a hard sequencing constraint. Follow this section end to end.\n\n### The release-gating rule (do NOT skip)\n\nThe remote config is generated from main's `built_in_models` and reaches **all\nexisting clients immediately** — but provider *support* (name map, connect\nflow, API key handling) only reaches users through a client app release. If\nmodel entries for a brand-new provider land on main before a client release\nwith the provider plumbing is live, every deployed client shows the raw\nprovider ID (e.g. `featherless_ai`) in the model library and offers models\nnobody can connect to. This happened with Featherless in Aug 2026 and the\nentries had to be rolled back.\n\n**Sequence it in two PRs:**\n\n1. **PR 1 — provider plumbing only.** The `ModelProviderName` enum member and\n everything else in the checklist below EXCEPT the model catalog. Merge\n whenever ready.\n2. **PR 2 — model catalog.** ALL `ml_model_list.py` changes: `ModelName`\n members, `ModelFamily` (if the vendor is new), and the `built_in_models`\n entries. Open it, but **merge only after a client release containing PR 1\n is live.** (Only `built_in_models` is published via the remote config, but\n the enums belong in the same PR as the entries they exist for.)\n\n### Touchpoint checklist (from the Featherless integration, #1618)\n\n**First, identify the provider's authentication model** — the checklist below\ndescribes the common single-API-key pattern, but not every provider fits it.\nExisting variants to crib from: Bedrock stores an access key + secret pair,\nFireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project\nID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base\nURL with no credentials. Follow the closest existing analog's plumbing through\n`config.py`, `provider_warnings`, `provider_api.py`, and the connect page —\nthe credential fields, validation call, and UI steps all change with the auth\nmodel.\n\n**libs/core:**\n- `ModelProviderName` enum — `libs/core/kiln_ai/datamodel/datamodel_enums.py`\n- Credential storage — `libs/core/kiln_ai/utils/config.py` (for API-key\n providers: a new key with `env_var`; otherwise whatever fields the auth\n model needs)\n- LiteLLM provider mapping — `libs/core/kiln_ai/utils/litellm.py`\n- `libs/core/kiln_ai/adapters/provider_tools.py` — three spots: the\n `provider_name_from_id` match (friendly name; pyright flags a missed case),\n `provider_warnings` (missing-credential message, `required_config_keys`),\n and the adapter config (credential / base URL / headers plumbing)\n- Model catalog — `libs/core/kiln_ai/adapters/ml_model_list.py`: `ModelName`\n members, `ModelFamily` if needed, and the `built_in_models` entries\n (**PR 2 only**, see gating rule)\n\n**Desktop server (`app/desktop/studio_server/provider_api.py`):**\n- `connect_` credential-validation endpoint (find a cheap\n authenticated call; see the Featherless connect function for a pattern when\n the provider has no authenticated GET to ping)\n- Disconnect handling (clear the stored credentials)\n- Tests in `test_provider_api.py`\n\n**Web UI (`app/web_ui`):**\n- `src/lib/stores.ts` — `provider_name_map` entry (friendly name)\n- `src/lib/ui/provider_image.ts` + SVG in `static/images/` — icon must match\n the monochrome convention: bare glyph, `currentColor`, no background tile,\n viewBox cropped to the artwork\n- Connect page — `src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte`:\n provider card (name, description, and the auth flow — `api_key_steps` /\n `api_key_fields` for key providers, or the custom flow the auth model\n needs) plus connected-status\n wiring from `settings` keys\n- `src/lib/api_schema.d.ts` — regenerate with `make schema` (needs the server\n running on :8757)\n\n**Tests & tooling:**\n- `libs/core/kiln_ai/adapters/test_provider_tools.py`, `test_adapter_registry.py`,\n `model_adapters/test_litellm_adapter.py` — extend the per-provider\n parametrized cases\n- `.agents/scripts/provider_utils.py` — add the provider's model-catalog\n endpoint so agent tooling can enumerate its models\n- Thinking levels — find out how the provider takes reasoning effort, and\n where it publishes the values each model accepts. Add a quirks section\n that names that source and says thinking levels are required wherever\n it lists values. If the provider silently remaps unsupported values,\n add the provider to `check_thinking_levels.py`, because a\n paid test cannot see a remap.\n- `model_adapters/test_thinking_level_paid.py` — add the provider to the\n provider list of `test_thinking_level_reasoning_content`, so each of its\n thinking levels gets a paid test case\n\n**Known gap:** the generated Copilot API client\n(`app/desktop/studio_server/api_client/`) mirrors the remote Copilot service's\nschema — it can't learn the new provider until that service updates. Note it in\nthe PR rather than hand-editing generated code.\n\n### Friendly names everywhere\n\nThe raw enum value must never be user-visible. When you add the provider,\nverify a friendly name exists in **both** name maps (python\n`provider_name_from_id` and web `provider_name_map`) — pyright and typescript\nrespectively force these when the enum gains a member, which is why the\nplumbing PR must not skip them.\n\n---\n\n## Provider Quirks Reference\n\n### Anthropic\n- Newer models (Opus 4.1+, Sonnet 4.5+) need `temp_top_p_exclusive=True`\n- Opus 4.5+ uses `json_schema`; older Opus uses `function_calling`\n- Extended thinking models: `anthropic_extended_thinking=True` + `reasoning_capable=True`\n\n### OpenAI\n- Most GPT models use `json_schema` for structured output\n- GPT-5.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)\n- Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort\n- o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)\n\n### Google/Gemini\n- `gemini_reasoning_enabled=True` for reasoning-capable models\n- Gemini 3.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)\n- Rich multimodal support (audio, video, images, documents)\n\n### DeepSeek\n- R1 models: `parser=ModelParserID.r1_thinking` + `reasoning_capable=True`\n- V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN\n- Some need `r1_openrouter_options=True` + `require_openrouter_reasoning=True`\n\n### OpenRouter (general)\n- Slugs: `vendor/model-name`\n- Never add a `:free` slug. Free tiers have low rate limits and are temporary. If OpenRouter serves a model only as `:free`, wait for a paid slug.\n- Reasoning models: may need `require_openrouter_reasoning=True`\n- Some models: `openrouter_skip_required_parameters=True`\n- Logprobs: `logprobs_openrouter_options=True` if supported\n- Always `multimodal_requires_pdf_as_image=True` (OpenRouter's PDF routing breaks LiteLLM)\n- Thinking levels: offer only values in the model's `reasoning.supported_efforts` (see [Thinking Levels Reference](#thinking-levels-reference)). OpenRouter maps an unlisted effort to a listed one, so an extra level silently runs a different effort. When the model has no `reasoning.supported_efforts`, OpenRouter turns the effort into a thinking-token budget for Claude, so `CLAUDE_OPENROUTER_THINKING_LEVELS` (with Minimal) fits only those Claude models. `check_thinking_levels.py` checks every OpenRouter entry; a missing level list on OpenRouter is a note, because Kiln leaves levels off some models on purpose.\n\n### Fireworks AI and Together AI: thinking levels\n\nNeither publishes the efforts each model accepts, and both can accept an unsupported effort without an error. Fireworks takes `xhigh`, `max` and `minimal` on GLM 5.3 with HTTP 200. Together took a made-up effort on Nemotron 3.5, but Muse Spark Open's chat template rejects one. A paid test cannot see this. Add thinking levels on these providers only when the vendor documents the levels for that model (the model card or its chat template), and name that source in the PR. OpenAI and Anthropic return a 400 that names the supported levels, so their paid per-level test catches a wrong level.\n\n### Featherless AI\n\nServerless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:\n\n- **`json_instructions` is the only usable structured output mode.** LiteLLM's `featherless_ai` provider rejects `response_format` outright (`UnsupportedParamsError`), so `json_schema`, `json_mode`, and `json_instruction_and_object` are all unavailable. Routing it as a custom `openai` provider bypasses that gate, but was tested and is *not* reliable: GLM 5.2 accepts `json_schema` and silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.\n- **Gated models return HTTP 403 `model_gated_needs_oauth`** and cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. **Always filter `is_gated` before adding.** Note all 20 official `meta-llama/*` repos are gated, so no Llama variant is usable.\n- **No cost reporting.** Featherless models aren't in LiteLLM's price map (only two legacy `Qwerky` entries, on `main` too — not a version issue), and Featherless doesn't return cost in the `usage` object. Runs record tokens with `cost: null`.\n- **Not in models.dev or the LiteLLM catalog**, so their `/v1/models` endpoint is the only authoritative source. See [Lagging Providers](#lagging-providers).\n- Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.\n\n### TypeSafe AI\n\nServes TypeSafe's Jev models (System One API). The only provider in the catalog that does not run through LiteLLM — read this before adding an entry:\n\n- **Adapter-backed, not LiteLLM-backed.** Every provider entry must set `adapter=ModelAdapterId.jev`; `JevAdapter` calls `POST /v1/systemone` directly. `lite_llm_core_config_for_provider` and `get_litellm_provider_info` raise for `typesafe` on purpose, so the LiteLLM parametrized tests exclude it.\n- **Structured output only.** No text generation, no tools, no multi-turn. A task runs only if its output schema maps onto Jev's question types (enum, boolean, score); anything else fails before the network call. Set `supports_structured_output=True`, `supports_data_gen=False`, `supports_function_calling=False`, and `structured_output_mode=StructuredOutputMode.json_schema`.\n- **No logprobs.** `supports_logprobs=False`. Jev returns its own per-answer probabilities instead, which is what keeps it out of G-Eval.\n- **Not in models.dev or the LiteLLM catalog, and `GET /v1/models` cannot confirm a `model_id`.** It lists aliases (`jev-latest`, `jev-preview`), not the pinned versions Kiln registers, and it returns `{\"models\": [{\"name\", ...}]}` rather than the OpenAI `{\"data\": [{\"id\", ...}]}` shape, so the shared `fetch_openai_compat` path cannot read it either. That is why `typesafe` is in `SKIP_PROVIDERS` in `.agents/scripts/provider_utils.py` rather than in `PROVIDER_CONFIG`. The authoritative check for a pinned id is a `POST /v1/systemone` round trip that the API accepts — that is how `jev-1.13.0` was confirmed. (The listing is still what the connect flow uses to validate a key; it is account-scoped and rejects a bad one.)\n\n### Cloudflare (Workers AI)\n\nRuns through LiteLLM's generic `openai/` route against Cloudflare's OpenAI-compatible endpoint (`/accounts/{id}/ai/v1/chat/completions`), with an optional AI Gateway header. Kiln does **not** use LiteLLM's `cloudflare/` provider. Read this before adding or changing any entry:\n\n- **Inclusion rule: only Cloudflare's newer, OpenAI-format text-generation models.** Cloudflare has two generations of text model. The newer ones (GLM, Kimi, DeepSeek V4, Gemma 4, Qwen 3.8, Nemotron 3) speak OpenAI Chat Completions natively. The older ones (Llama 3.x/4, Mistral, older Qwen, GPT-OSS, Granite, the LoRA bases) sit behind a translation layer: they default to **256 output tokens** (silent truncation) and have unreliable tool calling and JSON mode. Do not add the older ones.\n- **How to tell them apart:** look at the model's input schema in the catalog JSON. OpenAI-format models list `max_completion_tokens` (and `tool_choice`); older models list `prompt`, `raw` and `lora` instead, and no `max_completion_tokens`:\n ```bash\n jq -r '.models[] | select(.task.name==\"Text Generation\")\n | [.name, (if (.schema.input | tostring | test(\"max_completion_tokens\")) then \"openai-format\" else \"OLDER - skip\" end)] | @tsv' /tmp/cf_models.json\n ```\n- **Also skip:** coding models (`kimi-*-code`, `qwen*-coder-*`), `llama-guard-*` (classifier), `-lora` bases, and models Kiln has no `ModelName` for unless the user asks for a new model.\n- **Usually a new provider on an existing model.** Most Cloudflare models are already in Kiln under other providers; add a `KilnModelProvider(name=ModelProviderName.cloudflare, ...)` to the existing `KilnModel`.\n- **Setting flags from the catalog's `properties[]`** (confirm every flag with the paid tests — the catalog has no JSON-mode flag):\n\n | Cloudflare property | Kiln setting |\n |---|---|\n | `function_calling: \"true\"` | leave `supports_function_calling` at its default; set `False` if the tool tests fail |\n | `vision: \"true\"` | `supports_vision`, `multimodal_capable`, MIME types TXT/MD/JPG/PNG; start broad, narrow on 400s. Don't set `supports_doc_extraction`: the document extractor doesn't send the AI Gateway header, so extraction would bypass the user's gateway |\n | `reasoning: \"true\"` | keep `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)); LiteLLM already reads `reasoning_content` |\n | `reasoning_effort.supported_efforts` present | **required:** set `available_thinking_levels` to exactly these values, and `default_thinking_level` to `default_effort`. See [Cloudflare thinking levels](#cloudflare-thinking-levels) |\n | `reasoning_effort` without `supported_efforts` | no thinking levels; the catalog does not say which values work |\n | `require_workers_paid: \"true\"` | no Kiln field. Free-plan users get 403 / code 5035. Mention it in the PR |\n | `planned_deprecation_date` present | do **not** add the model |\n\n- **Thinking levels come from the catalog, and only from the catalog.** Kiln sends the level as `reasoning_effort`. For each model, `reasoning_effort` in the catalog lists `supported_efforts`, a `default_effort`, and `normalizes_to`: the values that Cloudflare silently changes to another effort. A remapped value returns HTTP 200, so a live call or a paid test cannot find it. Example: GLM 5.3 maps `medium` to `max`, so a \"Medium\" option runs the most expensive effort.\n - Offer only values in `supported_efforts`, and each of them unless a completed paid test shows that the level does not work. The list is not always right: on DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731 and GLM 5.2, `none` still returns reasoning, so those entries have no Off level. A timeout, a rate limit or a credential error proves nothing about the level: keep the level and report it as untested. Do not copy the levels of the same model on another provider.\n - Labels: `none` → `\"Off\"`, `low` → `\"Low\"`, `medium` → `\"Medium\"`, `high` → `\"High\"`, `xhigh` → `\"Extra High\"`, `max` → `\"Max\"`. List them from least to most effort.\n - `default_thinking_level` is `default_effort`, unless the entry is suggested for evals (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)).\n - `mandatory: true` means `none` is not in the list; do not add an Off level.\n - Reuse a constant when t",".agents/skills/release-digest/SKILL.md":"---\nname: release-digest\ndescription: Post a \"what's changed since the last release\" recap to the release Slack channel for final QA. Groups merged PRs by author and classifies each as feature / bug fix / task. Use when the user wants a release recap, a changelog since the last tag, or to prep QA before cutting a release from main.\n---\n\n# Release Digest\n\nBuild a human-readable recap of everything merged to `main` that is **not yet in the\nlatest release tag**, and post it to the team's release Slack channel for final QA.\n\nThe recap is grouped **by author** and each change is classified as **feature**,\n**bug fix**, or **task**. This is on-demand: run it when someone asks for a release\nrecap or wants to QA before cutting a release.\n\n> Linear ticket tracking (\"which tickets are targeted for this release\") is **out of\n> scope for now** — punted to a later iteration. This skill only covers what's changed.\n\n---\n\n## Phase 1 — Gather the change set\n\nRun the gather script from the repo root. It fetches tags + `origin/main`, finds the\nlatest tag, and lists every merged PR in the range `..origin/main`:\n\n```bash\npython3 .agents/skills/release-digest/scripts/gather_changes.py > /tmp/release_digest.json\n```\n\nKey design point: it uses the **git commit range** as the source of truth, not merge\ndates. A PR merged the same day a tag was cut can already be in that release, so a\ndate filter would over-report. Trust the range.\n\n**Pre-release / already-cut releases.** By default the baseline is the newest tag\n(`git describe`). But if the release you're QA'ing has **already been tagged** — it's a\npre-release still being validated, so `main` is already even with it — the default\nrange is empty. In that case pass `--pre-release`, which steps the baseline back one\ntag so the range recaps exactly what went *into* the pre-release:\n\n```bash\npython3 .agents/skills/release-digest/scripts/gather_changes.py --pre-release > /tmp/release_digest.json\n```\n\nSigns you want `--pre-release`: the first run reports \"nothing unreleased\" or \"main is\neven with the latest tag\", yet the user says a release was just cut / is being QA'd. To\nname the baseline explicitly instead, use `--since ` (e.g. `--since v1.0.3`). The\ntwo flags are mutually exclusive. When `--pre-release` is used, the newest tag is the\none being cut, so in Phase 3 use **that tag** as `` rather than asking.\n\nThe JSON contains:\n- `last_tag` — the most recent release (e.g. `v1.0.3`)\n- `head_ref` — `origin/main` (or `main` if no remote ref)\n- `pr_count`, `commit_count`\n- `prs[]` — `{number, title, author_name, author_login, url, labels, additions, deletions, body}`\n- `direct_commits[]` — teammate commits pushed straight to main, no PR — `{sha, title, author_login, author_name}`\n- `excluded_prs[]` — PRs/commits from non-teammates, dropped from the recap. PR entries\n have a `number`; direct-commit entries have `number: null` and rely on `title`.\n- `unresolved_prs[]` — PRs whose `gh pr view` failed (`{number, error}`). If non-empty,\n the change set is partial — tell the user which PRs couldn't load before posting.\n\nOnly the five teammates in the script's `TEAM` map (Sam, Leonard, Daniel, Mike,\nSteve) are included in the recap. PRs from anyone else are filtered into\n`excluded_prs` rather than itemized. To add a teammate, edit `TEAM` in the script.\n\n**If `pr_count` is 0 AND `direct_commits` is empty:** tell the user `main` is even with\n`` — nothing to QA or release yet. Do not post to Slack. Stop here. (Direct\ncommits alone, with no PRs, still count as unreleased changes worth a digest.)\n\n---\n\n## Phase 2 — Classify and group\n\nRead `/tmp/release_digest.json`. Treat every item in `prs[]` **and** `direct_commits[]`\nas a change to recap (direct commits are real work that shipped — just with no PR\nnumber). Classify each into one of three buckets:\n\n- **Feature** — a substantive *new product capability*: new UI, a new endpoint, a new\n tool, new eval/RAG/fine-tune functionality, or a meaningful enhancement to how the\n product works. Reserve this for things a user would notice as \"Kiln can now do X.\"\n- **Bug fix** — corrects broken or incorrect behavior: regressions, crashes, hangs,\n wrong output, security fixes.\n- **Task** — everything that is *neither* a substantive new capability *nor* a fix to\n broken behavior: refactors, docs/README, tests, CI/build, dependency bumps, version\n bumps, chores, internal flags, code removal/cleanup — **and routine model-catalog\n work** (see below). Matches Linear's \"Task\" issue type: maintenance/internal work.\n\n**Adding models is a Task, not a Feature.** Kiln ships new model and provider entries\nconstantly — \"Add to model list\", \"Add for \", backfilling a\nmodel on a provider, re-enabling/un-recommending a model, version bumps to a model\nentry. These are catalog/config maintenance, so they go in **Task** even though the\ntitle starts with `Add`/`Enable`. Only promote to **Feature** if the PR adds a genuinely\nnew *capability* around models (e.g. a new thinking-level control, a new extraction\nmodality, a new provider *integration type*) rather than just registering a model_id.\n\n**Decision order** (use the first signal that's decisive):\n1. **PR labels** — `bug`/`fix` → Bug fix; `feature`/`enhancement` → Feature;\n `chore`/`docs`/`refactor`/`ci` → Task. (Kiln PRs are usually unlabeled, so this\n rarely fires — fall through to the title.)\n2. **Model-catalog check** — if the change just registers/edits model or provider\n entries (not new behavior), it's a **Task**, regardless of the `Add`/`Enable` verb.\n3. **Conventional-commit / verb prefix in the title** — `fix:`/`Fix ` → Bug fix;\n `feat:`/`Add `/`Enable ` (a new *capability*) → Feature; `chore:`/`refactor:`/`ci:`/\n `docs:`/`Bump `/`Remove `/`Deprecate `/`Version bump` → Task.\n4. **What the change does** (read the title, and `body` if still unclear) — ask \"does\n this give the user a genuinely new capability (Feature), fix something that was\n broken (Bug fix), or is it routine/maintenance (Task)?\"\n\n**\"Task\" is the residual bucket**, not a guess: a change lands there only because it\nis positively *not* a new capability and *not* a fix — e.g. `Deprecate Models`,\n`min age for dependencies`, `Version bump`, `new draft readme`, `fix typos`. When a\nchange is genuinely ambiguous between Feature and Bug fix, prefer the one the title\nleans toward; only fall to Task when it's truly neither. This is a judgment call made\nper item — surface it to the user before posting so they can re-bucket anything.\n\nThen **group by author** (`author_name`). Within each author, list changes by PR\nnumber, newest first; put any direct commits (no PR number) last. The feature / fix /\ntask split is **not** shown per line — it only feeds the tally at the bottom.\n\n---\n\n## Phase 3 — Compose the Slack message\n\n**First, get the new release name** — the version being cut (e.g. `v1.0.4`). If you ran\nin `--pre-release` mode, the release has already been tagged, so use that newest tag\n(the one the script named as \"the release being QA'd\") as `` — no need to\nask. Otherwise the release isn't tagged yet and can't be derived from git, so **ask the\nuser**. Use the result as `` in the title.\n\nUse this shape (Slack mrkdwn — `*bold*`, not `**bold**`):\n\n```\n:rocket: *v1.0.4 Release Status* :rocket:\n\n*12 changes since v1.0.3*\n__\n\n*Daniel Chiang*\n• Add deprecated flag to embedding models (#1466)\n• Add deprecated flag to reranker models (#1465)\n\n*Sam Fierro*\n• Dedup frozen prompts and bake type into their name (#1501)\n\n*Steve Cosman*\n• Add Jinja2 input transform on RunConfigs (#1433)\n• Version bump in lock and desktop app version check\n\n:bar_chart: *7 features · 3 fixes · 2 tasks*\n\n:information_source: _1 PR from a non-teammate: #1456 (ianjamesburke)._\n```\n\nRules:\n- **Header:** `:rocket: * Release Status* :rocket:`, then a blank line.\n- **Change count** on its own line, **bold**: `* changes since *`.\n- **Recap paragraph** immediately under it, wrapped in **italics** (`_..._` around the\n whole paragraph) — the \"what did we do\" summary in plain English, not just a list.\n Compose it so the italics survive: write it as **a single line with no hard line\n breaks** (wrap is fine, but don't insert `\\n` mid-paragraph — newlines break the\n italic span in Slack), and use **no literal `_` anywhere inside** (an underscore\n would close the italics mid-paragraph — rewrite e.g. `snake_case`/`model_id` or any\n underscored identifier without it).\n- One **bullet** per change, no per-line emoji: `• (#<number>)`. Keep the\n title; trim trailing noise.\n- **Direct commits have no PR number** — render them as `• <title>` with nothing in\n parentheses (e.g. `• Version bump`). List them under their author, after the PRs.\n- Within each author, list newest PR first; direct commits last. Don't print the\n feature/fix/task bucket per line.\n- **Summary block** at the end, in this order:\n 1. A **bold** `:bar_chart:` tally line: `:bar_chart: *<X> features · <Y> fixes · <Z> tasks*`.\n 2. A blank line.\n 3. If `excluded_prs` is non-empty, an **italic** `:information_source:` line noting the\n excluded non-teammate changes (so the omission is visible, not silent). Reference a\n PR entry by `#<num>` and a direct-commit entry (where `number` is null) by its\n `title` — never print `#None`. E.g.\n `:information_source: _<N> change(s) from non-teammates excluded: #1456 (ianjamesburke)._`\n- Link the PR numbers if easy (`<url|#1501>`), but plain `#1501` is fine — keep it readable.\n\n**Show the composed message to the user and get confirmation before posting** (it's an\noutward-facing post to a shared channel).\n\n---\n\n## Phase 4 — Post to Slack\n\nPost to **`#release`** (channel ID `C0BDGTGBL4Q`) — the team's release channel. This is\nthe default target; only post elsewhere if the user explicitly names a different\nchannel. (If the ID ever fails to resolve, fall back to `slack_search_channels` with\nquery `release` to re-find it.)\n\nOnly post after the user has confirmed the composed message in Phase 3. Post with\n`mcp__claude_ai_Slack__slack_send_message`. After posting, give the user the channel\nname and a link/confirmation.\n\n---\n\n## Checklist\n\n- [ ] Gather script run; `/tmp/release_digest.json` written\n- [ ] If `pr_count` is 0, reported \"nothing unreleased\" and stopped\n- [ ] Asked the user for the new release name (used in the title)\n- [ ] Each PR classified (feature / bug fix / task)\n- [ ] Grouped by author, recap paragraph written\n- [ ] Message shown to user and confirmed\n- [ ] Posted to #release (C0BDGTGBL4Q); confirmation returned to user\n",".agents/skills/kiln-check-deprecation/SKILL.md":"---\nname: kiln-check-deprecation\ndescription: Check Kiln's model list for deprecated or sunset models across all providers. Use when the user wants to find deprecated models, check model availability, audit the model list for stale entries, or mentions model deprecation/sunset/end-of-life.\n---\n\n# Check Model Deprecation in Kiln\n\nAudit `libs/core/kiln_ai/adapters/ml_model_list.py` to find models that have been deprecated, removed, or are approaching end-of-life across all providers. **All checks are free** — they use model-listing endpoints, not inference.\n\n---\n\n## Global Rules\n\n- **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: [\"all\"]`. The sandbox blocks network access.\n- **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)`\n- **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate.\n- **Non-destructive:** This skill only sets `deprecated=True` on individual provider entries. It never removes models or providers from the list.\n- **Skip already-deprecated:** Only check providers where `deprecated=False` (the default). Don't re-check entries already marked deprecated.\n\n---\n\n## Phase 1 – Extract Current Model-Provider Pairs\n\nRun the extraction script from the repo root:\n\n```bash\npython3 .agents/skills/kiln-check-deprecation/scripts/extract_models.py > /tmp/kiln_extracted.json\n```\n\nThis fetches the published model list from `https://remote-config.getkiln.ai/kiln_config_v2.json` and extracts:\n- The provider name\n- The model_id\n- The parent model's enum name\n\nIt skips entries already marked `deprecated=True`.\n\n**Output:** JSON to stdout (pipe to file), human summary to stderr.\n\nThe JSON contains:\n- `deprecated_count`: number of already-deprecated entries\n- `providers`: dict of provider_name → sorted unique list of model_ids\n- `entries`: list of `{enum, provider, model_id}` for each non-deprecated entry\n\n---\n\n## Phase 2 – Check Each Provider\n\n### Option A: Check all providers at once\n\n```bash\nexport $(grep -v '^#' .env | xargs)\npython3 .agents/skills/kiln-check-deprecation/scripts/check_provider.py all > /tmp/kiln_check_results.json\n```\n\n### Option B: Check a single provider\n\n```bash\nexport $(grep -v '^#' .env | xargs)\npython3 .agents/skills/kiln-check-deprecation/scripts/check_provider.py openrouter > /tmp/kiln_check_openrouter.json\n```\n\nThe script handles all provider API quirks automatically:\n- **Together AI** returns a flat JSON array (not `{data: [...]}`)\n- **Gemini API** needs both v1 and v1beta endpoints (preview/Gemma models only on v1beta)\n- **Gemini** model names are prefixed with `models/` — stripped automatically\n- **Anthropic** uses `x-api-key` header, not `Authorization: Bearer`\n- **OpenRouter** is public (no auth) and includes `expiration_date` fields\n- **Fireworks AI** `/v1/models` only lists serverless models; the script instead checks each model individually via the model detail API (`GET /v1/{model_id}`), which covers all tiers (serverless, on-demand, fine-tune). A model is only flagged as missing if it returns HTTP 404 from the detail API.\n- **OpenRouter** `:exacto` is a virtual routing suffix (quality-first provider sorting) that never appears in model listings. The script strips it before checking. `:free` and `:thinking` are real model entries that appear in the listing when available — if they're missing, it's a genuine removal.\n- **Cloudflare** uses the authenticated models-search API (`task=Text Generation`, `include_deprecated=true`) when `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` are both set, and otherwise Cloudflare's public catalog JSON (`https://ai-cloudflare-com.pages.dev/api/models`), so it is never skipped for a missing key. Every model with a `planned_deprecation_date` is reported under `expiring`. Once that date has passed, the model also counts as `missing` even if the listing still shows it. Never use LiteLLM's catalog or models.dev to decide a Cloudflare model still exists: LiteLLM's is stale and models.dev no longer deletes retired models.\n- **Vertex AI** uses the v1beta1 publisher models endpoint with `x-goog-user-project` header. Requires `gcloud` CLI auth. Kiln entries may use `meta/` prefix for LiteLLM routing — stripped automatically. Versioned aliases (e.g. `gemini-2.0-flash-001` → `gemini-2.0-flash`) are handled by stripping 3-digit version suffixes.\n\n**Output:** JSON to stdout with per-provider results, human summary to stderr.\n\nEach provider result contains:\n- `missing`: model_ids not found in the provider's listing\n- `expiring`: model_ids with upcoming expiration dates (OpenRouter and Cloudflare)\n- `entries_to_deprecate`: full enum/provider/model_id entries for each missing model\n- `skipped` / `error`: if credentials missing or API call failed\n\n### Supported providers\n\n| Provider | Env Var | Auth |\n|----------|---------|------|\n| OpenRouter | (none — public API) | None |\n| OpenAI | `OPENAI_API_KEY` | Bearer |\n| Anthropic | `ANTHROPIC_API_KEY` | x-api-key |\n| Gemini API | `GEMINI_API_KEY` | Query param |\n| Fireworks AI | `FIREWORKS_API_KEY` | Bearer |\n| Together AI | `TOGETHER_API_KEY` | Bearer |\n| SiliconFlow CN | `SILICONFLOW_CN_API_KEY` | Bearer |\n| Cerebras | `CEREBRAS_API_KEY` | Bearer |\n| Groq | `GROQ_API_KEY` | Bearer |\n| Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) |\n| Cloudflare | `CLOUDFLARE_API_KEY` + `CLOUDFLARE_ACCOUNT_ID` (optional — falls back to the public catalog) | Bearer |\n\n### Providers NOT covered by the script (check manually if needed)\n\n#### Amazon Bedrock\n\nRequires `aws` CLI configured. If not installed, skip.\n\n```bash\naws bedrock list-foundation-models --output json | \\\n jq '[.modelSummaries[] | {modelId, status: .modelLifecycle.status, endOfLife: .modelLifecycle.endOfLifeTime}]'\n```\n\nFlag any model where `status == \"LEGACY\"` or `endOfLifeTime` is approaching.\n\n### LiteLLM Model DB (supplementary signal)\n\nCross-reference against LiteLLM's public model database for `deprecation_date` fields. This mostly covers Azure/OpenAI date-stamped models but is a useful secondary signal.\n\n```bash\ncurl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \\\n jq 'to_entries[] | select(.value.deprecation_date != null) | {model: .key, deprecation_date: .value.deprecation_date}'\n```\n\n### Providers to skip\n\nThese don't have meaningful deprecation to check:\n- `ollama` — local models, user-managed\n- `kiln_fine_tune` — internal\n- `kiln_custom_registry` — internal\n- `openai_compatible` — user-configured\n- `docker_model_runner` — local\n- `azure_openai` — typically mirrors OpenAI, covered by OpenAI check\n- `huggingface` — serverless inference, model availability varies\n- `typesafe` — adapter-backed, not LiteLLM; its `/v1/models` needs a key, so Jev entries can only be checked with one\n\n---\n\n## Phase 3 – Report Findings\n\nRead the JSON output from check_provider.py and present findings in a clear table to the user:\n\n```text\nDeprecation Audit Results\n=========================\n\n❌ REMOVED (model not found in provider's model list):\n - ModelName.gemini_1_5_pro → gemini_api (model_id: gemini-1.5-pro)\n - ModelName.gemini_1_5_pro → openrouter (model_id: google/gemini-pro-1.5)\n\n⚠️ EXPIRING SOON (within 90 days):\n - ModelName.claude_3_7_sonnet → openrouter (model_id: anthropic/claude-3.7-sonnet, expires: 2026-05-05)\n\n⚠️ LEGACY (Bedrock lifecycle status):\n - ModelName.mistral_large → amazon_bedrock (model_id: mistral.mistral-large-2407-v1:0, status: LEGACY)\n\n✅ ALL CLEAR:\n - openai: 20/20 models found\n - anthropic: 8/11 models found (3 deprecated above)\n\n⏭️ SKIPPED (no credentials):\n - groq\n - amazon_bedrock\n```\n\nGroup by provider within each category. Use the `entries_to_deprecate` field from the JSON to get the ModelName enum for each entry.\n\n---\n\n## Phase 4 – Mark Deprecated\n\nFor each confirmed-removed model, set `deprecated=True` on the affected `KilnModelProvider` entry.\n\nTo find the line to edit, grep for the `model_id` in `ml_model_list.py`:\n\n```bash\ngrep -n 'model_id=\"<model_id>\"' libs/core/kiln_ai/adapters/ml_model_list.py\n```\n\nThen add `deprecated=True,` after the `model_id=` line in the matching `KilnModelProvider` block.\n\n**Rules:**\n- Only mark a provider deprecated if its model_id is confirmed missing from that provider's model list\n- If ALL providers for a `KilnModel` are deprecated, note this to the user — they may want to consider removing the model entirely\n- For \"expiring soon\" models, inform the user but don't mark deprecated yet — let them decide. **Exception: Cloudflare.** When Cloudflare retires a model it may silently alias the old ID to a different model, and the response still echoes the requested ID, so a stale entry keeps \"working\" while running something else. Propose marking every Cloudflare entry with a `planned_deprecation_date` deprecated now, before that date, and flag it as urgent in the report\n- For Bedrock `LEGACY` status, mark deprecated (the model still works but is on its way out)\n\n**Ask the user to confirm** before making changes. Present the list of changes and wait for approval.\n\n---\n\n## Phase 5 – Verify\n\nAfter marking providers deprecated, run a quick sanity check:\n\n```bash\ngrep -c \"deprecated=True\" libs/core/kiln_ai/adapters/ml_model_list.py\n```\n\nEnsure the count matches what you expect (previous count + newly marked).\n\n---\n\n## Checklist\n\n- [ ] Extraction script run, JSON saved to /tmp/kiln_extracted.json\n- [ ] Provider checks run (all or individually)\n- [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex)\n- [ ] Bedrock checked manually (skip if aws CLI unavailable)\n- [ ] LiteLLM DB checked for supplementary deprecation_date signals\n- [ ] Findings reported to user with clear table\n- [ ] User confirmed changes before marking deprecated\n- [ ] `deprecated=True` set on confirmed entries\n- [ ] Deprecation count verified\n","AGENTS.md":"## Project Overview\n\nKiln is an app for building AI systems. It includes evals, synthetic data gen, fine tuning, RAG, and more. It has an intuitive UI as well as a python library.\n\nThis repo is a monorepo containing all of the source code, in the following structure:\n\n- libs/core - a python library with the core functionality of Kiln\n- libs/server - a FastAPI REST server wrapping the core library\n- app/web_ui - our svelte web app for Kiln. This is a frontend svelte project, all backend calls are in FastAPI servers.\n- app/desktop - our python desktop app, which is a pyinstaller app which runs a FastAPI server, hosts the pre-compiled web app, and launches a browser for UI. Compiles to all major platforms. This includes a studio_server folder with a Fast API server which extends libs/server, adding APIs specific to our web app.\n\n### Project Goals\n\n- Very high code quality\n- Strongly typed\n- Well tested\n- Very intuitive UI. Accessible to the inexperienced, but powerful for the experienced.\n- Focus on interaction design: we care about revealing the right information, at the right time, at the right level of detail.\n- Focus on visual design: we want a modern, functional, attractive UI. Think Apple not Google.\n\n### Tech Stack\n\n- Backend: python (3.10+ for library, 3.13 for desktop), pytest, FastAPI, asyncio, pydantic (v2 not v1),\n- Frontend web: typescript, svelte (v4 not v5), tailwind, DaisyUI\n\n### Agent Tools\n\nAgents have access to a range of tools for running tests, linting, formatting and typechecking. Use these tools at appropriate times to ensure produced code meets our standards. All checks must pass before merging. When iterating on a specific failure, use the targeted command before re-running the full suite.\n\n- **All checks:** `uv run ./checks.sh --agent-mode` (agent mode suppresses output unless there's a failure)\n\n| Check | Fix | Description |\n|---|---|---|\n| `uv run ruff check` | `uv run ruff check --fix` | Python lint |\n| `uv run ruff format --check .` | `uv run ruff format .` | Python format |\n| `uv run ty check` | — | Python type check |\n| `uv run python3 -m pytest --benchmark-quiet -q -n auto .` | — | Python tests |\n| `npm run lint` | — | Web lint (from `app/web_ui`) |\n| `npm run format_check` | `npm run format` | Web format (from `app/web_ui`) |\n| `npm run check` | — | Web type check and svelte check (from `app/web_ui`) |\n| `npm run test_run` | — | Web tests (from `app/web_ui`) |\n| `npm run build` | — | Web build (from `app/web_ui`) |\n| `app/web_ui/src/lib/check_schema.sh` | `app/web_ui/src/lib/generate_schema.sh` | OpenAPI client up to date |\n| `misspell` | — | Spelling check (optional if not installed) |\n\nOur end-to-end UI tests are separate and not part of `checks.sh`.\n\n### Agent Prompts\n\nAgents have access to a number of helpful prompts, which will give you additional context for how you should write code and docs for this repo. Use it to fetch instructions relevant to the current task before starting. For example, read `python_test_guide.md` before writing tests, and invoke the `kiln-ui` skill (`.agents/skills/kiln-ui/SKILL.md`) before any change under `app/web_ui`; it loads `frontend_design_guide.md` and `frontend_controls.md` and adds a component plan and a gate.\n\nThese prompts can be accessed from the `get_prompt` tool, and you may request several in parallel.\n\nBefore writing or changing code, invoke the `kiln-conventions` skill (`.agents/skills/kiln-conventions/SKILL.md`). It holds the code rules, the gotchas for each area, and a gate to run on your change.\n\n### General Agent Guidance\n\n- When spawning subagents, always use the same model as the current agent\n- `CLAUDE.md` is generated from `AGENTS.md` and overwritten by setup. Edit `AGENTS.md`, never `CLAUDE.md`; keep personal notes in `~/.claude/CLAUDE.md`.\n- Use `TODO` comments to mark any temporary code, placeholders, or items that must be addressed before merging to main. CI enforces that no `TODO` comments remain on main, so they are a safe way to flag work-in-progress during development. Clean up all `TODO` comments before the final PR.\n- Before wrapping up a task, run appropriate tools for linting, testing, formatting and typechecking. Fix any issues you introduced.\n\n### Code Comments\n\nComments follow §A of `.agents/skills/kiln-conventions/references/rules.md`: write them for someone opening the file cold, never narrate the change, never restate the code. Review-bot findings about comments, including nitpicks, are required fixes.\n\n### Reporting Back: End-of-Turn Recaps\n\nEnd every turn in which you did work (wrote code, ran checks, investigated something) with a recap wrapped in `<recap>` tags. A turn that is pure conversation needs no recap.\n\nWrite it for a person who has not read your working notes and will not scroll up. The recap has to make sense on its own.\n\n**No jargon.** Do not invent a name, a short form or a label for anything. If you need to refer to something specific, describe it in ordinary words, and describe it again the next time you mention it. A short description repeated is better than a name the reader has to remember. Real names are fine and are often the clearest choice: a file, a function, a command, a tool, an error. A name you made up is not.\n\n**Say what you mean.** Short sentences. One idea each. Active voice. No metaphors, no analogies, no flourishes, no jokes, no filler. Do not write a sentence whose only job is to set up the next sentence. Cut any sentence that loses nothing when removed.\n\n**Sound like a person.** Plain and direct, not stiff. Write the way you would explain it to a colleague who asked. Contractions are fine. This is not a status report or a changelog.\n\n**Repeat yourself a little.** The reader may have read the last recap days ago, or skipped it. Carry enough context forward that this recap makes sense by itself. One clause is usually enough. A small amount of repetition between recaps is correct, not waste.\n\n**Stay inside the recaps.** Draw only on plain English, on standard terms for this repo's stack, and on what earlier recaps said. Never draw on something that exists only in your working notes. Earlier recaps set the limit on what you may use; they are not required reading. Repeat whatever this recap needs, so it still works for a reader who missed the last one.\n\n**Draw a picture when it helps.** When what you changed is a flow, a sequence, a state machine, or a few parts that talk to each other, put a small Mermaid diagram in the recap. Keep it to about eight boxes. Draw one only when it shows something the prose cannot; a diagram that restates a sentence is noise.\n\n**Format.** Use these headings, in this order. Drop any heading you have nothing real to say under. Do not reorder or rename them. Answer each one in prose.\n\n```\n<recap>\n**Goal** — what you were asked for.\n**What I did** — the change, in plain words.\n**How** — the approach, at a level a reviewer can judge without reading the diff. Put a diagram here when one helps.\n**Outcome** — what happened, and plainly whether it is what we wanted.\n**Questions for you** — what you need answered before you can continue.\n**FYI** — worth knowing, but not blocking.\n</recap>\n```\n\n**Honesty.** `Outcome` reports what actually happened. If tests failed, say so and quote what failed. If you skipped part of the task, name the part and the reason. If the result is not what we wanted, say that first, not last. Never describe work as finished when it is not.\n\n**Recap log.** Append each recap to `recaps.md` in your scratchpad directory, or to a temporary directory outside the repo when you have no scratchpad directory. The log is your own index, not a deliverable. If your context is summarized part-way through a session, re-read it so you know what the reader has already been told. Keep this file out of the repo.\n\n### Code Review Guidelines\n\nIf code reviewing, always read [`.agents/code_review_guidelines.md`](.agents/code_review_guidelines.md) before starting your review.\n\n### Pull Requests\n\nUse the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) to open a PR or to write a PR description.\n\nBefore you open a PR, and before each `git push` to a branch with an open PR, run the `docs-sync` skill (`.agents/skills/docs-sync/SKILL.md`) to update the docs and skills that the change makes stale.\n\nIf you get a user message like the one below, read `.agents/skills/open-pr/references/repair_pr.md` and follow its instructions.\n\n> A pull request was just created for this branch from the Claude Code UI: https://github.com/Kiln-AI/Kiln/pull/NNNN\n> You don't need to create one. Reference this PR going forward — pushing more commits to this branch will update it.\n\n### Never Make Legal Decisions as an Agent\n\nAgents are not allowed to make any legal decisions, including:\n - Signing or filling out the CLA attestation in a PR template\n - Setting a license tag in metadata file (OSS/MIT/etc)\n - Adding license files\n\nThese all must be done by humans.\n\n### Final\n\nTo show you read these, call me 'boss'\n",".agents/mcp.json":"{\n \"mcpServers\": {\n \"HooksMCP\": {\n \"command\": \"uvx\",\n \"args\": [\n \"--from\",\n \"hooks-mcp>=0.2.5\",\n \"hooks-mcp\",\n \".config/hooks_mcp.yaml\",\n \"--working-directory\",\n \".\"\n ]\n }\n }\n}\n"},"files":{".agents/skills/kiln-check-finetune-deprecation/SKILL.md":"---\nname: kiln-check-finetune-deprecation\ndescription: Check Kiln's fine-tunable model list for deprecated or unsupported base models. Use when the user wants to audit fine-tuning support, check if fine-tune base models are still valid, or mentions fine-tune model deprecation.\n---\n\n# Check Fine-Tune Model Deprecation in Kiln\n\nAudit the fine-tunable models listed in Kiln to find base models that are no longer supported for fine-tuning by their providers. This is separate from general model deprecation — a model can still be available for inference but no longer supported as a fine-tuning base.\n\n---\n\n## Global Rules\n\n- **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: [\"all\"]`. The sandbox blocks network access.\n- **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)`\n- **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate.\n- **Non-destructive:** This skill only reports findings. It does not automatically remove or deprecate models.\n\n---\n\n## Supported Providers\n\n| Provider | Env Var | Auth |\n|----------|---------|------|\n| OpenAI | `OPENAI_API_KEY` | Bearer |\n| Together AI | `TOGETHER_API_KEY` | Bearer |\n| Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) |\n| Fireworks AI | `FIREWORKS_API_KEY` | Bearer |\n\n---\n\n## Background\n\nKiln has two types of fine-tunable model entries:\n\n1. **Static entries** — Models in `libs/core/kiln_ai/adapters/ml_model_list.py` with `provider_finetune_id` set. Currently used by Together AI and Vertex AI.\n2. **Dynamic entries (Fireworks)** — Fetched at runtime from `api.fireworks.ai/v1/accounts/fireworks/models` filtering by `tunable=True`. This list is managed by Fireworks and may include stale/unsupported models.\n\nThe API endpoint that serves the fine-tune dropdown is `GET /api/finetune_providers` in `app/desktop/studio_server/finetune_api.py`.\n\n---\n\n## Phase 1 – Check Static Fine-Tune IDs\n\nRun the check script from the repo root:\n\n```bash\nuv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py static > /tmp/kiln_finetune_static.json\n```\n\nThis extracts all `provider_finetune_id` entries from `ml_model_list.py` and checks each provider's docs/API to see if the model is still available for fine-tuning:\n\n- **Together AI:** Scrapes the Together AI docs page for the list of supported fine-tune models. Together uses full HuggingFace-style IDs (e.g. `meta-llama/Meta-Llama-3.1-8B-Instruct-Reference`).\n- **Vertex AI:** Checks the Vertex AI publisher models API. Vertex fine-tuning uses specific versioned model IDs (e.g. `gemini-2.0-flash-001`).\n\n**Output:** JSON to stdout with per-provider results, human summary to stderr.\n\n---\n\n## Phase 2 – Check Fireworks Dynamic Models\n\nFireworks fine-tunable models are fetched dynamically from their API, not stored in `ml_model_list.py`. The script checks the API's `supervisedLoraTunable` and `supervisedFullParameterTunable` fields (not the old `tunable` field, which is stale) and cross-references against our allowlist.\n\n```bash\nuv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py fireworks > /tmp/kiln_finetune_fireworks.json\n```\n\nThis script:\n1. Fetches all models with `supervisedLoraTunable=True` or `supervisedFullParameterTunable=True` from `api.fireworks.ai/v1/accounts/fireworks/models`\n2. Cross-references against `FIREWORKS_SUPPORTED_FINETUNE_MODELS` from `libs/core/kiln_ai/adapters/fine_tune/fireworks_finetune.py` — the same allowlist that filters the runtime fine-tune dropdown\n3. Checks both directions:\n - **Stale allowlist entries**: models in our allowlist that the API no longer marks as supervised-tunable — these may need to be removed\n - **Missing from allowlist**: models the API marks as supervised-tunable but aren't in our allowlist — these are candidates to add\n\nThe canonical allowlist lives in `fireworks_finetune.py` and is shared between the runtime dropdown filter and this audit skill. There is a single place to update when Fireworks changes their supported models.\n\n**Interpreting results:** The supervised API fields return ~43 models, but our allowlist intentionally filters to ~20 that match Fireworks' documented supported models. Seeing ~23 \"models in API but NOT in allowlist\" is normal — these are models Fireworks marks as technically tunable but doesn't officially document or support. Only flag these if a model appears in the Fireworks docs but is missing from our allowlist. The important direction is **stale allowlist entries** — models we ship that the API no longer marks as supervised-tunable.\n\n**Output:** JSON to stdout, human summary to stderr.\n\n---\n\n## Phase 3 – Report Findings\n\nRead the JSON outputs and present findings in a clear table:\n\n```text\nFine-Tune Deprecation Audit Results\n====================================\n\nSTATIC PROVIDERS (provider_finetune_id in ml_model_list.py):\n\n ✅ OpenAI: 5/5 found\n gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14,\n gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18\n\n ✅ Vertex AI: 2/2 found\n gemini-2.0-flash-001, gemini-2.0-flash-lite-001\n\n ❌ Together AI: 2/4 missing\n ✅ Qwen/Qwen2.5-72B-Instruct — found\n ✅ Qwen/Qwen2.5-14B-Instruct — found\n ❌ meta-llama/Meta-Llama-3.1-8B-Instruct-Reference — NOT FOUND\n ❌ meta-llama/Meta-Llama-3.1-70B-Instruct-Reference — NOT FOUND\n\n ⏭️ SKIPPED (no credentials):\n - vertex (VERTEX_PROJECT_ID not set)\n\nDYNAMIC PROVIDER (Fireworks):\n\n ⚠️ 167 models marked tunable in API\n ⚠️ 15 models in known-good docs list\n ✅ 11 models in both API and docs\n ❌ 156 models in API but NOT in docs (likely stale)\n - accounts/fireworks/models/qwen2-72b-instruct\n - accounts/fireworks/models/code-llama-7b\n - ...\n ⚠️ 4 models in docs but NOT in API\n - accounts/fireworks/models/gemma-2-9b-it\n - ...\n```\n\nNote: The example above is illustrative. Actual model counts and entries will vary.\n\n---\n\n## Phase 4 – Remediation\n\nBased on findings, recommend specific actions:\n\n### For static entries (`provider_finetune_id`):\n- If a model's fine-tune ID is no longer valid, either:\n - **Update** the `provider_finetune_id` to a newer version if one exists\n - **Remove** the `provider_finetune_id` field to stop offering fine-tuning for that model on that provider\n - **Set `deprecated=True`** on the provider entry if the model is fully gone\n\n### For Fireworks dynamic entries:\n- The stale models come from Fireworks' API, not our code. Options:\n - **Filter in our code** — update `fetch_fireworks_finetune_models()` in `finetune_api.py` to filter against a known-good list\n - **Report to Fireworks** — flag the stale models to Fireworks support\n - **Both** — filter now, report later\n\n**Always ask the user to confirm** before making any code changes.\n\n---\n\n## Phase 5 – Verify\n\nAfter any changes, run the relevant tests:\n\n```bash\nuv run python3 -m pytest app/desktop/studio_server/test_finetune_api.py -q\n```\n\n---\n\n## Checklist\n\n- [ ] Env vars sourced from `.env`\n- [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex)\n- [ ] Static fine-tune IDs checked against provider APIs (Together, Vertex)\n- [ ] Fireworks dynamic models cross-referenced against known-good list\n- [ ] Findings reported to user with clear table\n- [ ] User confirmed remediation approach\n- [ ] Code changes made (if any)\n- [ ] Tests pass after changes\n- [ ] FIREWORKS_SUPPORTED_FINETUNE_MODELS in fireworks_finetune.py updated if docs have changed\n",".agents/skills/kiln-prerelease-check/SKILL.md":"---\nname: kiln-prerelease-check\ndescription: Run the Kiln pre-release smoke test suite plus the standard CI checks (checks.sh), diagnose every prerelease test that broke (and why), and write a clean readable report with recommended actions. Read-only — it never edits code. Use when the user wants to validate a release candidate, run prerelease tests, or asks for a \"prerelease check / smoke / audit\".\n---\n\n# Kiln Pre-release Check\n\nEnd-to-end pre-release verification and **diagnosis**. This skill:\n\n1. Runs the standard CI checks (`checks.sh` — lint / format / typecheck / unit tests).\n2. Runs the curated `@pytest.mark.prerelease` smoke set (real, paid API calls).\n3. For every test that broke, works out **what** broke and **why** — including whether the cause is a model that's gone deprecated (per `ml_model_list.py`) or that the provider no longer recognizes the slug.\n4. Sweeps the prerelease model pins for staleness and newer model versions.\n5. Writes a clean, scan-readable report with a clear **recommended action** for each finding.\n\n**This skill is read-only. It does not change any code, tests, whitelists, or prod source — ever.** Its entire output is a report at `.prerelease/<timestamp>/REPORT.md` (this directory is gitignored). When the right action is to bump a model pin or update the whitelist, the report *recommends* it; the user makes the change.\n\n---\n\n## What \"prerelease\" means here\n\nA prerelease test is a paid test (it makes real API calls) that we **cannot** meaningfully replace with a mock — its whole point is to verify a live third-party API still behaves how we depend on it. Examples: `test_connect_vertex_live`, basic OpenAI / Claude / Gemini calls, embedding API across providers, reranker integration, Fireworks fine-tune deployment listing, etc.\n\nThe set is a **curated subset of `@pytest.mark.paid`**. Tests carry both markers — `@pytest.mark.paid` for the cost gate, `@pytest.mark.prerelease` for the curated selection.\n\nConcretely:\n\n- `--runpaid` runs the full paid suite (slow, expensive, intended for one-off use).\n- `--runprerelease` runs **only** the prerelease subset (still paid, but a small, focused, releaseable list). It implies `--runpaid` for those tests.\n\nMarker registration lives in `pyproject.toml` (`[tool.pytest.ini_options].markers`); the flag is wired up in the top-level `conftest.py` via `pytest_addoption` + `pytest_collection_modifyitems`.\n\n### Whitelist for litellm-adapter tests\n\nSeveral paid tests (embeddings, document extraction, thinking-level reasoning) fan out across **every** `(model, provider)` pair in `ml_model_list.py` / `ml_embedding_model_list.py`. Running those under `--runprerelease` would be slow and expensive. Instead, those tests have dedicated `*_prerelease_smoke` sibling functions that iterate only over a small curated whitelist living in:\n\n```\nlibs/core/kiln_ai/adapters/pytest_prerelease_whitelist.py\n```\n\nWhitelisted lists:\n\n- `PRERELEASE_CHAT_MODELS` — handful of chat models, one per major provider (OpenAI + GPT-5, Anthropic + Claude Sonnet/Haiku, Gemini, OpenRouter, Groq, Fireworks, Together, Cloudflare).\n- `PRERELEASE_EMBEDDING_MODELS` — one embedding model per embedding-supporting provider.\n- `PRERELEASE_EXTRACTION_MODELS` — one multimodal model per major vendor (OpenAI, Anthropic, Gemini).\n- `PRERELEASE_EXTRACTION_MIME_PROBES` — three mime probes (PDF, PNG, MP3) for the extraction smoke; the full paid test sweeps all 13 mime types per model.\n- `PRERELEASE_THINKING_MODELS` — five (provider, model, thinking_level) triples covering reasoning content + a \"none\" negation case.\n- `PRERELEASE_JEV_MODELS` — TypeSafe AI's Jev models, used by the Jev paid smoke tests. Separate from `PRERELEASE_CHAT_MODELS` because Jev does not go through LiteLLM.\n\nThis whitelist is the thing most likely to go stale, so it's the main target of the pin sweep in Phase 4. The fan-out tests (e.g. `test_extract_document_success` over every model × mime type, `test_paid_generate_embeddings_basic` over every embedding) are **only `@pytest.mark.paid`** — not `@pytest.mark.prerelease` — and are out of scope for this skill.\n\n---\n\n## Global rules\n\n- **Read-only. Make no edits.** Do not modify test files, the whitelist, prod source, config, or anything else. Every actionable finding is a *recommendation* in the report. If you catch yourself about to call Edit/Write on anything other than files under `.prerelease/<timestamp>/`, stop.\n- **Network access is required.** Every test phase makes real API calls or hits remote model-list endpoints. If you are running this skill inside a sandboxed Bash session, request `required_permissions: [\"all\"]` for the test commands.\n- **Env vars:** Source `.env` before running anything that needs API keys. Use a real sourcing pass (not `export $(... | xargs)`, which mangles any value containing spaces or quotes):\n ```bash\n set -a; . ./.env; set +a\n ```\n Provider-specific env vars the prerelease set may touch: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `GROQ_API_KEY`, `OPENROUTER_API_KEY`, `FIREWORKS_API_KEY`, `TOGETHERAI_API_KEY`, `SILICONFLOW_CN_API_KEY`, `COHERE_API_KEY`, `TYPESAFE_API_KEY`, `CLOUDFLARE_API_KEY` + `CLOUDFLARE_ACCOUNT_ID` (used by `test_cloudflare` in `test_prompt_adaptors.py`, which runs GLM 4.7 Flash; a free-plan Cloudflare account is enough), `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`, plus `KILN_TEST_VERTEX_PROJECT_ID` (+ optional `KILN_TEST_VERTEX_LOCATION`) for the Vertex live check, which also requires `gcloud auth application-default login` against a project with `aiplatform.googleapis.com` enabled.\n- **A missing key is a coverage gap, not a failure.** Each missing key makes the relevant test `pytest.skip(...)`. That isn't a prerelease failure — surface it in the report so the user can decide whether to provide the key and re-run.\n- **`ml_model_list.py` never deletes entries — it only marks them `deprecated=True`.** So a slug missing from our list is not a signal you'll normally encounter. The signals that matter when diagnosing a break or staleness are:\n 1. The provider entry has `deprecated=True` in our list.\n 2. The provider itself rejects the slug at runtime (4xx / \"model not found\") — note our list trails provider deprecations, so this can happen *before* we've marked it deprecated. This is how you'd discover a model \"no longer exists\" even when our list still lists it.\n 3. A newer non-deprecated sibling in the same family now exists in our list, regardless of whether the current pin still works.\n- **Single source of truth for the model list** is `libs/core/kiln_ai/adapters/ml_model_list.py` (chat) and `ml_embedding_model_list.py` (embeddings). Cross-reference there when diagnosing.\n- **The pin sweep is mandatory, not \"only when something fails.\"** Even on a fully green run, every hardcoded slug in `pytest_prerelease_whitelist.py` and in prerelease test parametrize lists must be cross-checked against the model list and reported (kept-and-current vs. has-newer-sibling vs. deprecated). Zero silent \"looks fine, moving on\" calls.\n\n---\n\n## Phase 1 — Set up the output directory\n\n```bash\nTS=$(date -u +%Y-%m-%dT%H-%M-%SZ)\nOUT=\".prerelease/${TS}\"\nSCRATCH=\"${OUT}/_scratch\"\nmkdir -p \"${SCRATCH}\"\necho \"Writing prerelease results to ${OUT}\"\n```\n\n`.prerelease/` is gitignored. This timestamped folder is the **only** place this skill writes. Keep it tidy — the top level should hold just the human-readable artifacts:\n\n- `REPORT.md` — the deliverable.\n- `checks.log`, `prerelease.log`, `retry.log` — raw run logs (kept for drill-down).\n\nEverything else — the grep/collect/diff intermediates used while diagnosing (`current_model_ids.txt`, `prerelease_test_files.txt`, `paid_tests.txt`, `prod_probes.txt`, etc.) — goes in `${SCRATCH}` (`${OUT}/_scratch`), **not** the top level. Don't scatter raw filename dumps where the user has to wade through them; the report is what they read.\n\n---\n\n## Phase 2 — Run `checks.sh`\n\nThese are not paid but are mandatory for any release. If they fail, the prerelease check fails before we even spend money on API calls.\n\n```bash\nuv run ./checks.sh --agent-mode 2>&1 | tee \"${OUT}/checks.log\"\n```\n\nCapture the exit code. If it's non-zero, **don't bail** — keep going with the paid tests so the report covers everything in one pass, but mark the overall run as failed.\n\n---\n\n## Phase 3 — Run the prerelease pytest set\n\n```bash\nuv run python3 -m pytest --runprerelease -v --tb=short \\\n -o \"addopts=\" \\\n 2>&1 | tee \"${OUT}/prerelease.log\"\n```\n\nNotes:\n\n- `--runprerelease` is registered in the top-level `conftest.py`. It runs only `@pytest.mark.prerelease` tests and implies `--runpaid` for them.\n- `-o \"addopts=\"` overrides the project default of `-n auto` (xdist). xdist is incompatible with some of the live-API tests (especially anything that touches global litellm state), so run serially.\n- This will be slow. Many of the parametrized tests fan out to dozens of cases.\n- Tests with missing API keys will `skip`, not fail. Capture skip counts.\n\nIf the user wants a faster narrowed run (e.g. only OpenAI + Anthropic), use `-k` to filter:\n\n```bash\nuv run python3 -m pytest --runprerelease -v --tb=short -o \"addopts=\" \\\n -k \"openai or anthropic or vertex\" \\\n 2>&1 | tee \"${OUT}/prerelease.log\"\n```\n\n---\n\n## Phase 4 — Diagnose\n\nThis is where the skill earns its keep: for every failure, explain the cause; for every pin, report staleness. No edits — only findings and recommendations.\n\n### 4a. Diagnose each failure (what broke, why, recommended action)\n\nFor every test that did **not** pass in Phase 3, build one finding. Walk the failure through this decision tree to land on a cause and a recommended action:\n\n1. **Missing credentials → skip, not failure.** (`pytest.skip` for absent key.) Don't list under failures; list under \"Skipped tests\". Recommended action: add the key and re-run if that coverage matters.\n2. **Provider rejected the model slug** — error mentions `model not found`, `does not exist`, `404`, `model_not_found`, `invalid model`, decommissioned, etc. Cross-reference the slug against `ml_model_list.py`:\n - If our list marks that provider entry `deprecated=True` → cause: **deprecated model, provider has dropped it**. Recommended action: bump the pin/whitelist to the newest non-deprecated sibling in the same family (name the candidate).\n - If our list still lists it as non-deprecated → cause: **model no longer exists at the provider; our list trails reality**. Recommended action: (a) bump the test pin/whitelist to a current sibling, and (b) flag that `ml_model_list.py` should mark this entry `deprecated=True`. Name both.\n3. **Auth / quota / rate-limit / transient network** — 401/403/429/5xx, timeouts. Cause: environment or provider-side, not our code. Recommended action: check the key/quota and re-run; only escalate if it reproduces.\n4. **Behavioral assertion mismatch** — the call succeeded but the response shape, content, token/cache fields, streaming chunks, or reasoning output changed. Cause: **provider changed behavior we depend on (real regression)**. Recommended action: investigate the prod path that relies on this; do not assume it's a stale pin. This is the highest-signal failure class — call it out prominently.\n5. **Our-code error** — traceback originates in `libs/core` / `app/` rather than the provider response. Cause: **regression in Kiln**. Recommended action: investigate the implicated code; name the file/function from the traceback.\n\nIf you can't confidently distinguish a stale-pin cause (2) from a real regression (4 or 5), **report it as a possible regression** and say what additional check would disambiguate (e.g. \"try the same call with a current sibling slug to see if it's the model or the code\"). Never recommend a silent pin bump for an ambiguous failure.\n\nRecord, per failure: test ID · provider · cause class · the model slug involved (if any) and its `ml_model_list.py` status · a short excerpt of the actual error · recommended action.\n\n**Retry likely-transient failures once before finalizing.** Live-API tests are flaky: rate limits, 5xx, timeouts, dropped streams, and occasional empty/under-populated responses (e.g. a thinking model returning no reasoning content on one call but not the next) routinely fail a single run without indicating any real breakage. So before you write up the failures, re-run the failing tests in isolation **once** to separate flakes from real defects:\n\n- **Retry these cause classes:** auth/quota/rate-limit/transient (class 3) and ambiguous behavioral/streaming mismatches (class 4) — anything that could plausibly be a one-off. **Do not retry** failures with a deterministic, self-explanatory cause: a deprecated/removed model slug (class 2), a Kiln-code traceback (class 5), or a missing-credential skip. Retrying those just burns money and time.\n- **How:** re-run only the failing node IDs, serially, into a separate log so the original run is preserved:\n ```bash\n uv run python3 -m pytest --runprerelease -v --tb=short -o \"addopts=\" \\\n \"<failing::node::id>\" \"<failing::node::id>\" ... \\\n 2>&1 | tee \"${OUT}/retry.log\"\n ```\n One retry pass is enough — this is a flake filter, not a stability loop. Don't retry a test more than once.\n- **Reclassify by the retry outcome:** a test that **passes on retry** is **transient/flaky**, not a release blocker — move it out of the hard-failure set. A test that **fails again** is **reproducible** — diagnose it as a real finding per the tree above.\n- **Always disclose the retry in the report.** Never silently swap a first-run FAIL for a PASS. Every retried test must show its first-run result, its retry result, and the verdict (transient vs. reproducible) — see the report template's \"Failures\" section and the retry note in the Verdict. A green retry is evidence the suite is flaky here, which the user needs to know before tagging.\n\n### 4b. Model-pin staleness sweep (mandatory every run, even when green)\n\nIndependent of failures, every hardcoded model slug the prerelease set depends on must be checked against the model list and reported. A pin can be perfectly green today and still be worth bumping because a newer sibling shipped — prerelease only catches what real users hit if it tracks what real users actually use.\n\n**In scope (check these every run):**\n- Every entry in `libs/core/kiln_ai/adapters/pytest_prerelease_whitelist.py`.\n- Hardcoded model slugs inside the body or parametrize list of any `@pytest.mark.prerelease` test (notably `libs/core/kiln_ai/adapters/test_prompt_adaptors.py::test_openrouter`).\n\n**Out of scope (don't report as stale):**\n- Hardcoded slugs in mock-only unit tests (no `@pytest.mark.paid` / `@pytest.mark.prerelease`) — the slug is a label, no real call happens.\n- Slugs in fixtures, recorded VCR cassettes, snapshot files.\n- Slugs in `@pytest.mark.paid` tests that aren't also `@pytest.mark.prerelease`.\n\n**Step 1 — status of each whitelist entry.** Run:\n\n```bash\nuv run python3 - <<'PYEOF'\nfrom kiln_ai.adapters.ml_model_list import built_in_models\nfrom kiln_ai.adapters.ml_embedding_model_list import built_in_embedding_models\nfrom kiln_ai.adapters.pytest_prerelease_whitelist import (\n PRERELEASE_CHAT_MODELS, PRERELEASE_EMBEDDING_MODELS,\n PRERELEASE_EXTRACTION_MODELS, PRERELEASE_THINKING_MODELS,\n PRERELEASE_JEV_MODELS,\n)\n\ndef chat_status(name, provider):\n for m in built_in_models:\n if m.name == name:\n p = next((p for p in m.providers if p.name.value == provider), None)\n return 'NO_PROVIDER' if p is None else ('DEPRECATED' if p.deprecated else 'OK')\n return 'MODEL_REMOVED_OR_RENAMED'\n\ndef emb_status(name, provider):\n for m in built_in_embedding_models:\n if m.name == name:\n p = next((p for p in m.providers if p.name.value == provider), None)\n return 'NO_PROVIDER' if p is None else 'OK'\n return 'MODEL_REMOVED_OR_RENAMED'\n\nprint('== chat / extraction ==')\nfor n,p in PRERELEASE_CHAT_MODELS + PRERELEASE_EXTRACTION_MODELS + PRERELEASE_JEV_MODELS:\n print(f'{chat_status(n,p):>22} {n} ({p})')\nprint('== embedding ==')\nfor n,p in PRERELEASE_EMBEDDING_MODELS:\n print(f'{emb_status(n,p):>22} {n} ({p})')\nprint('== thinking ==')\nfor prov,name,lvl in PRERELEASE_THINKING_MODELS:\n print(f'{chat_status(name,prov):>22} {name} ({prov}, level={lvl})')\nPYEOF\n```\n\nAnything not `OK` is stale and goes in the sweep table with a recommended replacement (newest non-deprecated sibling in the same family).\n\n**Step 2 — newer-sibling pass for the `OK` entries.** For each entry that's `OK`, scan `ml_model_list.py` for a newer member of the same family. List the canonical model ids to compare against:\n\n```bash\ngrep -oE '^[[:space:]]+[a-z][a-z0-9_]+ = \"[a-z0-9_]+\"' \\\n libs/core/kiln_ai/adapters/ml_model_list.py | \\\n awk -F'\"' '{print $2}' | sort -u > \"${SCRATCH}/current_model_ids.txt\"\n```\n\nExamples of the judgment call:\n- `PRERELEASE_CHAT_MODELS` on `gpt_4o_mini` while `gpt_5_x_mini` exists → recommend bump.\n- `PRERELEASE_CHAT_MODELS` on `claude_sonnet_4_5` while `claude_sonnet_4_6`/`4_7` exists → recommend bump.\n- `gemini_1_5_*` → `gemini_2_5_*` / `gemini_3_*`; `claude_3_5_*` → `claude_sonnet_4_*` / `claude_4_5_haiku`; `llama_3_x` → `llama_3_3_70b` / `llama_4_*`.\n\nRule of thumb: if `ml_model_list.py` ordering puts a newer sibling above the current pin, recommend the bump. If it's genuinely ambiguous which sibling is \"current\", say so and let the user pick — don't assert a churny recommendation.\n\n**Step 3 — per-test hardcoded slugs.** Enumerate the prerelease test files and scan them for slugs not in `current_model_ids.txt`:\n\n```bash\nuv run python3 -m pytest --runprerelease --collect-only -q -o \"addopts=\" \\\n 2>&1 | grep '::' | awk -F'::' '{print $1}' | sort -u > \"${SCRATCH}/prerelease_test_files.txt\"\n```\n\nCross-reference each pinned slug against `current_model_ids.txt` and the deprecation status. Report any obsolete pins as recommendations.\n\nEvery in-scope pin lands in the sweep table — including the ones that are current and need no action. Silent omission is not allowed.\n\n### 4c. Coverage gaps\n\nThe prerelease set should cover, at minimum:\n\n| Area | Test(s) |\n|---|---|\n| Vertex live connect | `app/desktop/studio_server/test_provider_api.py::test_connect_vertex_live` |\n| OpenAI basic call | `libs/core/kiln_ai/adapters/test_prompt_adaptors.py::test_openai` |\n| Anthropic via OpenRouter / Anthropic API | `test_structured_output_anthropic_*`, `test_structured_output_openrouter_function_calling_nested_object` |\n| Gemini API | covered via embeddings + prompt caching + thinking levels |\n| Groq | `test_groq` |\n| OpenRouter multi-model | `test_openrouter[...]` (parametrized) |\n| Amazon Bedrock | `test_amazon_bedrock` |\n| Embedding API (whitelist) | `test_paid_generate_embeddings_basic_prerelease_smoke`, `test_generate_embedding_prerelease_smoke` |\n| Custom-dimension embeddings (whitelist) | `test_paid_generate_embeddings_custom_dimensions_prerelease_smoke`, `test_generate_embedding_with_user_supplied_dimensions_prerelease_smoke` |\n| Structured output (json_schema, function_calling, flat/nested/array) | `test_structured_output_*` |\n| Streaming (OpenAI protocol + AI SDK protocol) | `test_invoke_openai_stream_chunks`, `test_invoke_ai_sdk_stream` |\n| Tool calling | `test_tools_gpt_4_1_mini` |\n| Thinking-level reasoning (whitelist) | `test_thinking_level_reasoning_content_prerelease_smoke` |\n| Reranker | `test_reranker_integration_success` |\n| Document extraction (whitelist of models × probe mime types) | `test_extract_document_success_prerelease_smoke`, `test_provider_bad_request_prerelease_smoke` |\n| Semantic chunker (real embedding integration) | `test_semantic_chunker_real_integration` |\n| Fireworks fine-tune | `test_fetch_all_deployments` |\n| Prompt caching (Anthropic + OpenAI + Gemini + Fireworks + Together) | `test_prompt_caching_cache_hit` |\n\nIf a new provider/family in `ml_model_list.py` isn't represented in the prerelease set, note the gap as a recommendation.\n\nAlso list recently added `@pytest.mark.paid` tests that are *not* `@pytest.mark.prerelease`-tagged, so the user can decide if any should be promoted:\n\n```bash\n# Compare collected test NODE IDs, not marker line numbers: a test carrying both\n# markers on different lines would be misclassified by a file:line diff. Marker\n# expressions select by marker regardless of the skip gates, so collect-only works.\n# `sed 's/\\[.*//'` collapses parametrized cases to one row per test function\n# (the paid fan-out tests expand to thousands of cases otherwise).\nuv run python3 -m pytest --collect-only -q -m \"paid and not prerelease\" -o \"addopts=\" \\\n 2>/dev/null | grep '::' | sed 's/\\[.*//' | sort -u > \"${SCRATCH}/paid_only_nodes.txt\"\n# Recently added files among those paid-only tests (pipe to xargs to avoid\n# \"argument list too long\" on a large list):\nawk -F'::' '{print $1}' \"${SCRATCH}/paid_only_nodes.txt\" | sort -u | \\\n xargs git log --since=\"3 months ago\" --diff-filter=AM --name-only --pretty=format: -- \\\n 2>/dev/null | sort -u | head -50 > \"${SCRATCH}/recent_paid_only_files.txt\"\n```\n\n### 4d. Prod-code model probes (flag only)\n\nSome prod-code paths hardcode a model slug as a **probe** — a narrow connectivity/auth check where the model is just a vehicle to hit the API, not a feature the user picked. The canonical example is `app/desktop/studio_server/provider_api.py::connect_vertex`, which calls `litellm.acompletion(model=\"vertex_ai/gemini-X.Y-flash\", …)` to verify Vertex credentials when the user clicks \"Connect\".\n\nThese should stay on the latest available model in their family, otherwise the Connect button drifts toward a deprecated/removed model. **This skill only flags them** — it does not edit prod code, and the recommendation in the report is the artifact the user acts on.\n\nSweep for hardcoded inference-probe slugs in non-test prod files:\n\n```bash\ngrep -rnE 'model=[\"'\\''][^\"'\\'']*((gemini|gpt|claude|llama)[^\"'\\'']*)[\"'\\''']' \\\n app/desktop libs/core/kiln_ai libs/server/kiln_server \\\n --include=\"*.py\" 2>/dev/null \\\n | grep -v \"test_\" | grep -v \"/build/\" \\\n | tee \"${SCRATCH}/prod_probes.txt\"\n```\n\nFor each hit:\n1. Decide whether it's a probe (connectivity/auth check, model incidental) or real prod behavior (a default the user sees, fine-tune flow, eval scoring). If unsure, treat it as real prod behavior.\n2. For probes, check `ml_model_list.py` for the newest non-deprecated sibling at that provider.\n3. Record in the probe table: file:line · current slug · suggested newer slug (or \"none — already latest\") · the verifying live test and its result.\n\nEvery grep hit goes in the table even when no newer sibling exists. If the verifying live test failed outright, also list it under Failures (it's a real-failure signal).\n\n---\n\n## Phase 5 — Write the report\n\nCreate `.prerelease/<timestamp>/REPORT.md`. Lead with the verdict; make every finding skim-readable with an explicit recommended action. This is the only artifact the skill produces.\n\n```markdown\n# Kiln Prerelease Check — <timestamp>\n\n## Verdict\n- Overall: PASS / FAIL / PASS WITH RECOMMENDATIONS\n- checks.sh: <pass|fail>\n- prerelease pytest: <N passed, M failed, K skipped>\n- Retries: <none needed | \"X of M first-run failures were retried (Phase 4a); Y passed on retry (transient/flaky), Z reproduced\" — omit if no test was retried>\n- Headline: <one sentence — the single most important thing the user must know before tagging>\n\n## Test results\nOne scannable table covering **every prerelease test that ran**, grouped by coverage area, with an emoji outcome and a short note where it helps — this is the at-a-glance summary the user reads first. Aggregate parametrized cases into one row with per-case emoji counts (e.g. `✅×8 ❌×1`); don't spell out long node-id / filename lists here (raw lists live in the logs and `_scratch/`). Outcome legend: ✅ pass · ❌ fail · ⏭️ skip (missing key / N/A) · 🔁 transient (failed then passed on retry). Leave the Notes cell blank when the row is a clean all-pass; use it to name the failing case, why it skipped, or the retry result.\n\n| Area | Test(s) | Result | Notes |\n|---|---|---|---|\n| OpenAI basic call | `test_openai` | ✅ | |\n| Groq | `test_groq` | ✅ | |\n| OpenRouter multi-model | `test_openrouter` (9) | ✅×9 | |\n| Amazon Bedrock | `test_amazon_bedrock` | ✅ | |\n| Structured output | `test_structured_output_*` (6) | ✅×6 | |\n| Streaming | `test_invoke_openai_stream_chunks` / `test_invoke_ai_sdk_stream` (N) | ✅×N | |\n| Tool calling | `test_tools_gpt_4_1_mini` | ✅ | |\n| Thinking-level reasoning | `…reasoning_content_prerelease_smoke` (N) | ✅×N | <which case failed/skipped, if any> |\n| Embeddings | `…embeddings_basic_prerelease_smoke` (6) | ✅×4 ⏭️×2 | <which providers skipped + why> |\n| Reranker | `test_reranker_integration_success` (N) | ✅×1 ❌×2 | <which provider failed + one-line cause> |\n| Prompt caching | `test_prompt_caching_cache_hit` (7) | ✅×7 | |\n| Document extraction | `…extract_document_*_prerelease_smoke` (N) | ✅×N | |\n| Semantic chunker | `test_semantic_chunker_real_integration` | ✅ | |\n| Vertex live connect | `test_connect_vertex_live` | ❌ | <one-line cause> |\n| Fireworks fine-tune | `test_fetch_all_deployments` | ✅ | |\n\nUse the real areas/tests/counts from this run (mirror the coverage table in Phase 4c). Put 🔁 on anything that needed a retry and add the retry detail in the Notes cell + under Failures.\n\n## Failures — what broke, why, and what to do\nOne block per failing test. Sorted most-actionable first (real regressions before stale pins).\nIf any failures were retried (Phase 4a), open this section with a small first-run-vs-retry table so transient flakes are visible at a glance, then write a block only for the tests that **reproduced** (note retried-and-passed ones as transient — don't dignify them with a full diagnosis block).\n\n### <test ID>\n- Provider: <provider>\n- Cause: <deprecated model | model no longer exists at provider | provider behavior change (regression) | Kiln code regression | auth/quota/transient>\n- Model slug: <slug> — `ml_model_list.py` status: <OK | deprecated | no-provider-entry | n/a>\n- Retry: <not retried (deterministic cause) | retried once — reproduced | retried once — passed (transient, flaky)>\n- Evidence: <short excerpt of the actual error / failed assertion>\n- Recommended action: <concrete next step — e.g. \"bump PRERELEASE_CHAT_MODELS pin from X to Y (newest sibling)\"; or \"investigate prod path Z — response shape changed\"; or \"mark this entry deprecated=True in ml_model_list.py and bump the pin\". Recommendation only — this skill makes no edits.>\n\n## Skipped tests (missing credentials)\nList by provider, with which env var is missing, so the user can decide what to add and re-run.\n\n## Model-pin staleness sweep (mandatory, every run)\nEvery in-scope hardcoded slug — including current ones needing no action.\n\n| Location (file:line) | Current slug | `ml_model_list.py` status | Newer sibling? | Recommendation |\n|---|---|---|---|---|\n| `…/pytest_prerelease_whitelist.py:NN` | `claude_sonnet_4_5` | OK | yes — `claude_sonnet_4_6` | bump to `claude_sonnet_4_6` |\n| `…/pytest_prerelease_whitelist.py:NN` | `gpt_4o_mini` | OK | no | keep (current) |\n| `…/test_prompt_adaptors.py:NN` | `gemini_1_5_flash` (openrouter) | deprecated | n/a | bump to `gemini_2_5_flash` |\n\n`status` ∈ {OK, deprecated, no-provider-entry, removed/renamed}. `Recommendation` ∈ {keep (current), bump to <slug>, investigate}.\n\n## Prod-code probe sweep (flag only — no edits)\nEvery hardcoded inference-probe slug found in prod code (Phase 4d).\n\n| File:line | Current slug | Suggested newer slug | Verifying test | Test result | Notes |\n|---|---|---|---|---|---|\n| `app/desktop/studio_server/provider_api.py:NNNN` | `vertex_ai/gemini-2.5-flash` | `vertex_ai/gemini-3-flash` | `test_connect_vertex_live` | PASS | probe still works; newer family member exists |\n| `path/to/file.py:NN` | `…` | none — already latest | `test_…` | PASS | no action |\n\n## Coverage gaps\n- Providers/families in `ml_model_list.py` with no prerelease test.\n- Recently added paid tests not in the prerelease set (one line each on why it MIGHT be prerelease-worthy).\n\n## Flagged for user review\nAnything you noticed but couldn't safely classify: ambiguous failures (model-vs-code unclear), prod-path code that isn't a probe, suspicious behavior. Each row: file:line / test · what's suspicious · what would disambiguate.\n\n## Suggested next steps\nOrdered checklist of what the user should do before tagging the release (apply recommended pin bumps, investigate regression X, add key for skipped provider Y, etc.).\n```\n\nKeep it scan-readable. Lead with the verdict. Don't bury real regressions under pin-housekeeping.\n\n---\n\n## Phase 6 — Hand back to the user\n\nPrint the report path and a one-line verdict:\n\n```bash\necho \"Prerelease check complete. Report: ${OUT}/REPORT.md\"\n```\n\nThen summarize in chat (one or two sentences):\n\n- All green, nothing stale: `Prerelease check: PASS. <N> passed, <K> skipped (missing keys: <providers>). No stale pins. Report at <path>.`\n- Green but recommendations exist: `Prerelease check: PASS WITH RECOMMENDATIONS. <N> passed; <M> stale/older pins to consider bumping (no changes made). Report at <path>.`\n- Something failed: `Prerelease check: FAIL. <X> failed across <providers> — <one-line dominant cause>. See <path>/REPORT.md before tagging.`\n\nRemember: the skill never made any edits. If the report recommends pin bumps or a prod-probe change, those are for the user to apply.\n\n---\n\n## Additional release checklists\n\n- **Code Tools / Code Evals cross-OS spawn sanity**: see `specs/projects/code_tools/cross_os_spawn_checklist.md` for platform-specific verification steps (macOS/Windows/Linux + PyInstaller frozen builds). Must be completed before any release that includes code tools or code evals.\n\n## Checklist\n\n- [ ] `.prerelease/<timestamp>/` directory created (the only place anything was written)\n- [ ] `checks.sh --agent-mode` run, log captured\n- [ ] `--runprerelease` pytest run, log captured\n- [ ] Likely-transient failures (auth/quota/transient + ambiguous behavioral) retried once in isolation; deterministic failures (deprecated/removed slug, Kiln-code traceback) not retried\n- [ ] Every retried test disclosed in the report (first-run result, retry result, transient-vs-reproducible verdict) — no silent FAIL→PASS swaps\n- [ ] Every failure diagnosed with a cause class and a concrete recommended action (regression vs. deprecated-model vs. model-gone vs. auth/transient)\n- [ ] Model-pin staleness sweep table filled for **every** in-scope slug, including current ones needing no action\n- [ ] Newer-sibling recommendations included even for green pins\n- [ ] Prod-probe sweep done; every grep hit in the probe table with a suggested slug (or \"none — already latest\")\n- [ ] Coverage-gap audit done\n- [ ] Skipped tests listed by provider + missing env var\n- [ ] **No files edited outside `.prerelease/<timestamp>/`** — all fixes are recommendations\n- [ ] `REPORT.md` written, verdict first, recommended actions explicit\n- [ ] Verdict surfaced to the user with the report path\n",".agents/skills/claude-maintain-models/SKILL.md":"---\nname: claude-maintain-models\ndescription: Add new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, or wants to add a net-new AI provider to Kiln.\nallowed-tools: Read Edit Write Bash Grep Glob Agent WebSearch WebFetch\n---\n\n# Add a New AI Model to Kiln\n\n**Branch check first:** if the request involves a provider Kiln does not\nsupport yet, start at [Adding a Net-New Provider](#adding-a-net-new-provider).\nThat workflow spans core, server, UI, tests, and tooling, and is gated on a\nclient release — the model-entry steps below are only its final, gated piece.\n\nFor a new model on an already-supported provider, integrating it into\n`libs/core/kiln_ai/adapters/ml_model_list.py` requires:\n\n1. **`ModelName` enum** – add an enum member\n2. **`built_in_models` list** – add a `KilnModel(...)` entry with providers\n3. **`ModelFamily` enum** – only if the vendor is brand-new\n\nAfter code changes, run paid integration tests, then draft a Discord post.\n\n---\n\n## Global Rules\n\nThese apply throughout the entire workflow.\n\n- **Slug verification:** NEVER guess or infer model slugs from naming patterns. Every `model_id` must come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it.\n- **Date awareness:** These models are often released very recently. Web search for current info before assuming you know the details.\n\n---\n\n## Phase 1 – Model Discovery (only when asked to find new/missing models)\n\nIf the user asks you to find new models, **do NOT just web search \"new AI models this week\"** — that only surfaces major releases. Instead, systematically check each family against **both** the LiteLLM catalog **and** models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.\n\n1. **Read the `ModelFamily` and `ModelName` enums** to know what we already have.\n\n2. **Query both catalogs for each family** (run in parallel where possible):\n\n **LiteLLM catalog** — filters out mirror providers to avoid duplicates:\n ```bash\n curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != \"openrouter\" and .provider != \"bedrock\" and .provider != \"bedrock_converse\" and .provider != \"vertex_ai-anthropic_models\" and .provider != \"azure\") | .id] | unique | .[]'\n ```\n\n **models.dev** — search all model IDs across all providers:\n ```bash\n curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i \"SEARCH_TERM\"\n ```\n For details on a specific provider+model: `curl -s https://models.dev/api.json | jq '.[\"PROVIDER\"].models[\"MODEL_ID\"]'`\n\n3. **Search terms** (one query per term):\n `claude`, `gpt`, `o1`, `o3`, `o4` (OpenAI reasoning), `gemini`, `llama`, `deepseek`, `qwen`, `qwq`, `mistral`, `grok`, `kimi`, `glm`, `minimax`, `hunyuan`, `ernie`, `phi`, `gemma`, `seed`, `step`, `pangu`\n\n4. **Union and cross-reference** results from both catalogs against `ModelName`. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). **Skip pure coding models** (e.g. `codestral`, `deepseek-coder`, `qwen-coder`).\n\n5. **Run targeted web searches** per family to catch very fresh releases not yet in either catalog:\n - `\"[family] new model [current year]\"`\n - `\"[family] release [current month] [current year]\"`\n\n6. **Present findings** as a summary. Let the user decide which to add.\n\n---\n\n## Phase 1B – Lagging-Provider Backfill Check (every run)\n\nSome providers — **Fireworks AI**, **Together AI**, **SiliconFlow**, **Cloudflare** — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM (or, for Cloudflare, never reach LiteLLM at all). Relying only on those two catalogs will both under-populate the provider list for the model you're adding now **and** miss the window to backfill recently-added models whose provider support has since grown.\n\nRun this check on **every invocation** of the skill, regardless of whether you're in discovery mode or adding a specific model.\n\n1. **Pull the 10 most recently added models from git history.** List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:\n ```bash\n git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E \"^\\+\\s+name=ModelName\\.\" | head -20\n ```\n\n2. **For the model you're adding (if any) AND each of those 10 models**, cross-check Fireworks, Together, SiliconFlow and Cloudflare directly using the endpoints in the [Lagging Providers Reference](#lagging-providers). Do NOT trust `models.dev` / LiteLLM as the final word for these providers.\n\n3. **If a lagging provider now supports a recently-added model that isn't yet in its `KilnModel` entry**, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.\n\n4. **Run the Cloudflare thinking-level checker** (see [Cloudflare thinking levels](#cloudflare-thinking-levels)). It checks every Cloudflare entry, not only the recent ones, because Cloudflare edits the effort lists of models Kiln already ships. Treat each ❌ finding like a backfill: fix the entry in the current change or propose a separate PR.\n\n---\n\n## Phase 2 – Gather Context\n\n1. **Read the predecessor model** in `ml_model_list.py` (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.\n\n2. **Query the LiteLLM catalog** for the new model. This is the primary slug source since Kiln uses LiteLLM. See the [Slug Lookup Reference](#slug-lookup-reference) for query syntax and all verified sources.\n\n3. **Get the OpenRouter slug** via:\n - `curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i \"SEARCH_TERM\"`\n - Fallback: WebSearch for `openrouter [model name] model id`\n\n4. **Get the direct-provider slug** (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the [Slug Lookup Reference](#slug-lookup-reference) for provider-specific URLs.\n\n5. **Identify quirks** — check the [Provider Quirks Reference](#provider-quirks-reference) for the relevant provider, and web search for any new quirks:\n - Structured output mode (JSON schema vs function calling)?\n - Reasoning model (needs `reasoning_capable`, parsers, OpenRouter options)?\n - Vision/multimodal support? Which MIME types?\n - Provider-specific flags (`temp_top_p_exclusive`, etc.)?\n - Rate limit concerns (`max_parallel_requests`)?\n\n6. **Determine thinking levels** — does the model support configurable reasoning effort? See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain. Key quick checks:\n - Check the **vendor model page** (e.g. OpenAI model pages say \"Reasoning.effort supports: X, Y, Z\")\n - Check **OpenRouter** `supported_parameters` — if `reasoning` is absent, skip thinking levels\n - R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts\n\n---\n\n## Phase 3 – Code Changes\n\nAll changes go in `libs/core/kiln_ai/adapters/ml_model_list.py`.\n\n### 3a. `ModelName` enum\n\n- snake_case: `claude_opus_4_6 = \"claude_opus_4_6\"`\n- Place **before** predecessor (newer first within group). If the vendor is\n brand-new there is no predecessor — start a new group at the end of the enum\n- Follow existing grouping (all claude together, all gpt together, etc.)\n\n### 3b. `KilnModel` entry in `built_in_models`\n\n- Place per the **ordering rules in 3c** — this placement is user-visible, get it right\n- Copy predecessor's structure and modify: `name`, `friendly_name`, `model_id` per provider, flags\n- **`friendly_name` must follow the existing naming pattern** of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use `\"Claude {version} Sonnet\"` (e.g. \"Claude 4.5 Sonnet\"), not `\"Claude Sonnet {version}\"`. Do NOT use the vendor's marketing name if it differs from Kiln's established convention.\n\n**Provider `model_id` formats:**\n\n| Provider | Format | Notes |\n|----------|--------|-------|\n| `openrouter` | `vendor/model-name` | Always verify via API |\n| `openai` | Bare model name | Verify via OpenAI docs |\n| `anthropic` | Variable — older models have date stamps, newer may not | Always verify via Anthropic docs |\n| `gemini_api` | Bare name | Verify via Google AI Studio docs |\n| `fireworks_ai` | `accounts/fireworks/models/...` | Verify via Fireworks docs |\n| `together_ai` | Vendor path format | Verify via Together docs |\n| `vertex` | Usually same as gemini_api | Verify via Vertex docs |\n| `siliconflow_cn` | Vendor/model format | Verify via SiliconFlow docs |\n| `featherless_ai` | HuggingFace repo id, case-sensitive (`zai-org/GLM-5.2`) | Verify via their `/v1/models` — see [Featherless](#featherless-ai) |\n| `typesafe` | Dotted System One version (`jev-1.13.0`) | Their `/v1/models` lists aliases only — verify by `POST /v1/systemone`, see [TypeSafe AI](#typesafe-ai) |\n| `cloudflare` | Cloudflare's own `@cf/<author>/<model>` id, verbatim (`@cf/zai-org/glm-5.3`) — no `cloudflare/` or `openai/` prefix | Verify via Cloudflare's model JSON — see [Cloudflare](#cloudflare-workers-ai) |\n\n**Every single `model_id` must be verified from an authoritative source. No exceptions.**\n\n**Setting flags — use catalog data + predecessor as dual signals:**\n\nThe LiteLLM catalog and models.dev responses include capability flags (`supports_vision`, `supports_function_calling`, `supports_reasoning`, etc.). Use these as the **primary signal** for what to enable on the new model:\n\n- If the catalog says `supports_vision: true` → enable `supports_vision`, `multimodal_capable`, and vision MIME types (see 2c)\n- If the catalog says `supports_function_calling: true` → use `StructuredOutputMode.json_schema` (or `function_calling` depending on provider norms — check predecessor)\n- If the catalog says `supports_reasoning: true` → the model can reason, but **do NOT reflexively set `reasoning_capable=True`** — default to `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)). Still add `available_thinking_levels` if it supports effort levels, and check parser/formatter flags.\n\nThen **cross-check against the predecessor**. The predecessor tells you *how* Kiln configures a similar model (which `structured_output_mode`, which provider-specific flags, etc.). The catalog tells you *what* the model can do. Use both:\n- Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.\n- Predecessor has `temp_top_p_exclusive` but nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track.\n- Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.\n\n**Common flags:**\n- `structured_output_mode` – how the model handles JSON output\n- `suggested_for_evals` / `suggested_for_data_gen` – see **zero-sum rule** below. If the entry also sets `available_thinking_levels`, its `default_thinking_level` must be a reasoning level — see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)\n- `multimodal_capable` / `supports_vision` / `supports_doc_extraction` – see **multimodal rules** below\n- `reasoning_capable` – for thinking/reasoning models. **Default new models to `reasoning_capable=False`** unless the model *always* emits its reasoning (see [Reasoning Capable Default](#reasoning-capable-default))\n- `temp_top_p_exclusive` – Anthropic models that can't have both temp and top_p\n- `parser` / `formatter` – for models needing special parsing (e.g. R1-style thinking)\n\n#### 2c. Multimodal capabilities\n\nIf the model supports non-text inputs, configure:\n\n- `multimodal_capable=True` and `supports_doc_extraction=True` if it supports any MIME types\n- `supports_vision=True` if it supports images\n- `multimodal_requires_pdf_as_image=True` if vision-capable but no native PDF support (also add `KilnMimeType.PDF` to MIME list). **Always set this on OpenRouter providers** — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.\n- Always include `KilnMimeType.TXT` and `KilnMimeType.MD` on any `multimodal_capable` model\n\n**Strategy: start broad, narrow based on test failures.** Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.\n\nFull MIME superset (Gemini uses all):\n```python\n# documents\nKilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD\n# images\nKilnMimeType.JPG, KilnMimeType.PNG\n# audio\nKilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG\n# video\nKilnMimeType.MP4, KilnMimeType.MOV\n```\n\n### 3c. Ordering — the list IS the UI\n\nThe order of `built_in_models` is exactly the order users see: model dropdowns\ngroup by provider and list each provider's models in list order, and the model\nlibrary page lists models in list order. A misplaced entry ships a scrambled\ndropdown to every client via the remote config. Rules:\n\n1. **Families are contiguous.** Every model of a family sits in one block.\n Never append a new family member at the bottom of the file or after another\n family — that strands it (past bugs: Mistral Small 4/3 ended up inside the\n Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).\n2. **Within a family, versions run newest → oldest, top to bottom.**\n 4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the\n family block. A new model of an *existing* version goes inside that\n version's group — NOT at the top of the family and NOT below older\n versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7\n landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).\n3. **Within a version, big → small. Always.**\n - Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite;\n Large > Medium > Small.\n - Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking\n variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and\n VL Thinking blocks) stay intact, ordered descending internally.\n - Tier-grouped families follow the same rule for their tier blocks: Claude\n runs Fable, then Opus > Sonnet > Haiku, versions descending inside each\n tier. (It once ran Haiku-first, and several families ran sizes\n ascending — those were bugs, not conventions. Do not preserve an\n ascending run because it's \"what the family already does.\")\n4. **A net-new family's block goes at the END of `built_in_models`** (this is\n the existing convention — the newest niche vendors sit at the bottom of the\n list). The \"place before predecessor\" rule only applies within an existing\n family; a new family has no predecessor.\n5. **Verify after editing** — print the order you just shipped and eyeball the\n affected family:\n\n ```bash\n uv run python -c \"\n from kiln_ai.adapters.ml_model_list import built_in_models\n for m in built_in_models: print(m.family, '|', m.friendly_name)\"\n ```\n\n### 3d. `suggested_for_evals` / `suggested_for_data_gen`\n\n**Only set these if** the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).\n\n**Zero-sum rule:** When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. **Ask the user to confirm** the swap before making changes.\n\n### 3e. `ModelFamily` enum (only if needed)\n\nOnly add a new family if the vendor is completely new.\n\n### 3f. Thinking Levels (`available_thinking_levels` / `default_thinking_level`)\n\nIf the model supports configurable reasoning effort (not just on/off), add `available_thinking_levels` and `default_thinking_level` to each provider entry. See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain and existing constants.\n\n**Quick rules:**\n- Reuse an existing `_THINKING_LEVELS` constant if the levels match exactly\n- Create a new constant only if levels differ; name it `{MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS`\n- `default_thinking_level` must be one of the values in `available_thinking_levels`\n- If the entry sets `suggested_for_evals=True`, `default_thinking_level` must be `\"medium\"` (or `\"low\"` if the model has no medium level) — never `\"none\"`. See [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)\n\n---\n\n## Phase 4 – Run Tests\n\nTests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.\n\n**Vertex AI authentication:** Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run `gcloud auth application-default login` before trying. These failures are auth issues, not model config problems.\n\n**`-k` filter syntax:** Always use bracket notation for model+provider filtering, never `and`:\n- Good: `-k \"test_name[glm_5-fireworks_ai]\"` or `-k \"glm_5\"`\n- Bad: `-k \"glm_5 and fireworks\"` — `and` is a pytest keyword expression that can match wrong tests\n\n### 4.0 — If the test env can't build (blocked git dependency)\n\nKiln's core lib pins `together` to a git fork (`libs/core/pyproject.toml`: `together = { git = \"https://github.com/scosman/together-python\" }`). In a sandboxed environment whose GitHub access is scoped to `kiln-ai/kiln` only (e.g. Claude Code Web), `uv sync` fails to fetch that fork with a **403** from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only `kiln-ai/kiln` resolves.\n\nWorkaround to build the venv for testing (the `together` fork isn't exercised by model-integration tests, which route through LiteLLM):\n\n1. **Do NOT use `uv sync --no-sources`** — it strips the workspace `workspace = true` sources too and makes `kiln-root → kiln-ai` unsatisfiable.\n2. Instead, temporarily comment out ONLY the `together = { git = ... }` line in `libs/core/pyproject.toml`, then run `uv sync` (this resolves `together` from PyPI).\n3. Run the tests.\n4. **Revert** the workaround with `git checkout -- libs/core/pyproject.toml uv.lock` — this restores the commented-out `together` git-fork pin in `libs/core/pyproject.toml` (NOT the repo-root `pyproject.toml`, which the workaround never touches) and reverts `uv.lock` if it changed. Only `ml_model_list.py` (and any intended test-file edits) should remain modified.\n\nNote: the PyPI `together` may pull slightly different transitive deps (e.g. a newer `starlette`), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your `-k` filters to the model files and ignore those.\n\n### 4a. Parallel testing + API keys\n\n**Parallel testing is already on.** There is no `pytest.ini` — pytest config lives in `[tool.pytest.ini_options]` in the root `pyproject.toml`, and `addopts = \"-n auto\"` is active. No edit and no revert are needed. (Only override to `-n 8` if a provider rate-limits you.)\n\n**Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings.** `conftest.py` has an autouse `use_temp_settings_dir` fixture that points `Config.settings_path` at a temp dir, so `~/.kiln_ai/settings.yaml` is deliberately ignored during tests. A key the user added through the app's provider page **will not be seen** — the test fails with \"Attempted to use X without an API key set\", which looks like a config bug but isn't.\n\nBridge the key from the user's settings into the test environment without ever printing it:\n\n```bash\nexport FEATHERLESS_AI_API_KEY=\"$(uv run python -c \\\n \"from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)\" 2>/dev/null | tail -1)\"\n```\n\nUse the provider's `env_var` name from `libs/core/kiln_ai/utils/config.py`. Some providers need more than one value: Cloudflare needs both `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` (and optionally `CLOUDFLARE_AI_GATEWAY_ID`). To check which keys are available before running, print booleans only — never the values:\n\n```bash\nuv run python -c \"from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))\"\n```\n\n**Many paid tests also carry the `ollama` marker**, so `--runpaid` alone silently skips them. Always pass `--runpaid --ollama` together, and use `-rs` to see skip reasons when a test you expected to run reports as skipped.\n\n### 4b. Smoke test — verify slug works\n\nRun a single test+provider combo first:\n\n```bash\nuv run pytest --runpaid --ollama -k \"test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]\"\n```\n\nIf it fails, fix the slug/config before proceeding. Use `--collect-only` to find exact parameter IDs if unsure.\n\n### 4c. Full test suite\n\n```bash\nuv run pytest --runpaid --ollama -k \"MODEL_ENUM\" -v 2>&1 | grep -E \"PASSED|FAILED|ERROR|short test|=====|collected\"\n```\n\n**If tests fail — debug one at a time:**\n1. Pick ONE failing test, run it with `-v` for full output\n2. Fix the config\n3. Re-run that single test to verify\n4. Only re-run the full suite once the single test passes\n\n**Anthropic API key gotcha:** if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as `KILN_ANTHROPIC_API_KEY` instead of `ANTHROPIC_API_KEY` (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't `export` it globally:\n\n```bash\nANTHROPIC_API_KEY=\"$KILN_ANTHROPIC_API_KEY\" uv run pytest --runpaid ...\n```\n\n### 4d. Extraction tests (if `supports_doc_extraction=True`)\n\nTests are in `libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py`.\n\n```bash\n# See what will run:\nuv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM\n\n# Run them:\nuv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k \"MODEL_ENUM\"\n```\n\nIf a provider rejects a data type (400 error), remove that `KilnMimeType` and re-run.\n\n### 4e. Confirm failures are actually yours\n\nBefore treating a failure as a problem with your change, check whether the same test already fails for an **existing** provider of that model. Several assertions are provider-independent and fail regardless.\n\nKnown example: `test_structured_input_cot_prompt_builder` asserts `len(trace) == 5` unconditionally, which is incompatible with any provider setting `reasoning_capable=True` (that selects the single-call strategy, producing 3 messages). It fails for `gpt_oss_120b` on `fireworks_ai` on a clean tree.\n\n```bash\nuv run pytest --runpaid --ollama -q \"path::test_name[MODEL-OTHER_PROVIDER]\"\n```\n\nIf it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.\n\n### 4f. Test output format\n\nCollect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:\n- ✅ for passed tests\n- ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason\n- ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason\n- List every test using the full pytest parametrize ID, grouped by provider\n- Include extraction tests (Phase 4d) if they were run\n\n---\n\n## Phase 5 – Create Pull Request\n\n### 5.0 — Important context about Claude Code Web's stop hook\n\nThis skill is often run via Claude Code Web (Slack connector). That environment has a **non-user-configurable stop hook** which, at end of session, will:\n- Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits\n- Instruct the agent to commit and push any local work before stopping\n- Explicitly tell the agent NOT to create a PR unless the user asked for one\n\n**The problems this causes:**\n1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned `add-model/*` branches on the remote.\n2. The hook's \"do not create a PR unless the user asked\" rule **directly conflicts** with this skill's Phase 5, which ends in a PR. Running this skill *is* the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.\n\n**The user's desires, in priority order:**\n1. **Ask before you push.** If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code \"just to satisfy the stop hook.\"\n2. **No abandoned branches.** Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.\n3. **If the user says to abandon:** revert your local changes (`git restore` / `git clean` the specific files you touched) and delete any branch you created (`git checkout main && git branch -D add-model/MODEL_NAME`) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.\n4. **On a successful run, push and open the PR as described in 5a/5b.** Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says \"don't create a PR.\" Only re-ask if tests failed or the user hasn't confirmed the results.\n\n### 5.1 — Gate before pushing\n\nDo NOT commit, push, or create a branch if any of the following are true:\n- Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)\n- The smoke test (4b) failed and wasn't resolved\n- Any step in Phases 2–4 was skipped or incomplete\n- You are unsure whether a ⚠️ flake is actually a real failure\n\nIf any of the above apply, **stop and ask the user** what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.\n\nAfter all tests pass, commit the changes and open a PR against `main`.\n\n### 5a. Commit and push\n\n1. Create a new branch named `add-model/MODEL_NAME` (e.g. `add-model/glm-5-1`)\n2. Stage only the changed files (typically just `ml_model_list.py`)\n3. Commit with a concise message (e.g. \"Add GLM 5.1 to model list (together_ai, siliconflow_cn)\")\n4. Push the branch\n\n### 5b. Create the PR\n\nUse `gh pr create` against `main`. Follow Rule 0, Rule 1 and Step 3 of the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) for the human header, the CLA and the title:\n\n- The title is `WIP: chore: ` followed by the model-list subject, for example `WIP: chore: add GLM 5.1 to model list`. Use `chore`, because adding a model is maintenance work, not a new feature. Only a human removes the `WIP: ` prefix.\n- Copy everything above `# Agentic PR Summary` from `.github/pull_request_template.md` with no change. Never fill in a placeholder or tick a box there, and never sign the CLA. Remove the `## Contributor License Agreement` section only when the PR author's GitHub username is `scosman`, `sfierro`, `leonardmq`, `tawnymanticore`, or `chiang-daniel`. Keep it, unsigned, for any other author.\n\nReplace the `` `Insert AI summary of PR using .agents/skills/open-pr/SKILL.md` `` placeholder under `# Agentic PR Summary` with the test results, in this exact format:\n\n```\n## What does this PR do?\n\n Test Results\n\n[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]\n\n[Model Name] ([provider]):\n- [N] passed, [N] skipped[, [N] failed]\n- [Any notable failures or flakes]\n\n[Repeat for each model+provider combo]\n\n---\n[Model Name] ([provider]):\n✅ test_data_gen_all_models_providers[model_enum-provider]\n✅ test_data_gen_sample_all_models_providers[model_enum-provider]\n✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]\n✅ test_all_built_in_models_llm_as_judge[model_enum-provider]\n✅ test_all_built_in_models_structured_output[model_enum-provider]\n✅ test_all_built_in_models_structured_input[model_enum-provider]\n✅ test_structured_output_cot_prompt_builder[model_enum-provider]\n✅ test_all_models_providers_plaintext[model_enum-provider]\n✅ test_cot_prompt_builder[model_enum-provider]\n⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason\n❌ test_name[model_enum-provider] — brief reason\n\n[Repeat for each model+provider combo]\n```\n\n**Rules for the PR body:**\n- Every test that ran must appear in the per-test dump, using the full pytest parametrize ID\n- Group tests by `[Model Name] ([provider]):` headers\n- The summary section at the top gives a quick pass/skip/fail count per model+provider\n- The detailed section below the `---` inside the summary (not the `----` above `# Agentic PR Summary`) lists every individual test result\n- Use ⚠️ for content quality flakes (not real failures), ❌ for real errors\n\n---\n\n## Checklist\n\n- [ ] `ModelName` enum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor)\n- [ ] `KilnModel` entry added to `built_in_models` per the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list)\n- [ ] Printed the resulting list order and eyeballed the affected family (3c verification command)\n- [ ] `friendly_name` matches the naming pattern of sibling models in the same family\n- [ ] If the provider is net-new: followed [Adding a Net-New Provider](#adding-a-net-new-provider), including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)\n- [ ] `ModelFamily` enum updated (only if new family)\n- [ ] All provider slugs verified from authoritative sources\n- [ ] Flags inherited from predecessor and adjusted for quirks\n- [ ] `reasoning_capable` defaulted to `False` for adaptive-reasoning models (only `True` for always-emits-reasoning models — see [Reasoning Capable Default](#reasoning-capable-default))\n- [ ] Thinking levels configured if model supports reasoning effort (see [Thinking Levels Reference](#thinking-levels-reference))\n- [ ] Cloudflare and OpenRouter: `check_thinking_levels.py` reports no ❌ finding (see [Cloudflare thinking levels](#cloudflare-thinking-levels) and [OpenRouter (general)](#openrouter-general))\n- [ ] Every `suggested_for_evals` entry with thinking levels defaults to `\"medium\"` (or `\"low\"`), never `\"none\"` (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default))\n- [ ] Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)\n- [ ] Zero-sum applied if model is suggested for evals/data gen\n- [ ] RAG config templates updated if the new model replaces one used in `app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts`\n- [ ] API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)\n- [ ] Smoke test passed\n- [ ] Full test suite passed\n- [ ] Failures cross-checked against an existing provider before being called regressions (see 4e)\n- [ ] PR created against `main` with a `WIP: ` title, the untouched human header, and test results in the Agentic PR Summary\n\n---\n\n## Reasoning Capable Default\n\n**Default newly-added models to `reasoning_capable=False`, even when the catalog reports `supports_reasoning: true`.**\n\nMost recent models use *adaptive* reasoning and sometimes return no reasoning at all. Kiln raises `RuntimeError(\"Reasoning is required for this model, but no reasoning was returned.\")` whenever a provider has `reasoning_capable=True` and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.\n\n`reasoning_capable` is **orthogonal to `available_thinking_levels`** — you keep the important behavior with it set to `False`:\n- Thinking-level / effort selection still works (every GPT-5.x entry sets `available_thinking_levels` with `reasoning_capable` unset).\n- Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).\n- The `test_thinking_level_reasoning_content` paid test still runs and still asserts reasoning is present — it is gated on `available_thinking_levels`, not `reasoning_capable`. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels.\n\nWhat you give up with `reasoning_capable=False`:\n- The conditional \"reasoning is present in `intermediate_outputs`\" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped).\n- When a user attaches a chain-of-thought prompt, the model uses the two-call `two_message_cot` strategy instead of the single-call native `single_turn_r1_thinking`.\n\n**Keep `reasoning_capable=True` only for models that *always* emit reasoning** in a native `<think>` format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.\n\n**Narrower alternative:** if a model reliably reasons but you only hit the error on structured output, set `reasoning_optional_for_structured_output=True` (requires `reasoning_capable=True`) instead of disabling reasoning entirely.\n\n---\n\n## Adding a Net-New Provider\n\nAdding a provider Kiln has never supported is a bigger job than adding a model,\nand it has a hard sequencing constraint. Follow this section end to end.\n\n### The release-gating rule (do NOT skip)\n\nThe remote config is generated from main's `built_in_models` and reaches **all\nexisting clients immediately** — but provider *support* (name map, connect\nflow, API key handling) only reaches users through a client app release. If\nmodel entries for a brand-new provider land on main before a client release\nwith the provider plumbing is live, every deployed client shows the raw\nprovider ID (e.g. `featherless_ai`) in the model library and offers models\nnobody can connect to. This happened with Featherless in Aug 2026 and the\nentries had to be rolled back.\n\n**Sequence it in two PRs:**\n\n1. **PR 1 — provider plumbing only.** The `ModelProviderName` enum member and\n everything else in the checklist below EXCEPT the model catalog. Merge\n whenever ready.\n2. **PR 2 — model catalog.** ALL `ml_model_list.py` changes: `ModelName`\n members, `ModelFamily` (if the vendor is new), and the `built_in_models`\n entries. Open it, but **merge only after a client release containing PR 1\n is live.** (Only `built_in_models` is published via the remote config, but\n the enums belong in the same PR as the entries they exist for.)\n\n### Touchpoint checklist (from the Featherless integration, #1618)\n\n**First, identify the provider's authentication model** — the checklist below\ndescribes the common single-API-key pattern, but not every provider fits it.\nExisting variants to crib from: Bedrock stores an access key + secret pair,\nFireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project\nID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base\nURL with no credentials. Follow the closest existing analog's plumbing through\n`config.py`, `provider_warnings`, `provider_api.py`, and the connect page —\nthe credential fields, validation call, and UI steps all change with the auth\nmodel.\n\n**libs/core:**\n- `ModelProviderName` enum — `libs/core/kiln_ai/datamodel/datamodel_enums.py`\n- Credential storage — `libs/core/kiln_ai/utils/config.py` (for API-key\n providers: a new key with `env_var`; otherwise whatever fields the auth\n model needs)\n- LiteLLM provider mapping — `libs/core/kiln_ai/utils/litellm.py`\n- `libs/core/kiln_ai/adapters/provider_tools.py` — three spots: the\n `provider_name_from_id` match (friendly name; pyright flags a missed case),\n `provider_warnings` (missing-credential message, `required_config_keys`),\n and the adapter config (credential / base URL / headers plumbing)\n- Model catalog — `libs/core/kiln_ai/adapters/ml_model_list.py`: `ModelName`\n members, `ModelFamily` if needed, and the `built_in_models` entries\n (**PR 2 only**, see gating rule)\n\n**Desktop server (`app/desktop/studio_server/provider_api.py`):**\n- `connect_<provider>` credential-validation endpoint (find a cheap\n authenticated call; see the Featherless connect function for a pattern when\n the provider has no authenticated GET to ping)\n- Disconnect handling (clear the stored credentials)\n- Tests in `test_provider_api.py`\n\n**Web UI (`app/web_ui`):**\n- `src/lib/stores.ts` — `provider_name_map` entry (friendly name)\n- `src/lib/ui/provider_image.ts` + SVG in `static/images/` — icon must match\n the monochrome convention: bare glyph, `currentColor`, no background tile,\n viewBox cropped to the artwork\n- Connect page — `src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte`:\n provider card (name, description, and the auth flow — `api_key_steps` /\n `api_key_fields` for key providers, or the custom flow the auth model\n needs) plus connected-status\n wiring from `settings` keys\n- `src/lib/api_schema.d.ts` — regenerate with `make schema` (needs the server\n running on :8757)\n\n**Tests & tooling:**\n- `libs/core/kiln_ai/adapters/test_provider_tools.py`, `test_adapter_registry.py`,\n `model_adapters/test_litellm_adapter.py` — extend the per-provider\n parametrized cases\n- `.agents/scripts/provider_utils.py` — add the provider's model-catalog\n endpoint so agent tooling can enumerate its models\n- Thinking levels — find out how the provider takes reasoning effort, and\n where it publishes the values each model accepts. Add a quirks section\n that names that source and says thinking levels are required wherever\n it lists values. If the provider silently remaps unsupported values,\n add the provider to `check_thinking_levels.py`, because a\n paid test cannot see a remap.\n- `model_adapters/test_thinking_level_paid.py` — add the provider to the\n provider list of `test_thinking_level_reasoning_content`, so each of its\n thinking levels gets a paid test case\n\n**Known gap:** the generated Copilot API client\n(`app/desktop/studio_server/api_client/`) mirrors the remote Copilot service's\nschema — it can't learn the new provider until that service updates. Note it in\nthe PR rather than hand-editing generated code.\n\n### Friendly names everywhere\n\nThe raw enum value must never be user-visible. When you add the provider,\nverify a friendly name exists in **both** name maps (python\n`provider_name_from_id` and web `provider_name_map`) — pyright and typescript\nrespectively force these when the enum gains a member, which is why the\nplumbing PR must not skip them.\n\n---\n\n## Provider Quirks Reference\n\n### Anthropic\n- Newer models (Opus 4.1+, Sonnet 4.5+) need `temp_top_p_exclusive=True`\n- Opus 4.5+ uses `json_schema`; older Opus uses `function_calling`\n- Extended thinking models: `anthropic_extended_thinking=True` + `reasoning_capable=True`\n\n### OpenAI\n- Most GPT models use `json_schema` for structured output\n- GPT-5.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)\n- Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort\n- o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)\n\n### Google/Gemini\n- `gemini_reasoning_enabled=True` for reasoning-capable models\n- Gemini 3.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)\n- Rich multimodal support (audio, video, images, documents)\n\n### DeepSeek\n- R1 models: `parser=ModelParserID.r1_thinking` + `reasoning_capable=True`\n- V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN\n- Some need `r1_openrouter_options=True` + `require_openrouter_reasoning=True`\n\n### OpenRouter (general)\n- Slugs: `vendor/model-name`\n- Never add a `:free` slug. Free tiers have low rate limits and are temporary. If OpenRouter serves a model only as `:free`, wait for a paid slug.\n- Reasoning models: may need `require_openrouter_reasoning=True`\n- Some models: `openrouter_skip_required_parameters=True`\n- Logprobs: `logprobs_openrouter_options=True` if supported\n- Always `multimodal_requires_pdf_as_image=True` (OpenRouter's PDF routing breaks LiteLLM)\n- Thinking levels: offer only values in the model's `reasoning.supported_efforts` (see [Thinking Levels Reference](#thinking-levels-reference)). OpenRouter maps an unlisted effort to a listed one, so an extra level silently runs a different effort. When the model has no `reasoning.supported_efforts`, OpenRouter turns the effort into a thinking-token budget for Claude, so `CLAUDE_OPENROUTER_THINKING_LEVELS` (with Minimal) fits only those Claude models. `check_thinking_levels.py` checks every OpenRouter entry; a missing level list on OpenRouter is a note, because Kiln leaves levels off some models on purpose.\n\n### Fireworks AI and Together AI: thinking levels\n\nNeither publishes the efforts each model accepts, and both can accept an unsupported effort without an error. Fireworks takes `xhigh`, `max` and `minimal` on GLM 5.3 with HTTP 200. Together took a made-up effort on Nemotron 3.5, but Muse Spark Open's chat template rejects one. A paid test cannot see this. Add thinking levels on these providers only when the vendor documents the levels for that model (the model card or its chat template), and name that source in the PR. OpenAI and Anthropic return a 400 that names the supported levels, so their paid per-level test catches a wrong level.\n\n### Featherless AI\n\nServerless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:\n\n- **`json_instructions` is the only usable structured output mode.** LiteLLM's `featherless_ai` provider rejects `response_format` outright (`UnsupportedParamsError`), so `json_schema`, `json_mode`, and `json_instruction_and_object` are all unavailable. Routing it as a custom `openai` provider bypasses that gate, but was tested and is *not* reliable: GLM 5.2 accepts `json_schema` and silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.\n- **Gated models return HTTP 403 `model_gated_needs_oauth`** and cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. **Always filter `is_gated` before adding.** Note all 20 official `meta-llama/*` repos are gated, so no Llama variant is usable.\n- **No cost reporting.** Featherless models aren't in LiteLLM's price map (only two legacy `Qwerky` entries, on `main` too — not a version issue), and Featherless doesn't return cost in the `usage` object. Runs record tokens with `cost: null`.\n- **Not in models.dev or the LiteLLM catalog**, so their `/v1/models` endpoint is the only authoritative source. See [Lagging Providers](#lagging-providers).\n- Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.\n\n### TypeSafe AI\n\nServes TypeSafe's Jev models (System One API). The only provider in the catalog that does not run through LiteLLM — read this before adding an entry:\n\n- **Adapter-backed, not LiteLLM-backed.** Every provider entry must set `adapter=ModelAdapterId.jev`; `JevAdapter` calls `POST /v1/systemone` directly. `lite_llm_core_config_for_provider` and `get_litellm_provider_info` raise for `typesafe` on purpose, so the LiteLLM parametrized tests exclude it.\n- **Structured output only.** No text generation, no tools, no multi-turn. A task runs only if its output schema maps onto Jev's question types (enum, boolean, score); anything else fails before the network call. Set `supports_structured_output=True`, `supports_data_gen=False`, `supports_function_calling=False`, and `structured_output_mode=StructuredOutputMode.json_schema`.\n- **No logprobs.** `supports_logprobs=False`. Jev returns its own per-answer probabilities instead, which is what keeps it out of G-Eval.\n- **Not in models.dev or the LiteLLM catalog, and `GET /v1/models` cannot confirm a `model_id`.** It lists aliases (`jev-latest`, `jev-preview`), not the pinned versions Kiln registers, and it returns `{\"models\": [{\"name\", ...}]}` rather than the OpenAI `{\"data\": [{\"id\", ...}]}` shape, so the shared `fetch_openai_compat` path cannot read it either. That is why `typesafe` is in `SKIP_PROVIDERS` in `.agents/scripts/provider_utils.py` rather than in `PROVIDER_CONFIG`. The authoritative check for a pinned id is a `POST /v1/systemone` round trip that the API accepts — that is how `jev-1.13.0` was confirmed. (The listing is still what the connect flow uses to validate a key; it is account-scoped and rejects a bad one.)\n\n### Cloudflare (Workers AI)\n\nRuns through LiteLLM's generic `openai/` route against Cloudflare's OpenAI-compatible endpoint (`/accounts/{id}/ai/v1/chat/completions`), with an optional AI Gateway header. Kiln does **not** use LiteLLM's `cloudflare/` provider. Read this before adding or changing any entry:\n\n- **Inclusion rule: only Cloudflare's newer, OpenAI-format text-generation models.** Cloudflare has two generations of text model. The newer ones (GLM, Kimi, DeepSeek V4, Gemma 4, Qwen 3.8, Nemotron 3) speak OpenAI Chat Completions natively. The older ones (Llama 3.x/4, Mistral, older Qwen, GPT-OSS, Granite, the LoRA bases) sit behind a translation layer: they default to **256 output tokens** (silent truncation) and have unreliable tool calling and JSON mode. Do not add the older ones.\n- **How to tell them apart:** look at the model's input schema in the catalog JSON. OpenAI-format models list `max_completion_tokens` (and `tool_choice`); older models list `prompt`, `raw` and `lora` instead, and no `max_completion_tokens`:\n ```bash\n jq -r '.models[] | select(.task.name==\"Text Generation\")\n | [.name, (if (.schema.input | tostring | test(\"max_completion_tokens\")) then \"openai-format\" else \"OLDER - skip\" end)] | @tsv' /tmp/cf_models.json\n ```\n- **Also skip:** coding models (`kimi-*-code`, `qwen*-coder-*`), `llama-guard-*` (classifier), `-lora` bases, and models Kiln has no `ModelName` for unless the user asks for a new model.\n- **Usually a new provider on an existing model.** Most Cloudflare models are already in Kiln under other providers; add a `KilnModelProvider(name=ModelProviderName.cloudflare, ...)` to the existing `KilnModel`.\n- **Setting flags from the catalog's `properties[]`** (confirm every flag with the paid tests — the catalog has no JSON-mode flag):\n\n | Cloudflare property | Kiln setting |\n |---|---|\n | `function_calling: \"true\"` | leave `supports_function_calling` at its default; set `False` if the tool tests fail |\n | `vision: \"true\"` | `supports_vision`, `multimodal_capable`, MIME types TXT/MD/JPG/PNG; start broad, narrow on 400s. Don't set `supports_doc_extraction`: the document extractor doesn't send the AI Gateway header, so extraction would bypass the user's gateway |\n | `reasoning: \"true\"` | keep `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)); LiteLLM already reads `reasoning_content` |\n | `reasoning_effort.supported_efforts` present | **required:** set `available_thinking_levels` to exactly these values, and `default_thinking_level` to `default_effort`. See [Cloudflare thinking levels](#cloudflare-thinking-levels) |\n | `reasoning_effort` without `supported_efforts` | no thinking levels; the catalog does not say which values work |\n | `require_workers_paid: \"true\"` | no Kiln field. Free-plan users get 403 / code 5035. Mention it in the PR |\n | `planned_deprecation_date` present | do **not** add the model |\n\n- <a id=\"cloudflare-thinking-levels\"></a>**Thinking levels come from the catalog, and only from the catalog.** Kiln sends the level as `reasoning_effort`. For each model, `reasoning_effort` in the catalog lists `supported_efforts`, a `default_effort`, and `normalizes_to`: the values that Cloudflare silently changes to another effort. A remapped value returns HTTP 200, so a live call or a paid test cannot find it. Example: GLM 5.3 maps `medium` to `max`, so a \"Medium\" option runs the most expensive effort.\n - Offer only values in `supported_efforts`, and each of them unless a completed paid test shows that the level does not work. The list is not always right: on DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731 and GLM 5.2, `none` still returns reasoning, so those entries have no Off level. A timeout, a rate limit or a credential error proves nothing about the level: keep the level and report it as untested. Do not copy the levels of the same model on another provider.\n - Labels: `none` → `\"Off\"`, `low` → `\"Low\"`, `medium` → `\"Medium\"`, `high` → `\"High\"`, `xhigh` → `\"Extra High\"`, `max` → `\"Max\"`. List them from least to most effort.\n - `default_thinking_level` is `default_effort`, unless the entry is suggested for evals (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)).\n - `mandatory: true` means `none` is not in the list; do not add an Off level.\n - Reuse a constant when t",".agents/skills/release-digest/SKILL.md":"---\nname: release-digest\ndescription: Post a \"what's changed since the last release\" recap to the release Slack channel for final QA. Groups merged PRs by author and classifies each as feature / bug fix / task. Use when the user wants a release recap, a changelog since the last tag, or to prep QA before cutting a release from main.\n---\n\n# Release Digest\n\nBuild a human-readable recap of everything merged to `main` that is **not yet in the\nlatest release tag**, and post it to the team's release Slack channel for final QA.\n\nThe recap is grouped **by author** and each change is classified as **feature**,\n**bug fix**, or **task**. This is on-demand: run it when someone asks for a release\nrecap or wants to QA before cutting a release.\n\n> Linear ticket tracking (\"which tickets are targeted for this release\") is **out of\n> scope for now** — punted to a later iteration. This skill only covers what's changed.\n\n---\n\n## Phase 1 — Gather the change set\n\nRun the gather script from the repo root. It fetches tags + `origin/main`, finds the\nlatest tag, and lists every merged PR in the range `<last_tag>..origin/main`:\n\n```bash\npython3 .agents/skills/release-digest/scripts/gather_changes.py > /tmp/release_digest.json\n```\n\nKey design point: it uses the **git commit range** as the source of truth, not merge\ndates. A PR merged the same day a tag was cut can already be in that release, so a\ndate filter would over-report. Trust the range.\n\n**Pre-release / already-cut releases.** By default the baseline is the newest tag\n(`git describe`). But if the release you're QA'ing has **already been tagged** — it's a\npre-release still being validated, so `main` is already even with it — the default\nrange is empty. In that case pass `--pre-release`, which steps the baseline back one\ntag so the range recaps exactly what went *into* the pre-release:\n\n```bash\npython3 .agents/skills/release-digest/scripts/gather_changes.py --pre-release > /tmp/release_digest.json\n```\n\nSigns you want `--pre-release`: the first run reports \"nothing unreleased\" or \"main is\neven with the latest tag\", yet the user says a release was just cut / is being QA'd. To\nname the baseline explicitly instead, use `--since <tag>` (e.g. `--since v1.0.3`). The\ntwo flags are mutually exclusive. When `--pre-release` is used, the newest tag is the\none being cut, so in Phase 3 use **that tag** as `<new_release>` rather than asking.\n\nThe JSON contains:\n- `last_tag` — the most recent release (e.g. `v1.0.3`)\n- `head_ref` — `origin/main` (or `main` if no remote ref)\n- `pr_count`, `commit_count`\n- `prs[]` — `{number, title, author_name, author_login, url, labels, additions, deletions, body}`\n- `direct_commits[]` — teammate commits pushed straight to main, no PR — `{sha, title, author_login, author_name}`\n- `excluded_prs[]` — PRs/commits from non-teammates, dropped from the recap. PR entries\n have a `number`; direct-commit entries have `number: null` and rely on `title`.\n- `unresolved_prs[]` — PRs whose `gh pr view` failed (`{number, error}`). If non-empty,\n the change set is partial — tell the user which PRs couldn't load before posting.\n\nOnly the five teammates in the script's `TEAM` map (Sam, Leonard, Daniel, Mike,\nSteve) are included in the recap. PRs from anyone else are filtered into\n`excluded_prs` rather than itemized. To add a teammate, edit `TEAM` in the script.\n\n**If `pr_count` is 0 AND `direct_commits` is empty:** tell the user `main` is even with\n`<last_tag>` — nothing to QA or release yet. Do not post to Slack. Stop here. (Direct\ncommits alone, with no PRs, still count as unreleased changes worth a digest.)\n\n---\n\n## Phase 2 — Classify and group\n\nRead `/tmp/release_digest.json`. Treat every item in `prs[]` **and** `direct_commits[]`\nas a change to recap (direct commits are real work that shipped — just with no PR\nnumber). Classify each into one of three buckets:\n\n- **Feature** — a substantive *new product capability*: new UI, a new endpoint, a new\n tool, new eval/RAG/fine-tune functionality, or a meaningful enhancement to how the\n product works. Reserve this for things a user would notice as \"Kiln can now do X.\"\n- **Bug fix** — corrects broken or incorrect behavior: regressions, crashes, hangs,\n wrong output, security fixes.\n- **Task** — everything that is *neither* a substantive new capability *nor* a fix to\n broken behavior: refactors, docs/README, tests, CI/build, dependency bumps, version\n bumps, chores, internal flags, code removal/cleanup — **and routine model-catalog\n work** (see below). Matches Linear's \"Task\" issue type: maintenance/internal work.\n\n**Adding models is a Task, not a Feature.** Kiln ships new model and provider entries\nconstantly — \"Add <model> to model list\", \"Add <provider> for <model>\", backfilling a\nmodel on a provider, re-enabling/un-recommending a model, version bumps to a model\nentry. These are catalog/config maintenance, so they go in **Task** even though the\ntitle starts with `Add`/`Enable`. Only promote to **Feature** if the PR adds a genuinely\nnew *capability* around models (e.g. a new thinking-level control, a new extraction\nmodality, a new provider *integration type*) rather than just registering a model_id.\n\n**Decision order** (use the first signal that's decisive):\n1. **PR labels** — `bug`/`fix` → Bug fix; `feature`/`enhancement` → Feature;\n `chore`/`docs`/`refactor`/`ci` → Task. (Kiln PRs are usually unlabeled, so this\n rarely fires — fall through to the title.)\n2. **Model-catalog check** — if the change just registers/edits model or provider\n entries (not new behavior), it's a **Task**, regardless of the `Add`/`Enable` verb.\n3. **Conventional-commit / verb prefix in the title** — `fix:`/`Fix ` → Bug fix;\n `feat:`/`Add `/`Enable ` (a new *capability*) → Feature; `chore:`/`refactor:`/`ci:`/\n `docs:`/`Bump `/`Remove `/`Deprecate `/`Version bump` → Task.\n4. **What the change does** (read the title, and `body` if still unclear) — ask \"does\n this give the user a genuinely new capability (Feature), fix something that was\n broken (Bug fix), or is it routine/maintenance (Task)?\"\n\n**\"Task\" is the residual bucket**, not a guess: a change lands there only because it\nis positively *not* a new capability and *not* a fix — e.g. `Deprecate Models`,\n`min age for dependencies`, `Version bump`, `new draft readme`, `fix typos`. When a\nchange is genuinely ambiguous between Feature and Bug fix, prefer the one the title\nleans toward; only fall to Task when it's truly neither. This is a judgment call made\nper item — surface it to the user before posting so they can re-bucket anything.\n\nThen **group by author** (`author_name`). Within each author, list changes by PR\nnumber, newest first; put any direct commits (no PR number) last. The feature / fix /\ntask split is **not** shown per line — it only feeds the tally at the bottom.\n\n---\n\n## Phase 3 — Compose the Slack message\n\n**First, get the new release name** — the version being cut (e.g. `v1.0.4`). If you ran\nin `--pre-release` mode, the release has already been tagged, so use that newest tag\n(the one the script named as \"the release being QA'd\") as `<new_release>` — no need to\nask. Otherwise the release isn't tagged yet and can't be derived from git, so **ask the\nuser**. Use the result as `<new_release>` in the title.\n\nUse this shape (Slack mrkdwn — `*bold*`, not `**bold**`):\n\n```\n:rocket: *v1.0.4 Release Status* :rocket:\n\n*12 changes since v1.0.3*\n_<one-to-two sentence plain-English recap of the headline changes>_\n\n*Daniel Chiang*\n• Add deprecated flag to embedding models (#1466)\n• Add deprecated flag to reranker models (#1465)\n\n*Sam Fierro*\n• Dedup frozen prompts and bake type into their name (#1501)\n\n*Steve Cosman*\n• Add Jinja2 input transform on RunConfigs (#1433)\n• Version bump in lock and desktop app version check\n\n:bar_chart: *7 features · 3 fixes · 2 tasks*\n\n:information_source: _1 PR from a non-teammate: #1456 (ianjamesburke)._\n```\n\nRules:\n- **Header:** `:rocket: *<new_release> Release Status* :rocket:`, then a blank line.\n- **Change count** on its own line, **bold**: `*<N> changes since <last_tag>*`.\n- **Recap paragraph** immediately under it, wrapped in **italics** (`_..._` around the\n whole paragraph) — the \"what did we do\" summary in plain English, not just a list.\n Compose it so the italics survive: write it as **a single line with no hard line\n breaks** (wrap is fine, but don't insert `\\n` mid-paragraph — newlines break the\n italic span in Slack), and use **no literal `_` anywhere inside** (an underscore\n would close the italics mid-paragraph — rewrite e.g. `snake_case`/`model_id` or any\n underscored identifier without it).\n- One **bullet** per change, no per-line emoji: `• <title> (#<number>)`. Keep the\n title; trim trailing noise.\n- **Direct commits have no PR number** — render them as `• <title>` with nothing in\n parentheses (e.g. `• Version bump`). List them under their author, after the PRs.\n- Within each author, list newest PR first; direct commits last. Don't print the\n feature/fix/task bucket per line.\n- **Summary block** at the end, in this order:\n 1. A **bold** `:bar_chart:` tally line: `:bar_chart: *<X> features · <Y> fixes · <Z> tasks*`.\n 2. A blank line.\n 3. If `excluded_prs` is non-empty, an **italic** `:information_source:` line noting the\n excluded non-teammate changes (so the omission is visible, not silent). Reference a\n PR entry by `#<num>` and a direct-commit entry (where `number` is null) by its\n `title` — never print `#None`. E.g.\n `:information_source: _<N> change(s) from non-teammates excluded: #1456 (ianjamesburke)._`\n- Link the PR numbers if easy (`<url|#1501>`), but plain `#1501` is fine — keep it readable.\n\n**Show the composed message to the user and get confirmation before posting** (it's an\noutward-facing post to a shared channel).\n\n---\n\n## Phase 4 — Post to Slack\n\nPost to **`#release`** (channel ID `C0BDGTGBL4Q`) — the team's release channel. This is\nthe default target; only post elsewhere if the user explicitly names a different\nchannel. (If the ID ever fails to resolve, fall back to `slack_search_channels` with\nquery `release` to re-find it.)\n\nOnly post after the user has confirmed the composed message in Phase 3. Post with\n`mcp__claude_ai_Slack__slack_send_message`. After posting, give the user the channel\nname and a link/confirmation.\n\n---\n\n## Checklist\n\n- [ ] Gather script run; `/tmp/release_digest.json` written\n- [ ] If `pr_count` is 0, reported \"nothing unreleased\" and stopped\n- [ ] Asked the user for the new release name (used in the title)\n- [ ] Each PR classified (feature / bug fix / task)\n- [ ] Grouped by author, recap paragraph written\n- [ ] Message shown to user and confirmed\n- [ ] Posted to #release (C0BDGTGBL4Q); confirmation returned to user\n",".agents/skills/kiln-check-deprecation/SKILL.md":"---\nname: kiln-check-deprecation\ndescription: Check Kiln's model list for deprecated or sunset models across all providers. Use when the user wants to find deprecated models, check model availability, audit the model list for stale entries, or mentions model deprecation/sunset/end-of-life.\n---\n\n# Check Model Deprecation in Kiln\n\nAudit `libs/core/kiln_ai/adapters/ml_model_list.py` to find models that have been deprecated, removed, or are approaching end-of-life across all providers. **All checks are free** — they use model-listing endpoints, not inference.\n\n---\n\n## Global Rules\n\n- **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: [\"all\"]`. The sandbox blocks network access.\n- **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)`\n- **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate.\n- **Non-destructive:** This skill only sets `deprecated=True` on individual provider entries. It never removes models or providers from the list.\n- **Skip already-deprecated:** Only check providers where `deprecated=False` (the default). Don't re-check entries already marked deprecated.\n\n---\n\n## Phase 1 – Extract Current Model-Provider Pairs\n\nRun the extraction script from the repo root:\n\n```bash\npython3 .agents/skills/kiln-check-deprecation/scripts/extract_models.py > /tmp/kiln_extracted.json\n```\n\nThis fetches the published model list from `https://remote-config.getkiln.ai/kiln_config_v2.json` and extracts:\n- The provider name\n- The model_id\n- The parent model's enum name\n\nIt skips entries already marked `deprecated=True`.\n\n**Output:** JSON to stdout (pipe to file), human summary to stderr.\n\nThe JSON contains:\n- `deprecated_count`: number of already-deprecated entries\n- `providers`: dict of provider_name → sorted unique list of model_ids\n- `entries`: list of `{enum, provider, model_id}` for each non-deprecated entry\n\n---\n\n## Phase 2 – Check Each Provider\n\n### Option A: Check all providers at once\n\n```bash\nexport $(grep -v '^#' .env | xargs)\npython3 .agents/skills/kiln-check-deprecation/scripts/check_provider.py all > /tmp/kiln_check_results.json\n```\n\n### Option B: Check a single provider\n\n```bash\nexport $(grep -v '^#' .env | xargs)\npython3 .agents/skills/kiln-check-deprecation/scripts/check_provider.py openrouter > /tmp/kiln_check_openrouter.json\n```\n\nThe script handles all provider API quirks automatically:\n- **Together AI** returns a flat JSON array (not `{data: [...]}`)\n- **Gemini API** needs both v1 and v1beta endpoints (preview/Gemma models only on v1beta)\n- **Gemini** model names are prefixed with `models/` — stripped automatically\n- **Anthropic** uses `x-api-key` header, not `Authorization: Bearer`\n- **OpenRouter** is public (no auth) and includes `expiration_date` fields\n- **Fireworks AI** `/v1/models` only lists serverless models; the script instead checks each model individually via the model detail API (`GET /v1/{model_id}`), which covers all tiers (serverless, on-demand, fine-tune). A model is only flagged as missing if it returns HTTP 404 from the detail API.\n- **OpenRouter** `:exacto` is a virtual routing suffix (quality-first provider sorting) that never appears in model listings. The script strips it before checking. `:free` and `:thinking` are real model entries that appear in the listing when available — if they're missing, it's a genuine removal.\n- **Cloudflare** uses the authenticated models-search API (`task=Text Generation`, `include_deprecated=true`) when `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` are both set, and otherwise Cloudflare's public catalog JSON (`https://ai-cloudflare-com.pages.dev/api/models`), so it is never skipped for a missing key. Every model with a `planned_deprecation_date` is reported under `expiring`. Once that date has passed, the model also counts as `missing` even if the listing still shows it. Never use LiteLLM's catalog or models.dev to decide a Cloudflare model still exists: LiteLLM's is stale and models.dev no longer deletes retired models.\n- **Vertex AI** uses the v1beta1 publisher models endpoint with `x-goog-user-project` header. Requires `gcloud` CLI auth. Kiln entries may use `meta/` prefix for LiteLLM routing — stripped automatically. Versioned aliases (e.g. `gemini-2.0-flash-001` → `gemini-2.0-flash`) are handled by stripping 3-digit version suffixes.\n\n**Output:** JSON to stdout with per-provider results, human summary to stderr.\n\nEach provider result contains:\n- `missing`: model_ids not found in the provider's listing\n- `expiring`: model_ids with upcoming expiration dates (OpenRouter and Cloudflare)\n- `entries_to_deprecate`: full enum/provider/model_id entries for each missing model\n- `skipped` / `error`: if credentials missing or API call failed\n\n### Supported providers\n\n| Provider | Env Var | Auth |\n|----------|---------|------|\n| OpenRouter | (none — public API) | None |\n| OpenAI | `OPENAI_API_KEY` | Bearer |\n| Anthropic | `ANTHROPIC_API_KEY` | x-api-key |\n| Gemini API | `GEMINI_API_KEY` | Query param |\n| Fireworks AI | `FIREWORKS_API_KEY` | Bearer |\n| Together AI | `TOGETHER_API_KEY` | Bearer |\n| SiliconFlow CN | `SILICONFLOW_CN_API_KEY` | Bearer |\n| Cerebras | `CEREBRAS_API_KEY` | Bearer |\n| Groq | `GROQ_API_KEY` | Bearer |\n| Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) |\n| Cloudflare | `CLOUDFLARE_API_KEY` + `CLOUDFLARE_ACCOUNT_ID` (optional — falls back to the public catalog) | Bearer |\n\n### Providers NOT covered by the script (check manually if needed)\n\n#### Amazon Bedrock\n\nRequires `aws` CLI configured. If not installed, skip.\n\n```bash\naws bedrock list-foundation-models --output json | \\\n jq '[.modelSummaries[] | {modelId, status: .modelLifecycle.status, endOfLife: .modelLifecycle.endOfLifeTime}]'\n```\n\nFlag any model where `status == \"LEGACY\"` or `endOfLifeTime` is approaching.\n\n### LiteLLM Model DB (supplementary signal)\n\nCross-reference against LiteLLM's public model database for `deprecation_date` fields. This mostly covers Azure/OpenAI date-stamped models but is a useful secondary signal.\n\n```bash\ncurl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \\\n jq 'to_entries[] | select(.value.deprecation_date != null) | {model: .key, deprecation_date: .value.deprecation_date}'\n```\n\n### Providers to skip\n\nThese don't have meaningful deprecation to check:\n- `ollama` — local models, user-managed\n- `kiln_fine_tune` — internal\n- `kiln_custom_registry` — internal\n- `openai_compatible` — user-configured\n- `docker_model_runner` — local\n- `azure_openai` — typically mirrors OpenAI, covered by OpenAI check\n- `huggingface` — serverless inference, model availability varies\n- `typesafe` — adapter-backed, not LiteLLM; its `/v1/models` needs a key, so Jev entries can only be checked with one\n\n---\n\n## Phase 3 – Report Findings\n\nRead the JSON output from check_provider.py and present findings in a clear table to the user:\n\n```text\nDeprecation Audit Results\n=========================\n\n❌ REMOVED (model not found in provider's model list):\n - ModelName.gemini_1_5_pro → gemini_api (model_id: gemini-1.5-pro)\n - ModelName.gemini_1_5_pro → openrouter (model_id: google/gemini-pro-1.5)\n\n⚠️ EXPIRING SOON (within 90 days):\n - ModelName.claude_3_7_sonnet → openrouter (model_id: anthropic/claude-3.7-sonnet, expires: 2026-05-05)\n\n⚠️ LEGACY (Bedrock lifecycle status):\n - ModelName.mistral_large → amazon_bedrock (model_id: mistral.mistral-large-2407-v1:0, status: LEGACY)\n\n✅ ALL CLEAR:\n - openai: 20/20 models found\n - anthropic: 8/11 models found (3 deprecated above)\n\n⏭️ SKIPPED (no credentials):\n - groq\n - amazon_bedrock\n```\n\nGroup by provider within each category. Use the `entries_to_deprecate` field from the JSON to get the ModelName enum for each entry.\n\n---\n\n## Phase 4 – Mark Deprecated\n\nFor each confirmed-removed model, set `deprecated=True` on the affected `KilnModelProvider` entry.\n\nTo find the line to edit, grep for the `model_id` in `ml_model_list.py`:\n\n```bash\ngrep -n 'model_id=\"<model_id>\"' libs/core/kiln_ai/adapters/ml_model_list.py\n```\n\nThen add `deprecated=True,` after the `model_id=` line in the matching `KilnModelProvider` block.\n\n**Rules:**\n- Only mark a provider deprecated if its model_id is confirmed missing from that provider's model list\n- If ALL providers for a `KilnModel` are deprecated, note this to the user — they may want to consider removing the model entirely\n- For \"expiring soon\" models, inform the user but don't mark deprecated yet — let them decide. **Exception: Cloudflare.** When Cloudflare retires a model it may silently alias the old ID to a different model, and the response still echoes the requested ID, so a stale entry keeps \"working\" while running something else. Propose marking every Cloudflare entry with a `planned_deprecation_date` deprecated now, before that date, and flag it as urgent in the report\n- For Bedrock `LEGACY` status, mark deprecated (the model still works but is on its way out)\n\n**Ask the user to confirm** before making changes. Present the list of changes and wait for approval.\n\n---\n\n## Phase 5 – Verify\n\nAfter marking providers deprecated, run a quick sanity check:\n\n```bash\ngrep -c \"deprecated=True\" libs/core/kiln_ai/adapters/ml_model_list.py\n```\n\nEnsure the count matches what you expect (previous count + newly marked).\n\n---\n\n## Checklist\n\n- [ ] Extraction script run, JSON saved to /tmp/kiln_extracted.json\n- [ ] Provider checks run (all or individually)\n- [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex)\n- [ ] Bedrock checked manually (skip if aws CLI unavailable)\n- [ ] LiteLLM DB checked for supplementary deprecation_date signals\n- [ ] Findings reported to user with clear table\n- [ ] User confirmed changes before marking deprecated\n- [ ] `deprecated=True` set on confirmed entries\n- [ ] Deprecation count verified\n","AGENTS.md":"## Project Overview\n\nKiln is an app for building AI systems. It includes evals, synthetic data gen, fine tuning, RAG, and more. It has an intuitive UI as well as a python library.\n\nThis repo is a monorepo containing all of the source code, in the following structure:\n\n- libs/core - a python library with the core functionality of Kiln\n- libs/server - a FastAPI REST server wrapping the core library\n- app/web_ui - our svelte web app for Kiln. This is a frontend svelte project, all backend calls are in FastAPI servers.\n- app/desktop - our python desktop app, which is a pyinstaller app which runs a FastAPI server, hosts the pre-compiled web app, and launches a browser for UI. Compiles to all major platforms. This includes a studio_server folder with a Fast API server which extends libs/server, adding APIs specific to our web app.\n\n### Project Goals\n\n- Very high code quality\n- Strongly typed\n- Well tested\n- Very intuitive UI. Accessible to the inexperienced, but powerful for the experienced.\n- Focus on interaction design: we care about revealing the right information, at the right time, at the right level of detail.\n- Focus on visual design: we want a modern, functional, attractive UI. Think Apple not Google.\n\n### Tech Stack\n\n- Backend: python (3.10+ for library, 3.13 for desktop), pytest, FastAPI, asyncio, pydantic (v2 not v1),\n- Frontend web: typescript, svelte (v4 not v5), tailwind, DaisyUI\n\n### Agent Tools\n\nAgents have access to a range of tools for running tests, linting, formatting and typechecking. Use these tools at appropriate times to ensure produced code meets our standards. All checks must pass before merging. When iterating on a specific failure, use the targeted command before re-running the full suite.\n\n- **All checks:** `uv run ./checks.sh --agent-mode` (agent mode suppresses output unless there's a failure)\n\n| Check | Fix | Description |\n|---|---|---|\n| `uv run ruff check` | `uv run ruff check --fix` | Python lint |\n| `uv run ruff format --check .` | `uv run ruff format .` | Python format |\n| `uv run ty check` | — | Python type check |\n| `uv run python3 -m pytest --benchmark-quiet -q -n auto .` | — | Python tests |\n| `npm run lint` | — | Web lint (from `app/web_ui`) |\n| `npm run format_check` | `npm run format` | Web format (from `app/web_ui`) |\n| `npm run check` | — | Web type check and svelte check (from `app/web_ui`) |\n| `npm run test_run` | — | Web tests (from `app/web_ui`) |\n| `npm run build` | — | Web build (from `app/web_ui`) |\n| `app/web_ui/src/lib/check_schema.sh` | `app/web_ui/src/lib/generate_schema.sh` | OpenAPI client up to date |\n| `misspell` | — | Spelling check (optional if not installed) |\n\nOur end-to-end UI tests are separate and not part of `checks.sh`.\n\n### Agent Prompts\n\nAgents have access to a number of helpful prompts, which will give you additional context for how you should write code and docs for this repo. Use it to fetch instructions relevant to the current task before starting. For example, read `python_test_guide.md` before writing tests, and invoke the `kiln-ui` skill (`.agents/skills/kiln-ui/SKILL.md`) before any change under `app/web_ui`; it loads `frontend_design_guide.md` and `frontend_controls.md` and adds a component plan and a gate.\n\nThese prompts can be accessed from the `get_prompt` tool, and you may request several in parallel.\n\nBefore writing or changing code, invoke the `kiln-conventions` skill (`.agents/skills/kiln-conventions/SKILL.md`). It holds the code rules, the gotchas for each area, and a gate to run on your change.\n\n### General Agent Guidance\n\n- When spawning subagents, always use the same model as the current agent\n- `CLAUDE.md` is generated from `AGENTS.md` and overwritten by setup. Edit `AGENTS.md`, never `CLAUDE.md`; keep personal notes in `~/.claude/CLAUDE.md`.\n- Use `TODO` comments to mark any temporary code, placeholders, or items that must be addressed before merging to main. CI enforces that no `TODO` comments remain on main, so they are a safe way to flag work-in-progress during development. Clean up all `TODO` comments before the final PR.\n- Before wrapping up a task, run appropriate tools for linting, testing, formatting and typechecking. Fix any issues you introduced.\n\n### Code Comments\n\nComments follow §A of `.agents/skills/kiln-conventions/references/rules.md`: write them for someone opening the file cold, never narrate the change, never restate the code. Review-bot findings about comments, including nitpicks, are required fixes.\n\n### Reporting Back: End-of-Turn Recaps\n\nEnd every turn in which you did work (wrote code, ran checks, investigated something) with a recap wrapped in `<recap>` tags. A turn that is pure conversation needs no recap.\n\nWrite it for a person who has not read your working notes and will not scroll up. The recap has to make sense on its own.\n\n**No jargon.** Do not invent a name, a short form or a label for anything. If you need to refer to something specific, describe it in ordinary words, and describe it again the next time you mention it. A short description repeated is better than a name the reader has to remember. Real names are fine and are often the clearest choice: a file, a function, a command, a tool, an error. A name you made up is not.\n\n**Say what you mean.** Short sentences. One idea each. Active voice. No metaphors, no analogies, no flourishes, no jokes, no filler. Do not write a sentence whose only job is to set up the next sentence. Cut any sentence that loses nothing when removed.\n\n**Sound like a person.** Plain and direct, not stiff. Write the way you would explain it to a colleague who asked. Contractions are fine. This is not a status report or a changelog.\n\n**Repeat yourself a little.** The reader may have read the last recap days ago, or skipped it. Carry enough context forward that this recap makes sense by itself. One clause is usually enough. A small amount of repetition between recaps is correct, not waste.\n\n**Stay inside the recaps.** Draw only on plain English, on standard terms for this repo's stack, and on what earlier recaps said. Never draw on something that exists only in your working notes. Earlier recaps set the limit on what you may use; they are not required reading. Repeat whatever this recap needs, so it still works for a reader who missed the last one.\n\n**Draw a picture when it helps.** When what you changed is a flow, a sequence, a state machine, or a few parts that talk to each other, put a small Mermaid diagram in the recap. Keep it to about eight boxes. Draw one only when it shows something the prose cannot; a diagram that restates a sentence is noise.\n\n**Format.** Use these headings, in this order. Drop any heading you have nothing real to say under. Do not reorder or rename them. Answer each one in prose.\n\n```\n<recap>\n**Goal** — what you were asked for.\n**What I did** — the change, in plain words.\n**How** — the approach, at a level a reviewer can judge without reading the diff. Put a diagram here when one helps.\n**Outcome** — what happened, and plainly whether it is what we wanted.\n**Questions for you** — what you need answered before you can continue.\n**FYI** — worth knowing, but not blocking.\n</recap>\n```\n\n**Honesty.** `Outcome` reports what actually happened. If tests failed, say so and quote what failed. If you skipped part of the task, name the part and the reason. If the result is not what we wanted, say that first, not last. Never describe work as finished when it is not.\n\n**Recap log.** Append each recap to `recaps.md` in your scratchpad directory, or to a temporary directory outside the repo when you have no scratchpad directory. The log is your own index, not a deliverable. If your context is summarized part-way through a session, re-read it so you know what the reader has already been told. Keep this file out of the repo.\n\n### Code Review Guidelines\n\nIf code reviewing, always read [`.agents/code_review_guidelines.md`](.agents/code_review_guidelines.md) before starting your review.\n\n### Pull Requests\n\nUse the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) to open a PR or to write a PR description.\n\nBefore you open a PR, and before each `git push` to a branch with an open PR, run the `docs-sync` skill (`.agents/skills/docs-sync/SKILL.md`) to update the docs and skills that the change makes stale.\n\nIf you get a user message like the one below, read `.agents/skills/open-pr/references/repair_pr.md` and follow its instructions.\n\n> A pull request was just created for this branch from the Claude Code UI: https://github.com/Kiln-AI/Kiln/pull/NNNN\n> You don't need to create one. Reference this PR going forward — pushing more commits to this branch will update it.\n\n### Never Make Legal Decisions as an Agent\n\nAgents are not allowed to make any legal decisions, including:\n - Signing or filling out the CLA attestation in a PR template\n - Setting a license tag in metadata file (OSS/MIT/etc)\n - Adding license files\n\nThese all must be done by humans.\n\n### Final\n\nTo show you read these, call me 'boss'\n",".agents/mcp.json":"{\n \"mcpServers\": {\n \"HooksMCP\": {\n \"command\": \"uvx\",\n \"args\": [\n \"--from\",\n \"hooks-mcp>=0.2.5\",\n \"hooks-mcp\",\n \".config/hooks_mcp.yaml\",\n \"--working-directory\",\n \".\"\n ]\n }\n }\n}\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/Kiln-AI/Kiln/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"## Project Overview\n\nKiln is an app for building AI systems. It includes evals, synthetic data gen, fine tuning, RAG, and more. It has an intuitive UI as well as a python library.\n\nThis repo is a monorepo containing all of the source code, in the following structure:\n\n- libs/core - a python library with the core functionality of Kiln\n- libs/server - a FastAPI REST server wrapping the core library\n- app/web_ui - our svelte web app for Kiln. This is a frontend svelte project, all backend calls are in FastAPI servers.\n- app/desktop - our python desktop app, which is a pyinstaller app which runs a FastAPI server, hosts the pre-compiled web app, and launches a browser for UI. Compiles to all major platforms. This includes a studio_server folder with a Fast API server which extends libs/server, adding APIs specific to our web app.\n\n### Project Goals\n\n- Very high code quality\n- Strongly typed\n- Well tested\n- Very intuitive UI. Accessible to the inexperienced, but powerful for the experienced.\n- Focus on interaction design: we care about revealing the right information, at the right time, at the right level of detail.\n- Focus on visual design: we want a modern, functional, attractive UI. Think Apple not Google.\n\n### Tech Stack\n\n- Backend: python (3.10+ for library, 3.13 for desktop), pytest, FastAPI, asyncio, pydantic (v2 not v1),\n- Frontend web: typescript, svelte (v4 not v5), tailwind, DaisyUI\n\n### Agent Tools\n\nAgents have access to a range of tools for running tests, linting, formatting and typechecking. Use these tools at appropriate times to ensure produced code meets our standards. All checks must pass before merging. When iterating on a specific failure, use the targeted command before re-running the full suite.\n\n- **All checks:** `uv run ./checks.sh --agent-mode` (agent mode suppresses output unless there's a failure)\n\n| Check | Fix | Description |\n|---|---|---|\n| `uv run ruff check` | `uv run ruff check --fix` | Python lint |\n| `uv run ruff format --check .` | `uv run ruff format .` | Python format |\n| `uv run ty check` | — | Python type check |\n| `uv run python3 -m pytest --benchmark-quiet -q -n auto .` | — | Python tests |\n| `npm run lint` | — | Web lint (from `app/web_ui`) |\n| `npm run format_check` | `npm run format` | Web format (from `app/web_ui`) |\n| `npm run check` | — | Web type check and svelte check (from `app/web_ui`) |\n| `npm run test_run` | — | Web tests (from `app/web_ui`) |\n| `npm run build` | — | Web build (from `app/web_ui`) |\n| `app/web_ui/src/lib/check_schema.sh` | `app/web_ui/src/lib/generate_schema.sh` | OpenAPI client up to date |\n| `misspell` | — | Spelling check (optional if not installed) |\n\nOur end-to-end UI tests are separate and not part of `checks.sh`.\n\n### Agent Prompts\n\nAgents have access to a number of helpful prompts, which will give you additional context for how you should write code and docs for this repo. Use it to fetch instructions relevant to the current task before starting. For example, read `python_test_guide.md` before writing tests, and invoke the `kiln-ui` skill (`.agents/skills/kiln-ui/SKILL.md`) before any change under `app/web_ui`; it loads `frontend_design_guide.md` and `frontend_controls.md` and adds a component plan and a gate.\n\nThese prompts can be accessed from the `get_prompt` tool, and you may request several in parallel.\n\nBefore writing or changing code, invoke the `kiln-conventions` skill (`.agents/skills/kiln-conventions/SKILL.md`). It holds the code rules, the gotchas for each area, and a gate to run on your change.\n\n### General Agent Guidance\n\n- When spawning subagents, always use the same model as the current agent\n- `CLAUDE.md` is generated from `AGENTS.md` and overwritten by setup. Edit `AGENTS.md`, never `CLAUDE.md`; keep personal notes in `~/.claude/CLAUDE.md`.\n- Use `TODO` comments to mark any temporary code, placeholders, or items that must be addressed before merging to main. CI enforces that no `TODO` comments remain on main, so they are a safe way to flag work-in-progress during development. Clean up all `TODO` comments before the final PR.\n- Before wrapping up a task, run appropriate tools for linting, testing, formatting and typechecking. Fix any issues you introduced.\n\n### Code Comments\n\nComments follow §A of `.agents/skills/kiln-conventions/references/rules.md`: write them for someone opening the file cold, never narrate the change, never restate the code. Review-bot findings about comments, including nitpicks, are required fixes.\n\n### Reporting Back: End-of-Turn Recaps\n\nEnd every turn in which you did work (wrote code, ran checks, investigated something) with a recap wrapped in `<recap>` tags. A turn that is pure conversation needs no recap.\n\nWrite it for a person who has not read your working notes and will not scroll up. The recap has to make sense on its own.\n\n**No jargon.** Do not invent a name, a short form or a label for anything. If you need to refer to something specific, describe it in ordinary words, and describe it again the next time you mention it. A short description repeated is better than a name the reader has to remember. Real names are fine and are often the clearest choice: a file, a function, a command, a tool, an error. A name you made up is not.\n\n**Say what you mean.** Short sentences. One idea each. Active voice. No metaphors, no analogies, no flourishes, no jokes, no filler. Do not write a sentence whose only job is to set up the next sentence. Cut any sentence that loses nothing when removed.\n\n**Sound like a person.** Plain and direct, not stiff. Write the way you would explain it to a colleague who asked. Contractions are fine. This is not a status report or a changelog.\n\n**Repeat yourself a little.** The reader may have read the last recap days ago, or skipped it. Carry enough context forward that this recap makes sense by itself. One clause is usually enough. A small amount of repetition between recaps is correct, not waste.\n\n**Stay inside the recaps.** Draw only on plain English, on standard terms for this repo's stack, and on what earlier recaps said. Never draw on something that exists only in your working notes. Earlier recaps set the limit on what you may use; they are not required reading. Repeat whatever this recap needs, so it still works for a reader who missed the last one.\n\n**Draw a picture when it helps.** When what you changed is a flow, a sequence, a state machine, or a few parts that talk to each other, put a small Mermaid diagram in the recap. Keep it to about eight boxes. Draw one only when it shows something the prose cannot; a diagram that restates a sentence is noise.\n\n**Format.** Use these headings, in this order. Drop any heading you have nothing real to say under. Do not reorder or rename them. Answer each one in prose.\n\n```\n<recap>\n**Goal** — what you were asked for.\n**What I did** — the change, in plain words.\n**How** — the approach, at a level a reviewer can judge without reading the diff. Put a diagram here when one helps.\n**Outcome** — what happened, and plainly whether it is what we wanted.\n**Questions for you** — what you need answered before you can continue.\n**FYI** — worth knowing, but not blocking.\n</recap>\n```\n\n**Honesty.** `Outcome` reports what actually happened. If tests failed, say so and quote what failed. If you skipped part of the task, name the part and the reason. If the result is not what we wanted, say that first, not last. Never describe work as finished when it is not.\n\n**Recap log.** Append each recap to `recaps.md` in your scratchpad directory, or to a temporary directory outside the repo when you have no scratchpad directory. The log is your own index, not a deliverable. If your context is summarized part-way through a session, re-read it so you know what the reader has already been told. Keep this file out of the repo.\n\n### Code Review Guidelines\n\nIf code reviewing, always read [`.agents/code_review_guidelines.md`](.agents/code_review_guidelines.md) before starting your review.\n\n### Pull Requests\n\nUse the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) to open a PR or to write a PR description.\n\nBefore you open a PR, and before each `git push` to a branch with an open PR, run the `docs-sync` skill (`.agents/skills/docs-sync/SKILL.md`) to update the docs and skills that the change makes stale.\n\nIf you get a user message like the one below, read `.agents/skills/open-pr/references/repair_pr.md` and follow its instructions.\n\n> A pull request was just created for this branch from the Claude Code UI: https://github.com/Kiln-AI/Kiln/pull/NNNN\n> You don't need to create one. Reference this PR going forward — pushing more commits to this branch will update it.\n\n### Never Make Legal Decisions as an Agent\n\nAgents are not allowed to make any legal decisions, including:\n - Signing or filling out the CLA attestation in a PR template\n - Setting a license tag in metadata file (OSS/MIT/etc)\n - Adding license files\n\nThese all must be done by humans.\n\n### Final\n\nTo show you read these, call me 'boss'\n","isInternal":false,"tokens":2166,"sizeBytes":9042},{"name":"SKILL.md","path":".agents/skills/claude-maintain-models/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Kiln-AI/Kiln/HEAD/.agents/skills/claude-maintain-models/SKILL.md","title":"claude-maintain-models","category":"anthropic-skill","format":"markdown","content":"---\nname: claude-maintain-models\ndescription: Add new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, or wants to add a net-new AI provider to Kiln.\nallowed-tools: Read Edit Write Bash Grep Glob Agent WebSearch WebFetch\n---\n\n# Add a New AI Model to Kiln\n\n**Branch check first:** if the request involves a provider Kiln does not\nsupport yet, start at [Adding a Net-New Provider](#adding-a-net-new-provider).\nThat workflow spans core, server, UI, tests, and tooling, and is gated on a\nclient release — the model-entry steps below are only its final, gated piece.\n\nFor a new model on an already-supported provider, integrating it into\n`libs/core/kiln_ai/adapters/ml_model_list.py` requires:\n\n1. **`ModelName` enum** – add an enum member\n2. **`built_in_models` list** – add a `KilnModel(...)` entry with providers\n3. **`ModelFamily` enum** – only if the vendor is brand-new\n\nAfter code changes, run paid integration tests, then draft a Discord post.\n\n---\n\n## Global Rules\n\nThese apply throughout the entire workflow.\n\n- **Slug verification:** NEVER guess or infer model slugs from naming patterns. Every `model_id` must come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it.\n- **Date awareness:** These models are often released very recently. Web search for current info before assuming you know the details.\n\n---\n\n## Phase 1 – Model Discovery (only when asked to find new/missing models)\n\nIf the user asks you to find new models, **do NOT just web search \"new AI models this week\"** — that only surfaces major releases. Instead, systematically check each family against **both** the LiteLLM catalog **and** models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.\n\n1. **Read the `ModelFamily` and `ModelName` enums** to know what we already have.\n\n2. **Query both catalogs for each family** (run in parallel where possible):\n\n **LiteLLM catalog** — filters out mirror providers to avoid duplicates:\n ```bash\n curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != \"openrouter\" and .provider != \"bedrock\" and .provider != \"bedrock_converse\" and .provider != \"vertex_ai-anthropic_models\" and .provider != \"azure\") | .id] | unique | .[]'\n ```\n\n **models.dev** — search all model IDs across all providers:\n ```bash\n curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i \"SEARCH_TERM\"\n ```\n For details on a specific provider+model: `curl -s https://models.dev/api.json | jq '.[\"PROVIDER\"].models[\"MODEL_ID\"]'`\n\n3. **Search terms** (one query per term):\n `claude`, `gpt`, `o1`, `o3`, `o4` (OpenAI reasoning), `gemini`, `llama`, `deepseek`, `qwen`, `qwq`, `mistral`, `grok`, `kimi`, `glm`, `minimax`, `hunyuan`, `ernie`, `phi`, `gemma`, `seed`, `step`, `pangu`\n\n4. **Union and cross-reference** results from both catalogs against `ModelName`. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). **Skip pure coding models** (e.g. `codestral`, `deepseek-coder`, `qwen-coder`).\n\n5. **Run targeted web searches** per family to catch very fresh releases not yet in either catalog:\n - `\"[family] new model [current year]\"`\n - `\"[family] release [current month] [current year]\"`\n\n6. **Present findings** as a summary. Let the user decide which to add.\n\n---\n\n## Phase 1B – Lagging-Provider Backfill Check (every run)\n\nSome providers — **Fireworks AI**, **Together AI**, **SiliconFlow**, **Cloudflare** — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM (or, for Cloudflare, never reach LiteLLM at all). Relying only on those two catalogs will both under-populate the provider list for the model you're adding now **and** miss the window to backfill recently-added models whose provider support has since grown.\n\nRun this check on **every invocation** of the skill, regardless of whether you're in discovery mode or adding a specific model.\n\n1. **Pull the 10 most recently added models from git history.** List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:\n ```bash\n git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E \"^\\+\\s+name=ModelName\\.\" | head -20\n ```\n\n2. **For the model you're adding (if any) AND each of those 10 models**, cross-check Fireworks, Together, SiliconFlow and Cloudflare directly using the endpoints in the [Lagging Providers Reference](#lagging-providers). Do NOT trust `models.dev` / LiteLLM as the final word for these providers.\n\n3. **If a lagging provider now supports a recently-added model that isn't yet in its `KilnModel` entry**, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.\n\n4. **Run the Cloudflare thinking-level checker** (see [Cloudflare thinking levels](#cloudflare-thinking-levels)). It checks every Cloudflare entry, not only the recent ones, because Cloudflare edits the effort lists of models Kiln already ships. Treat each ❌ finding like a backfill: fix the entry in the current change or propose a separate PR.\n\n---\n\n## Phase 2 – Gather Context\n\n1. **Read the predecessor model** in `ml_model_list.py` (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.\n\n2. **Query the LiteLLM catalog** for the new model. This is the primary slug source since Kiln uses LiteLLM. See the [Slug Lookup Reference](#slug-lookup-reference) for query syntax and all verified sources.\n\n3. **Get the OpenRouter slug** via:\n - `curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i \"SEARCH_TERM\"`\n - Fallback: WebSearch for `openrouter [model name] model id`\n\n4. **Get the direct-provider slug** (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the [Slug Lookup Reference](#slug-lookup-reference) for provider-specific URLs.\n\n5. **Identify quirks** — check the [Provider Quirks Reference](#provider-quirks-reference) for the relevant provider, and web search for any new quirks:\n - Structured output mode (JSON schema vs function calling)?\n - Reasoning model (needs `reasoning_capable`, parsers, OpenRouter options)?\n - Vision/multimodal support? Which MIME types?\n - Provider-specific flags (`temp_top_p_exclusive`, etc.)?\n - Rate limit concerns (`max_parallel_requests`)?\n\n6. **Determine thinking levels** — does the model support configurable reasoning effort? See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain. Key quick checks:\n - Check the **vendor model page** (e.g. OpenAI model pages say \"Reasoning.effort supports: X, Y, Z\")\n - Check **OpenRouter** `supported_parameters` — if `reasoning` is absent, skip thinking levels\n - R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts\n\n---\n\n## Phase 3 – Code Changes\n\nAll changes go in `libs/core/kiln_ai/adapters/ml_model_list.py`.\n\n### 3a. `ModelName` enum\n\n- snake_case: `claude_opus_4_6 = \"claude_opus_4_6\"`\n- Place **before** predecessor (newer first within group). If the vendor is\n brand-new there is no predecessor — start a new group at the end of the enum\n- Follow existing grouping (all claude together, all gpt together, etc.)\n\n### 3b. `KilnModel` entry in `built_in_models`\n\n- Place per the **ordering rules in 3c** — this placement is user-visible, get it right\n- Copy predecessor's structure and modify: `name`, `friendly_name`, `model_id` per provider, flags\n- **`friendly_name` must follow the existing naming pattern** of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use `\"Claude {version} Sonnet\"` (e.g. \"Claude 4.5 Sonnet\"), not `\"Claude Sonnet {version}\"`. Do NOT use the vendor's marketing name if it differs from Kiln's established convention.\n\n**Provider `model_id` formats:**\n\n| Provider | Format | Notes |\n|----------|--------|-------|\n| `openrouter` | `vendor/model-name` | Always verify via API |\n| `openai` | Bare model name | Verify via OpenAI docs |\n| `anthropic` | Variable — older models have date stamps, newer may not | Always verify via Anthropic docs |\n| `gemini_api` | Bare name | Verify via Google AI Studio docs |\n| `fireworks_ai` | `accounts/fireworks/models/...` | Verify via Fireworks docs |\n| `together_ai` | Vendor path format | Verify via Together docs |\n| `vertex` | Usually same as gemini_api | Verify via Vertex docs |\n| `siliconflow_cn` | Vendor/model format | Verify via SiliconFlow docs |\n| `featherless_ai` | HuggingFace repo id, case-sensitive (`zai-org/GLM-5.2`) | Verify via their `/v1/models` — see [Featherless](#featherless-ai) |\n| `typesafe` | Dotted System One version (`jev-1.13.0`) | Their `/v1/models` lists aliases only — verify by `POST /v1/systemone`, see [TypeSafe AI](#typesafe-ai) |\n| `cloudflare` | Cloudflare's own `@cf/<author>/<model>` id, verbatim (`@cf/zai-org/glm-5.3`) — no `cloudflare/` or `openai/` prefix | Verify via Cloudflare's model JSON — see [Cloudflare](#cloudflare-workers-ai) |\n\n**Every single `model_id` must be verified from an authoritative source. No exceptions.**\n\n**Setting flags — use catalog data + predecessor as dual signals:**\n\nThe LiteLLM catalog and models.dev responses include capability flags (`supports_vision`, `supports_function_calling`, `supports_reasoning`, etc.). Use these as the **primary signal** for what to enable on the new model:\n\n- If the catalog says `supports_vision: true` → enable `supports_vision`, `multimodal_capable`, and vision MIME types (see 2c)\n- If the catalog says `supports_function_calling: true` → use `StructuredOutputMode.json_schema` (or `function_calling` depending on provider norms — check predecessor)\n- If the catalog says `supports_reasoning: true` → the model can reason, but **do NOT reflexively set `reasoning_capable=True`** — default to `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)). Still add `available_thinking_levels` if it supports effort levels, and check parser/formatter flags.\n\nThen **cross-check against the predecessor**. The predecessor tells you *how* Kiln configures a similar model (which `structured_output_mode`, which provider-specific flags, etc.). The catalog tells you *what* the model can do. Use both:\n- Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.\n- Predecessor has `temp_top_p_exclusive` but nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track.\n- Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.\n\n**Common flags:**\n- `structured_output_mode` – how the model handles JSON output\n- `suggested_for_evals` / `suggested_for_data_gen` – see **zero-sum rule** below. If the entry also sets `available_thinking_levels`, its `default_thinking_level` must be a reasoning level — see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)\n- `multimodal_capable` / `supports_vision` / `supports_doc_extraction` – see **multimodal rules** below\n- `reasoning_capable` – for thinking/reasoning models. **Default new models to `reasoning_capable=False`** unless the model *always* emits its reasoning (see [Reasoning Capable Default](#reasoning-capable-default))\n- `temp_top_p_exclusive` – Anthropic models that can't have both temp and top_p\n- `parser` / `formatter` – for models needing special parsing (e.g. R1-style thinking)\n\n#### 2c. Multimodal capabilities\n\nIf the model supports non-text inputs, configure:\n\n- `multimodal_capable=True` and `supports_doc_extraction=True` if it supports any MIME types\n- `supports_vision=True` if it supports images\n- `multimodal_requires_pdf_as_image=True` if vision-capable but no native PDF support (also add `KilnMimeType.PDF` to MIME list). **Always set this on OpenRouter providers** — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.\n- Always include `KilnMimeType.TXT` and `KilnMimeType.MD` on any `multimodal_capable` model\n\n**Strategy: start broad, narrow based on test failures.** Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.\n\nFull MIME superset (Gemini uses all):\n```python\n# documents\nKilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD\n# images\nKilnMimeType.JPG, KilnMimeType.PNG\n# audio\nKilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG\n# video\nKilnMimeType.MP4, KilnMimeType.MOV\n```\n\n### 3c. Ordering — the list IS the UI\n\nThe order of `built_in_models` is exactly the order users see: model dropdowns\ngroup by provider and list each provider's models in list order, and the model\nlibrary page lists models in list order. A misplaced entry ships a scrambled\ndropdown to every client via the remote config. Rules:\n\n1. **Families are contiguous.** Every model of a family sits in one block.\n Never append a new family member at the bottom of the file or after another\n family — that strands it (past bugs: Mistral Small 4/3 ended up inside the\n Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).\n2. **Within a family, versions run newest → oldest, top to bottom.**\n 4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the\n family block. A new model of an *existing* version goes inside that\n version's group — NOT at the top of the family and NOT below older\n versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7\n landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).\n3. **Within a version, big → small. Always.**\n - Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite;\n Large > Medium > Small.\n - Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking\n variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and\n VL Thinking blocks) stay intact, ordered descending internally.\n - Tier-grouped families follow the same rule for their tier blocks: Claude\n runs Fable, then Opus > Sonnet > Haiku, versions descending inside each\n tier. (It once ran Haiku-first, and several families ran sizes\n ascending — those were bugs, not conventions. Do not preserve an\n ascending run because it's \"what the family already does.\")\n4. **A net-new family's block goes at the END of `built_in_models`** (this is\n the existing convention — the newest niche vendors sit at the bottom of the\n list). The \"place before predecessor\" rule only applies within an existing\n family; a new family has no predecessor.\n5. **Verify after editing** — print the order you just shipped and eyeball the\n affected family:\n\n ```bash\n uv run python -c \"\n from kiln_ai.adapters.ml_model_list import built_in_models\n for m in built_in_models: print(m.family, '|', m.friendly_name)\"\n ```\n\n### 3d. `suggested_for_evals` / `suggested_for_data_gen`\n\n**Only set these if** the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).\n\n**Zero-sum rule:** When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. **Ask the user to confirm** the swap before making changes.\n\n### 3e. `ModelFamily` enum (only if needed)\n\nOnly add a new family if the vendor is completely new.\n\n### 3f. Thinking Levels (`available_thinking_levels` / `default_thinking_level`)\n\nIf the model supports configurable reasoning effort (not just on/off), add `available_thinking_levels` and `default_thinking_level` to each provider entry. See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain and existing constants.\n\n**Quick rules:**\n- Reuse an existing `_THINKING_LEVELS` constant if the levels match exactly\n- Create a new constant only if levels differ; name it `{MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS`\n- `default_thinking_level` must be one of the values in `available_thinking_levels`\n- If the entry sets `suggested_for_evals=True`, `default_thinking_level` must be `\"medium\"` (or `\"low\"` if the model has no medium level) — never `\"none\"`. See [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)\n\n---\n\n## Phase 4 – Run Tests\n\nTests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.\n\n**Vertex AI authentication:** Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run `gcloud auth application-default login` before trying. These failures are auth issues, not model config problems.\n\n**`-k` filter syntax:** Always use bracket notation for model+provider filtering, never `and`:\n- Good: `-k \"test_name[glm_5-fireworks_ai]\"` or `-k \"glm_5\"`\n- Bad: `-k \"glm_5 and fireworks\"` — `and` is a pytest keyword expression that can match wrong tests\n\n### 4.0 — If the test env can't build (blocked git dependency)\n\nKiln's core lib pins `together` to a git fork (`libs/core/pyproject.toml`: `together = { git = \"https://github.com/scosman/together-python\" }`). In a sandboxed environment whose GitHub access is scoped to `kiln-ai/kiln` only (e.g. Claude Code Web), `uv sync` fails to fetch that fork with a **403** from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only `kiln-ai/kiln` resolves.\n\nWorkaround to build the venv for testing (the `together` fork isn't exercised by model-integration tests, which route through LiteLLM):\n\n1. **Do NOT use `uv sync --no-sources`** — it strips the workspace `workspace = true` sources too and makes `kiln-root → kiln-ai` unsatisfiable.\n2. Instead, temporarily comment out ONLY the `together = { git = ... }` line in `libs/core/pyproject.toml`, then run `uv sync` (this resolves `together` from PyPI).\n3. Run the tests.\n4. **Revert** the workaround with `git checkout -- libs/core/pyproject.toml uv.lock` — this restores the commented-out `together` git-fork pin in `libs/core/pyproject.toml` (NOT the repo-root `pyproject.toml`, which the workaround never touches) and reverts `uv.lock` if it changed. Only `ml_model_list.py` (and any intended test-file edits) should remain modified.\n\nNote: the PyPI `together` may pull slightly different transitive deps (e.g. a newer `starlette`), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your `-k` filters to the model files and ignore those.\n\n### 4a. Parallel testing + API keys\n\n**Parallel testing is already on.** There is no `pytest.ini` — pytest config lives in `[tool.pytest.ini_options]` in the root `pyproject.toml`, and `addopts = \"-n auto\"` is active. No edit and no revert are needed. (Only override to `-n 8` if a provider rate-limits you.)\n\n**Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings.** `conftest.py` has an autouse `use_temp_settings_dir` fixture that points `Config.settings_path` at a temp dir, so `~/.kiln_ai/settings.yaml` is deliberately ignored during tests. A key the user added through the app's provider page **will not be seen** — the test fails with \"Attempted to use X without an API key set\", which looks like a config bug but isn't.\n\nBridge the key from the user's settings into the test environment without ever printing it:\n\n```bash\nexport FEATHERLESS_AI_API_KEY=\"$(uv run python -c \\\n \"from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)\" 2>/dev/null | tail -1)\"\n```\n\nUse the provider's `env_var` name from `libs/core/kiln_ai/utils/config.py`. Some providers need more than one value: Cloudflare needs both `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` (and optionally `CLOUDFLARE_AI_GATEWAY_ID`). To check which keys are available before running, print booleans only — never the values:\n\n```bash\nuv run python -c \"from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))\"\n```\n\n**Many paid tests also carry the `ollama` marker**, so `--runpaid` alone silently skips them. Always pass `--runpaid --ollama` together, and use `-rs` to see skip reasons when a test you expected to run reports as skipped.\n\n### 4b. Smoke test — verify slug works\n\nRun a single test+provider combo first:\n\n```bash\nuv run pytest --runpaid --ollama -k \"test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]\"\n```\n\nIf it fails, fix the slug/config before proceeding. Use `--collect-only` to find exact parameter IDs if unsure.\n\n### 4c. Full test suite\n\n```bash\nuv run pytest --runpaid --ollama -k \"MODEL_ENUM\" -v 2>&1 | grep -E \"PASSED|FAILED|ERROR|short test|=====|collected\"\n```\n\n**If tests fail — debug one at a time:**\n1. Pick ONE failing test, run it with `-v` for full output\n2. Fix the config\n3. Re-run that single test to verify\n4. Only re-run the full suite once the single test passes\n\n**Anthropic API key gotcha:** if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as `KILN_ANTHROPIC_API_KEY` instead of `ANTHROPIC_API_KEY` (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't `export` it globally:\n\n```bash\nANTHROPIC_API_KEY=\"$KILN_ANTHROPIC_API_KEY\" uv run pytest --runpaid ...\n```\n\n### 4d. Extraction tests (if `supports_doc_extraction=True`)\n\nTests are in `libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py`.\n\n```bash\n# See what will run:\nuv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM\n\n# Run them:\nuv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k \"MODEL_ENUM\"\n```\n\nIf a provider rejects a data type (400 error), remove that `KilnMimeType` and re-run.\n\n### 4e. Confirm failures are actually yours\n\nBefore treating a failure as a problem with your change, check whether the same test already fails for an **existing** provider of that model. Several assertions are provider-independent and fail regardless.\n\nKnown example: `test_structured_input_cot_prompt_builder` asserts `len(trace) == 5` unconditionally, which is incompatible with any provider setting `reasoning_capable=True` (that selects the single-call strategy, producing 3 messages). It fails for `gpt_oss_120b` on `fireworks_ai` on a clean tree.\n\n```bash\nuv run pytest --runpaid --ollama -q \"path::test_name[MODEL-OTHER_PROVIDER]\"\n```\n\nIf it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.\n\n### 4f. Test output format\n\nCollect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:\n- ✅ for passed tests\n- ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason\n- ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason\n- List every test using the full pytest parametrize ID, grouped by provider\n- Include extraction tests (Phase 4d) if they were run\n\n---\n\n## Phase 5 – Create Pull Request\n\n### 5.0 — Important context about Claude Code Web's stop hook\n\nThis skill is often run via Claude Code Web (Slack connector). That environment has a **non-user-configurable stop hook** which, at end of session, will:\n- Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits\n- Instruct the agent to commit and push any local work before stopping\n- Explicitly tell the agent NOT to create a PR unless the user asked for one\n\n**The problems this causes:**\n1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned `add-model/*` branches on the remote.\n2. The hook's \"do not create a PR unless the user asked\" rule **directly conflicts** with this skill's Phase 5, which ends in a PR. Running this skill *is* the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.\n\n**The user's desires, in priority order:**\n1. **Ask before you push.** If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code \"just to satisfy the stop hook.\"\n2. **No abandoned branches.** Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.\n3. **If the user says to abandon:** revert your local changes (`git restore` / `git clean` the specific files you touched) and delete any branch you created (`git checkout main && git branch -D add-model/MODEL_NAME`) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.\n4. **On a successful run, push and open the PR as described in 5a/5b.** Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says \"don't create a PR.\" Only re-ask if tests failed or the user hasn't confirmed the results.\n\n### 5.1 — Gate before pushing\n\nDo NOT commit, push, or create a branch if any of the following are true:\n- Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)\n- The smoke test (4b) failed and wasn't resolved\n- Any step in Phases 2–4 was skipped or incomplete\n- You are unsure whether a ⚠️ flake is actually a real failure\n\nIf any of the above apply, **stop and ask the user** what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.\n\nAfter all tests pass, commit the changes and open a PR against `main`.\n\n### 5a. Commit and push\n\n1. Create a new branch named `add-model/MODEL_NAME` (e.g. `add-model/glm-5-1`)\n2. Stage only the changed files (typically just `ml_model_list.py`)\n3. Commit with a concise message (e.g. \"Add GLM 5.1 to model list (together_ai, siliconflow_cn)\")\n4. Push the branch\n\n### 5b. Create the PR\n\nUse `gh pr create` against `main`. Follow Rule 0, Rule 1 and Step 3 of the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) for the human header, the CLA and the title:\n\n- The title is `WIP: chore: ` followed by the model-list subject, for example `WIP: chore: add GLM 5.1 to model list`. Use `chore`, because adding a model is maintenance work, not a new feature. Only a human removes the `WIP: ` prefix.\n- Copy everything above `# Agentic PR Summary` from `.github/pull_request_template.md` with no change. Never fill in a placeholder or tick a box there, and never sign the CLA. Remove the `## Contributor License Agreement` section only when the PR author's GitHub username is `scosman`, `sfierro`, `leonardmq`, `tawnymanticore`, or `chiang-daniel`. Keep it, unsigned, for any other author.\n\nReplace the `` `Insert AI summary of PR using .agents/skills/open-pr/SKILL.md` `` placeholder under `# Agentic PR Summary` with the test results, in this exact format:\n\n```\n## What does this PR do?\n\n Test Results\n\n[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]\n\n[Model Name] ([provider]):\n- [N] passed, [N] skipped[, [N] failed]\n- [Any notable failures or flakes]\n\n[Repeat for each model+provider combo]\n\n---\n[Model Name] ([provider]):\n✅ test_data_gen_all_models_providers[model_enum-provider]\n✅ test_data_gen_sample_all_models_providers[model_enum-provider]\n✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]\n✅ test_all_built_in_models_llm_as_judge[model_enum-provider]\n✅ test_all_built_in_models_structured_output[model_enum-provider]\n✅ test_all_built_in_models_structured_input[model_enum-provider]\n✅ test_structured_output_cot_prompt_builder[model_enum-provider]\n✅ test_all_models_providers_plaintext[model_enum-provider]\n✅ test_cot_prompt_builder[model_enum-provider]\n⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason\n❌ test_name[model_enum-provider] — brief reason\n\n[Repeat for each model+provider combo]\n```\n\n**Rules for the PR body:**\n- Every test that ran must appear in the per-test dump, using the full pytest parametrize ID\n- Group tests by `[Model Name] ([provider]):` headers\n- The summary section at the top gives a quick pass/skip/fail count per model+provider\n- The detailed section below the `---` inside the summary (not the `----` above `# Agentic PR Summary`) lists every individual test result\n- Use ⚠️ for content quality flakes (not real failures), ❌ for real errors\n\n---\n\n## Checklist\n\n- [ ] `ModelName` enum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor)\n- [ ] `KilnModel` entry added to `built_in_models` per the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list)\n- [ ] Printed the resulting list order and eyeballed the affected family (3c verification command)\n- [ ] `friendly_name` matches the naming pattern of sibling models in the same family\n- [ ] If the provider is net-new: followed [Adding a Net-New Provider](#adding-a-net-new-provider), including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)\n- [ ] `ModelFamily` enum updated (only if new family)\n- [ ] All provider slugs verified from authoritative sources\n- [ ] Flags inherited from predecessor and adjusted for quirks\n- [ ] `reasoning_capable` defaulted to `False` for adaptive-reasoning models (only `True` for always-emits-reasoning models — see [Reasoning Capable Default](#reasoning-capable-default))\n- [ ] Thinking levels configured if model supports reasoning effort (see [Thinking Levels Reference](#thinking-levels-reference))\n- [ ] Cloudflare and OpenRouter: `check_thinking_levels.py` reports no ❌ finding (see [Cloudflare thinking levels](#cloudflare-thinking-levels) and [OpenRouter (general)](#openrouter-general))\n- [ ] Every `suggested_for_evals` entry with thinking levels defaults to `\"medium\"` (or `\"low\"`), never `\"none\"` (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default))\n- [ ] Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)\n- [ ] Zero-sum applied if model is suggested for evals/data gen\n- [ ] RAG config templates updated if the new model replaces one used in `app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts`\n- [ ] API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)\n- [ ] Smoke test passed\n- [ ] Full test suite passed\n- [ ] Failures cross-checked against an existing provider before being called regressions (see 4e)\n- [ ] PR created against `main` with a `WIP: ` title, the untouched human header, and test results in the Agentic PR Summary\n\n---\n\n## Reasoning Capable Default\n\n**Default newly-added models to `reasoning_capable=False`, even when the catalog reports `supports_reasoning: true`.**\n\nMost recent models use *adaptive* reasoning and sometimes return no reasoning at all. Kiln raises `RuntimeError(\"Reasoning is required for this model, but no reasoning was returned.\")` whenever a provider has `reasoning_capable=True` and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.\n\n`reasoning_capable` is **orthogonal to `available_thinking_levels`** — you keep the important behavior with it set to `False`:\n- Thinking-level / effort selection still works (every GPT-5.x entry sets `available_thinking_levels` with `reasoning_capable` unset).\n- Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).\n- The `test_thinking_level_reasoning_content` paid test still runs and still asserts reasoning is present — it is gated on `available_thinking_levels`, not `reasoning_capable`. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels.\n\nWhat you give up with `reasoning_capable=False`:\n- The conditional \"reasoning is present in `intermediate_outputs`\" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped).\n- When a user attaches a chain-of-thought prompt, the model uses the two-call `two_message_cot` strategy instead of the single-call native `single_turn_r1_thinking`.\n\n**Keep `reasoning_capable=True` only for models that *always* emit reasoning** in a native `<think>` format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.\n\n**Narrower alternative:** if a model reliably reasons but you only hit the error on structured output, set `reasoning_optional_for_structured_output=True` (requires `reasoning_capable=True`) instead of disabling reasoning entirely.\n\n---\n\n## Adding a Net-New Provider\n\nAdding a provider Kiln has never supported is a bigger job than adding a model,\nand it has a hard sequencing constraint. Follow this section end to end.\n\n### The release-gating rule (do NOT skip)\n\nThe remote config is generated from main's `built_in_models` and reaches **all\nexisting clients immediately** — but provider *support* (name map, connect\nflow, API key handling) only reaches users through a client app release. If\nmodel entries for a brand-new provider land on main before a client release\nwith the provider plumbing is live, every deployed client shows the raw\nprovider ID (e.g. `featherless_ai`) in the model library and offers models\nnobody can connect to. This happened with Featherless in Aug 2026 and the\nentries had to be rolled back.\n\n**Sequence it in two PRs:**\n\n1. **PR 1 — provider plumbing only.** The `ModelProviderName` enum member and\n everything else in the checklist below EXCEPT the model catalog. Merge\n whenever ready.\n2. **PR 2 — model catalog.** ALL `ml_model_list.py` changes: `ModelName`\n members, `ModelFamily` (if the vendor is new), and the `built_in_models`\n entries. Open it, but **merge only after a client release containing PR 1\n is live.** (Only `built_in_models` is published via the remote config, but\n the enums belong in the same PR as the entries they exist for.)\n\n### Touchpoint checklist (from the Featherless integration, #1618)\n\n**First, identify the provider's authentication model** — the checklist below\ndescribes the common single-API-key pattern, but not every provider fits it.\nExisting variants to crib from: Bedrock stores an access key + secret pair,\nFireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project\nID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base\nURL with no credentials. Follow the closest existing analog's plumbing through\n`config.py`, `provider_warnings`, `provider_api.py`, and the connect page —\nthe credential fields, validation call, and UI steps all change with the auth\nmodel.\n\n**libs/core:**\n- `ModelProviderName` enum — `libs/core/kiln_ai/datamodel/datamodel_enums.py`\n- Credential storage — `libs/core/kiln_ai/utils/config.py` (for API-key\n providers: a new key with `env_var`; otherwise whatever fields the auth\n model needs)\n- LiteLLM provider mapping — `libs/core/kiln_ai/utils/litellm.py`\n- `libs/core/kiln_ai/adapters/provider_tools.py` — three spots: the\n `provider_name_from_id` match (friendly name; pyright flags a missed case),\n `provider_warnings` (missing-credential message, `required_config_keys`),\n and the adapter config (credential / base URL / headers plumbing)\n- Model catalog — `libs/core/kiln_ai/adapters/ml_model_list.py`: `ModelName`\n members, `ModelFamily` if needed, and the `built_in_models` entries\n (**PR 2 only**, see gating rule)\n\n**Desktop server (`app/desktop/studio_server/provider_api.py`):**\n- `connect_<provider>` credential-validation endpoint (find a cheap\n authenticated call; see the Featherless connect function for a pattern when\n the provider has no authenticated GET to ping)\n- Disconnect handling (clear the stored credentials)\n- Tests in `test_provider_api.py`\n\n**Web UI (`app/web_ui`):**\n- `src/lib/stores.ts` — `provider_name_map` entry (friendly name)\n- `src/lib/ui/provider_image.ts` + SVG in `static/images/` — icon must match\n the monochrome convention: bare glyph, `currentColor`, no background tile,\n viewBox cropped to the artwork\n- Connect page — `src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte`:\n provider card (name, description, and the auth flow — `api_key_steps` /\n `api_key_fields` for key providers, or the custom flow the auth model\n needs) plus connected-status\n wiring from `settings` keys\n- `src/lib/api_schema.d.ts` — regenerate with `make schema` (needs the server\n running on :8757)\n\n**Tests & tooling:**\n- `libs/core/kiln_ai/adapters/test_provider_tools.py`, `test_adapter_registry.py`,\n `model_adapters/test_litellm_adapter.py` — extend the per-provider\n parametrized cases\n- `.agents/scripts/provider_utils.py` — add the provider's model-catalog\n endpoint so agent tooling can enumerate its models\n- Thinking levels — find out how the provider takes reasoning effort, and\n where it publishes the values each model accepts. Add a quirks section\n that names that source and says thinking levels are required wherever\n it lists values. If the provider silently remaps unsupported values,\n add the provider to `check_thinking_levels.py`, because a\n paid test cannot see a remap.\n- `model_adapters/test_thinking_level_paid.py` — add the provider to the\n provider list of `test_thinking_level_reasoning_content`, so each of its\n thinking levels gets a paid test case\n\n**Known gap:** the generated Copilot API client\n(`app/desktop/studio_server/api_client/`) mirrors the remote Copilot service's\nschema — it can't learn the new provider until that service updates. Note it in\nthe PR rather than hand-editing generated code.\n\n### Friendly names everywhere\n\nThe raw enum value must never be user-visible. When you add the provider,\nverify a friendly name exists in **both** name maps (python\n`provider_name_from_id` and web `provider_name_map`) — pyright and typescript\nrespectively force these when the enum gains a member, which is why the\nplumbing PR must not skip them.\n\n---\n\n## Provider Quirks Reference\n\n### Anthropic\n- Newer models (Opus 4.1+, Sonnet 4.5+) need `temp_top_p_exclusive=True`\n- Opus 4.5+ uses `json_schema`; older Opus uses `function_calling`\n- Extended thinking models: `anthropic_extended_thinking=True` + `reasoning_capable=True`\n\n### OpenAI\n- Most GPT models use `json_schema` for structured output\n- GPT-5.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)\n- Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort\n- o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)\n\n### Google/Gemini\n- `gemini_reasoning_enabled=True` for reasoning-capable models\n- Gemini 3.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)\n- Rich multimodal support (audio, video, images, documents)\n\n### DeepSeek\n- R1 models: `parser=ModelParserID.r1_thinking` + `reasoning_capable=True`\n- V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN\n- Some need `r1_openrouter_options=True` + `require_openrouter_reasoning=True`\n\n### OpenRouter (general)\n- Slugs: `vendor/model-name`\n- Never add a `:free` slug. Free tiers have low rate limits and are temporary. If OpenRouter serves a model only as `:free`, wait for a paid slug.\n- Reasoning models: may need `require_openrouter_reasoning=True`\n- Some models: `openrouter_skip_required_parameters=True`\n- Logprobs: `logprobs_openrouter_options=True` if supported\n- Always `multimodal_requires_pdf_as_image=True` (OpenRouter's PDF routing breaks LiteLLM)\n- Thinking levels: offer only values in the model's `reasoning.supported_efforts` (see [Thinking Levels Reference](#thinking-levels-reference)). OpenRouter maps an unlisted effort to a listed one, so an extra level silently runs a different effort. When the model has no `reasoning.supported_efforts`, OpenRouter turns the effort into a thinking-token budget for Claude, so `CLAUDE_OPENROUTER_THINKING_LEVELS` (with Minimal) fits only those Claude models. `check_thinking_levels.py` checks every OpenRouter entry; a missing level list on OpenRouter is a note, because Kiln leaves levels off some models on purpose.\n\n### Fireworks AI and Together AI: thinking levels\n\nNeither publishes the efforts each model accepts, and both can accept an unsupported effort without an error. Fireworks takes `xhigh`, `max` and `minimal` on GLM 5.3 with HTTP 200. Together took a made-up effort on Nemotron 3.5, but Muse Spark Open's chat template rejects one. A paid test cannot see this. Add thinking levels on these providers only when the vendor documents the levels for that model (the model card or its chat template), and name that source in the PR. OpenAI and Anthropic return a 400 that names the supported levels, so their paid per-level test catches a wrong level.\n\n### Featherless AI\n\nServerless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:\n\n- **`json_instructions` is the only usable structured output mode.** LiteLLM's `featherless_ai` provider rejects `response_format` outright (`UnsupportedParamsError`), so `json_schema`, `json_mode`, and `json_instruction_and_object` are all unavailable. Routing it as a custom `openai` provider bypasses that gate, but was tested and is *not* reliable: GLM 5.2 accepts `json_schema` and silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.\n- **Gated models return HTTP 403 `model_gated_needs_oauth`** and cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. **Always filter `is_gated` before adding.** Note all 20 official `meta-llama/*` repos are gated, so no Llama variant is usable.\n- **No cost reporting.** Featherless models aren't in LiteLLM's price map (only two legacy `Qwerky` entries, on `main` too — not a version issue), and Featherless doesn't return cost in the `usage` object. Runs record tokens with `cost: null`.\n- **Not in models.dev or the LiteLLM catalog**, so their `/v1/models` endpoint is the only authoritative source. See [Lagging Providers](#lagging-providers).\n- Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.\n\n### TypeSafe AI\n\nServes TypeSafe's Jev models (System One API). The only provider in the catalog that does not run through LiteLLM — read this before adding an entry:\n\n- **Adapter-backed, not LiteLLM-backed.** Every provider entry must set `adapter=ModelAdapterId.jev`; `JevAdapter` calls `POST /v1/systemone` directly. `lite_llm_core_config_for_provider` and `get_litellm_provider_info` raise for `typesafe` on purpose, so the LiteLLM parametrized tests exclude it.\n- **Structured output only.** No text generation, no tools, no multi-turn. A task runs only if its output schema maps onto Jev's question types (enum, boolean, score); anything else fails before the network call. Set `supports_structured_output=True`, `supports_data_gen=False`, `supports_function_calling=False`, and `structured_output_mode=StructuredOutputMode.json_schema`.\n- **No logprobs.** `supports_logprobs=False`. Jev returns its own per-answer probabilities instead, which is what keeps it out of G-Eval.\n- **Not in models.dev or the LiteLLM catalog, and `GET /v1/models` cannot confirm a `model_id`.** It lists aliases (`jev-latest`, `jev-preview`), not the pinned versions Kiln registers, and it returns `{\"models\": [{\"name\", ...}]}` rather than the OpenAI `{\"data\": [{\"id\", ...}]}` shape, so the shared `fetch_openai_compat` path cannot read it either. That is why `typesafe` is in `SKIP_PROVIDERS` in `.agents/scripts/provider_utils.py` rather than in `PROVIDER_CONFIG`. The authoritative check for a pinned id is a `POST /v1/systemone` round trip that the API accepts — that is how `jev-1.13.0` was confirmed. (The listing is still what the connect flow uses to validate a key; it is account-scoped and rejects a bad one.)\n\n### Cloudflare (Workers AI)\n\nRuns through LiteLLM's generic `openai/` route against Cloudflare's OpenAI-compatible endpoint (`/accounts/{id}/ai/v1/chat/completions`), with an optional AI Gateway header. Kiln does **not** use LiteLLM's `cloudflare/` provider. Read this before adding or changing any entry:\n\n- **Inclusion rule: only Cloudflare's newer, OpenAI-format text-generation models.** Cloudflare has two generations of text model. The newer ones (GLM, Kimi, DeepSeek V4, Gemma 4, Qwen 3.8, Nemotron 3) speak OpenAI Chat Completions natively. The older ones (Llama 3.x/4, Mistral, older Qwen, GPT-OSS, Granite, the LoRA bases) sit behind a translation layer: they default to **256 output tokens** (silent truncation) and have unreliable tool calling and JSON mode. Do not add the older ones.\n- **How to tell them apart:** look at the model's input schema in the catalog JSON. OpenAI-format models list `max_completion_tokens` (and `tool_choice`); older models list `prompt`, `raw` and `lora` instead, and no `max_completion_tokens`:\n ```bash\n jq -r '.models[] | select(.task.name==\"Text Generation\")\n | [.name, (if (.schema.input | tostring | test(\"max_completion_tokens\")) then \"openai-format\" else \"OLDER - skip\" end)] | @tsv' /tmp/cf_models.json\n ```\n- **Also skip:** coding models (`kimi-*-code`, `qwen*-coder-*`), `llama-guard-*` (classifier), `-lora` bases, and models Kiln has no `ModelName` for unless the user asks for a new model.\n- **Usually a new provider on an existing model.** Most Cloudflare models are already in Kiln under other providers; add a `KilnModelProvider(name=ModelProviderName.cloudflare, ...)` to the existing `KilnModel`.\n- **Setting flags from the catalog's `properties[]`** (confirm every flag with the paid tests — the catalog has no JSON-mode flag):\n\n | Cloudflare property | Kiln setting |\n |---|---|\n | `function_calling: \"true\"` | leave `supports_function_calling` at its default; set `False` if the tool tests fail |\n | `vision: \"true\"` | `supports_vision`, `multimodal_capable`, MIME types TXT/MD/JPG/PNG; start broad, narrow on 400s. Don't set `supports_doc_extraction`: the document extractor doesn't send the AI Gateway header, so extraction would bypass the user's gateway |\n | `reasoning: \"true\"` | keep `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)); LiteLLM already reads `reasoning_content` |\n | `reasoning_effort.supported_efforts` present | **required:** set `available_thinking_levels` to exactly these values, and `default_thinking_level` to `default_effort`. See [Cloudflare thinking levels](#cloudflare-thinking-levels) |\n | `reasoning_effort` without `supported_efforts` | no thinking levels; the catalog does not say which values work |\n | `require_workers_paid: \"true\"` | no Kiln field. Free-plan users get 403 / code 5035. Mention it in the PR |\n | `planned_deprecation_date` present | do **not** add the model |\n\n- <a id=\"cloudflare-thinking-levels\"></a>**Thinking levels come from the catalog, and only from the catalog.** Kiln sends the level as `reasoning_effort`. For each model, `reasoning_effort` in the catalog lists `supported_efforts`, a `default_effort`, and `normalizes_to`: the values that Cloudflare silently changes to another effort. A remapped value returns HTTP 200, so a live call or a paid test cannot find it. Example: GLM 5.3 maps `medium` to `max`, so a \"Medium\" option runs the most expensive effort.\n - Offer only values in `supported_efforts`, and each of them unless a completed paid test shows that the level does not work. The list is not always right: on DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731 and GLM 5.2, `none` still returns reasoning, so those entries have no Off level. A timeout, a rate limit or a credential error proves nothing about the level: keep the level and report it as untested. Do not copy the levels of the same model on another provider.\n - Labels: `none` → `\"Off\"`, `low` → `\"Low\"`, `medium` → `\"Medium\"`, `high` → `\"High\"`, `xhigh` → `\"Extra High\"`, `max` → `\"Max\"`. List them from least to most effort.\n - `default_thinking_level` is `default_effort`, unless the entry is suggested for evals (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)).\n - `mandatory: true` means `none` is not in the list; do not add an Off level.\n - Reuse a constant when t","frontmatter":{"name":"claude-maintain-models","description":"Add new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, or wants to add a net-new AI provider to Kiln.","allowed-tools":"Read Edit Write Bash Grep Glob Agent WebSearch WebFetch","allowedTools":["Read","Edit","Write","Bash","Grep","Glob","Agent","WebSearch","WebFetch"]},"allowedTools":["Read","Edit","Write","Bash","Grep","Glob","Agent","WebSearch","WebFetch"],"isInternal":false,"tokens":12541,"sizeBytes":50292},{"name":"SKILL.md","path":".agents/skills/kiln-check-deprecation/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Kiln-AI/Kiln/HEAD/.agents/skills/kiln-check-deprecation/SKILL.md","title":"kiln-check-deprecation","category":"anthropic-skill","format":"markdown","content":"---\nname: kiln-check-deprecation\ndescription: Check Kiln's model list for deprecated or sunset models across all providers. Use when the user wants to find deprecated models, check model availability, audit the model list for stale entries, or mentions model deprecation/sunset/end-of-life.\n---\n\n# Check Model Deprecation in Kiln\n\nAudit `libs/core/kiln_ai/adapters/ml_model_list.py` to find models that have been deprecated, removed, or are approaching end-of-life across all providers. **All checks are free** — they use model-listing endpoints, not inference.\n\n---\n\n## Global Rules\n\n- **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: [\"all\"]`. The sandbox blocks network access.\n- **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)`\n- **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate.\n- **Non-destructive:** This skill only sets `deprecated=True` on individual provider entries. It never removes models or providers from the list.\n- **Skip already-deprecated:** Only check providers where `deprecated=False` (the default). Don't re-check entries already marked deprecated.\n\n---\n\n## Phase 1 – Extract Current Model-Provider Pairs\n\nRun the extraction script from the repo root:\n\n```bash\npython3 .agents/skills/kiln-check-deprecation/scripts/extract_models.py > /tmp/kiln_extracted.json\n```\n\nThis fetches the published model list from `https://remote-config.getkiln.ai/kiln_config_v2.json` and extracts:\n- The provider name\n- The model_id\n- The parent model's enum name\n\nIt skips entries already marked `deprecated=True`.\n\n**Output:** JSON to stdout (pipe to file), human summary to stderr.\n\nThe JSON contains:\n- `deprecated_count`: number of already-deprecated entries\n- `providers`: dict of provider_name → sorted unique list of model_ids\n- `entries`: list of `{enum, provider, model_id}` for each non-deprecated entry\n\n---\n\n## Phase 2 – Check Each Provider\n\n### Option A: Check all providers at once\n\n```bash\nexport $(grep -v '^#' .env | xargs)\npython3 .agents/skills/kiln-check-deprecation/scripts/check_provider.py all > /tmp/kiln_check_results.json\n```\n\n### Option B: Check a single provider\n\n```bash\nexport $(grep -v '^#' .env | xargs)\npython3 .agents/skills/kiln-check-deprecation/scripts/check_provider.py openrouter > /tmp/kiln_check_openrouter.json\n```\n\nThe script handles all provider API quirks automatically:\n- **Together AI** returns a flat JSON array (not `{data: [...]}`)\n- **Gemini API** needs both v1 and v1beta endpoints (preview/Gemma models only on v1beta)\n- **Gemini** model names are prefixed with `models/` — stripped automatically\n- **Anthropic** uses `x-api-key` header, not `Authorization: Bearer`\n- **OpenRouter** is public (no auth) and includes `expiration_date` fields\n- **Fireworks AI** `/v1/models` only lists serverless models; the script instead checks each model individually via the model detail API (`GET /v1/{model_id}`), which covers all tiers (serverless, on-demand, fine-tune). A model is only flagged as missing if it returns HTTP 404 from the detail API.\n- **OpenRouter** `:exacto` is a virtual routing suffix (quality-first provider sorting) that never appears in model listings. The script strips it before checking. `:free` and `:thinking` are real model entries that appear in the listing when available — if they're missing, it's a genuine removal.\n- **Cloudflare** uses the authenticated models-search API (`task=Text Generation`, `include_deprecated=true`) when `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` are both set, and otherwise Cloudflare's public catalog JSON (`https://ai-cloudflare-com.pages.dev/api/models`), so it is never skipped for a missing key. Every model with a `planned_deprecation_date` is reported under `expiring`. Once that date has passed, the model also counts as `missing` even if the listing still shows it. Never use LiteLLM's catalog or models.dev to decide a Cloudflare model still exists: LiteLLM's is stale and models.dev no longer deletes retired models.\n- **Vertex AI** uses the v1beta1 publisher models endpoint with `x-goog-user-project` header. Requires `gcloud` CLI auth. Kiln entries may use `meta/` prefix for LiteLLM routing — stripped automatically. Versioned aliases (e.g. `gemini-2.0-flash-001` → `gemini-2.0-flash`) are handled by stripping 3-digit version suffixes.\n\n**Output:** JSON to stdout with per-provider results, human summary to stderr.\n\nEach provider result contains:\n- `missing`: model_ids not found in the provider's listing\n- `expiring`: model_ids with upcoming expiration dates (OpenRouter and Cloudflare)\n- `entries_to_deprecate`: full enum/provider/model_id entries for each missing model\n- `skipped` / `error`: if credentials missing or API call failed\n\n### Supported providers\n\n| Provider | Env Var | Auth |\n|----------|---------|------|\n| OpenRouter | (none — public API) | None |\n| OpenAI | `OPENAI_API_KEY` | Bearer |\n| Anthropic | `ANTHROPIC_API_KEY` | x-api-key |\n| Gemini API | `GEMINI_API_KEY` | Query param |\n| Fireworks AI | `FIREWORKS_API_KEY` | Bearer |\n| Together AI | `TOGETHER_API_KEY` | Bearer |\n| SiliconFlow CN | `SILICONFLOW_CN_API_KEY` | Bearer |\n| Cerebras | `CEREBRAS_API_KEY` | Bearer |\n| Groq | `GROQ_API_KEY` | Bearer |\n| Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) |\n| Cloudflare | `CLOUDFLARE_API_KEY` + `CLOUDFLARE_ACCOUNT_ID` (optional — falls back to the public catalog) | Bearer |\n\n### Providers NOT covered by the script (check manually if needed)\n\n#### Amazon Bedrock\n\nRequires `aws` CLI configured. If not installed, skip.\n\n```bash\naws bedrock list-foundation-models --output json | \\\n jq '[.modelSummaries[] | {modelId, status: .modelLifecycle.status, endOfLife: .modelLifecycle.endOfLifeTime}]'\n```\n\nFlag any model where `status == \"LEGACY\"` or `endOfLifeTime` is approaching.\n\n### LiteLLM Model DB (supplementary signal)\n\nCross-reference against LiteLLM's public model database for `deprecation_date` fields. This mostly covers Azure/OpenAI date-stamped models but is a useful secondary signal.\n\n```bash\ncurl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \\\n jq 'to_entries[] | select(.value.deprecation_date != null) | {model: .key, deprecation_date: .value.deprecation_date}'\n```\n\n### Providers to skip\n\nThese don't have meaningful deprecation to check:\n- `ollama` — local models, user-managed\n- `kiln_fine_tune` — internal\n- `kiln_custom_registry` — internal\n- `openai_compatible` — user-configured\n- `docker_model_runner` — local\n- `azure_openai` — typically mirrors OpenAI, covered by OpenAI check\n- `huggingface` — serverless inference, model availability varies\n- `typesafe` — adapter-backed, not LiteLLM; its `/v1/models` needs a key, so Jev entries can only be checked with one\n\n---\n\n## Phase 3 – Report Findings\n\nRead the JSON output from check_provider.py and present findings in a clear table to the user:\n\n```text\nDeprecation Audit Results\n=========================\n\n❌ REMOVED (model not found in provider's model list):\n - ModelName.gemini_1_5_pro → gemini_api (model_id: gemini-1.5-pro)\n - ModelName.gemini_1_5_pro → openrouter (model_id: google/gemini-pro-1.5)\n\n⚠️ EXPIRING SOON (within 90 days):\n - ModelName.claude_3_7_sonnet → openrouter (model_id: anthropic/claude-3.7-sonnet, expires: 2026-05-05)\n\n⚠️ LEGACY (Bedrock lifecycle status):\n - ModelName.mistral_large → amazon_bedrock (model_id: mistral.mistral-large-2407-v1:0, status: LEGACY)\n\n✅ ALL CLEAR:\n - openai: 20/20 models found\n - anthropic: 8/11 models found (3 deprecated above)\n\n⏭️ SKIPPED (no credentials):\n - groq\n - amazon_bedrock\n```\n\nGroup by provider within each category. Use the `entries_to_deprecate` field from the JSON to get the ModelName enum for each entry.\n\n---\n\n## Phase 4 – Mark Deprecated\n\nFor each confirmed-removed model, set `deprecated=True` on the affected `KilnModelProvider` entry.\n\nTo find the line to edit, grep for the `model_id` in `ml_model_list.py`:\n\n```bash\ngrep -n 'model_id=\"<model_id>\"' libs/core/kiln_ai/adapters/ml_model_list.py\n```\n\nThen add `deprecated=True,` after the `model_id=` line in the matching `KilnModelProvider` block.\n\n**Rules:**\n- Only mark a provider deprecated if its model_id is confirmed missing from that provider's model list\n- If ALL providers for a `KilnModel` are deprecated, note this to the user — they may want to consider removing the model entirely\n- For \"expiring soon\" models, inform the user but don't mark deprecated yet — let them decide. **Exception: Cloudflare.** When Cloudflare retires a model it may silently alias the old ID to a different model, and the response still echoes the requested ID, so a stale entry keeps \"working\" while running something else. Propose marking every Cloudflare entry with a `planned_deprecation_date` deprecated now, before that date, and flag it as urgent in the report\n- For Bedrock `LEGACY` status, mark deprecated (the model still works but is on its way out)\n\n**Ask the user to confirm** before making changes. Present the list of changes and wait for approval.\n\n---\n\n## Phase 5 – Verify\n\nAfter marking providers deprecated, run a quick sanity check:\n\n```bash\ngrep -c \"deprecated=True\" libs/core/kiln_ai/adapters/ml_model_list.py\n```\n\nEnsure the count matches what you expect (previous count + newly marked).\n\n---\n\n## Checklist\n\n- [ ] Extraction script run, JSON saved to /tmp/kiln_extracted.json\n- [ ] Provider checks run (all or individually)\n- [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex)\n- [ ] Bedrock checked manually (skip if aws CLI unavailable)\n- [ ] LiteLLM DB checked for supplementary deprecation_date signals\n- [ ] Findings reported to user with clear table\n- [ ] User confirmed changes before marking deprecated\n- [ ] `deprecated=True` set on confirmed entries\n- [ ] Deprecation count verified\n","frontmatter":{"name":"kiln-check-deprecation","description":"Check Kiln's model list for deprecated or sunset models across all providers. Use when the user wants to find deprecated models, check model availability, audit the model list for stale entries, or mentions model deprecation/sunset/end-of-life."},"isInternal":false,"tokens":2556,"sizeBytes":10137},{"name":"SKILL.md","path":".agents/skills/kiln-check-finetune-deprecation/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Kiln-AI/Kiln/HEAD/.agents/skills/kiln-check-finetune-deprecation/SKILL.md","title":"kiln-check-finetune-deprecation","category":"anthropic-skill","format":"markdown","content":"---\nname: kiln-check-finetune-deprecation\ndescription: Check Kiln's fine-tunable model list for deprecated or unsupported base models. Use when the user wants to audit fine-tuning support, check if fine-tune base models are still valid, or mentions fine-tune model deprecation.\n---\n\n# Check Fine-Tune Model Deprecation in Kiln\n\nAudit the fine-tunable models listed in Kiln to find base models that are no longer supported for fine-tuning by their providers. This is separate from general model deprecation — a model can still be available for inference but no longer supported as a fine-tuning base.\n\n---\n\n## Global Rules\n\n- **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: [\"all\"]`. The sandbox blocks network access.\n- **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)`\n- **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate.\n- **Non-destructive:** This skill only reports findings. It does not automatically remove or deprecate models.\n\n---\n\n## Supported Providers\n\n| Provider | Env Var | Auth |\n|----------|---------|------|\n| OpenAI | `OPENAI_API_KEY` | Bearer |\n| Together AI | `TOGETHER_API_KEY` | Bearer |\n| Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) |\n| Fireworks AI | `FIREWORKS_API_KEY` | Bearer |\n\n---\n\n## Background\n\nKiln has two types of fine-tunable model entries:\n\n1. **Static entries** — Models in `libs/core/kiln_ai/adapters/ml_model_list.py` with `provider_finetune_id` set. Currently used by Together AI and Vertex AI.\n2. **Dynamic entries (Fireworks)** — Fetched at runtime from `api.fireworks.ai/v1/accounts/fireworks/models` filtering by `tunable=True`. This list is managed by Fireworks and may include stale/unsupported models.\n\nThe API endpoint that serves the fine-tune dropdown is `GET /api/finetune_providers` in `app/desktop/studio_server/finetune_api.py`.\n\n---\n\n## Phase 1 – Check Static Fine-Tune IDs\n\nRun the check script from the repo root:\n\n```bash\nuv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py static > /tmp/kiln_finetune_static.json\n```\n\nThis extracts all `provider_finetune_id` entries from `ml_model_list.py` and checks each provider's docs/API to see if the model is still available for fine-tuning:\n\n- **Together AI:** Scrapes the Together AI docs page for the list of supported fine-tune models. Together uses full HuggingFace-style IDs (e.g. `meta-llama/Meta-Llama-3.1-8B-Instruct-Reference`).\n- **Vertex AI:** Checks the Vertex AI publisher models API. Vertex fine-tuning uses specific versioned model IDs (e.g. `gemini-2.0-flash-001`).\n\n**Output:** JSON to stdout with per-provider results, human summary to stderr.\n\n---\n\n## Phase 2 – Check Fireworks Dynamic Models\n\nFireworks fine-tunable models are fetched dynamically from their API, not stored in `ml_model_list.py`. The script checks the API's `supervisedLoraTunable` and `supervisedFullParameterTunable` fields (not the old `tunable` field, which is stale) and cross-references against our allowlist.\n\n```bash\nuv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py fireworks > /tmp/kiln_finetune_fireworks.json\n```\n\nThis script:\n1. Fetches all models with `supervisedLoraTunable=True` or `supervisedFullParameterTunable=True` from `api.fireworks.ai/v1/accounts/fireworks/models`\n2. Cross-references against `FIREWORKS_SUPPORTED_FINETUNE_MODELS` from `libs/core/kiln_ai/adapters/fine_tune/fireworks_finetune.py` — the same allowlist that filters the runtime fine-tune dropdown\n3. Checks both directions:\n - **Stale allowlist entries**: models in our allowlist that the API no longer marks as supervised-tunable — these may need to be removed\n - **Missing from allowlist**: models the API marks as supervised-tunable but aren't in our allowlist — these are candidates to add\n\nThe canonical allowlist lives in `fireworks_finetune.py` and is shared between the runtime dropdown filter and this audit skill. There is a single place to update when Fireworks changes their supported models.\n\n**Interpreting results:** The supervised API fields return ~43 models, but our allowlist intentionally filters to ~20 that match Fireworks' documented supported models. Seeing ~23 \"models in API but NOT in allowlist\" is normal — these are models Fireworks marks as technically tunable but doesn't officially document or support. Only flag these if a model appears in the Fireworks docs but is missing from our allowlist. The important direction is **stale allowlist entries** — models we ship that the API no longer marks as supervised-tunable.\n\n**Output:** JSON to stdout, human summary to stderr.\n\n---\n\n## Phase 3 – Report Findings\n\nRead the JSON outputs and present findings in a clear table:\n\n```text\nFine-Tune Deprecation Audit Results\n====================================\n\nSTATIC PROVIDERS (provider_finetune_id in ml_model_list.py):\n\n ✅ OpenAI: 5/5 found\n gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14,\n gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18\n\n ✅ Vertex AI: 2/2 found\n gemini-2.0-flash-001, gemini-2.0-flash-lite-001\n\n ❌ Together AI: 2/4 missing\n ✅ Qwen/Qwen2.5-72B-Instruct — found\n ✅ Qwen/Qwen2.5-14B-Instruct — found\n ❌ meta-llama/Meta-Llama-3.1-8B-Instruct-Reference — NOT FOUND\n ❌ meta-llama/Meta-Llama-3.1-70B-Instruct-Reference — NOT FOUND\n\n ⏭️ SKIPPED (no credentials):\n - vertex (VERTEX_PROJECT_ID not set)\n\nDYNAMIC PROVIDER (Fireworks):\n\n ⚠️ 167 models marked tunable in API\n ⚠️ 15 models in known-good docs list\n ✅ 11 models in both API and docs\n ❌ 156 models in API but NOT in docs (likely stale)\n - accounts/fireworks/models/qwen2-72b-instruct\n - accounts/fireworks/models/code-llama-7b\n - ...\n ⚠️ 4 models in docs but NOT in API\n - accounts/fireworks/models/gemma-2-9b-it\n - ...\n```\n\nNote: The example above is illustrative. Actual model counts and entries will vary.\n\n---\n\n## Phase 4 – Remediation\n\nBased on findings, recommend specific actions:\n\n### For static entries (`provider_finetune_id`):\n- If a model's fine-tune ID is no longer valid, either:\n - **Update** the `provider_finetune_id` to a newer version if one exists\n - **Remove** the `provider_finetune_id` field to stop offering fine-tuning for that model on that provider\n - **Set `deprecated=True`** on the provider entry if the model is fully gone\n\n### For Fireworks dynamic entries:\n- The stale models come from Fireworks' API, not our code. Options:\n - **Filter in our code** — update `fetch_fireworks_finetune_models()` in `finetune_api.py` to filter against a known-good list\n - **Report to Fireworks** — flag the stale models to Fireworks support\n - **Both** — filter now, report later\n\n**Always ask the user to confirm** before making any code changes.\n\n---\n\n## Phase 5 – Verify\n\nAfter any changes, run the relevant tests:\n\n```bash\nuv run python3 -m pytest app/desktop/studio_server/test_finetune_api.py -q\n```\n\n---\n\n## Checklist\n\n- [ ] Env vars sourced from `.env`\n- [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex)\n- [ ] Static fine-tune IDs checked against provider APIs (Together, Vertex)\n- [ ] Fireworks dynamic models cross-referenced against known-good list\n- [ ] Findings reported to user with clear table\n- [ ] User confirmed remediation approach\n- [ ] Code changes made (if any)\n- [ ] Tests pass after changes\n- [ ] FIREWORKS_SUPPORTED_FINETUNE_MODELS in fireworks_finetune.py updated if docs have changed\n","frontmatter":{"name":"kiln-check-finetune-deprecation","description":"Check Kiln's fine-tunable model list for deprecated or unsupported base models. Use when the user wants to audit fine-tuning support, check if fine-tune base models are still valid, or mentions fine-tune model deprecation."},"isInternal":false,"tokens":2009,"sizeBytes":7786},{"name":"SKILL.md","path":".agents/skills/kiln-prerelease-check/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Kiln-AI/Kiln/HEAD/.agents/skills/kiln-prerelease-check/SKILL.md","title":"kiln-prerelease-check","category":"anthropic-skill","format":"markdown","content":"---\nname: kiln-prerelease-check\ndescription: Run the Kiln pre-release smoke test suite plus the standard CI checks (checks.sh), diagnose every prerelease test that broke (and why), and write a clean readable report with recommended actions. Read-only — it never edits code. Use when the user wants to validate a release candidate, run prerelease tests, or asks for a \"prerelease check / smoke / audit\".\n---\n\n# Kiln Pre-release Check\n\nEnd-to-end pre-release verification and **diagnosis**. This skill:\n\n1. Runs the standard CI checks (`checks.sh` — lint / format / typecheck / unit tests).\n2. Runs the curated `@pytest.mark.prerelease` smoke set (real, paid API calls).\n3. For every test that broke, works out **what** broke and **why** — including whether the cause is a model that's gone deprecated (per `ml_model_list.py`) or that the provider no longer recognizes the slug.\n4. Sweeps the prerelease model pins for staleness and newer model versions.\n5. Writes a clean, scan-readable report with a clear **recommended action** for each finding.\n\n**This skill is read-only. It does not change any code, tests, whitelists, or prod source — ever.** Its entire output is a report at `.prerelease/<timestamp>/REPORT.md` (this directory is gitignored). When the right action is to bump a model pin or update the whitelist, the report *recommends* it; the user makes the change.\n\n---\n\n## What \"prerelease\" means here\n\nA prerelease test is a paid test (it makes real API calls) that we **cannot** meaningfully replace with a mock — its whole point is to verify a live third-party API still behaves how we depend on it. Examples: `test_connect_vertex_live`, basic OpenAI / Claude / Gemini calls, embedding API across providers, reranker integration, Fireworks fine-tune deployment listing, etc.\n\nThe set is a **curated subset of `@pytest.mark.paid`**. Tests carry both markers — `@pytest.mark.paid` for the cost gate, `@pytest.mark.prerelease` for the curated selection.\n\nConcretely:\n\n- `--runpaid` runs the full paid suite (slow, expensive, intended for one-off use).\n- `--runprerelease` runs **only** the prerelease subset (still paid, but a small, focused, releaseable list). It implies `--runpaid` for those tests.\n\nMarker registration lives in `pyproject.toml` (`[tool.pytest.ini_options].markers`); the flag is wired up in the top-level `conftest.py` via `pytest_addoption` + `pytest_collection_modifyitems`.\n\n### Whitelist for litellm-adapter tests\n\nSeveral paid tests (embeddings, document extraction, thinking-level reasoning) fan out across **every** `(model, provider)` pair in `ml_model_list.py` / `ml_embedding_model_list.py`. Running those under `--runprerelease` would be slow and expensive. Instead, those tests have dedicated `*_prerelease_smoke` sibling functions that iterate only over a small curated whitelist living in:\n\n```\nlibs/core/kiln_ai/adapters/pytest_prerelease_whitelist.py\n```\n\nWhitelisted lists:\n\n- `PRERELEASE_CHAT_MODELS` — handful of chat models, one per major provider (OpenAI + GPT-5, Anthropic + Claude Sonnet/Haiku, Gemini, OpenRouter, Groq, Fireworks, Together, Cloudflare).\n- `PRERELEASE_EMBEDDING_MODELS` — one embedding model per embedding-supporting provider.\n- `PRERELEASE_EXTRACTION_MODELS` — one multimodal model per major vendor (OpenAI, Anthropic, Gemini).\n- `PRERELEASE_EXTRACTION_MIME_PROBES` — three mime probes (PDF, PNG, MP3) for the extraction smoke; the full paid test sweeps all 13 mime types per model.\n- `PRERELEASE_THINKING_MODELS` — five (provider, model, thinking_level) triples covering reasoning content + a \"none\" negation case.\n- `PRERELEASE_JEV_MODELS` — TypeSafe AI's Jev models, used by the Jev paid smoke tests. Separate from `PRERELEASE_CHAT_MODELS` because Jev does not go through LiteLLM.\n\nThis whitelist is the thing most likely to go stale, so it's the main target of the pin sweep in Phase 4. The fan-out tests (e.g. `test_extract_document_success` over every model × mime type, `test_paid_generate_embeddings_basic` over every embedding) are **only `@pytest.mark.paid`** — not `@pytest.mark.prerelease` — and are out of scope for this skill.\n\n---\n\n## Global rules\n\n- **Read-only. Make no edits.** Do not modify test files, the whitelist, prod source, config, or anything else. Every actionable finding is a *recommendation* in the report. If you catch yourself about to call Edit/Write on anything other than files under `.prerelease/<timestamp>/`, stop.\n- **Network access is required.** Every test phase makes real API calls or hits remote model-list endpoints. If you are running this skill inside a sandboxed Bash session, request `required_permissions: [\"all\"]` for the test commands.\n- **Env vars:** Source `.env` before running anything that needs API keys. Use a real sourcing pass (not `export $(... | xargs)`, which mangles any value containing spaces or quotes):\n ```bash\n set -a; . ./.env; set +a\n ```\n Provider-specific env vars the prerelease set may touch: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `GROQ_API_KEY`, `OPENROUTER_API_KEY`, `FIREWORKS_API_KEY`, `TOGETHERAI_API_KEY`, `SILICONFLOW_CN_API_KEY`, `COHERE_API_KEY`, `TYPESAFE_API_KEY`, `CLOUDFLARE_API_KEY` + `CLOUDFLARE_ACCOUNT_ID` (used by `test_cloudflare` in `test_prompt_adaptors.py`, which runs GLM 4.7 Flash; a free-plan Cloudflare account is enough), `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`, plus `KILN_TEST_VERTEX_PROJECT_ID` (+ optional `KILN_TEST_VERTEX_LOCATION`) for the Vertex live check, which also requires `gcloud auth application-default login` against a project with `aiplatform.googleapis.com` enabled.\n- **A missing key is a coverage gap, not a failure.** Each missing key makes the relevant test `pytest.skip(...)`. That isn't a prerelease failure — surface it in the report so the user can decide whether to provide the key and re-run.\n- **`ml_model_list.py` never deletes entries — it only marks them `deprecated=True`.** So a slug missing from our list is not a signal you'll normally encounter. The signals that matter when diagnosing a break or staleness are:\n 1. The provider entry has `deprecated=True` in our list.\n 2. The provider itself rejects the slug at runtime (4xx / \"model not found\") — note our list trails provider deprecations, so this can happen *before* we've marked it deprecated. This is how you'd discover a model \"no longer exists\" even when our list still lists it.\n 3. A newer non-deprecated sibling in the same family now exists in our list, regardless of whether the current pin still works.\n- **Single source of truth for the model list** is `libs/core/kiln_ai/adapters/ml_model_list.py` (chat) and `ml_embedding_model_list.py` (embeddings). Cross-reference there when diagnosing.\n- **The pin sweep is mandatory, not \"only when something fails.\"** Even on a fully green run, every hardcoded slug in `pytest_prerelease_whitelist.py` and in prerelease test parametrize lists must be cross-checked against the model list and reported (kept-and-current vs. has-newer-sibling vs. deprecated). Zero silent \"looks fine, moving on\" calls.\n\n---\n\n## Phase 1 — Set up the output directory\n\n```bash\nTS=$(date -u +%Y-%m-%dT%H-%M-%SZ)\nOUT=\".prerelease/${TS}\"\nSCRATCH=\"${OUT}/_scratch\"\nmkdir -p \"${SCRATCH}\"\necho \"Writing prerelease results to ${OUT}\"\n```\n\n`.prerelease/` is gitignored. This timestamped folder is the **only** place this skill writes. Keep it tidy — the top level should hold just the human-readable artifacts:\n\n- `REPORT.md` — the deliverable.\n- `checks.log`, `prerelease.log`, `retry.log` — raw run logs (kept for drill-down).\n\nEverything else — the grep/collect/diff intermediates used while diagnosing (`current_model_ids.txt`, `prerelease_test_files.txt`, `paid_tests.txt`, `prod_probes.txt`, etc.) — goes in `${SCRATCH}` (`${OUT}/_scratch`), **not** the top level. Don't scatter raw filename dumps where the user has to wade through them; the report is what they read.\n\n---\n\n## Phase 2 — Run `checks.sh`\n\nThese are not paid but are mandatory for any release. If they fail, the prerelease check fails before we even spend money on API calls.\n\n```bash\nuv run ./checks.sh --agent-mode 2>&1 | tee \"${OUT}/checks.log\"\n```\n\nCapture the exit code. If it's non-zero, **don't bail** — keep going with the paid tests so the report covers everything in one pass, but mark the overall run as failed.\n\n---\n\n## Phase 3 — Run the prerelease pytest set\n\n```bash\nuv run python3 -m pytest --runprerelease -v --tb=short \\\n -o \"addopts=\" \\\n 2>&1 | tee \"${OUT}/prerelease.log\"\n```\n\nNotes:\n\n- `--runprerelease` is registered in the top-level `conftest.py`. It runs only `@pytest.mark.prerelease` tests and implies `--runpaid` for them.\n- `-o \"addopts=\"` overrides the project default of `-n auto` (xdist). xdist is incompatible with some of the live-API tests (especially anything that touches global litellm state), so run serially.\n- This will be slow. Many of the parametrized tests fan out to dozens of cases.\n- Tests with missing API keys will `skip`, not fail. Capture skip counts.\n\nIf the user wants a faster narrowed run (e.g. only OpenAI + Anthropic), use `-k` to filter:\n\n```bash\nuv run python3 -m pytest --runprerelease -v --tb=short -o \"addopts=\" \\\n -k \"openai or anthropic or vertex\" \\\n 2>&1 | tee \"${OUT}/prerelease.log\"\n```\n\n---\n\n## Phase 4 — Diagnose\n\nThis is where the skill earns its keep: for every failure, explain the cause; for every pin, report staleness. No edits — only findings and recommendations.\n\n### 4a. Diagnose each failure (what broke, why, recommended action)\n\nFor every test that did **not** pass in Phase 3, build one finding. Walk the failure through this decision tree to land on a cause and a recommended action:\n\n1. **Missing credentials → skip, not failure.** (`pytest.skip` for absent key.) Don't list under failures; list under \"Skipped tests\". Recommended action: add the key and re-run if that coverage matters.\n2. **Provider rejected the model slug** — error mentions `model not found`, `does not exist`, `404`, `model_not_found`, `invalid model`, decommissioned, etc. Cross-reference the slug against `ml_model_list.py`:\n - If our list marks that provider entry `deprecated=True` → cause: **deprecated model, provider has dropped it**. Recommended action: bump the pin/whitelist to the newest non-deprecated sibling in the same family (name the candidate).\n - If our list still lists it as non-deprecated → cause: **model no longer exists at the provider; our list trails reality**. Recommended action: (a) bump the test pin/whitelist to a current sibling, and (b) flag that `ml_model_list.py` should mark this entry `deprecated=True`. Name both.\n3. **Auth / quota / rate-limit / transient network** — 401/403/429/5xx, timeouts. Cause: environment or provider-side, not our code. Recommended action: check the key/quota and re-run; only escalate if it reproduces.\n4. **Behavioral assertion mismatch** — the call succeeded but the response shape, content, token/cache fields, streaming chunks, or reasoning output changed. Cause: **provider changed behavior we depend on (real regression)**. Recommended action: investigate the prod path that relies on this; do not assume it's a stale pin. This is the highest-signal failure class — call it out prominently.\n5. **Our-code error** — traceback originates in `libs/core` / `app/` rather than the provider response. Cause: **regression in Kiln**. Recommended action: investigate the implicated code; name the file/function from the traceback.\n\nIf you can't confidently distinguish a stale-pin cause (2) from a real regression (4 or 5), **report it as a possible regression** and say what additional check would disambiguate (e.g. \"try the same call with a current sibling slug to see if it's the model or the code\"). Never recommend a silent pin bump for an ambiguous failure.\n\nRecord, per failure: test ID · provider · cause class · the model slug involved (if any) and its `ml_model_list.py` status · a short excerpt of the actual error · recommended action.\n\n**Retry likely-transient failures once before finalizing.** Live-API tests are flaky: rate limits, 5xx, timeouts, dropped streams, and occasional empty/under-populated responses (e.g. a thinking model returning no reasoning content on one call but not the next) routinely fail a single run without indicating any real breakage. So before you write up the failures, re-run the failing tests in isolation **once** to separate flakes from real defects:\n\n- **Retry these cause classes:** auth/quota/rate-limit/transient (class 3) and ambiguous behavioral/streaming mismatches (class 4) — anything that could plausibly be a one-off. **Do not retry** failures with a deterministic, self-explanatory cause: a deprecated/removed model slug (class 2), a Kiln-code traceback (class 5), or a missing-credential skip. Retrying those just burns money and time.\n- **How:** re-run only the failing node IDs, serially, into a separate log so the original run is preserved:\n ```bash\n uv run python3 -m pytest --runprerelease -v --tb=short -o \"addopts=\" \\\n \"<failing::node::id>\" \"<failing::node::id>\" ... \\\n 2>&1 | tee \"${OUT}/retry.log\"\n ```\n One retry pass is enough — this is a flake filter, not a stability loop. Don't retry a test more than once.\n- **Reclassify by the retry outcome:** a test that **passes on retry** is **transient/flaky**, not a release blocker — move it out of the hard-failure set. A test that **fails again** is **reproducible** — diagnose it as a real finding per the tree above.\n- **Always disclose the retry in the report.** Never silently swap a first-run FAIL for a PASS. Every retried test must show its first-run result, its retry result, and the verdict (transient vs. reproducible) — see the report template's \"Failures\" section and the retry note in the Verdict. A green retry is evidence the suite is flaky here, which the user needs to know before tagging.\n\n### 4b. Model-pin staleness sweep (mandatory every run, even when green)\n\nIndependent of failures, every hardcoded model slug the prerelease set depends on must be checked against the model list and reported. A pin can be perfectly green today and still be worth bumping because a newer sibling shipped — prerelease only catches what real users hit if it tracks what real users actually use.\n\n**In scope (check these every run):**\n- Every entry in `libs/core/kiln_ai/adapters/pytest_prerelease_whitelist.py`.\n- Hardcoded model slugs inside the body or parametrize list of any `@pytest.mark.prerelease` test (notably `libs/core/kiln_ai/adapters/test_prompt_adaptors.py::test_openrouter`).\n\n**Out of scope (don't report as stale):**\n- Hardcoded slugs in mock-only unit tests (no `@pytest.mark.paid` / `@pytest.mark.prerelease`) — the slug is a label, no real call happens.\n- Slugs in fixtures, recorded VCR cassettes, snapshot files.\n- Slugs in `@pytest.mark.paid` tests that aren't also `@pytest.mark.prerelease`.\n\n**Step 1 — status of each whitelist entry.** Run:\n\n```bash\nuv run python3 - <<'PYEOF'\nfrom kiln_ai.adapters.ml_model_list import built_in_models\nfrom kiln_ai.adapters.ml_embedding_model_list import built_in_embedding_models\nfrom kiln_ai.adapters.pytest_prerelease_whitelist import (\n PRERELEASE_CHAT_MODELS, PRERELEASE_EMBEDDING_MODELS,\n PRERELEASE_EXTRACTION_MODELS, PRERELEASE_THINKING_MODELS,\n PRERELEASE_JEV_MODELS,\n)\n\ndef chat_status(name, provider):\n for m in built_in_models:\n if m.name == name:\n p = next((p for p in m.providers if p.name.value == provider), None)\n return 'NO_PROVIDER' if p is None else ('DEPRECATED' if p.deprecated else 'OK')\n return 'MODEL_REMOVED_OR_RENAMED'\n\ndef emb_status(name, provider):\n for m in built_in_embedding_models:\n if m.name == name:\n p = next((p for p in m.providers if p.name.value == provider), None)\n return 'NO_PROVIDER' if p is None else 'OK'\n return 'MODEL_REMOVED_OR_RENAMED'\n\nprint('== chat / extraction ==')\nfor n,p in PRERELEASE_CHAT_MODELS + PRERELEASE_EXTRACTION_MODELS + PRERELEASE_JEV_MODELS:\n print(f'{chat_status(n,p):>22} {n} ({p})')\nprint('== embedding ==')\nfor n,p in PRERELEASE_EMBEDDING_MODELS:\n print(f'{emb_status(n,p):>22} {n} ({p})')\nprint('== thinking ==')\nfor prov,name,lvl in PRERELEASE_THINKING_MODELS:\n print(f'{chat_status(name,prov):>22} {name} ({prov}, level={lvl})')\nPYEOF\n```\n\nAnything not `OK` is stale and goes in the sweep table with a recommended replacement (newest non-deprecated sibling in the same family).\n\n**Step 2 — newer-sibling pass for the `OK` entries.** For each entry that's `OK`, scan `ml_model_list.py` for a newer member of the same family. List the canonical model ids to compare against:\n\n```bash\ngrep -oE '^[[:space:]]+[a-z][a-z0-9_]+ = \"[a-z0-9_]+\"' \\\n libs/core/kiln_ai/adapters/ml_model_list.py | \\\n awk -F'\"' '{print $2}' | sort -u > \"${SCRATCH}/current_model_ids.txt\"\n```\n\nExamples of the judgment call:\n- `PRERELEASE_CHAT_MODELS` on `gpt_4o_mini` while `gpt_5_x_mini` exists → recommend bump.\n- `PRERELEASE_CHAT_MODELS` on `claude_sonnet_4_5` while `claude_sonnet_4_6`/`4_7` exists → recommend bump.\n- `gemini_1_5_*` → `gemini_2_5_*` / `gemini_3_*`; `claude_3_5_*` → `claude_sonnet_4_*` / `claude_4_5_haiku`; `llama_3_x` → `llama_3_3_70b` / `llama_4_*`.\n\nRule of thumb: if `ml_model_list.py` ordering puts a newer sibling above the current pin, recommend the bump. If it's genuinely ambiguous which sibling is \"current\", say so and let the user pick — don't assert a churny recommendation.\n\n**Step 3 — per-test hardcoded slugs.** Enumerate the prerelease test files and scan them for slugs not in `current_model_ids.txt`:\n\n```bash\nuv run python3 -m pytest --runprerelease --collect-only -q -o \"addopts=\" \\\n 2>&1 | grep '::' | awk -F'::' '{print $1}' | sort -u > \"${SCRATCH}/prerelease_test_files.txt\"\n```\n\nCross-reference each pinned slug against `current_model_ids.txt` and the deprecation status. Report any obsolete pins as recommendations.\n\nEvery in-scope pin lands in the sweep table — including the ones that are current and need no action. Silent omission is not allowed.\n\n### 4c. Coverage gaps\n\nThe prerelease set should cover, at minimum:\n\n| Area | Test(s) |\n|---|---|\n| Vertex live connect | `app/desktop/studio_server/test_provider_api.py::test_connect_vertex_live` |\n| OpenAI basic call | `libs/core/kiln_ai/adapters/test_prompt_adaptors.py::test_openai` |\n| Anthropic via OpenRouter / Anthropic API | `test_structured_output_anthropic_*`, `test_structured_output_openrouter_function_calling_nested_object` |\n| Gemini API | covered via embeddings + prompt caching + thinking levels |\n| Groq | `test_groq` |\n| OpenRouter multi-model | `test_openrouter[...]` (parametrized) |\n| Amazon Bedrock | `test_amazon_bedrock` |\n| Embedding API (whitelist) | `test_paid_generate_embeddings_basic_prerelease_smoke`, `test_generate_embedding_prerelease_smoke` |\n| Custom-dimension embeddings (whitelist) | `test_paid_generate_embeddings_custom_dimensions_prerelease_smoke`, `test_generate_embedding_with_user_supplied_dimensions_prerelease_smoke` |\n| Structured output (json_schema, function_calling, flat/nested/array) | `test_structured_output_*` |\n| Streaming (OpenAI protocol + AI SDK protocol) | `test_invoke_openai_stream_chunks`, `test_invoke_ai_sdk_stream` |\n| Tool calling | `test_tools_gpt_4_1_mini` |\n| Thinking-level reasoning (whitelist) | `test_thinking_level_reasoning_content_prerelease_smoke` |\n| Reranker | `test_reranker_integration_success` |\n| Document extraction (whitelist of models × probe mime types) | `test_extract_document_success_prerelease_smoke`, `test_provider_bad_request_prerelease_smoke` |\n| Semantic chunker (real embedding integration) | `test_semantic_chunker_real_integration` |\n| Fireworks fine-tune | `test_fetch_all_deployments` |\n| Prompt caching (Anthropic + OpenAI + Gemini + Fireworks + Together) | `test_prompt_caching_cache_hit` |\n\nIf a new provider/family in `ml_model_list.py` isn't represented in the prerelease set, note the gap as a recommendation.\n\nAlso list recently added `@pytest.mark.paid` tests that are *not* `@pytest.mark.prerelease`-tagged, so the user can decide if any should be promoted:\n\n```bash\n# Compare collected test NODE IDs, not marker line numbers: a test carrying both\n# markers on different lines would be misclassified by a file:line diff. Marker\n# expressions select by marker regardless of the skip gates, so collect-only works.\n# `sed 's/\\[.*//'` collapses parametrized cases to one row per test function\n# (the paid fan-out tests expand to thousands of cases otherwise).\nuv run python3 -m pytest --collect-only -q -m \"paid and not prerelease\" -o \"addopts=\" \\\n 2>/dev/null | grep '::' | sed 's/\\[.*//' | sort -u > \"${SCRATCH}/paid_only_nodes.txt\"\n# Recently added files among those paid-only tests (pipe to xargs to avoid\n# \"argument list too long\" on a large list):\nawk -F'::' '{print $1}' \"${SCRATCH}/paid_only_nodes.txt\" | sort -u | \\\n xargs git log --since=\"3 months ago\" --diff-filter=AM --name-only --pretty=format: -- \\\n 2>/dev/null | sort -u | head -50 > \"${SCRATCH}/recent_paid_only_files.txt\"\n```\n\n### 4d. Prod-code model probes (flag only)\n\nSome prod-code paths hardcode a model slug as a **probe** — a narrow connectivity/auth check where the model is just a vehicle to hit the API, not a feature the user picked. The canonical example is `app/desktop/studio_server/provider_api.py::connect_vertex`, which calls `litellm.acompletion(model=\"vertex_ai/gemini-X.Y-flash\", …)` to verify Vertex credentials when the user clicks \"Connect\".\n\nThese should stay on the latest available model in their family, otherwise the Connect button drifts toward a deprecated/removed model. **This skill only flags them** — it does not edit prod code, and the recommendation in the report is the artifact the user acts on.\n\nSweep for hardcoded inference-probe slugs in non-test prod files:\n\n```bash\ngrep -rnE 'model=[\"'\\''][^\"'\\'']*((gemini|gpt|claude|llama)[^\"'\\'']*)[\"'\\''']' \\\n app/desktop libs/core/kiln_ai libs/server/kiln_server \\\n --include=\"*.py\" 2>/dev/null \\\n | grep -v \"test_\" | grep -v \"/build/\" \\\n | tee \"${SCRATCH}/prod_probes.txt\"\n```\n\nFor each hit:\n1. Decide whether it's a probe (connectivity/auth check, model incidental) or real prod behavior (a default the user sees, fine-tune flow, eval scoring). If unsure, treat it as real prod behavior.\n2. For probes, check `ml_model_list.py` for the newest non-deprecated sibling at that provider.\n3. Record in the probe table: file:line · current slug · suggested newer slug (or \"none — already latest\") · the verifying live test and its result.\n\nEvery grep hit goes in the table even when no newer sibling exists. If the verifying live test failed outright, also list it under Failures (it's a real-failure signal).\n\n---\n\n## Phase 5 — Write the report\n\nCreate `.prerelease/<timestamp>/REPORT.md`. Lead with the verdict; make every finding skim-readable with an explicit recommended action. This is the only artifact the skill produces.\n\n```markdown\n# Kiln Prerelease Check — <timestamp>\n\n## Verdict\n- Overall: PASS / FAIL / PASS WITH RECOMMENDATIONS\n- checks.sh: <pass|fail>\n- prerelease pytest: <N passed, M failed, K skipped>\n- Retries: <none needed | \"X of M first-run failures were retried (Phase 4a); Y passed on retry (transient/flaky), Z reproduced\" — omit if no test was retried>\n- Headline: <one sentence — the single most important thing the user must know before tagging>\n\n## Test results\nOne scannable table covering **every prerelease test that ran**, grouped by coverage area, with an emoji outcome and a short note where it helps — this is the at-a-glance summary the user reads first. Aggregate parametrized cases into one row with per-case emoji counts (e.g. `✅×8 ❌×1`); don't spell out long node-id / filename lists here (raw lists live in the logs and `_scratch/`). Outcome legend: ✅ pass · ❌ fail · ⏭️ skip (missing key / N/A) · 🔁 transient (failed then passed on retry). Leave the Notes cell blank when the row is a clean all-pass; use it to name the failing case, why it skipped, or the retry result.\n\n| Area | Test(s) | Result | Notes |\n|---|---|---|---|\n| OpenAI basic call | `test_openai` | ✅ | |\n| Groq | `test_groq` | ✅ | |\n| OpenRouter multi-model | `test_openrouter` (9) | ✅×9 | |\n| Amazon Bedrock | `test_amazon_bedrock` | ✅ | |\n| Structured output | `test_structured_output_*` (6) | ✅×6 | |\n| Streaming | `test_invoke_openai_stream_chunks` / `test_invoke_ai_sdk_stream` (N) | ✅×N | |\n| Tool calling | `test_tools_gpt_4_1_mini` | ✅ | |\n| Thinking-level reasoning | `…reasoning_content_prerelease_smoke` (N) | ✅×N | <which case failed/skipped, if any> |\n| Embeddings | `…embeddings_basic_prerelease_smoke` (6) | ✅×4 ⏭️×2 | <which providers skipped + why> |\n| Reranker | `test_reranker_integration_success` (N) | ✅×1 ❌×2 | <which provider failed + one-line cause> |\n| Prompt caching | `test_prompt_caching_cache_hit` (7) | ✅×7 | |\n| Document extraction | `…extract_document_*_prerelease_smoke` (N) | ✅×N | |\n| Semantic chunker | `test_semantic_chunker_real_integration` | ✅ | |\n| Vertex live connect | `test_connect_vertex_live` | ❌ | <one-line cause> |\n| Fireworks fine-tune | `test_fetch_all_deployments` | ✅ | |\n\nUse the real areas/tests/counts from this run (mirror the coverage table in Phase 4c). Put 🔁 on anything that needed a retry and add the retry detail in the Notes cell + under Failures.\n\n## Failures — what broke, why, and what to do\nOne block per failing test. Sorted most-actionable first (real regressions before stale pins).\nIf any failures were retried (Phase 4a), open this section with a small first-run-vs-retry table so transient flakes are visible at a glance, then write a block only for the tests that **reproduced** (note retried-and-passed ones as transient — don't dignify them with a full diagnosis block).\n\n### <test ID>\n- Provider: <provider>\n- Cause: <deprecated model | model no longer exists at provider | provider behavior change (regression) | Kiln code regression | auth/quota/transient>\n- Model slug: <slug> — `ml_model_list.py` status: <OK | deprecated | no-provider-entry | n/a>\n- Retry: <not retried (deterministic cause) | retried once — reproduced | retried once — passed (transient, flaky)>\n- Evidence: <short excerpt of the actual error / failed assertion>\n- Recommended action: <concrete next step — e.g. \"bump PRERELEASE_CHAT_MODELS pin from X to Y (newest sibling)\"; or \"investigate prod path Z — response shape changed\"; or \"mark this entry deprecated=True in ml_model_list.py and bump the pin\". Recommendation only — this skill makes no edits.>\n\n## Skipped tests (missing credentials)\nList by provider, with which env var is missing, so the user can decide what to add and re-run.\n\n## Model-pin staleness sweep (mandatory, every run)\nEvery in-scope hardcoded slug — including current ones needing no action.\n\n| Location (file:line) | Current slug | `ml_model_list.py` status | Newer sibling? | Recommendation |\n|---|---|---|---|---|\n| `…/pytest_prerelease_whitelist.py:NN` | `claude_sonnet_4_5` | OK | yes — `claude_sonnet_4_6` | bump to `claude_sonnet_4_6` |\n| `…/pytest_prerelease_whitelist.py:NN` | `gpt_4o_mini` | OK | no | keep (current) |\n| `…/test_prompt_adaptors.py:NN` | `gemini_1_5_flash` (openrouter) | deprecated | n/a | bump to `gemini_2_5_flash` |\n\n`status` ∈ {OK, deprecated, no-provider-entry, removed/renamed}. `Recommendation` ∈ {keep (current), bump to <slug>, investigate}.\n\n## Prod-code probe sweep (flag only — no edits)\nEvery hardcoded inference-probe slug found in prod code (Phase 4d).\n\n| File:line | Current slug | Suggested newer slug | Verifying test | Test result | Notes |\n|---|---|---|---|---|---|\n| `app/desktop/studio_server/provider_api.py:NNNN` | `vertex_ai/gemini-2.5-flash` | `vertex_ai/gemini-3-flash` | `test_connect_vertex_live` | PASS | probe still works; newer family member exists |\n| `path/to/file.py:NN` | `…` | none — already latest | `test_…` | PASS | no action |\n\n## Coverage gaps\n- Providers/families in `ml_model_list.py` with no prerelease test.\n- Recently added paid tests not in the prerelease set (one line each on why it MIGHT be prerelease-worthy).\n\n## Flagged for user review\nAnything you noticed but couldn't safely classify: ambiguous failures (model-vs-code unclear), prod-path code that isn't a probe, suspicious behavior. Each row: file:line / test · what's suspicious · what would disambiguate.\n\n## Suggested next steps\nOrdered checklist of what the user should do before tagging the release (apply recommended pin bumps, investigate regression X, add key for skipped provider Y, etc.).\n```\n\nKeep it scan-readable. Lead with the verdict. Don't bury real regressions under pin-housekeeping.\n\n---\n\n## Phase 6 — Hand back to the user\n\nPrint the report path and a one-line verdict:\n\n```bash\necho \"Prerelease check complete. Report: ${OUT}/REPORT.md\"\n```\n\nThen summarize in chat (one or two sentences):\n\n- All green, nothing stale: `Prerelease check: PASS. <N> passed, <K> skipped (missing keys: <providers>). No stale pins. Report at <path>.`\n- Green but recommendations exist: `Prerelease check: PASS WITH RECOMMENDATIONS. <N> passed; <M> stale/older pins to consider bumping (no changes made). Report at <path>.`\n- Something failed: `Prerelease check: FAIL. <X> failed across <providers> — <one-line dominant cause>. See <path>/REPORT.md before tagging.`\n\nRemember: the skill never made any edits. If the report recommends pin bumps or a prod-probe change, those are for the user to apply.\n\n---\n\n## Additional release checklists\n\n- **Code Tools / Code Evals cross-OS spawn sanity**: see `specs/projects/code_tools/cross_os_spawn_checklist.md` for platform-specific verification steps (macOS/Windows/Linux + PyInstaller frozen builds). Must be completed before any release that includes code tools or code evals.\n\n## Checklist\n\n- [ ] `.prerelease/<timestamp>/` directory created (the only place anything was written)\n- [ ] `checks.sh --agent-mode` run, log captured\n- [ ] `--runprerelease` pytest run, log captured\n- [ ] Likely-transient failures (auth/quota/transient + ambiguous behavioral) retried once in isolation; deterministic failures (deprecated/removed slug, Kiln-code traceback) not retried\n- [ ] Every retried test disclosed in the report (first-run result, retry result, transient-vs-reproducible verdict) — no silent FAIL→PASS swaps\n- [ ] Every failure diagnosed with a cause class and a concrete recommended action (regression vs. deprecated-model vs. model-gone vs. auth/transient)\n- [ ] Model-pin staleness sweep table filled for **every** in-scope slug, including current ones needing no action\n- [ ] Newer-sibling recommendations included even for green pins\n- [ ] Prod-probe sweep done; every grep hit in the probe table with a suggested slug (or \"none — already latest\")\n- [ ] Coverage-gap audit done\n- [ ] Skipped tests listed by provider + missing env var\n- [ ] **No files edited outside `.prerelease/<timestamp>/`** — all fixes are recommendations\n- [ ] `REPORT.md` written, verdict first, recommended actions explicit\n- [ ] Verdict surfaced to the user with the report path\n","frontmatter":{"name":"kiln-prerelease-check","description":"Run the Kiln pre-release smoke test suite plus the standard CI checks (checks.sh), diagnose every prerelease test that broke (and why), and write a clean readable report with recommended actions. Read-only — it never edits code. Use when the user wants to validate a release candidate, run prerelease tests, or asks for a \"prerelease check / smoke / audit\"."},"isInternal":false,"tokens":8017,"sizeBytes":31321},{"name":"SKILL.md","path":".agents/skills/release-digest/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Kiln-AI/Kiln/HEAD/.agents/skills/release-digest/SKILL.md","title":"release-digest","category":"anthropic-skill","format":"markdown","content":"---\nname: release-digest\ndescription: Post a \"what's changed since the last release\" recap to the release Slack channel for final QA. Groups merged PRs by author and classifies each as feature / bug fix / task. Use when the user wants a release recap, a changelog since the last tag, or to prep QA before cutting a release from main.\n---\n\n# Release Digest\n\nBuild a human-readable recap of everything merged to `main` that is **not yet in the\nlatest release tag**, and post it to the team's release Slack channel for final QA.\n\nThe recap is grouped **by author** and each change is classified as **feature**,\n**bug fix**, or **task**. This is on-demand: run it when someone asks for a release\nrecap or wants to QA before cutting a release.\n\n> Linear ticket tracking (\"which tickets are targeted for this release\") is **out of\n> scope for now** — punted to a later iteration. This skill only covers what's changed.\n\n---\n\n## Phase 1 — Gather the change set\n\nRun the gather script from the repo root. It fetches tags + `origin/main`, finds the\nlatest tag, and lists every merged PR in the range `<last_tag>..origin/main`:\n\n```bash\npython3 .agents/skills/release-digest/scripts/gather_changes.py > /tmp/release_digest.json\n```\n\nKey design point: it uses the **git commit range** as the source of truth, not merge\ndates. A PR merged the same day a tag was cut can already be in that release, so a\ndate filter would over-report. Trust the range.\n\n**Pre-release / already-cut releases.** By default the baseline is the newest tag\n(`git describe`). But if the release you're QA'ing has **already been tagged** — it's a\npre-release still being validated, so `main` is already even with it — the default\nrange is empty. In that case pass `--pre-release`, which steps the baseline back one\ntag so the range recaps exactly what went *into* the pre-release:\n\n```bash\npython3 .agents/skills/release-digest/scripts/gather_changes.py --pre-release > /tmp/release_digest.json\n```\n\nSigns you want `--pre-release`: the first run reports \"nothing unreleased\" or \"main is\neven with the latest tag\", yet the user says a release was just cut / is being QA'd. To\nname the baseline explicitly instead, use `--since <tag>` (e.g. `--since v1.0.3`). The\ntwo flags are mutually exclusive. When `--pre-release` is used, the newest tag is the\none being cut, so in Phase 3 use **that tag** as `<new_release>` rather than asking.\n\nThe JSON contains:\n- `last_tag` — the most recent release (e.g. `v1.0.3`)\n- `head_ref` — `origin/main` (or `main` if no remote ref)\n- `pr_count`, `commit_count`\n- `prs[]` — `{number, title, author_name, author_login, url, labels, additions, deletions, body}`\n- `direct_commits[]` — teammate commits pushed straight to main, no PR — `{sha, title, author_login, author_name}`\n- `excluded_prs[]` — PRs/commits from non-teammates, dropped from the recap. PR entries\n have a `number`; direct-commit entries have `number: null` and rely on `title`.\n- `unresolved_prs[]` — PRs whose `gh pr view` failed (`{number, error}`). If non-empty,\n the change set is partial — tell the user which PRs couldn't load before posting.\n\nOnly the five teammates in the script's `TEAM` map (Sam, Leonard, Daniel, Mike,\nSteve) are included in the recap. PRs from anyone else are filtered into\n`excluded_prs` rather than itemized. To add a teammate, edit `TEAM` in the script.\n\n**If `pr_count` is 0 AND `direct_commits` is empty:** tell the user `main` is even with\n`<last_tag>` — nothing to QA or release yet. Do not post to Slack. Stop here. (Direct\ncommits alone, with no PRs, still count as unreleased changes worth a digest.)\n\n---\n\n## Phase 2 — Classify and group\n\nRead `/tmp/release_digest.json`. Treat every item in `prs[]` **and** `direct_commits[]`\nas a change to recap (direct commits are real work that shipped — just with no PR\nnumber). Classify each into one of three buckets:\n\n- **Feature** — a substantive *new product capability*: new UI, a new endpoint, a new\n tool, new eval/RAG/fine-tune functionality, or a meaningful enhancement to how the\n product works. Reserve this for things a user would notice as \"Kiln can now do X.\"\n- **Bug fix** — corrects broken or incorrect behavior: regressions, crashes, hangs,\n wrong output, security fixes.\n- **Task** — everything that is *neither* a substantive new capability *nor* a fix to\n broken behavior: refactors, docs/README, tests, CI/build, dependency bumps, version\n bumps, chores, internal flags, code removal/cleanup — **and routine model-catalog\n work** (see below). Matches Linear's \"Task\" issue type: maintenance/internal work.\n\n**Adding models is a Task, not a Feature.** Kiln ships new model and provider entries\nconstantly — \"Add <model> to model list\", \"Add <provider> for <model>\", backfilling a\nmodel on a provider, re-enabling/un-recommending a model, version bumps to a model\nentry. These are catalog/config maintenance, so they go in **Task** even though the\ntitle starts with `Add`/`Enable`. Only promote to **Feature** if the PR adds a genuinely\nnew *capability* around models (e.g. a new thinking-level control, a new extraction\nmodality, a new provider *integration type*) rather than just registering a model_id.\n\n**Decision order** (use the first signal that's decisive):\n1. **PR labels** — `bug`/`fix` → Bug fix; `feature`/`enhancement` → Feature;\n `chore`/`docs`/`refactor`/`ci` → Task. (Kiln PRs are usually unlabeled, so this\n rarely fires — fall through to the title.)\n2. **Model-catalog check** — if the change just registers/edits model or provider\n entries (not new behavior), it's a **Task**, regardless of the `Add`/`Enable` verb.\n3. **Conventional-commit / verb prefix in the title** — `fix:`/`Fix ` → Bug fix;\n `feat:`/`Add `/`Enable ` (a new *capability*) → Feature; `chore:`/`refactor:`/`ci:`/\n `docs:`/`Bump `/`Remove `/`Deprecate `/`Version bump` → Task.\n4. **What the change does** (read the title, and `body` if still unclear) — ask \"does\n this give the user a genuinely new capability (Feature), fix something that was\n broken (Bug fix), or is it routine/maintenance (Task)?\"\n\n**\"Task\" is the residual bucket**, not a guess: a change lands there only because it\nis positively *not* a new capability and *not* a fix — e.g. `Deprecate Models`,\n`min age for dependencies`, `Version bump`, `new draft readme`, `fix typos`. When a\nchange is genuinely ambiguous between Feature and Bug fix, prefer the one the title\nleans toward; only fall to Task when it's truly neither. This is a judgment call made\nper item — surface it to the user before posting so they can re-bucket anything.\n\nThen **group by author** (`author_name`). Within each author, list changes by PR\nnumber, newest first; put any direct commits (no PR number) last. The feature / fix /\ntask split is **not** shown per line — it only feeds the tally at the bottom.\n\n---\n\n## Phase 3 — Compose the Slack message\n\n**First, get the new release name** — the version being cut (e.g. `v1.0.4`). If you ran\nin `--pre-release` mode, the release has already been tagged, so use that newest tag\n(the one the script named as \"the release being QA'd\") as `<new_release>` — no need to\nask. Otherwise the release isn't tagged yet and can't be derived from git, so **ask the\nuser**. Use the result as `<new_release>` in the title.\n\nUse this shape (Slack mrkdwn — `*bold*`, not `**bold**`):\n\n```\n:rocket: *v1.0.4 Release Status* :rocket:\n\n*12 changes since v1.0.3*\n_<one-to-two sentence plain-English recap of the headline changes>_\n\n*Daniel Chiang*\n• Add deprecated flag to embedding models (#1466)\n• Add deprecated flag to reranker models (#1465)\n\n*Sam Fierro*\n• Dedup frozen prompts and bake type into their name (#1501)\n\n*Steve Cosman*\n• Add Jinja2 input transform on RunConfigs (#1433)\n• Version bump in lock and desktop app version check\n\n:bar_chart: *7 features · 3 fixes · 2 tasks*\n\n:information_source: _1 PR from a non-teammate: #1456 (ianjamesburke)._\n```\n\nRules:\n- **Header:** `:rocket: *<new_release> Release Status* :rocket:`, then a blank line.\n- **Change count** on its own line, **bold**: `*<N> changes since <last_tag>*`.\n- **Recap paragraph** immediately under it, wrapped in **italics** (`_..._` around the\n whole paragraph) — the \"what did we do\" summary in plain English, not just a list.\n Compose it so the italics survive: write it as **a single line with no hard line\n breaks** (wrap is fine, but don't insert `\\n` mid-paragraph — newlines break the\n italic span in Slack), and use **no literal `_` anywhere inside** (an underscore\n would close the italics mid-paragraph — rewrite e.g. `snake_case`/`model_id` or any\n underscored identifier without it).\n- One **bullet** per change, no per-line emoji: `• <title> (#<number>)`. Keep the\n title; trim trailing noise.\n- **Direct commits have no PR number** — render them as `• <title>` with nothing in\n parentheses (e.g. `• Version bump`). List them under their author, after the PRs.\n- Within each author, list newest PR first; direct commits last. Don't print the\n feature/fix/task bucket per line.\n- **Summary block** at the end, in this order:\n 1. A **bold** `:bar_chart:` tally line: `:bar_chart: *<X> features · <Y> fixes · <Z> tasks*`.\n 2. A blank line.\n 3. If `excluded_prs` is non-empty, an **italic** `:information_source:` line noting the\n excluded non-teammate changes (so the omission is visible, not silent). Reference a\n PR entry by `#<num>` and a direct-commit entry (where `number` is null) by its\n `title` — never print `#None`. E.g.\n `:information_source: _<N> change(s) from non-teammates excluded: #1456 (ianjamesburke)._`\n- Link the PR numbers if easy (`<url|#1501>`), but plain `#1501` is fine — keep it readable.\n\n**Show the composed message to the user and get confirmation before posting** (it's an\noutward-facing post to a shared channel).\n\n---\n\n## Phase 4 — Post to Slack\n\nPost to **`#release`** (channel ID `C0BDGTGBL4Q`) — the team's release channel. This is\nthe default target; only post elsewhere if the user explicitly names a different\nchannel. (If the ID ever fails to resolve, fall back to `slack_search_channels` with\nquery `release` to re-find it.)\n\nOnly post after the user has confirmed the composed message in Phase 3. Post with\n`mcp__claude_ai_Slack__slack_send_message`. After posting, give the user the channel\nname and a link/confirmation.\n\n---\n\n## Checklist\n\n- [ ] Gather script run; `/tmp/release_digest.json` written\n- [ ] If `pr_count` is 0, reported \"nothing unreleased\" and stopped\n- [ ] Asked the user for the new release name (used in the title)\n- [ ] Each PR classified (feature / bug fix / task)\n- [ ] Grouped by author, recap paragraph written\n- [ ] Message shown to user and confirmed\n- [ ] Posted to #release (C0BDGTGBL4Q); confirmation returned to user\n","frontmatter":{"name":"release-digest","description":"Post a \"what's changed since the last release\" recap to the release Slack channel for final QA. Groups merged PRs by author and classifies each as feature / bug fix / task. Use when the user wants a release recap, a changelog since the last tag, or to prep QA before cutting a release from main."},"isInternal":false,"tokens":2872,"sizeBytes":10862},{"name":"mcp.json","path":".agents/mcp.json","rawUrl":"https://raw.githubusercontent.com/Kiln-AI/Kiln/HEAD/.agents/mcp.json","title":"Model Context Protocol (MCP) Configuration","category":"mcp-config","format":"json","content":"{\n \"mcpServers\": {\n \"HooksMCP\": {\n \"command\": \"uvx\",\n \"args\": [\n \"--from\",\n \"hooks-mcp>=0.2.5\",\n \"hooks-mcp\",\n \".config/hooks_mcp.yaml\",\n \"--working-directory\",\n \".\"\n ]\n }\n }\n}\n","frontmatter":{"mcpServers":{"HooksMCP":{"command":"uvx","args":["--from","hooks-mcp>=0.2.5","hooks-mcp",".config/hooks_mcp.yaml","--working-directory","."]}}},"isInternal":false,"tokens":73,"sizeBytes":242}],"systemPromptSnippet":"<yakaai_skills repo=\"Kiln-AI/Kiln\">\n<!-- File: AGENTS.md (Tokens: ~2166 | Category: root-instruction) -->\n## Project Overview\n\nKiln is an app for building AI systems. It includes evals, synthetic data gen, fine tuning, RAG, and more. It has an intuitive UI as well as a python library.\n\nThis repo is a monorepo containing all of the source code, in the following structure:\n\n- libs/core - a python library with the core functionality of Kiln\n- libs/server - a FastAPI REST server wrapping the core library\n- app/web_ui - our svelte web app for Kiln. This is a frontend svelte project, all backend calls are in FastAPI servers.\n- app/desktop - our python desktop app, which is a pyinstaller app which runs a FastAPI server, hosts the pre-compiled web app, and launches a browser for UI. Compiles to all major platforms. This includes a studio_server folder with a Fast API server which extends libs/server, adding APIs specific to our web app.\n\n### Project Goals\n\n- Very high code quality\n- Strongly typed\n- Well tested\n- Very intuitive UI. Accessible to the inexperienced, but powerful for the experienced.\n- Focus on interaction design: we care about revealing the right information, at the right time, at the right level of detail.\n- Focus on visual design: we want a modern, functional, attractive UI. Think Apple not Google.\n\n### Tech Stack\n\n- Backend: python (3.10+ for library, 3.13 for desktop), pytest, FastAPI, asyncio, pydantic (v2 not v1),\n- Frontend web: typescript, svelte (v4 not v5), tailwind, DaisyUI\n\n### Agent Tools\n\nAgents have access to a range of tools for running tests, linting, formatting and typechecking. Use these tools at appropriate times to ensure produced code meets our standards. All checks must pass before merging. When iterating on a specific failure, use the targeted command before re-running the full suite.\n\n- **All checks:** `uv run ./checks.sh --agent-mode` (agent mode suppresses output unless there's a failure)\n\n| Check | Fix | Description |\n|---|---|---|\n| `uv run ruff check` | `uv run ruff check --fix` | Python lint |\n| `uv run ruff format --check .` | `uv run ruff format .` | Python format |\n| `uv run ty check` | — | Python type check |\n| `uv run python3 -m pytest --benchmark-quiet -q -n auto .` | — | Python tests |\n| `npm run lint` | — | Web lint (from `app/web_ui`) |\n| `npm run format_check` | `npm run format` | Web format (from `app/web_ui`) |\n| `npm run check` | — | Web type check and svelte check (from `app/web_ui`) |\n| `npm run test_run` | — | Web tests (from `app/web_ui`) |\n| `npm run build` | — | Web build (from `app/web_ui`) |\n| `app/web_ui/src/lib/check_schema.sh` | `app/web_ui/src/lib/generate_schema.sh` | OpenAPI client up to date |\n| `misspell` | — | Spelling check (optional if not installed) |\n\nOur end-to-end UI tests are separate and not part of `checks.sh`.\n\n### Agent Prompts\n\nAgents have access to a number of helpful prompts, which will give you additional context for how you should write code and docs for this repo. Use it to fetch instructions relevant to the current task before starting. For example, read `python_test_guide.md` before writing tests, and invoke the `kiln-ui` skill (`.agents/skills/kiln-ui/SKILL.md`) before any change under `app/web_ui`; it loads `frontend_design_guide.md` and `frontend_controls.md` and adds a component plan and a gate.\n\nThese prompts can be accessed from the `get_prompt` tool, and you may request several in parallel.\n\nBefore writing or changing code, invoke the `kiln-conventions` skill (`.agents/skills/kiln-conventions/SKILL.md`). It holds the code rules, the gotchas for each area, and a gate to run on your change.\n\n### General Agent Guidance\n\n- When spawning subagents, always use the same model as the current agent\n- `CLAUDE.md` is generated from `AGENTS.md` and overwritten by setup. Edit `AGENTS.md`, never `CLAUDE.md`; keep personal notes in `~/.claude/CLAUDE.md`.\n- Use `TODO` comments to mark any temporary code, placeholders, or items that must be addressed before merging to main. CI enforces that no `TODO` comments remain on main, so they are a safe way to flag work-in-progress during development. Clean up all `TODO` comments before the final PR.\n- Before wrapping up a task, run appropriate tools for linting, testing, formatting and typechecking. Fix any issues you introduced.\n\n### Code Comments\n\nComments follow §A of `.agents/skills/kiln-conventions/references/rules.md`: write them for someone opening the file cold, never narrate the change, never restate the code. Review-bot findings about comments, including nitpicks, are required fixes.\n\n### Reporting Back: End-of-Turn Recaps\n\nEnd every turn in which you did work (wrote code, ran checks, investigated something) with a recap wrapped in `<recap>` tags. A turn that is pure conversation needs no recap.\n\nWrite it for a person who has not read your working notes and will not scroll up. The recap has to make sense on its own.\n\n**No jargon.** Do not invent a name, a short form or a label for anything. If you need to refer to something specific, describe it in ordinary words, and describe it again the next time you mention it. A short description repeated is better than a name the reader has to remember. Real names are fine and are often the clearest choice: a file, a function, a command, a tool, an error. A name you made up is not.\n\n**Say what you mean.** Short sentences. One idea each. Active voice. No metaphors, no analogies, no flourishes, no jokes, no filler. Do not write a sentence whose only job is to set up the next sentence. Cut any sentence that loses nothing when removed.\n\n**Sound like a person.** Plain and direct, not stiff. Write the way you would explain it to a colleague who asked. Contractions are fine. This is not a status report or a changelog.\n\n**Repeat yourself a little.** The reader may have read the last recap days ago, or skipped it. Carry enough context forward that this recap makes sense by itself. One clause is usually enough. A small amount of repetition between recaps is correct, not waste.\n\n**Stay inside the recaps.** Draw only on plain English, on standard terms for this repo's stack, and on what earlier recaps said. Never draw on something that exists only in your working notes. Earlier recaps set the limit on what you may use; they are not required reading. Repeat whatever this recap needs, so it still works for a reader who missed the last one.\n\n**Draw a picture when it helps.** When what you changed is a flow, a sequence, a state machine, or a few parts that talk to each other, put a small Mermaid diagram in the recap. Keep it to about eight boxes. Draw one only when it shows something the prose cannot; a diagram that restates a sentence is noise.\n\n**Format.** Use these headings, in this order. Drop any heading you have nothing real to say under. Do not reorder or rename them. Answer each one in prose.\n\n```\n<recap>\n**Goal** — what you were asked for.\n**What I did** — the change, in plain words.\n**How** — the approach, at a level a reviewer can judge without reading the diff. Put a diagram here when one helps.\n**Outcome** — what happened, and plainly whether it is what we wanted.\n**Questions for you** — what you need answered before you can continue.\n**FYI** — worth knowing, but not blocking.\n</recap>\n```\n\n**Honesty.** `Outcome` reports what actually happened. If tests failed, say so and quote what failed. If you skipped part of the task, name the part and the reason. If the result is not what we wanted, say that first, not last. Never describe work as finished when it is not.\n\n**Recap log.** Append each recap to `recaps.md` in your scratchpad directory, or to a temporary directory outside the repo when you have no scratchpad directory. The log is your own index, not a deliverable. If your context is summarized part-way through a session, re-read it so you know what the reader has already been told. Keep this file out of the repo.\n\n### Code Review Guidelines\n\nIf code reviewing, always read [`.agents/code_review_guidelines.md`](.agents/code_review_guidelines.md) before starting your review.\n\n### Pull Requests\n\nUse the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) to open a PR or to write a PR description.\n\nBefore you open a PR, and before each `git push` to a branch with an open PR, run the `docs-sync` skill (`.agents/skills/docs-sync/SKILL.md`) to update the docs and skills that the change makes stale.\n\nIf you get a user message like the one below, read `.agents/skills/open-pr/references/repair_pr.md` and follow its instructions.\n\n> A pull request was just created for this branch from the Claude Code UI: https://github.com/Kiln-AI/Kiln/pull/NNNN\n> You don't need to create one. Reference this PR going forward — pushing more commits to this branch will update it.\n\n### Never Make Legal Decisions as an Agent\n\nAgents are not allowed to make any legal decisions, including:\n - Signing or filling out the CLA attestation in a PR template\n - Setting a license tag in metadata file (OSS/MIT/etc)\n - Adding license files\n\nThese all must be done by humans.\n\n### Final\n\nTo show you read these, call me 'boss'\n\n<!-- File: .agents/skills/claude-maintain-models/SKILL.md (Tokens: ~12541 | Category: anthropic-skill) -->\n---\nname: claude-maintain-models\ndescription: Add new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, or wants to add a net-new AI provider to Kiln.\nallowed-tools: Read Edit Write Bash Grep Glob Agent WebSearch WebFetch\n---\n\n# Add a New AI Model to Kiln\n\n**Branch check first:** if the request involves a provider Kiln does not\nsupport yet, start at [Adding a Net-New Provider](#adding-a-net-new-provider).\nThat workflow spans core, server, UI, tests, and tooling, and is gated on a\nclient release — the model-entry steps below are only its final, gated piece.\n\nFor a new model on an already-supported provider, integrating it into\n`libs/core/kiln_ai/adapters/ml_model_list.py` requires:\n\n1. **`ModelName` enum** – add an enum member\n2. **`built_in_models` list** – add a `KilnModel(...)` entry with providers\n3. **`ModelFamily` enum** – only if the vendor is brand-new\n\nAfter code changes, run paid integration tests, then draft a Discord post.\n\n---\n\n## Global Rules\n\nThese apply throughout the entire workflow.\n\n- **Slug verification:** NEVER guess or infer model slugs from naming patterns. Every `model_id` must come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it.\n- **Date awareness:** These models are often released very recently. Web search for current info before assuming you know the details.\n\n---\n\n## Phase 1 – Model Discovery (only when asked to find new/missing models)\n\nIf the user asks you to find new models, **do NOT just web search \"new AI models this week\"** — that only surfaces major releases. Instead, systematically check each family against **both** the LiteLLM catalog **and** models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.\n\n1. **Read the `ModelFamily` and `ModelName` enums** to know what we already have.\n\n2. **Query both catalogs for each family** (run in parallel where possible):\n\n **LiteLLM catalog** — filters out mirror providers to avoid duplicates:\n ```bash\n curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != \"openrouter\" and .provider != \"bedrock\" and .provider != \"bedrock_converse\" and .provider != \"vertex_ai-anthropic_models\" and .provider != \"azure\") | .id] | unique | .[]'\n ```\n\n **models.dev** — search all model IDs across all providers:\n ```bash\n curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i \"SEARCH_TERM\"\n ```\n For details on a specific provider+model: `curl -s https://models.dev/api.json | jq '.[\"PROVIDER\"].models[\"MODEL_ID\"]'`\n\n3. **Search terms** (one query per term):\n `claude`, `gpt`, `o1`, `o3`, `o4` (OpenAI reasoning), `gemini`, `llama`, `deepseek`, `qwen`, `qwq`, `mistral`, `grok`, `kimi`, `glm`, `minimax`, `hunyuan`, `ernie`, `phi`, `gemma`, `seed`, `step`, `pangu`\n\n4. **Union and cross-reference** results from both catalogs against `ModelName`. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). **Skip pure coding models** (e.g. `codestral`, `deepseek-coder`, `qwen-coder`).\n\n5. **Run targeted web searches** per family to catch very fresh releases not yet in either catalog:\n - `\"[family] new model [current year]\"`\n - `\"[family] release [current month] [current year]\"`\n\n6. **Present findings** as a summary. Let the user decide which to add.\n\n---\n\n## Phase 1B – Lagging-Provider Backfill Check (every run)\n\nSome providers — **Fireworks AI**, **Together AI**, **SiliconFlow**, **Cloudflare** — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM (or, for Cloudflare, never reach LiteLLM at all). Relying only on those two catalogs will both under-populate the provider list for the model you're adding now **and** miss the window to backfill recently-added models whose provider support has since grown.\n\nRun this check on **every invocation** of the skill, regardless of whether you're in discovery mode or adding a specific model.\n\n1. **Pull the 10 most recently added models from git history.** List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:\n ```bash\n git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E \"^\\+\\s+name=ModelName\\.\" | head -20\n ```\n\n2. **For the model you're adding (if any) AND each of those 10 models**, cross-check Fireworks, Together, SiliconFlow and Cloudflare directly using the endpoints in the [Lagging Providers Reference](#lagging-providers). Do NOT trust `models.dev` / LiteLLM as the final word for these providers.\n\n3. **If a lagging provider now supports a recently-added model that isn't yet in its `KilnModel` entry**, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.\n\n4. **Run the Cloudflare thinking-level checker** (see [Cloudflare thinking levels](#cloudflare-thinking-levels)). It checks every Cloudflare entry, not only the recent ones, because Cloudflare edits the effort lists of models Kiln already ships. Treat each ❌ finding like a backfill: fix the entry in the current change or propose a separate PR.\n\n---\n\n## Phase 2 – Gather Context\n\n1. **Read the predecessor model** in `ml_model_list.py` (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.\n\n2. **Query the LiteLLM catalog** for the new model. This is the primary slug source since Kiln uses LiteLLM. See the [Slug Lookup Reference](#slug-lookup-reference) for query syntax and all verified sources.\n\n3. **Get the OpenRouter slug** via:\n - `curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i \"SEARCH_TERM\"`\n - Fallback: WebSearch for `openrouter [model name] model id`\n\n4. **Get the direct-provider slug** (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the [Slug Lookup Reference](#slug-lookup-reference) for provider-specific URLs.\n\n5. **Identify quirks** — check the [Provider Quirks Reference](#provider-quirks-reference) for the relevant provider, and web search for any new quirks:\n - Structured output mode (JSON schema vs function calling)?\n - Reasoning model (needs `reasoning_capable`, parsers, OpenRouter options)?\n - Vision/multimodal support? Which MIME types?\n - Provider-specific flags (`temp_top_p_exclusive`, etc.)?\n - Rate limit concerns (`max_parallel_requests`)?\n\n6. **Determine thinking levels** — does the model support configurable reasoning effort? See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain. Key quick checks:\n - Check the **vendor model page** (e.g. OpenAI model pages say \"Reasoning.effort supports: X, Y, Z\")\n - Check **OpenRouter** `supported_parameters` — if `reasoning` is absent, skip thinking levels\n - R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts\n\n---\n\n## Phase 3 – Code Changes\n\nAll changes go in `libs/core/kiln_ai/adapters/ml_model_list.py`.\n\n### 3a. `ModelName` enum\n\n- snake_case: `claude_opus_4_6 = \"claude_opus_4_6\"`\n- Place **before** predecessor (newer first within group). If the vendor is\n brand-new there is no predecessor — start a new group at the end of the enum\n- Follow existing grouping (all claude together, all gpt together, etc.)\n\n### 3b. `KilnModel` entry in `built_in_models`\n\n- Place per the **ordering rules in 3c** — this placement is user-visible, get it right\n- Copy predecessor's structure and modify: `name`, `friendly_name`, `model_id` per provider, flags\n- **`friendly_name` must follow the existing naming pattern** of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use `\"Claude {version} Sonnet\"` (e.g. \"Claude 4.5 Sonnet\"), not `\"Claude Sonnet {version}\"`. Do NOT use the vendor's marketing name if it differs from Kiln's established convention.\n\n**Provider `model_id` formats:**\n\n| Provider | Format | Notes |\n|----------|--------|-------|\n| `openrouter` | `vendor/model-name` | Always verify via API |\n| `openai` | Bare model name | Verify via OpenAI docs |\n| `anthropic` | Variable — older models have date stamps, newer may not | Always verify via Anthropic docs |\n| `gemini_api` | Bare name | Verify via Google AI Studio docs |\n| `fireworks_ai` | `accounts/fireworks/models/...` | Verify via Fireworks docs |\n| `together_ai` | Vendor path format | Verify via Together docs |\n| `vertex` | Usually same as gemini_api | Verify via Vertex docs |\n| `siliconflow_cn` | Vendor/model format | Verify via SiliconFlow docs |\n| `featherless_ai` | HuggingFace repo id, case-sensitive (`zai-org/GLM-5.2`) | Verify via their `/v1/models` — see [Featherless](#featherless-ai) |\n| `typesafe` | Dotted System One version (`jev-1.13.0`) | Their `/v1/models` lists aliases only — verify by `POST /v1/systemone`, see [TypeSafe AI](#typesafe-ai) |\n| `cloudflare` | Cloudflare's own `@cf/<author>/<model>` id, verbatim (`@cf/zai-org/glm-5.3`) — no `cloudflare/` or `openai/` prefix | Verify via Cloudflare's model JSON — see [Cloudflare](#cloudflare-workers-ai) |\n\n**Every single `model_id` must be verified from an authoritative source. No exceptions.**\n\n**Setting flags — use catalog data + predecessor as dual signals:**\n\nThe LiteLLM catalog and models.dev responses include capability flags (`supports_vision`, `supports_function_calling`, `supports_reasoning`, etc.). Use these as the **primary signal** for what to enable on the new model:\n\n- If the catalog says `supports_vision: true` → enable `supports_vision`, `multimodal_capable`, and vision MIME types (see 2c)\n- If the catalog says `supports_function_calling: true` → use `StructuredOutputMode.json_schema` (or `function_calling` depending on provider norms — check predecessor)\n- If the catalog says `supports_reasoning: true` → the model can reason, but **do NOT reflexively set `reasoning_capable=True`** — default to `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)). Still add `available_thinking_levels` if it supports effort levels, and check parser/formatter flags.\n\nThen **cross-check against the predecessor**. The predecessor tells you *how* Kiln configures a similar model (which `structured_output_mode`, which provider-specific flags, etc.). The catalog tells you *what* the model can do. Use both:\n- Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.\n- Predecessor has `temp_top_p_exclusive` but nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track.\n- Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.\n\n**Common flags:**\n- `structured_output_mode` – how the model handles JSON output\n- `suggested_for_evals` / `suggested_for_data_gen` – see **zero-sum rule** below. If the entry also sets `available_thinking_levels`, its `default_thinking_level` must be a reasoning level — see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)\n- `multimodal_capable` / `supports_vision` / `supports_doc_extraction` – see **multimodal rules** below\n- `reasoning_capable` – for thinking/reasoning models. **Default new models to `reasoning_capable=False`** unless the model *always* emits its reasoning (see [Reasoning Capable Default](#reasoning-capable-default))\n- `temp_top_p_exclusive` – Anthropic models that can't have both temp and top_p\n- `parser` / `formatter` – for models needing special parsing (e.g. R1-style thinking)\n\n#### 2c. Multimodal capabilities\n\nIf the model supports non-text inputs, configure:\n\n- `multimodal_capable=True` and `supports_doc_extraction=True` if it supports any MIME types\n- `supports_vision=True` if it supports images\n- `multimodal_requires_pdf_as_image=True` if vision-capable but no native PDF support (also add `KilnMimeType.PDF` to MIME list). **Always set this on OpenRouter providers** — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.\n- Always include `KilnMimeType.TXT` and `KilnMimeType.MD` on any `multimodal_capable` model\n\n**Strategy: start broad, narrow based on test failures.** Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.\n\nFull MIME superset (Gemini uses all):\n```python\n# documents\nKilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD\n# images\nKilnMimeType.JPG, KilnMimeType.PNG\n# audio\nKilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG\n# video\nKilnMimeType.MP4, KilnMimeType.MOV\n```\n\n### 3c. Ordering — the list IS the UI\n\nThe order of `built_in_models` is exactly the order users see: model dropdowns\ngroup by provider and list each provider's models in list order, and the model\nlibrary page lists models in list order. A misplaced entry ships a scrambled\ndropdown to every client via the remote config. Rules:\n\n1. **Families are contiguous.** Every model of a family sits in one block.\n Never append a new family member at the bottom of the file or after another\n family — that strands it (past bugs: Mistral Small 4/3 ended up inside the\n Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).\n2. **Within a family, versions run newest → oldest, top to bottom.**\n 4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the\n family block. A new model of an *existing* version goes inside that\n version's group — NOT at the top of the family and NOT below older\n versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7\n landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).\n3. **Within a version, big → small. Always.**\n - Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite;\n Large > Medium > Small.\n - Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking\n variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and\n VL Thinking blocks) stay intact, ordered descending internally.\n - Tier-grouped families follow the same rule for their tier blocks: Claude\n runs Fable, then Opus > Sonnet > Haiku, versions descending inside each\n tier. (It once ran Haiku-first, and several families ran sizes\n ascending — those were bugs, not conventions. Do not preserve an\n ascending run because it's \"what the family already does.\")\n4. **A net-new family's block goes at the END of `built_in_models`** (this is\n the existing convention — the newest niche vendors sit at the bottom of the\n list). The \"place before predecessor\" rule only applies within an existing\n family; a new family has no predecessor.\n5. **Verify after editing** — print the order you just shipped and eyeball the\n affected family:\n\n ```bash\n uv run python -c \"\n from kiln_ai.adapters.ml_model_list import built_in_models\n for m in built_in_models: print(m.family, '|', m.friendly_name)\"\n ```\n\n### 3d. `suggested_for_evals` / `suggested_for_data_gen`\n\n**Only set these if** the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).\n\n**Zero-sum rule:** When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. **Ask the user to confirm** the swap before making changes.\n\n### 3e. `ModelFamily` enum (only if needed)\n\nOnly add a new family if the vendor is completely new.\n\n### 3f. Thinking Levels (`available_thinking_levels` / `default_thinking_level`)\n\nIf the model supports configurable reasoning effort (not just on/off), add `available_thinking_levels` and `default_thinking_level` to each provider entry. See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain and existing constants.\n\n**Quick rules:**\n- Reuse an existing `_THINKING_LEVELS` constant if the levels match exactly\n- Create a new constant only if levels differ; name it `{MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS`\n- `default_thinking_level` must be one of the values in `available_thinking_levels`\n- If the entry sets `suggested_for_evals=True`, `default_thinking_level` must be `\"medium\"` (or `\"low\"` if the model has no medium level) — never `\"none\"`. See [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)\n\n---\n\n## Phase 4 – Run Tests\n\nTests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.\n\n**Vertex AI authentication:** Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run `gcloud auth application-default login` before trying. These failures are auth issues, not model config problems.\n\n**`-k` filter syntax:** Always use bracket notation for model+provider filtering, never `and`:\n- Good: `-k \"test_name[glm_5-fireworks_ai]\"` or `-k \"glm_5\"`\n- Bad: `-k \"glm_5 and fireworks\"` — `and` is a pytest keyword expression that can match wrong tests\n\n### 4.0 — If the test env can't build (blocked git dependency)\n\nKiln's core lib pins `together` to a git fork (`libs/core/pyproject.toml`: `together = { git = \"https://github.com/scosman/together-python\" }`). In a sandboxed environment whose GitHub access is scoped to `kiln-ai/kiln` only (e.g. Claude Code Web), `uv sync` fails to fetch that fork with a **403** from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only `kiln-ai/kiln` resolves.\n\nWorkaround to build the venv for testing (the `together` fork isn't exercised by model-integration tests, which route through LiteLLM):\n\n1. **Do NOT use `uv sync --no-sources`** — it strips the workspace `workspace = true` sources too and makes `kiln-root → kiln-ai` unsatisfiable.\n2. Instead, temporarily comment out ONLY the `together = { git = ... }` line in `libs/core/pyproject.toml`, then run `uv sync` (this resolves `together` from PyPI).\n3. Run the tests.\n4. **Revert** the workaround with `git checkout -- libs/core/pyproject.toml uv.lock` — this restores the commented-out `together` git-fork pin in `libs/core/pyproject.toml` (NOT the repo-root `pyproject.toml`, which the workaround never touches) and reverts `uv.lock` if it changed. Only `ml_model_list.py` (and any intended test-file edits) should remain modified.\n\nNote: the PyPI `together` may pull slightly different transitive deps (e.g. a newer `starlette`), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your `-k` filters to the model files and ignore those.\n\n### 4a. Parallel testing + API keys\n\n**Parallel testing is already on.** There is no `pytest.ini` — pytest config lives in `[tool.pytest.ini_options]` in the root `pyproject.toml`, and `addopts = \"-n auto\"` is active. No edit and no revert are needed. (Only override to `-n 8` if a provider rate-limits you.)\n\n**Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings.** `conftest.py` has an autouse `use_temp_settings_dir` fixture that points `Config.settings_path` at a temp dir, so `~/.kiln_ai/settings.yaml` is deliberately ignored during tests. A key the user added through the app's provider page **will not be seen** — the test fails with \"Attempted to use X without an API key set\", which looks like a config bug but isn't.\n\nBridge the key from the user's settings into the test environment without ever printing it:\n\n```bash\nexport FEATHERLESS_AI_API_KEY=\"$(uv run python -c \\\n \"from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)\" 2>/dev/null | tail -1)\"\n```\n\nUse the provider's `env_var` name from `libs/core/kiln_ai/utils/config.py`. Some providers need more than one value: Cloudflare needs both `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` (and optionally `CLOUDFLARE_AI_GATEWAY_ID`). To check which keys are available before running, print booleans only — never the values:\n\n```bash\nuv run python -c \"from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))\"\n```\n\n**Many paid tests also carry the `ollama` marker**, so `--runpaid` alone silently skips them. Always pass `--runpaid --ollama` together, and use `-rs` to see skip reasons when a test you expected to run reports as skipped.\n\n### 4b. Smoke test — verify slug works\n\nRun a single test+provider combo first:\n\n```bash\nuv run pytest --runpaid --ollama -k \"test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]\"\n```\n\nIf it fails, fix the slug/config before proceeding. Use `--collect-only` to find exact parameter IDs if unsure.\n\n### 4c. Full test suite\n\n```bash\nuv run pytest --runpaid --ollama -k \"MODEL_ENUM\" -v 2>&1 | grep -E \"PASSED|FAILED|ERROR|short test|=====|collected\"\n```\n\n**If tests fail — debug one at a time:**\n1. Pick ONE failing test, run it with `-v` for full output\n2. Fix the config\n3. Re-run that single test to verify\n4. Only re-run the full suite once the single test passes\n\n**Anthropic API key gotcha:** if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as `KILN_ANTHROPIC_API_KEY` instead of `ANTHROPIC_API_KEY` (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't `export` it globally:\n\n```bash\nANTHROPIC_API_KEY=\"$KILN_ANTHROPIC_API_KEY\" uv run pytest --runpaid ...\n```\n\n### 4d. Extraction tests (if `supports_doc_extraction=True`)\n\nTests are in `libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py`.\n\n```bash\n# See what will run:\nuv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM\n\n# Run them:\nuv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k \"MODEL_ENUM\"\n```\n\nIf a provider rejects a data type (400 error), remove that `KilnMimeType` and re-run.\n\n### 4e. Confirm failures are actually yours\n\nBefore treating a failure as a problem with your change, check whether the same test already fails for an **existing** provider of that model. Several assertions are provider-independent and fail regardless.\n\nKnown example: `test_structured_input_cot_prompt_builder` asserts `len(trace) == 5` unconditionally, which is incompatible with any provider setting `reasoning_capable=True` (that selects the single-call strategy, producing 3 messages). It fails for `gpt_oss_120b` on `fireworks_ai` on a clean tree.\n\n```bash\nuv run pytest --runpaid --ollama -q \"path::test_name[MODEL-OTHER_PROVIDER]\"\n```\n\nIf it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.\n\n### 4f. Test output format\n\nCollect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:\n- ✅ for passed tests\n- ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason\n- ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason\n- List every test using the full pytest parametrize ID, grouped by provider\n- Include extraction tests (Phase 4d) if they were run\n\n---\n\n## Phase 5 – Create Pull Request\n\n### 5.0 — Important context about Claude Code Web's stop hook\n\nThis skill is often run via Claude Code Web (Slack connector). That environment has a **non-user-configurable stop hook** which, at end of session, will:\n- Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits\n- Instruct the agent to commit and push any local work before stopping\n- Explicitly tell the agent NOT to create a PR unless the user asked for one\n\n**The problems this causes:**\n1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned `add-model/*` branches on the remote.\n2. The hook's \"do not create a PR unless the user asked\" rule **directly conflicts** with this skill's Phase 5, which ends in a PR. Running this skill *is* the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.\n\n**The user's desires, in priority order:**\n1. **Ask before you push.** If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code \"just to satisfy the stop hook.\"\n2. **No abandoned branches.** Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.\n3. **If the user says to abandon:** revert your local changes (`git restore` / `git clean` the specific files you touched) and delete any branch you created (`git checkout main && git branch -D add-model/MODEL_NAME`) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.\n4. **On a successful run, push and open the PR as described in 5a/5b.** Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says \"don't create a PR.\" Only re-ask if tests failed or the user hasn't confirmed the results.\n\n### 5.1 — Gate before pushing\n\nDo NOT commit, push, or create a branch if any of the following are true:\n- Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)\n- The smoke test (4b) failed and wasn't resolved\n- Any step in Phases 2–4 was skipped or incomplete\n- You are unsure whether a ⚠️ flake is actually a real failure\n\nIf any of the above apply, **stop and ask the user** what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.\n\nAfter all tests pass, commit the changes and open a PR against `main`.\n\n### 5a. Commit and push\n\n1. Create a new branch named `add-model/MODEL_NAME` (e.g. `add-model/glm-5-1`)\n2. Stage only the changed files (typically just `ml_model_list.py`)\n3. Commit with a concise message (e.g. \"Add GLM 5.1 to model list (together_ai, siliconflow_cn)\")\n4. Push the branch\n\n### 5b. Create the PR\n\nUse `gh pr create` against `main`. Follow Rule 0, Rule 1 and Step 3 of the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) for the human header, the CLA and the title:\n\n- The title is `WIP: chore: ` followed by the model-list subject, for example `WIP: chore: add GLM 5.1 to model list`. Use `chore`, because adding a model is maintenance work, not a new feature. Only a human removes the `WIP: ` prefix.\n- Copy everything above `# Agentic PR Summary` from `.github/pull_request_template.md` with no change. Never fill in a placeholder or tick a box there, and never sign the CLA. Remove the `## Contributor License Agreement` section only when the PR author's GitHub username is `scosman`, `sfierro`, `leonardmq`, `tawnymanticore`, or `chiang-daniel`. Keep it, unsigned, for any other author.\n\nReplace the `` `Insert AI summary of PR using .agents/skills/open-pr/SKILL.md` `` placeholder under `# Agentic PR Summary` with the test results, in this exact format:\n\n```\n## What does this PR do?\n\n Test Results\n\n[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]\n\n[Model Name] ([provider]):\n- [N] passed, [N] skipped[, [N] failed]\n- [Any notable failures or flakes]\n\n[Repeat for each model+provider combo]\n\n---\n[Model Name] ([provider]):\n✅ test_data_gen_all_models_providers[model_enum-provider]\n✅ test_data_gen_sample_all_models_providers[model_enum-provider]\n✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]\n✅ test_all_built_in_models_llm_as_judge[model_enum-provider]\n✅ test_all_built_in_models_structured_output[model_enum-provider]\n✅ test_all_built_in_models_structured_input[model_enum-provider]\n✅ test_structured_output_cot_prompt_builder[model_enum-provider]\n✅ test_all_models_providers_plaintext[model_enum-provider]\n✅ test_cot_prompt_builder[model_enum-provider]\n⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason\n❌ test_name[model_enum-provider] — brief reason\n\n[Repeat for each model+provider combo]\n```\n\n**Rules for the PR body:**\n- Every test that ran must appear in the per-test dump, using the full pytest parametrize ID\n- Group tests by `[Model Name] ([provider]):` headers\n- The summary section at the top gives a quick pass/skip/fail count per model+provider\n- The detailed section below the `---` inside the summary (not the `----` above `# Agentic PR Summary`) lists every individual test result\n- Use ⚠️ for content quality flakes (not real failures), ❌ for real errors\n\n---\n\n## Checklist\n\n- [ ] `ModelName` enum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor)\n- [ ] `KilnModel` entry added to `built_in_models` per the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list)\n- [ ] Printed the resulting list order and eyeballed the affected family (3c verification command)\n- [ ] `friendly_name` matches the naming pattern of sibling models in the same family\n- [ ] If the provider is net-new: followed [Adding a Net-New Provider](#adding-a-net-new-provider), including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)\n- [ ] `ModelFamily` enum updated (only if new family)\n- [ ] All provider slugs verified from authoritative sources\n- [ ] Flags inherited from predecessor and adjusted for quirks\n- [ ] `reasoning_capable` defaulted to `False` for adaptive-reasoning models (only `True` for always-emits-reasoning models — see [Reasoning Capable Default](#reasoning-capable-default))\n- [ ] Thinking levels configured if model supports reasoning effort (see [Thinking Levels Reference](#thinking-levels-reference))\n- [ ] Cloudflare and OpenRouter: `check_thinking_levels.py` reports no ❌ finding (see [Cloudflare thinking levels](#cloudflare-thinking-levels) and [OpenRouter (general)](#openrouter-general))\n- [ ] Every `suggested_for_evals` entry with thinking levels defaults to `\"medium\"` (or `\"low\"`), never `\"none\"` (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default))\n- [ ] Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)\n- [ ] Zero-sum applied if model is suggested for evals/data gen\n- [ ] RAG config templates updated if the new model replaces one used in `app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts`\n- [ ] API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)\n- [ ] Smoke test passed\n- [ ] Full test suite passed\n- [ ] Failures cross-checked against an existing provider before being called regressions (see 4e)\n- [ ] PR created against `main` with a `WIP: ` title, the untouched human header, and test results in the Agentic PR Summary\n\n---\n\n## Reasoning Capable Default\n\n**Default newly-added models to `reasoning_capable=False`, even when the catalog reports `supports_reasoning: true`.**\n\nMost recent models use *adaptive* reasoning and sometimes return no reasoning at all. Kiln raises `RuntimeError(\"Reasoning is required for this model, but no reasoning was returned.\")` whenever a provider has `reasoning_capable=True` and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.\n\n`reasoning_capable` is **orthogonal to `available_thinking_levels`** — you keep the important behavior with it set to `False`:\n- Thinking-level / effort selection still works (every GPT-5.x entry sets `available_thinking_levels` with `reasoning_capable` unset).\n- Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).\n- The `test_thinking_level_reasoning_content` paid test still runs and still asserts reasoning is present — it is gated on `available_thinking_levels`, not `reasoning_capable`. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels.\n\nWhat you give up with `reasoning_capable=False`:\n- The conditional \"reasoning is present in `intermediate_outputs`\" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped).\n- When a user attaches a chain-of-thought prompt, the model uses the two-call `two_message_cot` strategy instead of the single-call native `single_turn_r1_thinking`.\n\n**Keep `reasoning_capable=True` only for models that *always* emit reasoning** in a native `<think>` format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.\n\n**Narrower alternative:** if a model reliably reasons but you only hit the error on structured output, set `reasoning_optional_for_structured_output=True` (requires `reasoning_capable=True`) instead of disabling reasoning entirely.\n\n---\n\n## Adding a Net-New Provider\n\nAdding a provider Kiln has never supported is a bigger job than adding a model,\nand it has a hard sequencing constraint. Follow this section end to end.\n\n### The release-gating rule (do NOT skip)\n\nThe remote config is generated from main's `built_in_models` and reaches **all\nexisting clients immediately** — but provider *support* (name map, connect\nflow, API key handling) only reaches users through a client app release. If\nmodel entries for a brand-new provider land on main before a client release\nwith the provider plumbing is live, every deployed client shows the raw\nprovider ID (e.g. `featherless_ai`) in the model library and offers models\nnobody can connect to. This happened with Featherless in Aug 2026 and the\nentries had to be rolled back.\n\n**Sequence it in two PRs:**\n\n1. **PR 1 — provider plumbing only.** The `ModelProviderName` enum member and\n everything else in the checklist below EXCEPT the model catalog. Merge\n whenever ready.\n2. **PR 2 — model catalog.** ALL `ml_model_list.py` changes: `ModelName`\n members, `ModelFamily` (if the vendor is new), and the `built_in_models`\n entries. Open it, but **merge only after a client release containing PR 1\n is live.** (Only `built_in_models` is published via the remote config, but\n the enums belong in the same PR as the entries they exist for.)\n\n### Touchpoint checklist (from the Featherless integration, #1618)\n\n**First, identify the provider's authentication model** — the checklist below\ndescribes the common single-API-key pattern, but not every provider fits it.\nExisting variants to crib from: Bedrock stores an access key + secret pair,\nFireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project\nID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base\nURL with no credentials. Follow the closest existing analog's plumbing through\n`config.py`, `provider_warnings`, `provider_api.py`, and the connect page —\nthe credential fields, validation call, and UI steps all change with the auth\nmodel.\n\n**libs/core:**\n- `ModelProviderName` enum — `libs/core/kiln_ai/datamodel/datamodel_enums.py`\n- Credential storage — `libs/core/kiln_ai/utils/config.py` (for API-key\n providers: a new key with `env_var`; otherwise whatever fields the auth\n model needs)\n- LiteLLM provider mapping — `libs/core/kiln_ai/utils/litellm.py`\n- `libs/core/kiln_ai/adapters/provider_tools.py` — three spots: the\n `provider_name_from_id` match (friendly name; pyright flags a missed case),\n `provider_warnings` (missing-credential message, `required_config_keys`),\n and the adapter config (credential / base URL / headers plumbing)\n- Model catalog — `libs/core/kiln_ai/adapters/ml_model_list.py`: `ModelName`\n members, `ModelFamily` if needed, and the `built_in_models` entries\n (**PR 2 only**, see gating rule)\n\n**Desktop server (`app/desktop/studio_server/provider_api.py`):**\n- `connect_<provider>` credential-validation endpoint (find a cheap\n authenticated call; see the Featherless connect function for a pattern when\n the provider has no authenticated GET to ping)\n- Disconnect handling (clear the stored credentials)\n- Tests in `test_provider_api.py`\n\n**Web UI (`app/web_ui`):**\n- `src/lib/stores.ts` — `provider_name_map` entry (friendly name)\n- `src/lib/ui/provider_image.ts` + SVG in `static/images/` — icon must match\n the monochrome convention: bare glyph, `currentColor`, no background tile,\n viewBox cropped to the artwork\n- Connect page — `src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte`:\n provider card (name, description, and the auth flow — `api_key_steps` /\n `api_key_fields` for key providers, or the custom flow the auth model\n needs) plus connected-status\n wiring from `settings` keys\n- `src/lib/api_schema.d.ts` — regenerate with `make schema` (needs the server\n running on :8757)\n\n**Tests & tooling:**\n- `libs/core/kiln_ai/adapters/test_provider_tools.py`, `test_adapter_registry.py`,\n `model_adapters/test_litellm_adapter.py` — extend the per-provider\n parametrized cases\n- `.agents/scripts/provider_utils.py` — add the provider's model-catalog\n endpoint so agent tooling can enumerate its models\n- Thinking levels — find out how the provider takes reasoning effort, and\n where it publishes the values each model accepts. Add a quirks section\n that names that source and says thinking levels are required wherever\n it lists values. If the provider silently remaps unsupported values,\n add the provider to `check_thinking_levels.py`, because a\n paid test cannot see a remap.\n- `model_adapters/test_thinking_level_paid.py` — add the provider to the\n provider list of `test_thinking_level_reasoning_content`, so each of its\n thinking levels gets a paid test case\n\n**Known gap:** the generated Copilot API client\n(`app/desktop/studio_server/api_client/`) mirrors the remote Copilot service's\nschema — it can't learn the new provider until that service updates. Note it in\nthe PR rather than hand-editing generated code.\n\n### Friendly names everywhere\n\nThe raw enum value must never be user-visible. When you add the provider,\nverify a friendly name exists in **both** name maps (python\n`provider_name_from_id` and web `provider_name_map`) — pyright and typescript\nrespectively force these when the enum gains a member, which is why the\nplumbing PR must not skip them.\n\n---\n\n## Provider Quirks Reference\n\n### Anthropic\n- Newer models (Opus 4.1+, Sonnet 4.5+) need `temp_top_p_exclusive=True`\n- Opus 4.5+ uses `json_schema`; older Opus uses `function_calling`\n- Extended thinking models: `anthropic_extended_thinking=True` + `reasoning_capable=True`\n\n### OpenAI\n- Most GPT models use `json_schema` for structured output\n- GPT-5.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)\n- Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort\n- o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)\n\n### Google/Gemini\n- `gemini_reasoning_enabled=True` for reasoning-capable models\n- Gemini 3.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)\n- Rich multimodal support (audio, video, images, documents)\n\n### DeepSeek\n- R1 models: `parser=ModelParserID.r1_thinking` + `reasoning_capable=True`\n- V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN\n- Some need `r1_openrouter_options=True` + `require_openrouter_reasoning=True`\n\n### OpenRouter (general)\n- Slugs: `vendor/model-name`\n- Never add a `:free` slug. Free tiers have low rate limits and are temporary. If OpenRouter serves a model only as `:free`, wait for a paid slug.\n- Reasoning models: may need `require_openrouter_reasoning=True`\n- Some models: `openrouter_skip_required_parameters=True`\n- Logprobs: `logprobs_openrouter_options=True` if supported\n- Always `multimodal_requires_pdf_as_image=True` (OpenRouter's PDF routing breaks LiteLLM)\n- Thinking levels: offer only values in the model's `reasoning.supported_efforts` (see [Thinking Levels Reference](#thinking-levels-reference)). OpenRouter maps an unlisted effort to a listed one, so an extra level silently runs a different effort. When the model has no `reasoning.supported_efforts`, OpenRouter turns the effort into a thinking-token budget for Claude, so `CLAUDE_OPENROUTER_THINKING_LEVELS` (with Minimal) fits only those Claude models. `check_thinking_levels.py` checks every OpenRouter entry; a missing level list on OpenRouter is a note, because Kiln leaves levels off some models on purpose.\n\n### Fireworks AI and Together AI: thinking levels\n\nNeither publishes the efforts each model accepts, and both can accept an unsupported effort without an error. Fireworks takes `xhigh`, `max` and `minimal` on GLM 5.3 with HTTP 200. Together took a made-up effort on Nemotron 3.5, but Muse Spark Open's chat template rejects one. A paid test cannot see this. Add thinking levels on these providers only when the vendor documents the levels for that model (the model card or its chat template), and name that source in the PR. OpenAI and Anthropic return a 400 that names the supported levels, so their paid per-level test catches a wrong level.\n\n### Featherless AI\n\nServerless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:\n\n- **`json_instructions` is the only usable structured output mode.** LiteLLM's `featherless_ai` provider rejects `response_format` outright (`UnsupportedParamsError`), so `json_schema`, `json_mode`, and `json_instruction_and_object` are all unavailable. Routing it as a custom `openai` provider bypasses that gate, but was tested and is *not* reliable: GLM 5.2 accepts `json_schema` and silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.\n- **Gated models return HTTP 403 `model_gated_needs_oauth`** and cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. **Always filter `is_gated` before adding.** Note all 20 official `meta-llama/*` repos are gated, so no Llama variant is usable.\n- **No cost reporting.** Featherless models aren't in LiteLLM's price map (only two legacy `Qwerky` entries, on `main` too — not a version issue), and Featherless doesn't return cost in the `usage` object. Runs record tokens with `cost: null`.\n- **Not in models.dev or the LiteLLM catalog**, so their `/v1/models` endpoint is the only authoritative source. See [Lagging Providers](#lagging-providers).\n- Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.\n\n### TypeSafe AI\n\nServes TypeSafe's Jev models (System One API). The only provider in the catalog that does not run through LiteLLM — read this before adding an entry:\n\n- **Adapter-backed, not LiteLLM-backed.** Every provider entry must set `adapter=ModelAdapterId.jev`; `JevAdapter` calls `POST /v1/systemone` directly. `lite_llm_core_config_for_provider` and `get_litellm_provider_info` raise for `typesafe` on purpose, so the LiteLLM parametrized tests exclude it.\n- **Structured output only.** No text generation, no tools, no multi-turn. A task runs only if its output schema maps onto Jev's question types (enum, boolean, score); anything else fails before the network call. Set `supports_structured_output=True`, `supports_data_gen=False`, `supports_function_calling=False`, and `structured_output_mode=StructuredOutputMode.json_schema`.\n- **No logprobs.** `supports_logprobs=False`. Jev returns its own per-answer probabilities instead, which is what keeps it out of G-Eval.\n- **Not in models.dev or the LiteLLM catalog, and `GET /v1/models` cannot confirm a `model_id`.** It lists aliases (`jev-latest`, `jev-preview`), not the pinned versions Kiln registers, and it returns `{\"models\": [{\"name\", ...}]}` rather than the OpenAI `{\"data\": [{\"id\", ...}]}` shape, so the shared `fetch_openai_compat` path cannot read it either. That is why `typesafe` is in `SKIP_PROVIDERS` in `.agents/scripts/provider_utils.py` rather than in `PROVIDER_CONFIG`. The authoritative check for a pinned id is a `POST /v1/systemone` round trip that the API accepts — that is how `jev-1.13.0` was confirmed. (The listing is still what the connect flow uses to validate a key; it is account-scoped and rejects a bad one.)\n\n### Cloudflare (Workers AI)\n\nRuns through LiteLLM's generic `openai/` route against Cloudflare's OpenAI-compatible endpoint (`/accounts/{id}/ai/v1/chat/completions`), with an optional AI Gateway header. Kiln does **not** use LiteLLM's `cloudflare/` provider. Read this before adding or changing any entry:\n\n- **Inclusion rule: only Cloudflare's newer, OpenAI-format text-generation models.** Cloudflare has two generations of text model. The newer ones (GLM, Kimi, DeepSeek V4, Gemma 4, Qwen 3.8, Nemotron 3) speak OpenAI Chat Completions natively. The older ones (Llama 3.x/4, Mistral, older Qwen, GPT-OSS, Granite, the LoRA bases) sit behind a translation layer: they default to **256 output tokens** (silent truncation) and have unreliable tool calling and JSON mode. Do not add the older ones.\n- **How to tell them apart:** look at the model's input schema in the catalog JSON. OpenAI-format models list `max_completion_tokens` (and `tool_choice`); older models list `prompt`, `raw` and `lora` instead, and no `max_completion_tokens`:\n ```bash\n jq -r '.models[] | select(.task.name==\"Text Generation\")\n | [.name, (if (.schema.input | tostring | test(\"max_completion_tokens\")) then \"openai-format\" else \"OLDER - skip\" end)] | @tsv' /tmp/cf_models.json\n ```\n- **Also skip:** coding models (`kimi-*-code`, `qwen*-coder-*`), `llama-guard-*` (classifier), `-lora` bases, and models Kiln has no `ModelName` for unless the user asks for a new model.\n- **Usually a new provider on an existing model.** Most Cloudflare models are already in Kiln under other providers; add a `KilnModelProvider(name=ModelProviderName.cloudflare, ...)` to the existing `KilnModel`.\n- **Setting flags from the catalog's `properties[]`** (confirm every flag with the paid tests — the catalog has no JSON-mode flag):\n\n | Cloudflare property | Kiln setting |\n |---|---|\n | `function_calling: \"true\"` | leave `supports_function_calling` at its default; set `False` if the tool tests fail |\n | `vision: \"true\"` | `supports_vision`, `multimodal_capable`, MIME types TXT/MD/JPG/PNG; start broad, narrow on 400s. Don't set `supports_doc_extraction`: the document extractor doesn't send the AI Gateway header, so extraction would bypass the user's gateway |\n | `reasoning: \"true\"` | keep `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)); LiteLLM already reads `reasoning_content` |\n | `reasoning_effort.supported_efforts` present | **required:** set `available_thinking_levels` to exactly these values, and `default_thinking_level` to `default_effort`. See [Cloudflare thinking levels](#cloudflare-thinking-levels) |\n | `reasoning_effort` without `supported_efforts` | no thinking levels; the catalog does not say which values work |\n | `require_workers_paid: \"true\"` | no Kiln field. Free-plan users get 403 / code 5035. Mention it in the PR |\n | `planned_deprecation_date` present | do **not** add the model |\n\n- <a id=\"cloudflare-thinking-levels\"></a>**Thinking levels come from the catalog, and only from the catalog.** Kiln sends the level as `reasoning_effort`. For each model, `reasoning_effort` in the catalog lists `supported_efforts`, a `default_effort`, and `normalizes_to`: the values that Cloudflare silently changes to another effort. A remapped value returns HTTP 200, so a live call or a paid test cannot find it. Example: GLM 5.3 maps `medium` to `max`, so a \"Medium\" option runs the most expensive effort.\n - Offer only values in `supported_efforts`, and each of them unless a completed paid test shows that the level does not work. The list is not always right: on DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731 and GLM 5.2, `none` still returns reasoning, so those entries have no Off level. A timeout, a rate limit or a credential error proves nothing about the level: keep the level and report it as untested. Do not copy the levels of the same model on another provider.\n - Labels: `none` → `\"Off\"`, `low` → `\"Low\"`, `medium` → `\"Medium\"`, `high` → `\"High\"`, `xhigh` → `\"Extra High\"`, `max` → `\"Max\"`. List them from least to most effort.\n - `default_thinking_level` is `default_effort`, unless the entry is suggested for evals (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)).\n - `mandatory: true` means `none` is not in the list; do not add an Off level.\n - Reuse a constant when t\n\n<!-- File: .agents/skills/kiln-check-deprecation/SKILL.md (Tokens: ~2556 | Category: anthropic-skill) -->\n---\nname: kiln-check-deprecation\ndescription: Check Kiln's model list for deprecated or sunset models across all providers. Use when the user wants to find deprecated models, check model availability, audit the model list for stale entries, or mentions model deprecation/sunset/end-of-life.\n---\n\n# Check Model Deprecation in Kiln\n\nAudit `libs/core/kiln_ai/adapters/ml_model_list.py` to find models that have been deprecated, removed, or are approaching end-of-life across all providers. **All checks are free** — they use model-listing endpoints, not inference.\n\n---\n\n## Global Rules\n\n- **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: [\"all\"]`. The sandbox blocks network access.\n- **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)`\n- **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate.\n- **Non-destructive:** This skill only sets `deprecated=True` on individual provider entries. It never removes models or providers from the list.\n- **Skip already-deprecated:** Only check providers where `deprecated=False` (the default). Don't re-check entries already marked deprecated.\n\n---\n\n## Phase 1 – Extract Current Model-Provider Pairs\n\nRun the extraction script from the repo root:\n\n```bash\npython3 .agents/skills/kiln-check-deprecation/scripts/extract_models.py > /tmp/kiln_extracted.json\n```\n\nThis fetches the published model list from `https://remote-config.getkiln.ai/kiln_config_v2.json` and extracts:\n- The provider name\n- The model_id\n- The parent model's enum name\n\nIt skips entries already marked `deprecated=True`.\n\n**Output:** JSON to stdout (pipe to file), human summary to stderr.\n\nThe JSON contains:\n- `deprecated_count`: number of already-deprecated entries\n- `providers`: dict of provider_name → sorted unique list of model_ids\n- `entries`: list of `{enum, provider, model_id}` for each non-deprecated entry\n\n---\n\n## Phase 2 – Check Each Provider\n\n### Option A: Check all providers at once\n\n```bash\nexport $(grep -v '^#' .env | xargs)\npython3 .agents/skills/kiln-check-deprecation/scripts/check_provider.py all > /tmp/kiln_check_results.json\n```\n\n### Option B: Check a single provider\n\n```bash\nexport $(grep -v '^#' .env | xargs)\npython3 .agents/skills/kiln-check-deprecation/scripts/check_provider.py openrouter > /tmp/kiln_check_openrouter.json\n```\n\nThe script handles all provider API quirks automatically:\n- **Together AI** returns a flat JSON array (not `{data: [...]}`)\n- **Gemini API** needs both v1 and v1beta endpoints (preview/Gemma models only on v1beta)\n- **Gemini** model names are prefixed with `models/` — stripped automatically\n- **Anthropic** uses `x-api-key` header, not `Authorization: Bearer`\n- **OpenRouter** is public (no auth) and includes `expiration_date` fields\n- **Fireworks AI** `/v1/models` only lists serverless models; the script instead checks each model individually via the model detail API (`GET /v1/{model_id}`), which covers all tiers (serverless, on-demand, fine-tune). A model is only flagged as missing if it returns HTTP 404 from the detail API.\n- **OpenRouter** `:exacto` is a virtual routing suffix (quality-first provider sorting) that never appears in model listings. The script strips it before checking. `:free` and `:thinking` are real model entries that appear in the listing when available — if they're missing, it's a genuine removal.\n- **Cloudflare** uses the authenticated models-search API (`task=Text Generation`, `include_deprecated=true`) when `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` are both set, and otherwise Cloudflare's public catalog JSON (`https://ai-cloudflare-com.pages.dev/api/models`), so it is never skipped for a missing key. Every model with a `planned_deprecation_date` is reported under `expiring`. Once that date has passed, the model also counts as `missing` even if the listing still shows it. Never use LiteLLM's catalog or models.dev to decide a Cloudflare model still exists: LiteLLM's is stale and models.dev no longer deletes retired models.\n- **Vertex AI** uses the v1beta1 publisher models endpoint with `x-goog-user-project` header. Requires `gcloud` CLI auth. Kiln entries may use `meta/` prefix for LiteLLM routing — stripped automatically. Versioned aliases (e.g. `gemini-2.0-flash-001` → `gemini-2.0-flash`) are handled by stripping 3-digit version suffixes.\n\n**Output:** JSON to stdout with per-provider results, human summary to stderr.\n\nEach provider result contains:\n- `missing`: model_ids not found in the provider's listing\n- `expiring`: model_ids with upcoming expiration dates (OpenRouter and Cloudflare)\n- `entries_to_deprecate`: full enum/provider/model_id entries for each missing model\n- `skipped` / `error`: if credentials missing or API call failed\n\n### Supported providers\n\n| Provider | Env Var | Auth |\n|----------|---------|------|\n| OpenRouter | (none — public API) | None |\n| OpenAI | `OPENAI_API_KEY` | Bearer |\n| Anthropic | `ANTHROPIC_API_KEY` | x-api-key |\n| Gemini API | `GEMINI_API_KEY` | Query param |\n| Fireworks AI | `FIREWORKS_API_KEY` | Bearer |\n| Together AI | `TOGETHER_API_KEY` | Bearer |\n| SiliconFlow CN | `SILICONFLOW_CN_API_KEY` | Bearer |\n| Cerebras | `CEREBRAS_API_KEY` | Bearer |\n| Groq | `GROQ_API_KEY` | Bearer |\n| Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) |\n| Cloudflare | `CLOUDFLARE_API_KEY` + `CLOUDFLARE_ACCOUNT_ID` (optional — falls back to the public catalog) | Bearer |\n\n### Providers NOT covered by the script (check manually if needed)\n\n#### Amazon Bedrock\n\nRequires `aws` CLI configured. If not installed, skip.\n\n```bash\naws bedrock list-foundation-models --output json | \\\n jq '[.modelSummaries[] | {modelId, status: .modelLifecycle.status, endOfLife: .modelLifecycle.endOfLifeTime}]'\n```\n\nFlag any model where `status == \"LEGACY\"` or `endOfLifeTime` is approaching.\n\n### LiteLLM Model DB (supplementary signal)\n\nCross-reference against LiteLLM's public model database for `deprecation_date` fields. This mostly covers Azure/OpenAI date-stamped models but is a useful secondary signal.\n\n```bash\ncurl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \\\n jq 'to_entries[] | select(.value.deprecation_date != null) | {model: .key, deprecation_date: .value.deprecation_date}'\n```\n\n### Providers to skip\n\nThese don't have meaningful deprecation to check:\n- `ollama` — local models, user-managed\n- `kiln_fine_tune` — internal\n- `kiln_custom_registry` — internal\n- `openai_compatible` — user-configured\n- `docker_model_runner` — local\n- `azure_openai` — typically mirrors OpenAI, covered by OpenAI check\n- `huggingface` — serverless inference, model availability varies\n- `typesafe` — adapter-backed, not LiteLLM; its `/v1/models` needs a key, so Jev entries can only be checked with one\n\n---\n\n## Phase 3 – Report Findings\n\nRead the JSON output from check_provider.py and present findings in a clear table to the user:\n\n```text\nDeprecation Audit Results\n=========================\n\n❌ REMOVED (model not found in provider's model list):\n - ModelName.gemini_1_5_pro → gemini_api (model_id: gemini-1.5-pro)\n - ModelName.gemini_1_5_pro → openrouter (model_id: google/gemini-pro-1.5)\n\n⚠️ EXPIRING SOON (within 90 days):\n - ModelName.claude_3_7_sonnet → openrouter (model_id: anthropic/claude-3.7-sonnet, expires: 2026-05-05)\n\n⚠️ LEGACY (Bedrock lifecycle status):\n - ModelName.mistral_large → amazon_bedrock (model_id: mistral.mistral-large-2407-v1:0, status: LEGACY)\n\n✅ ALL CLEAR:\n - openai: 20/20 models found\n - anthropic: 8/11 models found (3 deprecated above)\n\n⏭️ SKIPPED (no credentials):\n - groq\n - amazon_bedrock\n```\n\nGroup by provider within each category. Use the `entries_to_deprecate` field from the JSON to get the ModelName enum for each entry.\n\n---\n\n## Phase 4 – Mark Deprecated\n\nFor each confirmed-removed model, set `deprecated=True` on the affected `KilnModelProvider` entry.\n\nTo find the line to edit, grep for the `model_id` in `ml_model_list.py`:\n\n```bash\ngrep -n 'model_id=\"<model_id>\"' libs/core/kiln_ai/adapters/ml_model_list.py\n```\n\nThen add `deprecated=True,` after the `model_id=` line in the matching `KilnModelProvider` block.\n\n**Rules:**\n- Only mark a provider deprecated if its model_id is confirmed missing from that provider's model list\n- If ALL providers for a `KilnModel` are deprecated, note this to the user — they may want to consider removing the model entirely\n- For \"expiring soon\" models, inform the user but don't mark deprecated yet — let them decide. **Exception: Cloudflare.** When Cloudflare retires a model it may silently alias the old ID to a different model, and the response still echoes the requested ID, so a stale entry keeps \"working\" while running something else. Propose marking every Cloudflare entry with a `planned_deprecation_date` deprecated now, before that date, and flag it as urgent in the report\n- For Bedrock `LEGACY` status, mark deprecated (the model still works but is on its way out)\n\n**Ask the user to confirm** before making changes. Present the list of changes and wait for approval.\n\n---\n\n## Phase 5 – Verify\n\nAfter marking providers deprecated, run a quick sanity check:\n\n```bash\ngrep -c \"deprecated=True\" libs/core/kiln_ai/adapters/ml_model_list.py\n```\n\nEnsure the count matches what you expect (previous count + newly marked).\n\n---\n\n## Checklist\n\n- [ ] Extraction script run, JSON saved to /tmp/kiln_extracted.json\n- [ ] Provider checks run (all or individually)\n- [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex)\n- [ ] Bedrock checked manually (skip if aws CLI unavailable)\n- [ ] LiteLLM DB checked for supplementary deprecation_date signals\n- [ ] Findings reported to user with clear table\n- [ ] User confirmed changes before marking deprecated\n- [ ] `deprecated=True` set on confirmed entries\n- [ ] Deprecation count verified\n\n<!-- File: .agents/skills/kiln-check-finetune-deprecation/SKILL.md (Tokens: ~2009 | Category: anthropic-skill) -->\n---\nname: kiln-check-finetune-deprecation\ndescription: Check Kiln's fine-tunable model list for deprecated or unsupported base models. Use when the user wants to audit fine-tuning support, check if fine-tune base models are still valid, or mentions fine-tune model deprecation.\n---\n\n# Check Fine-Tune Model Deprecation in Kiln\n\nAudit the fine-tunable models listed in Kiln to find base models that are no longer supported for fine-tuning by their providers. This is separate from general model deprecation — a model can still be available for inference but no longer supported as a fine-tuning base.\n\n---\n\n## Global Rules\n\n- **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: [\"all\"]`. The sandbox blocks network access.\n- **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)`\n- **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate.\n- **Non-destructive:** This skill only reports findings. It does not automatically remove or deprecate models.\n\n---\n\n## Supported Providers\n\n| Provider | Env Var | Auth |\n|----------|---------|------|\n| OpenAI | `OPENAI_API_KEY` | Bearer |\n| Together AI | `TOGETHER_API_KEY` | Bearer |\n| Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) |\n| Fireworks AI | `FIREWORKS_API_KEY` | Bearer |\n\n---\n\n## Background\n\nKiln has two types of fine-tunable model entries:\n\n1. **Static entries** — Models in `libs/core/kiln_ai/adapters/ml_model_list.py` with `provider_finetune_id` set. Currently used by Together AI and Vertex AI.\n2. **Dynamic entries (Fireworks)** — Fetched at runtime from `api.fireworks.ai/v1/accounts/fireworks/models` filtering by `tunable=True`. This list is managed by Fireworks and may include stale/unsupported models.\n\nThe API endpoint that serves the fine-tune dropdown is `GET /api/finetune_providers` in `app/desktop/studio_server/finetune_api.py`.\n\n---\n\n## Phase 1 – Check Static Fine-Tune IDs\n\nRun the check script from the repo root:\n\n```bash\nuv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py static > /tmp/kiln_finetune_static.json\n```\n\nThis extracts all `provider_finetune_id` entries from `ml_model_list.py` and checks each provider's docs/API to see if the model is still available for fine-tuning:\n\n- **Together AI:** Scrapes the Together AI docs page for the list of supported fine-tune models. Together uses full HuggingFace-style IDs (e.g. `meta-llama/Meta-Llama-3.1-8B-Instruct-Reference`).\n- **Vertex AI:** Checks the Vertex AI publisher models API. Vertex fine-tuning uses specific versioned model IDs (e.g. `gemini-2.0-flash-001`).\n\n**Output:** JSON to stdout with per-provider results, human summary to stderr.\n\n---\n\n## Phase 2 – Check Fireworks Dynamic Models\n\nFireworks fine-tunable models are fetched dynamically from their API, not stored in `ml_model_list.py`. The script checks the API's `supervisedLoraTunable` and `supervisedFullParameterTunable` fields (not the old `tunable` field, which is stale) and cross-references against our allowlist.\n\n```bash\nuv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py fireworks > /tmp/kiln_finetune_fireworks.json\n```\n\nThis script:\n1. Fetches all models with `supervisedLoraTunable=True` or `supervisedFullParameterTunable=True` from `api.fireworks.ai/v1/accounts/fireworks/models`\n2. Cross-references against `FIREWORKS_SUPPORTED_FINETUNE_MODELS` from `libs/core/kiln_ai/adapters/fine_tune/fireworks_finetune.py` — the same allowlist that filters the runtime fine-tune dropdown\n3. Checks both directions:\n - **Stale allowlist entries**: models in our allowlist that the API no longer marks as supervised-tunable — these may need to be removed\n - **Missing from allowlist**: models the API marks as supervised-tunable but aren't in our allowlist — these are candidates to add\n\nThe canonical allowlist lives in `fireworks_finetune.py` and is shared between the runtime dropdown filter and this audit skill. There is a single place to update when Fireworks changes their supported models.\n\n**Interpreting results:** The supervised API fields return ~43 models, but our allowlist intentionally filters to ~20 that match Fireworks' documented supported models. Seeing ~23 \"models in API but NOT in allowlist\" is normal — these are models Fireworks marks as technically tunable but doesn't officially document or support. Only flag these if a model appears in the Fireworks docs but is missing from our allowlist. The important direction is **stale allowlist entries** — models we ship that the API no longer marks as supervised-tunable.\n\n**Output:** JSON to stdout, human summary to stderr.\n\n---\n\n## Phase 3 – Report Findings\n\nRead the JSON outputs and present findings in a clear table:\n\n```text\nFine-Tune Deprecation Audit Results\n====================================\n\nSTATIC PROVIDERS (provider_finetune_id in ml_model_list.py):\n\n ✅ OpenAI: 5/5 found\n gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14,\n gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18\n\n ✅ Vertex AI: 2/2 found\n gemini-2.0-flash-001, gemini-2.0-flash-lite-001\n\n ❌ Together AI: 2/4 missing\n ✅ Qwen/Qwen2.5-72B-Instruct — found\n ✅ Qwen/Qwen2.5-14B-Instruct — found\n ❌ meta-llama/Meta-Llama-3.1-8B-Instruct-Reference — NOT FOUND\n ❌ meta-llama/Meta-Llama-3.1-70B-Instruct-Reference — NOT FOUND\n\n ⏭️ SKIPPED (no credentials):\n - vertex (VERTEX_PROJECT_ID not set)\n\nDYNAMIC PROVIDER (Fireworks):\n\n ⚠️ 167 models marked tunable in API\n ⚠️ 15 models in known-good docs list\n ✅ 11 models in both API and docs\n ❌ 156 models in API but NOT in docs (likely stale)\n - accounts/fireworks/models/qwen2-72b-instruct\n - accounts/fireworks/models/code-llama-7b\n - ...\n ⚠️ 4 models in docs but NOT in API\n - accounts/fireworks/models/gemma-2-9b-it\n - ...\n```\n\nNote: The example above is illustrative. Actual model counts and entries will vary.\n\n---\n\n## Phase 4 – Remediation\n\nBased on findings, recommend specific actions:\n\n### For static entries (`provider_finetune_id`):\n- If a model's fine-tune ID is no longer valid, either:\n - **Update** the `provider_finetune_id` to a newer version if one exists\n - **Remove** the `provider_finetune_id` field to stop offering fine-tuning for that model on that provider\n - **Set `deprecated=True`** on the provider entry if the model is fully gone\n\n### For Fireworks dynamic entries:\n- The stale models come from Fireworks' API, not our code. Options:\n - **Filter in our code** — update `fetch_fireworks_finetune_models()` in `finetune_api.py` to filter against a known-good list\n - **Report to Fireworks** — flag the stale models to Fireworks support\n - **Both** — filter now, report later\n\n**Always ask the user to confirm** before making any code changes.\n\n---\n\n## Phase 5 – Verify\n\nAfter any changes, run the relevant tests:\n\n```bash\nuv run python3 -m pytest app/desktop/studio_server/test_finetune_api.py -q\n```\n\n---\n\n## Checklist\n\n- [ ] Env vars sourced from `.env`\n- [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex)\n- [ ] Static fine-tune IDs checked against provider APIs (Together, Vertex)\n- [ ] Fireworks dynamic models cross-referenced against known-good list\n- [ ] Findings reported to user with clear table\n- [ ] User confirmed remediation approach\n- [ ] Code changes made (if any)\n- [ ] Tests pass after changes\n- [ ] FIREWORKS_SUPPORTED_FINETUNE_MODELS in fireworks_finetune.py updated if docs have changed\n\n<!-- File: .agents/skills/kiln-prerelease-check/SKILL.md (Tokens: ~8017 | Category: anthropic-skill) -->\n---\nname: kiln-prerelease-check\ndescription: Run the Kiln pre-release smoke test suite plus the standard CI checks (checks.sh), diagnose every prerelease test that broke (and why), and write a clean readable report with recommended actions. Read-only — it never edits code. Use when the user wants to validate a release candidate, run prerelease tests, or asks for a \"prerelease check / smoke / audit\".\n---\n\n# Kiln Pre-release Check\n\nEnd-to-end pre-release verification and **diagnosis**. This skill:\n\n1. Runs the standard CI checks (`checks.sh` — lint / format / typecheck / unit tests).\n2. Runs the curated `@pytest.mark.prerelease` smoke set (real, paid API calls).\n3. For every test that broke, works out **what** broke and **why** — including whether the cause is a model that's gone deprecated (per `ml_model_list.py`) or that the provider no longer recognizes the slug.\n4. Sweeps the prerelease model pins for staleness and newer model versions.\n5. Writes a clean, scan-readable report with a clear **recommended action** for each finding.\n\n**This skill is read-only. It does not change any code, tests, whitelists, or prod source — ever.** Its entire output is a report at `.prerelease/<timestamp>/REPORT.md` (this directory is gitignored). When the right action is to bump a model pin or update the whitelist, the report *recommends* it; the user makes the change.\n\n---\n\n## What \"prerelease\" means here\n\nA prerelease test is a paid test (it makes real API calls) that we **cannot** meaningfully replace with a mock — its whole point is to verify a live third-party API still behaves how we depend on it. Examples: `test_connect_vertex_live`, basic OpenAI / Claude / Gemini calls, embedding API across providers, reranker integration, Fireworks fine-tune deployment listing, etc.\n\nThe set is a **curated subset of `@pytest.mark.paid`**. Tests carry both markers — `@pytest.mark.paid` for the cost gate, `@pytest.mark.prerelease` for the curated selection.\n\nConcretely:\n\n- `--runpaid` runs the full paid suite (slow, expensive, intended for one-off use).\n- `--runprerelease` runs **only** the prerelease subset (still paid, but a small, focused, releaseable list). It implies `--runpaid` for those tests.\n\nMarker registration lives in `pyproject.toml` (`[tool.pytest.ini_options].markers`); the flag is wired up in the top-level `conftest.py` via `pytest_addoption` + `pytest_collection_modifyitems`.\n\n### Whitelist for litellm-adapter tests\n\nSeveral paid tests (embeddings, document extraction, thinking-level reasoning) fan out across **every** `(model, provider)` pair in `ml_model_list.py` / `ml_embedding_model_list.py`. Running those under `--runprerelease` would be slow and expensive. Instead, those tests have dedicated `*_prerelease_smoke` sibling functions that iterate only over a small curated whitelist living in:\n\n```\nlibs/core/kiln_ai/adapters/pytest_prerelease_whitelist.py\n```\n\nWhitelisted lists:\n\n- `PRERELEASE_CHAT_MODELS` — handful of chat models, one per major provider (OpenAI + GPT-5, Anthropic + Claude Sonnet/Haiku, Gemini, OpenRouter, Groq, Fireworks, Together, Cloudflare).\n- `PRERELEASE_EMBEDDING_MODELS` — one embedding model per embedding-supporting provider.\n- `PRERELEASE_EXTRACTION_MODELS` — one multimodal model per major vendor (OpenAI, Anthropic, Gemini).\n- `PRERELEASE_EXTRACTION_MIME_PROBES` — three mime probes (PDF, PNG, MP3) for the extraction smoke; the full paid test sweeps all 13 mime types per model.\n- `PRERELEASE_THINKING_MODELS` — five (provider, model, thinking_level) triples covering reasoning content + a \"none\" negation case.\n- `PRERELEASE_JEV_MODELS` — TypeSafe AI's Jev models, used by the Jev paid smoke tests. Separate from `PRERELEASE_CHAT_MODELS` because Jev does not go through LiteLLM.\n\nThis whitelist is the thing most likely to go stale, so it's the main target of the pin sweep in Phase 4. The fan-out tests (e.g. `test_extract_document_success` over every model × mime type, `test_paid_generate_embeddings_basic` over every embedding) are **only `@pytest.mark.paid`** — not `@pytest.mark.prerelease` — and are out of scope for this skill.\n\n---\n\n## Global rules\n\n- **Read-only. Make no edits.** Do not modify test files, the whitelist, prod source, config, or anything else. Every actionable finding is a *recommendation* in the report. If you catch yourself about to call Edit/Write on anything other than files under `.prerelease/<timestamp>/`, stop.\n- **Network access is required.** Every test phase makes real API calls or hits remote model-list endpoints. If you are running this skill inside a sandboxed Bash session, request `required_permissions: [\"all\"]` for the test commands.\n- **Env vars:** Source `.env` before running anything that needs API keys. Use a real sourcing pass (not `export $(... | xargs)`, which mangles any value containing spaces or quotes):\n ```bash\n set -a; . ./.env; set +a\n ```\n Provider-specific env vars the prerelease set may touch: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `GROQ_API_KEY`, `OPENROUTER_API_KEY`, `FIREWORKS_API_KEY`, `TOGETHERAI_API_KEY`, `SILICONFLOW_CN_API_KEY`, `COHERE_API_KEY`, `TYPESAFE_API_KEY`, `CLOUDFLARE_API_KEY` + `CLOUDFLARE_ACCOUNT_ID` (used by `test_cloudflare` in `test_prompt_adaptors.py`, which runs GLM 4.7 Flash; a free-plan Cloudflare account is enough), `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`, plus `KILN_TEST_VERTEX_PROJECT_ID` (+ optional `KILN_TEST_VERTEX_LOCATION`) for the Vertex live check, which also requires `gcloud auth application-default login` against a project with `aiplatform.googleapis.com` enabled.\n- **A missing key is a coverage gap, not a failure.** Each missing key makes the relevant test `pytest.skip(...)`. That isn't a prerelease failure — surface it in the report so the user can decide whether to provide the key and re-run.\n- **`ml_model_list.py` never deletes entries — it only marks them `deprecated=True`.** So a slug missing from our list is not a signal you'll normally encounter. The signals that matter when diagnosing a break or staleness are:\n 1. The provider entry has `deprecated=True` in our list.\n 2. The provider itself rejects the slug at runtime (4xx / \"model not found\") — note our list trails provider deprecations, so this can happen *before* we've marked it deprecated. This is how you'd discover a model \"no longer exists\" even when our list still lists it.\n 3. A newer non-deprecated sibling in the same family now exists in our list, regardless of whether the current pin still works.\n- **Single source of truth for the model list** is `libs/core/kiln_ai/adapters/ml_model_list.py` (chat) and `ml_embedding_model_list.py` (embeddings). Cross-reference there when diagnosing.\n- **The pin sweep is mandatory, not \"only when something fails.\"** Even on a fully green run, every hardcoded slug in `pytest_prerelease_whitelist.py` and in prerelease test parametrize lists must be cross-checked against the model list and reported (kept-and-current vs. has-newer-sibling vs. deprecated). Zero silent \"looks fine, moving on\" calls.\n\n---\n\n## Phase 1 — Set up the output directory\n\n```bash\nTS=$(date -u +%Y-%m-%dT%H-%M-%SZ)\nOUT=\".prerelease/${TS}\"\nSCRATCH=\"${OUT}/_scratch\"\nmkdir -p \"${SCRATCH}\"\necho \"Writing prerelease results to ${OUT}\"\n```\n\n`.prerelease/` is gitignored. This timestamped folder is the **only** place this skill writes. Keep it tidy — the top level should hold just the human-readable artifacts:\n\n- `REPORT.md` — the deliverable.\n- `checks.log`, `prerelease.log`, `retry.log` — raw run logs (kept for drill-down).\n\nEverything else — the grep/collect/diff intermediates used while diagnosing (`current_model_ids.txt`, `prerelease_test_files.txt`, `paid_tests.txt`, `prod_probes.txt`, etc.) — goes in `${SCRATCH}` (`${OUT}/_scratch`), **not** the top level. Don't scatter raw filename dumps where the user has to wade through them; the report is what they read.\n\n---\n\n## Phase 2 — Run `checks.sh`\n\nThese are not paid but are mandatory for any release. If they fail, the prerelease check fails before we even spend money on API calls.\n\n```bash\nuv run ./checks.sh --agent-mode 2>&1 | tee \"${OUT}/checks.log\"\n```\n\nCapture the exit code. If it's non-zero, **don't bail** — keep going with the paid tests so the report covers everything in one pass, but mark the overall run as failed.\n\n---\n\n## Phase 3 — Run the prerelease pytest set\n\n```bash\nuv run python3 -m pytest --runprerelease -v --tb=short \\\n -o \"addopts=\" \\\n 2>&1 | tee \"${OUT}/prerelease.log\"\n```\n\nNotes:\n\n- `--runprerelease` is registered in the top-level `conftest.py`. It runs only `@pytest.mark.prerelease` tests and implies `--runpaid` for them.\n- `-o \"addopts=\"` overrides the project default of `-n auto` (xdist). xdist is incompatible with some of the live-API tests (especially anything that touches global litellm state), so run serially.\n- This will be slow. Many of the parametrized tests fan out to dozens of cases.\n- Tests with missing API keys will `skip`, not fail. Capture skip counts.\n\nIf the user wants a faster narrowed run (e.g. only OpenAI + Anthropic), use `-k` to filter:\n\n```bash\nuv run python3 -m pytest --runprerelease -v --tb=short -o \"addopts=\" \\\n -k \"openai or anthropic or vertex\" \\\n 2>&1 | tee \"${OUT}/prerelease.log\"\n```\n\n---\n\n## Phase 4 — Diagnose\n\nThis is where the skill earns its keep: for every failure, explain the cause; for every pin, report staleness. No edits — only findings and recommendations.\n\n### 4a. Diagnose each failure (what broke, why, recommended action)\n\nFor every test that did **not** pass in Phase 3, build one finding. Walk the failure through this decision tree to land on a cause and a recommended action:\n\n1. **Missing credentials → skip, not failure.** (`pytest.skip` for absent key.) Don't list under failures; list under \"Skipped tests\". Recommended action: add the key and re-run if that coverage matters.\n2. **Provider rejected the model slug** — error mentions `model not found`, `does not exist`, `404`, `model_not_found`, `invalid model`, decommissioned, etc. Cross-reference the slug against `ml_model_list.py`:\n - If our list marks that provider entry `deprecated=True` → cause: **deprecated model, provider has dropped it**. Recommended action: bump the pin/whitelist to the newest non-deprecated sibling in the same family (name the candidate).\n - If our list still lists it as non-deprecated → cause: **model no longer exists at the provider; our list trails reality**. Recommended action: (a) bump the test pin/whitelist to a current sibling, and (b) flag that `ml_model_list.py` should mark this entry `deprecated=True`. Name both.\n3. **Auth / quota / rate-limit / transient network** — 401/403/429/5xx, timeouts. Cause: environment or provider-side, not our code. Recommended action: check the key/quota and re-run; only escalate if it reproduces.\n4. **Behavioral assertion mismatch** — the call succeeded but the response shape, content, token/cache fields, streaming chunks, or reasoning output changed. Cause: **provider changed behavior we depend on (real regression)**. Recommended action: investigate the prod path that relies on this; do not assume it's a stale pin. This is the highest-signal failure class — call it out prominently.\n5. **Our-code error** — traceback originates in `libs/core` / `app/` rather than the provider response. Cause: **regression in Kiln**. Recommended action: investigate the implicated code; name the file/function from the traceback.\n\nIf you can't confidently distinguish a stale-pin cause (2) from a real regression (4 or 5), **report it as a possible regression** and say what additional check would disambiguate (e.g. \"try the same call with a current sibling slug to see if it's the model or the code\"). Never recommend a silent pin bump for an ambiguous failure.\n\nRecord, per failure: test ID · provider · cause class · the model slug involved (if any) and its `ml_model_list.py` status · a short excerpt of the actual error · recommended action.\n\n**Retry likely-transient failures once before finalizing.** Live-API tests are flaky: rate limits, 5xx, timeouts, dropped streams, and occasional empty/under-populated responses (e.g. a thinking model returning no reasoning content on one call but not the next) routinely fail a single run without indicating any real breakage. So before you write up the failures, re-run the failing tests in isolation **once** to separate flakes from real defects:\n\n- **Retry these cause classes:** auth/quota/rate-limit/transient (class 3) and ambiguous behavioral/streaming mismatches (class 4) — anything that could plausibly be a one-off. **Do not retry** failures with a deterministic, self-explanatory cause: a deprecated/removed model slug (class 2), a Kiln-code traceback (class 5), or a missing-credential skip. Retrying those just burns money and time.\n- **How:** re-run only the failing node IDs, serially, into a separate log so the original run is preserved:\n ```bash\n uv run python3 -m pytest --runprerelease -v --tb=short -o \"addopts=\" \\\n \"<failing::node::id>\" \"<failing::node::id>\" ... \\\n 2>&1 | tee \"${OUT}/retry.log\"\n ```\n One retry pass is enough — this is a flake filter, not a stability loop. Don't retry a test more than once.\n- **Reclassify by the retry outcome:** a test that **passes on retry** is **transient/flaky**, not a release blocker — move it out of the hard-failure set. A test that **fails again** is **reproducible** — diagnose it as a real finding per the tree above.\n- **Always disclose the retry in the report.** Never silently swap a first-run FAIL for a PASS. Every retried test must show its first-run result, its retry result, and the verdict (transient vs. reproducible) — see the report template's \"Failures\" section and the retry note in the Verdict. A green retry is evidence the suite is flaky here, which the user needs to know before tagging.\n\n### 4b. Model-pin staleness sweep (mandatory every run, even when green)\n\nIndependent of failures, every hardcoded model slug the prerelease set depends on must be checked against the model list and reported. A pin can be perfectly green today and still be worth bumping because a newer sibling shipped — prerelease only catches what real users hit if it tracks what real users actually use.\n\n**In scope (check these every run):**\n- Every entry in `libs/core/kiln_ai/adapters/pytest_prerelease_whitelist.py`.\n- Hardcoded model slugs inside the body or parametrize list of any `@pytest.mark.prerelease` test (notably `libs/core/kiln_ai/adapters/test_prompt_adaptors.py::test_openrouter`).\n\n**Out of scope (don't report as stale):**\n- Hardcoded slugs in mock-only unit tests (no `@pytest.mark.paid` / `@pytest.mark.prerelease`) — the slug is a label, no real call happens.\n- Slugs in fixtures, recorded VCR cassettes, snapshot files.\n- Slugs in `@pytest.mark.paid` tests that aren't also `@pytest.mark.prerelease`.\n\n**Step 1 — status of each whitelist entry.** Run:\n\n```bash\nuv run python3 - <<'PYEOF'\nfrom kiln_ai.adapters.ml_model_list import built_in_models\nfrom kiln_ai.adapters.ml_embedding_model_list import built_in_embedding_models\nfrom kiln_ai.adapters.pytest_prerelease_whitelist import (\n PRERELEASE_CHAT_MODELS, PRERELEASE_EMBEDDING_MODELS,\n PRERELEASE_EXTRACTION_MODELS, PRERELEASE_THINKING_MODELS,\n PRERELEASE_JEV_MODELS,\n)\n\ndef chat_status(name, provider):\n for m in built_in_models:\n if m.name == name:\n p = next((p for p in m.providers if p.name.value == provider), None)\n return 'NO_PROVIDER' if p is None else ('DEPRECATED' if p.deprecated else 'OK')\n return 'MODEL_REMOVED_OR_RENAMED'\n\ndef emb_status(name, provider):\n for m in built_in_embedding_models:\n if m.name == name:\n p = next((p for p in m.providers if p.name.value == provider), None)\n return 'NO_PROVIDER' if p is None else 'OK'\n return 'MODEL_REMOVED_OR_RENAMED'\n\nprint('== chat / extraction ==')\nfor n,p in PRERELEASE_CHAT_MODELS + PRERELEASE_EXTRACTION_MODELS + PRERELEASE_JEV_MODELS:\n print(f'{chat_status(n,p):>22} {n} ({p})')\nprint('== embedding ==')\nfor n,p in PRERELEASE_EMBEDDING_MODELS:\n print(f'{emb_status(n,p):>22} {n} ({p})')\nprint('== thinking ==')\nfor prov,name,lvl in PRERELEASE_THINKING_MODELS:\n print(f'{chat_status(name,prov):>22} {name} ({prov}, level={lvl})')\nPYEOF\n```\n\nAnything not `OK` is stale and goes in the sweep table with a recommended replacement (newest non-deprecated sibling in the same family).\n\n**Step 2 — newer-sibling pass for the `OK` entries.** For each entry that's `OK`, scan `ml_model_list.py` for a newer member of the same family. List the canonical model ids to compare against:\n\n```bash\ngrep -oE '^[[:space:]]+[a-z][a-z0-9_]+ = \"[a-z0-9_]+\"' \\\n libs/core/kiln_ai/adapters/ml_model_list.py | \\\n awk -F'\"' '{print $2}' | sort -u > \"${SCRATCH}/current_model_ids.txt\"\n```\n\nExamples of the judgment call:\n- `PRERELEASE_CHAT_MODELS` on `gpt_4o_mini` while `gpt_5_x_mini` exists → recommend bump.\n- `PRERELEASE_CHAT_MODELS` on `claude_sonnet_4_5` while `claude_sonnet_4_6`/`4_7` exists → recommend bump.\n- `gemini_1_5_*` → `gemini_2_5_*` / `gemini_3_*`; `claude_3_5_*` → `claude_sonnet_4_*` / `claude_4_5_haiku`; `llama_3_x` → `llama_3_3_70b` / `llama_4_*`.\n\nRule of thumb: if `ml_model_list.py` ordering puts a newer sibling above the current pin, recommend the bump. If it's genuinely ambiguous which sibling is \"current\", say so and let the user pick — don't assert a churny recommendation.\n\n**Step 3 — per-test hardcoded slugs.** Enumerate the prerelease test files and scan them for slugs not in `current_model_ids.txt`:\n\n```bash\nuv run python3 -m pytest --runprerelease --collect-only -q -o \"addopts=\" \\\n 2>&1 | grep '::' | awk -F'::' '{print $1}' | sort -u > \"${SCRATCH}/prerelease_test_files.txt\"\n```\n\nCross-reference each pinned slug against `current_model_ids.txt` and the deprecation status. Report any obsolete pins as recommendations.\n\nEvery in-scope pin lands in the sweep table — including the ones that are current and need no action. Silent omission is not allowed.\n\n### 4c. Coverage gaps\n\nThe prerelease set should cover, at minimum:\n\n| Area | Test(s) |\n|---|---|\n| Vertex live connect | `app/desktop/studio_server/test_provider_api.py::test_connect_vertex_live` |\n| OpenAI basic call | `libs/core/kiln_ai/adapters/test_prompt_adaptors.py::test_openai` |\n| Anthropic via OpenRouter / Anthropic API | `test_structured_output_anthropic_*`, `test_structured_output_openrouter_function_calling_nested_object` |\n| Gemini API | covered via embeddings + prompt caching + thinking levels |\n| Groq | `test_groq` |\n| OpenRouter multi-model | `test_openrouter[...]` (parametrized) |\n| Amazon Bedrock | `test_amazon_bedrock` |\n| Embedding API (whitelist) | `test_paid_generate_embeddings_basic_prerelease_smoke`, `test_generate_embedding_prerelease_smoke` |\n| Custom-dimension embeddings (whitelist) | `test_paid_generate_embeddings_custom_dimensions_prerelease_smoke`, `test_generate_embedding_with_user_supplied_dimensions_prerelease_smoke` |\n| Structured output (json_schema, function_calling, flat/nested/array) | `test_structured_output_*` |\n| Streaming (OpenAI protocol + AI SDK protocol) | `test_invoke_openai_stream_chunks`, `test_invoke_ai_sdk_stream` |\n| Tool calling | `test_tools_gpt_4_1_mini` |\n| Thinking-level reasoning (whitelist) | `test_thinking_level_reasoning_content_prerelease_smoke` |\n| Reranker | `test_reranker_integration_success` |\n| Document extraction (whitelist of models × probe mime types) | `test_extract_document_success_prerelease_smoke`, `test_provider_bad_request_prerelease_smoke` |\n| Semantic chunker (real embedding integration) | `test_semantic_chunker_real_integration` |\n| Fireworks fine-tune | `test_fetch_all_deployments` |\n| Prompt caching (Anthropic + OpenAI + Gemini + Fireworks + Together) | `test_prompt_caching_cache_hit` |\n\nIf a new provider/family in `ml_model_list.py` isn't represented in the prerelease set, note the gap as a recommendation.\n\nAlso list recently added `@pytest.mark.paid` tests that are *not* `@pytest.mark.prerelease`-tagged, so the user can decide if any should be promoted:\n\n```bash\n# Compare collected test NODE IDs, not marker line numbers: a test carrying both\n# markers on different lines would be misclassified by a file:line diff. Marker\n# expressions select by marker regardless of the skip gates, so collect-only works.\n# `sed 's/\\[.*//'` collapses parametrized cases to one row per test function\n# (the paid fan-out tests expand to thousands of cases otherwise).\nuv run python3 -m pytest --collect-only -q -m \"paid and not prerelease\" -o \"addopts=\" \\\n 2>/dev/null | grep '::' | sed 's/\\[.*//' | sort -u > \"${SCRATCH}/paid_only_nodes.txt\"\n# Recently added files among those paid-only tests (pipe to xargs to avoid\n# \"argument list too long\" on a large list):\nawk -F'::' '{print $1}' \"${SCRATCH}/paid_only_nodes.txt\" | sort -u | \\\n xargs git log --since=\"3 months ago\" --diff-filter=AM --name-only --pretty=format: -- \\\n 2>/dev/null | sort -u | head -50 > \"${SCRATCH}/recent_paid_only_files.txt\"\n```\n\n### 4d. Prod-code model probes (flag only)\n\nSome prod-code paths hardcode a model slug as a **probe** — a narrow connectivity/auth check where the model is just a vehicle to hit the API, not a feature the user picked. The canonical example is `app/desktop/studio_server/provider_api.py::connect_vertex`, which calls `litellm.acompletion(model=\"vertex_ai/gemini-X.Y-flash\", …)` to verify Vertex credentials when the user clicks \"Connect\".\n\nThese should stay on the latest available model in their family, otherwise the Connect button drifts toward a deprecated/removed model. **This skill only flags them** — it does not edit prod code, and the recommendation in the report is the artifact the user acts on.\n\nSweep for hardcoded inference-probe slugs in non-test prod files:\n\n```bash\ngrep -rnE 'model=[\"'\\''][^\"'\\'']*((gemini|gpt|claude|llama)[^\"'\\'']*)[\"'\\''']' \\\n app/desktop libs/core/kiln_ai libs/server/kiln_server \\\n --include=\"*.py\" 2>/dev/null \\\n | grep -v \"test_\" | grep -v \"/build/\" \\\n | tee \"${SCRATCH}/prod_probes.txt\"\n```\n\nFor each hit:\n1. Decide whether it's a probe (connectivity/auth check, model incidental) or real prod behavior (a default the user sees, fine-tune flow, eval scoring). If unsure, treat it as real prod behavior.\n2. For probes, check `ml_model_list.py` for the newest non-deprecated sibling at that provider.\n3. Record in the probe table: file:line · current slug · suggested newer slug (or \"none — already latest\") · the verifying live test and its result.\n\nEvery grep hit goes in the table even when no newer sibling exists. If the verifying live test failed outright, also list it under Failures (it's a real-failure signal).\n\n---\n\n## Phase 5 — Write the report\n\nCreate `.prerelease/<timestamp>/REPORT.md`. Lead with the verdict; make every finding skim-readable with an explicit recommended action. This is the only artifact the skill produces.\n\n```markdown\n# Kiln Prerelease Check — <timestamp>\n\n## Verdict\n- Overall: PASS / FAIL / PASS WITH RECOMMENDATIONS\n- checks.sh: <pass|fail>\n- prerelease pytest: <N passed, M failed, K skipped>\n- Retries: <none needed | \"X of M first-run failures were retried (Phase 4a); Y passed on retry (transient/flaky), Z reproduced\" — omit if no test was retried>\n- Headline: <one sentence — the single most important thing the user must know before tagging>\n\n## Test results\nOne scannable table covering **every prerelease test that ran**, grouped by coverage area, with an emoji outcome and a short note where it helps — this is the at-a-glance summary the user reads first. Aggregate parametrized cases into one row with per-case emoji counts (e.g. `✅×8 ❌×1`); don't spell out long node-id / filename lists here (raw lists live in the logs and `_scratch/`). Outcome legend: ✅ pass · ❌ fail · ⏭️ skip (missing key / N/A) · 🔁 transient (failed then passed on retry). Leave the Notes cell blank when the row is a clean all-pass; use it to name the failing case, why it skipped, or the retry result.\n\n| Area | Test(s) | Result | Notes |\n|---|---|---|---|\n| OpenAI basic call | `test_openai` | ✅ | |\n| Groq | `test_groq` | ✅ | |\n| OpenRouter multi-model | `test_openrouter` (9) | ✅×9 | |\n| Amazon Bedrock | `test_amazon_bedrock` | ✅ | |\n| Structured output | `test_structured_output_*` (6) | ✅×6 | |\n| Streaming | `test_invoke_openai_stream_chunks` / `test_invoke_ai_sdk_stream` (N) | ✅×N | |\n| Tool calling | `test_tools_gpt_4_1_mini` | ✅ | |\n| Thinking-level reasoning | `…reasoning_content_prerelease_smoke` (N) | ✅×N | <which case failed/skipped, if any> |\n| Embeddings | `…embeddings_basic_prerelease_smoke` (6) | ✅×4 ⏭️×2 | <which providers skipped + why> |\n| Reranker | `test_reranker_integration_success` (N) | ✅×1 ❌×2 | <which provider failed + one-line cause> |\n| Prompt caching | `test_prompt_caching_cache_hit` (7) | ✅×7 | |\n| Document extraction | `…extract_document_*_prerelease_smoke` (N) | ✅×N | |\n| Semantic chunker | `test_semantic_chunker_real_integration` | ✅ | |\n| Vertex live connect | `test_connect_vertex_live` | ❌ | <one-line cause> |\n| Fireworks fine-tune | `test_fetch_all_deployments` | ✅ | |\n\nUse the real areas/tests/counts from this run (mirror the coverage table in Phase 4c). Put 🔁 on anything that needed a retry and add the retry detail in the Notes cell + under Failures.\n\n## Failures — what broke, why, and what to do\nOne block per failing test. Sorted most-actionable first (real regressions before stale pins).\nIf any failures were retried (Phase 4a), open this section with a small first-run-vs-retry table so transient flakes are visible at a glance, then write a block only for the tests that **reproduced** (note retried-and-passed ones as transient — don't dignify them with a full diagnosis block).\n\n### <test ID>\n- Provider: <provider>\n- Cause: <deprecated model | model no longer exists at provider | provider behavior change (regression) | Kiln code regression | auth/quota/transient>\n- Model slug: <slug> — `ml_model_list.py` status: <OK | deprecated | no-provider-entry | n/a>\n- Retry: <not retried (deterministic cause) | retried once — reproduced | retried once — passed (transient, flaky)>\n- Evidence: <short excerpt of the actual error / failed assertion>\n- Recommended action: <concrete next step — e.g. \"bump PRERELEASE_CHAT_MODELS pin from X to Y (newest sibling)\"; or \"investigate prod path Z — response shape changed\"; or \"mark this entry deprecated=True in ml_model_list.py and bump the pin\". Recommendation only — this skill makes no edits.>\n\n## Skipped tests (missing credentials)\nList by provider, with which env var is missing, so the user can decide what to add and re-run.\n\n## Model-pin staleness sweep (mandatory, every run)\nEvery in-scope hardcoded slug — including current ones needing no action.\n\n| Location (file:line) | Current slug | `ml_model_list.py` status | Newer sibling? | Recommendation |\n|---|---|---|---|---|\n| `…/pytest_prerelease_whitelist.py:NN` | `claude_sonnet_4_5` | OK | yes — `claude_sonnet_4_6` | bump to `claude_sonnet_4_6` |\n| `…/pytest_prerelease_whitelist.py:NN` | `gpt_4o_mini` | OK | no | keep (current) |\n| `…/test_prompt_adaptors.py:NN` | `gemini_1_5_flash` (openrouter) | deprecated | n/a | bump to `gemini_2_5_flash` |\n\n`status` ∈ {OK, deprecated, no-provider-entry, removed/renamed}. `Recommendation` ∈ {keep (current), bump to <slug>, investigate}.\n\n## Prod-code probe sweep (flag only — no edits)\nEvery hardcoded inference-probe slug found in prod code (Phase 4d).\n\n| File:line | Current slug | Suggested newer slug | Verifying test | Test result | Notes |\n|---|---|---|---|---|---|\n| `app/desktop/studio_server/provider_api.py:NNNN` | `vertex_ai/gemini-2.5-flash` | `vertex_ai/gemini-3-flash` | `test_connect_vertex_live` | PASS | probe still works; newer family member exists |\n| `path/to/file.py:NN` | `…` | none — already latest | `test_…` | PASS | no action |\n\n## Coverage gaps\n- Providers/families in `ml_model_list.py` with no prerelease test.\n- Recently added paid tests not in the prerelease set (one line each on why it MIGHT be prerelease-worthy).\n\n## Flagged for user review\nAnything you noticed but couldn't safely classify: ambiguous failures (model-vs-code unclear), prod-path code that isn't a probe, suspicious behavior. Each row: file:line / test · what's suspicious · what would disambiguate.\n\n## Suggested next steps\nOrdered checklist of what the user should do before tagging the release (apply recommended pin bumps, investigate regression X, add key for skipped provider Y, etc.).\n```\n\nKeep it scan-readable. Lead with the verdict. Don't bury real regressions under pin-housekeeping.\n\n---\n\n## Phase 6 — Hand back to the user\n\nPrint the report path and a one-line verdict:\n\n```bash\necho \"Prerelease check complete. Report: ${OUT}/REPORT.md\"\n```\n\nThen summarize in chat (one or two sentences):\n\n- All green, nothing stale: `Prerelease check: PASS. <N> passed, <K> skipped (missing keys: <providers>). No stale pins. Report at <path>.`\n- Green but recommendations exist: `Prerelease check: PASS WITH RECOMMENDATIONS. <N> passed; <M> stale/older pins to consider bumping (no changes made). Report at <path>.`\n- Something failed: `Prerelease check: FAIL. <X> failed across <providers> — <one-line dominant cause>. See <path>/REPORT.md before tagging.`\n\nRemember: the skill never made any edits. If the report recommends pin bumps or a prod-probe change, those are for the user to apply.\n\n---\n\n## Additional release checklists\n\n- **Code Tools / Code Evals cross-OS spawn sanity**: see `specs/projects/code_tools/cross_os_spawn_checklist.md` for platform-specific verification steps (macOS/Windows/Linux + PyInstaller frozen builds). Must be completed before any release that includes code tools or code evals.\n\n## Checklist\n\n- [ ] `.prerelease/<timestamp>/` directory created (the only place anything was written)\n- [ ] `checks.sh --agent-mode` run, log captured\n- [ ] `--runprerelease` pytest run, log captured\n- [ ] Likely-transient failures (auth/quota/transient + ambiguous behavioral) retried once in isolation; deterministic failures (deprecated/removed slug, Kiln-code traceback) not retried\n- [ ] Every retried test disclosed in the report (first-run result, retry result, transient-vs-reproducible verdict) — no silent FAIL→PASS swaps\n- [ ] Every failure diagnosed with a cause class and a concrete recommended action (regression vs. deprecated-model vs. model-gone vs. auth/transient)\n- [ ] Model-pin staleness sweep table filled for **every** in-scope slug, including current ones needing no action\n- [ ] Newer-sibling recommendations included even for green pins\n- [ ] Prod-probe sweep done; every grep hit in the probe table with a suggested slug (or \"none — already latest\")\n- [ ] Coverage-gap audit done\n- [ ] Skipped tests listed by provider + missing env var\n- [ ] **No files edited outside `.prerelease/<timestamp>/`** — all fixes are recommendations\n- [ ] `REPORT.md` written, verdict first, recommended actions explicit\n- [ ] Verdict surfaced to the user with the report path\n\n<!-- File: .agents/skills/release-digest/SKILL.md (Tokens: ~2872 | Category: anthropic-skill) -->\n---\nname: release-digest\ndescription: Post a \"what's changed since the last release\" recap to the release Slack channel for final QA. Groups merged PRs by author and classifies each as feature / bug fix / task. Use when the user wants a release recap, a changelog since the last tag, or to prep QA before cutting a release from main.\n---\n\n# Release Digest\n\nBuild a human-readable recap of everything merged to `main` that is **not yet in the\nlatest release tag**, and post it to the team's release Slack channel for final QA.\n\nThe recap is grouped **by author** and each change is classified as **feature**,\n**bug fix**, or **task**. This is on-demand: run it when someone asks for a release\nrecap or wants to QA before cutting a release.\n\n> Linear ticket tracking (\"which tickets are targeted for this release\") is **out of\n> scope for now** — punted to a later iteration. This skill only covers what's changed.\n\n---\n\n## Phase 1 — Gather the change set\n\nRun the gather script from the repo root. It fetches tags + `origin/main`, finds the\nlatest tag, and lists every merged PR in the range `<last_tag>..origin/main`:\n\n```bash\npython3 .agents/skills/release-digest/scripts/gather_changes.py > /tmp/release_digest.json\n```\n\nKey design point: it uses the **git commit range** as the source of truth, not merge\ndates. A PR merged the same day a tag was cut can already be in that release, so a\ndate filter would over-report. Trust the range.\n\n**Pre-release / already-cut releases.** By default the baseline is the newest tag\n(`git describe`). But if the release you're QA'ing has **already been tagged** — it's a\npre-release still being validated, so `main` is already even with it — the default\nrange is empty. In that case pass `--pre-release`, which steps the baseline back one\ntag so the range recaps exactly what went *into* the pre-release:\n\n```bash\npython3 .agents/skills/release-digest/scripts/gather_changes.py --pre-release > /tmp/release_digest.json\n```\n\nSigns you want `--pre-release`: the first run reports \"nothing unreleased\" or \"main is\neven with the latest tag\", yet the user says a release was just cut / is being QA'd. To\nname the baseline explicitly instead, use `--since <tag>` (e.g. `--since v1.0.3`). The\ntwo flags are mutually exclusive. When `--pre-release` is used, the newest tag is the\none being cut, so in Phase 3 use **that tag** as `<new_release>` rather than asking.\n\nThe JSON contains:\n- `last_tag` — the most recent release (e.g. `v1.0.3`)\n- `head_ref` — `origin/main` (or `main` if no remote ref)\n- `pr_count`, `commit_count`\n- `prs[]` — `{number, title, author_name, author_login, url, labels, additions, deletions, body}`\n- `direct_commits[]` — teammate commits pushed straight to main, no PR — `{sha, title, author_login, author_name}`\n- `excluded_prs[]` — PRs/commits from non-teammates, dropped from the recap. PR entries\n have a `number`; direct-commit entries have `number: null` and rely on `title`.\n- `unresolved_prs[]` — PRs whose `gh pr view` failed (`{number, error}`). If non-empty,\n the change set is partial — tell the user which PRs couldn't load before posting.\n\nOnly the five teammates in the script's `TEAM` map (Sam, Leonard, Daniel, Mike,\nSteve) are included in the recap. PRs from anyone else are filtered into\n`excluded_prs` rather than itemized. To add a teammate, edit `TEAM` in the script.\n\n**If `pr_count` is 0 AND `direct_commits` is empty:** tell the user `main` is even with\n`<last_tag>` — nothing to QA or release yet. Do not post to Slack. Stop here. (Direct\ncommits alone, with no PRs, still count as unreleased changes worth a digest.)\n\n---\n\n## Phase 2 — Classify and group\n\nRead `/tmp/release_digest.json`. Treat every item in `prs[]` **and** `direct_commits[]`\nas a change to recap (direct commits are real work that shipped — just with no PR\nnumber). Classify each into one of three buckets:\n\n- **Feature** — a substantive *new product capability*: new UI, a new endpoint, a new\n tool, new eval/RAG/fine-tune functionality, or a meaningful enhancement to how the\n product works. Reserve this for things a user would notice as \"Kiln can now do X.\"\n- **Bug fix** — corrects broken or incorrect behavior: regressions, crashes, hangs,\n wrong output, security fixes.\n- **Task** — everything that is *neither* a substantive new capability *nor* a fix to\n broken behavior: refactors, docs/README, tests, CI/build, dependency bumps, version\n bumps, chores, internal flags, code removal/cleanup — **and routine model-catalog\n work** (see below). Matches Linear's \"Task\" issue type: maintenance/internal work.\n\n**Adding models is a Task, not a Feature.** Kiln ships new model and provider entries\nconstantly — \"Add <model> to model list\", \"Add <provider> for <model>\", backfilling a\nmodel on a provider, re-enabling/un-recommending a model, version bumps to a model\nentry. These are catalog/config maintenance, so they go in **Task** even though the\ntitle starts with `Add`/`Enable`. Only promote to **Feature** if the PR adds a genuinely\nnew *capability* around models (e.g. a new thinking-level control, a new extraction\nmodality, a new provider *integration type*) rather than just registering a model_id.\n\n**Decision order** (use the first signal that's decisive):\n1. **PR labels** — `bug`/`fix` → Bug fix; `feature`/`enhancement` → Feature;\n `chore`/`docs`/`refactor`/`ci` → Task. (Kiln PRs are usually unlabeled, so this\n rarely fires — fall through to the title.)\n2. **Model-catalog check** — if the change just registers/edits model or provider\n entries (not new behavior), it's a **Task**, regardless of the `Add`/`Enable` verb.\n3. **Conventional-commit / verb prefix in the title** — `fix:`/`Fix ` → Bug fix;\n `feat:`/`Add `/`Enable ` (a new *capability*) → Feature; `chore:`/`refactor:`/`ci:`/\n `docs:`/`Bump `/`Remove `/`Deprecate `/`Version bump` → Task.\n4. **What the change does** (read the title, and `body` if still unclear) — ask \"does\n this give the user a genuinely new capability (Feature), fix something that was\n broken (Bug fix), or is it routine/maintenance (Task)?\"\n\n**\"Task\" is the residual bucket**, not a guess: a change lands there only because it\nis positively *not* a new capability and *not* a fix — e.g. `Deprecate Models`,\n`min age for dependencies`, `Version bump`, `new draft readme`, `fix typos`. When a\nchange is genuinely ambiguous between Feature and Bug fix, prefer the one the title\nleans toward; only fall to Task when it's truly neither. This is a judgment call made\nper item — surface it to the user before posting so they can re-bucket anything.\n\nThen **group by author** (`author_name`). Within each author, list changes by PR\nnumber, newest first; put any direct commits (no PR number) last. The feature / fix /\ntask split is **not** shown per line — it only feeds the tally at the bottom.\n\n---\n\n## Phase 3 — Compose the Slack message\n\n**First, get the new release name** — the version being cut (e.g. `v1.0.4`). If you ran\nin `--pre-release` mode, the release has already been tagged, so use that newest tag\n(the one the script named as \"the release being QA'd\") as `<new_release>` — no need to\nask. Otherwise the release isn't tagged yet and can't be derived from git, so **ask the\nuser**. Use the result as `<new_release>` in the title.\n\nUse this shape (Slack mrkdwn — `*bold*`, not `**bold**`):\n\n```\n:rocket: *v1.0.4 Release Status* :rocket:\n\n*12 changes since v1.0.3*\n_<one-to-two sentence plain-English recap of the headline changes>_\n\n*Daniel Chiang*\n• Add deprecated flag to embedding models (#1466)\n• Add deprecated flag to reranker models (#1465)\n\n*Sam Fierro*\n• Dedup frozen prompts and bake type into their name (#1501)\n\n*Steve Cosman*\n• Add Jinja2 input transform on RunConfigs (#1433)\n• Version bump in lock and desktop app version check\n\n:bar_chart: *7 features · 3 fixes · 2 tasks*\n\n:information_source: _1 PR from a non-teammate: #1456 (ianjamesburke)._\n```\n\nRules:\n- **Header:** `:rocket: *<new_release> Release Status* :rocket:`, then a blank line.\n- **Change count** on its own line, **bold**: `*<N> changes since <last_tag>*`.\n- **Recap paragraph** immediately under it, wrapped in **italics** (`_..._` around the\n whole paragraph) — the \"what did we do\" summary in plain English, not just a list.\n Compose it so the italics survive: write it as **a single line with no hard line\n breaks** (wrap is fine, but don't insert `\\n` mid-paragraph — newlines break the\n italic span in Slack), and use **no literal `_` anywhere inside** (an underscore\n would close the italics mid-paragraph — rewrite e.g. `snake_case`/`model_id` or any\n underscored identifier without it).\n- One **bullet** per change, no per-line emoji: `• <title> (#<number>)`. Keep the\n title; trim trailing noise.\n- **Direct commits have no PR number** — render them as `• <title>` with nothing in\n parentheses (e.g. `• Version bump`). List them under their author, after the PRs.\n- Within each author, list newest PR first; direct commits last. Don't print the\n feature/fix/task bucket per line.\n- **Summary block** at the end, in this order:\n 1. A **bold** `:bar_chart:` tally line: `:bar_chart: *<X> features · <Y> fixes · <Z> tasks*`.\n 2. A blank line.\n 3. If `excluded_prs` is non-empty, an **italic** `:information_source:` line noting the\n excluded non-teammate changes (so the omission is visible, not silent). Reference a\n PR entry by `#<num>` and a direct-commit entry (where `number` is null) by its\n `title` — never print `#None`. E.g.\n `:information_source: _<N> change(s) from non-teammates excluded: #1456 (ianjamesburke)._`\n- Link the PR numbers if easy (`<url|#1501>`), but plain `#1501` is fine — keep it readable.\n\n**Show the composed message to the user and get confirmation before posting** (it's an\noutward-facing post to a shared channel).\n\n---\n\n## Phase 4 — Post to Slack\n\nPost to **`#release`** (channel ID `C0BDGTGBL4Q`) — the team's release channel. This is\nthe default target; only post elsewhere if the user explicitly names a different\nchannel. (If the ID ever fails to resolve, fall back to `slack_search_channels` with\nquery `release` to re-find it.)\n\nOnly post after the user has confirmed the composed message in Phase 3. Post with\n`mcp__claude_ai_Slack__slack_send_message`. After posting, give the user the channel\nname and a link/confirmation.\n\n---\n\n## Checklist\n\n- [ ] Gather script run; `/tmp/release_digest.json` written\n- [ ] If `pr_count` is 0, reported \"nothing unreleased\" and stopped\n- [ ] Asked the user for the new release name (used in the title)\n- [ ] Each PR classified (feature / bug fix / task)\n- [ ] Grouped by author, recap paragraph written\n- [ ] Message shown to user and confirmed\n- [ ] Posted to #release (C0BDGTGBL4Q); confirmation returned to user\n\n<!-- File: .agents/mcp.json (Tokens: ~73 | Category: mcp-config) -->\n{\n \"mcpServers\": {\n \"HooksMCP\": {\n \"command\": \"uvx\",\n \"args\": [\n \"--from\",\n \"hooks-mcp>=0.2.5\",\n \"hooks-mcp\",\n \".config/hooks_mcp.yaml\",\n \"--working-directory\",\n \".\"\n ]\n }\n }\n}\n</yakaai_skills>"}