# Technical Documentation: Andyyyy64/whichllm > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/Andyyyy64/whichllm) ### Overview (`README.md`) # whichllm [](https://pypi.org/project/whichllm/) [](https://www.python.org/downloads/) [](https://opensource.org/licenses/MIT) [](https://github.com/Andyyyy64/whichllm/actions/workflows/test.yml) [](https://github.com/sponsors/Andyyyy64)

Andyyyy64%2Fwhichllm | Trendshift

**Find the best local LLM that actually runs on your hardware.** Auto-detects your GPU/CPU/RAM and ranks the top models from HuggingFace that fit your system. [日本語版はこちら](docs/README.ja.md) ## Quick start Run the recommendation command once, with no project setup. ```bash uvx whichllm@latest ``` Simulate a GPU before you buy hardware. ```bash uvx whichllm@latest --gpu "RTX 4090" ``` Install it when you use it often. ```bash uv tool install whichllm uv tool upgrade whichllm # update an existing install ``` Other install paths. ```bash brew install andyyyy64/whichllm/whichllm pip install whichllm ``` ## Want a safer pick? By default, whichllm is ambitious. It ranks the best model that looks runnable on your machine, including partial RAM offload and near-edge VRAM fits when they seem usable. If you want a more comfortable LM Studio-style recommendation, start with: ```bash uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1GB ``` This keeps only models that fit fully in GPU VRAM, filters out slow estimates, and leaves extra VRAM for runtime overhead. If LM Studio still says the model is slightly too large, increase the headroom: ```bash uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1.5GB ``` ## Common workflows After install, run `whichllm` directly. For one-off runs, replace `whichllm` with `uvx whichllm@latest`. ```bash # Best models for this machine whichllm # Pretend you have a specific GPU whichllm --gpu "RTX 4090" # Override detected iGPU/unified-memory limits whichllm --vram 8 --ram-bandwidth 68 # Only show models that fit fully in GPU VRAM whichllm --gpu-only whichllm --fit gpu # Simulate a multi-GPU workstation whichllm --gpu "2x RTX 4090" # Hide models that are technically runnable but too slow whichllm --speed usable whichllm --speed fast # Pasteable GitHub / Slack / Discord output whichllm --markdown # Compare upgrade candidates whichllm upgrade "RTX 4090" "RTX 5090" "H100" # Find the GPU needed for a model whichllm plan "llama 3 70b" # Start a chat with a model whichllm run "qwen 2.5 1.5b gguf" # Print copy-paste Python whichllm snippet "qwen 7b" # Return JSON for scripts whichllm --top 1 --json ``` ## See it ```text $ whichllm --gpu "RTX 4090" #1 Qwen/Qwen3.6-27B 27.8B Q5_K_M score 92.8 27 t/s #2 Qwen/Qwen3-32B 32.0B Q4_K_M score 83.0 31 t/s #3 Qwen/Qwen3-30B-A3B 30.0B Q5_K_M score 82.7 102 t/s ``` The 32B model **fits your card fine** — whichllm still ranks the 27B #1, because it scores higher on real benchmarks and is a newer generation. A size-only "what fits?" tool would hand you the bigger one. That gap is the whole point of whichllm. (Note #3: a MoE model at 102 t/s — speed is ranked on *active* params, quality on *total*.) ## What can I run? Real top picks (snapshot 2026-05 — your results track **live** HuggingFace data, this is not a static list): | Hardware | VRAM | Top pick | Speed | |---|---|---|---| | RTX 5090 | 32 GB | `Qwen3.6-27B` · Q6_K · score 94.7 | ~40 t/s | | RTX 4090 / 3090 | 24 GB | `Qwen3.6-27B` · Q5_K_M · score 92.8 | ~27 t/s | | RTX 4060 | 8 GB | `Qwen3-14B` · Q3_K_M · score 71.0 | ~22 t/s | | Apple M3 Max | 36 GB | `Qwen3.6-27B` · Q5_K_M · score 89.4 | ~9 t/s | | CPU only | — | `gpt-oss-20b` (MoE) · Q4_K_M · score 45.2 | ~6 t/s | `whichllm --gpu ""` simulates any of these before you buy. By default, rankings include full-GPU, partial-offload, and CPU-only candidates when they are usable. Use `--gpu-only` or `--fit full-gpu` when you only want models that fit entirely in GPU VRAM. The default table shows memory, estimated generation speed, fit type, and published date. Speed is colored by practical usability: under 4 tok/s is red, 4-10 is yellow, 10-30 is green, and 30+ is bright green. `~` / `?` still mark estimate confidence. ## Why whichllm? Fitting a model into your VRAM is the easy part. The hard part is knowing **which of the models that fit is actually the best** — and that is what whichllm is built to get right. - **Evidence-based ranking, not a size heuristic** — The top pick is chosen from merged real benchmarks (LiveBench, Artificial Analysis, Aider, multimodal/vision, Chatbot Arena ELO, Open LLM Leaderboard) — never "the biggest model that happens to fit." - **Recency-aware** — Stale leaderboards are demoted along each model's lineage, so a 2024 model can't outrank a current-generation one on an outdated score. The benchmark snapshot date is printed under every ranking, so a stale recommendation is self-evident instead of silently trusted. - **Evidence-graded and guarded** — Every score is tagged `direct` / `variant` / `base` / `interpolated` / `self-reported` and discounted by confidence. Fabricated uploader claims and cross-family inheritance (a small fork borrowing its much larger base's score) are actively rejected. - **Architecture-aware estimates** — VRAM = weights + GQA KV cache + activation + overhead; speed is bandwidth-bound with per-quant efficiency, per-backend factors, MoE active-vs-total split, and unified-memory vs discrete-PCIe partial-offload modeling. - **One command, scriptable** — `whichllm` prints the answer; add `--json | jq` for pipelines. No TUI, no keybindings to memorize. - **Live data** — Models fetched directly from the HuggingFace API, with curated frozen fallbacks for offline or rate-limited use. ## Features - **Auto-detect hardware** — NVIDIA, AMD, Intel, Apple Silicon, CPU-only - **Smart ranking** — Scores models by VRAM fit, speed, and benchmark quality - **One-command chat** — `whichllm run` downloads and starts a chat session instantly - **Code snippets** — `whichllm snippet` prints ready-to-run Python for any model - **Live data** — Fetches models directly from HuggingFace (cached for performance) - **Benchmark-aware** — Integrates real eval scores with confidence-based dampening - **Task profiles** — Filter by general, coding, vision, or math use cases - **GPU simulation** — Test with any GPU: `whichllm --gpu "RTX 4090"` - **Multi-GPU simulation** — Repeat `--gpu`, use commas, or write `2x RTX 4090` - **Full-GPU filter** — `--gpu-only` / `--fit full-gpu` hides offload candidates - **Speed-aware filtering** — `--speed usable|fast` hides slow rows by threshold - **Markdown output** — `--markdown` / `-m` prints pasteable GFM tables - **Runtime memory budgets** — `--vram-headroom` and `--ram-budget` avoid edge fits - **LM Studio detection** — Marks exact, complete GGUF artifacts already installed - **Hardware planning** — Reverse lookup: `whichllm plan "llama 3 70b"` - **Upgrade planning** — Compare your current machine with candidate GPUs - **JSON output** — Pipe-friendly: `whichllm --json` ## Run & Snippet Try any model with a single command. No manual installs needed — whichllm creates an isolated environment via `uv`, installs dependencies, downloads the model, and starts an interactive chat. ```bash # Chat with a model (auto-picks the best GGUF variant) whichllm run "qwen 2.5 1.5b gguf" # Auto-pick the best model for your hardware and chat whichllm run # CPU-only mode whichllm run "phi 3 mini gguf" --cpu-only ``` Works with **all model formats**: - **GGUF** — via `llama-cpp-python` (lightweight, fast) - **AWQ / GPTQ** — via `transformers` + `autoawq` / `auto-gptq` - **FP16 / BF16** — via `transformers` Get a **copy-paste Python snippet** instead: ```bash whichllm snippet "qwen 7b" ``` ```python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="Qwen/Qwen2.5-7B-Instruct-GGUF", filename="qwen2.5-7b-instruct-q4_k_m.gguf", n_ctx=4096, n_gpu_layers=-1, verbose=False, ) output = llm.create_chat_completion( messages=[{"role": "user", "content": "Hello!"}], ) print(output["choices"][0]["message"]["content"]) ``` ## Usage ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Markdown output is intended for GitHub issues, READMEs, Slack, Discord, and blog posts: ```bash whichllm --markdown whichllm -m --top 5 --gpu "RTX 4090" ``` JSON model rows include `fit_type`, `vram_required_bytes`, `vram_available_bytes`, `uses_multi_gpu`, `multi_gpu_effective_vram_bytes`, `estimated_tok_per_sec`, `speed_confidence`, `speed_range_tok_per_sec`, `speed_notes`, `benchmark_source`, and `benchmark_confidence`. The speed range is a planning range, not a live benchmark. ## Integrations ### Ollama Use JSON output to feed scripts that map HuggingFace IDs to your local Ollama model names: ```bash # Pick the top HuggingFace model ID whichllm --top 1 --json | jq -r '.models[0].model_id' # Find the best coding model ID whichllm --profile coding --top 1 --json | jq -r '.models[0].model_id' ``` Ollama model names do not always match HuggingFace repo IDs, so a small mapping step is usually needed before `ollama run`. ### Shell alias Add to your `.bashrc` / `.zshrc`: ```bash alias bestllm='whichllm --top 1 --json | jq -r ".models[0].model_id"' # Usage: ollama run $(bestllm) ``` ## Scoring Each model gets a 0-100 score. Benchmark quality and size form the core; evidence confidence and runtime fit then scale it, with speed, source trust, and popularity as adjustments. | Factor | Effect | Description | |--------|--------|-------------| | Benchmark quality | core | Merged LiveBench / Artificial Analysis / Aider / Vision / Arena ELO / Open LLM Leaderboard, weighted by source confidence | | Model size | up to 35 | `log2`-scaled world-knowledge proxy (MoE uses total params) | | Quantization | × penalty | Lower-bit quants discounted multiplicatively | | Evidence confidence | ×0.55–1.0 | none / self-reported ×0.55, inherited ×0.78, direct full | | Runtime fit | ×0.50–1.0 | partial-offload ×0.72, CPU-only ×0.50 | | Speed | -8 to +8 | Usability gate vs a fit-dependent tok/s floor; reported with confidence and range metadata | | Source trust | -5 to +5 | Official-org bonus, known-repackager penalty | | Popularity | tie-breaker | Downloads/likes; weight shrinks as evidence strengthens | Score markers: - **`~`** (yellow) — No direct benchmark; score inherited/interpolated from the model family - **`!sr`** (bright yellow) — Uploader-reported benchmark only, not independently verified - **`?`** (red) — No benchmark data available Speed display: - **red** — Slow generation speed (`<4 tok/s`) - **yellow** — Marginal generation speed (`4-10 tok/s`) - **green** — Usable generation speed (`10-30 tok/s`) - **bright green** — Fast local generation speed (`>=30 tok/s`) - **`~`** (yellow) — Estimated tok/s range is available - **`?`** (red) — Low-confidence speed estimate; backend/runtime sensitivity is high ## Documentation - [CLI reference](docs/cli.md) - [How it works](docs/how-it-works.md) - [Scoring](docs/scoring.md) - [Hardware detection and simulation](docs/hardware.md) - [Run and snippet](docs/run-snippet.md) - [Troubleshooting](docs/troubleshooting.md) ## How it works ### Data pipeline 1. **Model fetching** — Fetches popular models from HuggingFace API: - Text-generation (downloads + recently updated) - GGUF-filtered (separate query for coverage) - Vision models (`image-text-to-text`) when `--profile vision` or `any` 2. **Benchmark sources** — *Current tier* (LiveBench, Artificial Analysis Index, Aider) merged live when reachable, plus a curated multimodal / vision index; *frozen tier* (Open LLM Leaderboard v2, Chatbot Arena ELO). Tiers have separate caps and lineage-aware recency demotion so stale leaderboards stop over-rewarding older generations. 3. **Benchmark evidence** — Five resolution levels, increasingly discounted: - `direct` — Exact model ID match - `variant` — Suffix-stripped or -Instruct variant - `base_model` — Base model from cardData - `line_interp` — Size-aware interpolation within model family - `self_reported` — Uploader-claimed eval (heavily discounted) Inheritance is rejected when a model's params diverge more than 2× from its family's dominant member, catching draft / MTP / abliterated forks that share a `family_id` with a much larger base. 4. **Cache** — normally `~/.cache/whichllm/`, or `$XDG_CACHE_HOME/whichllm/` when `XDG_CACHE_HOME` is set to an absolute path: - `models.json` — 6h TTL - `benchmark.json` — 24h TTL ### Ranking engine 1. **Hardware detection** — NVIDIA (nvidia-ml-py), AMD (ROCm/dbgpu), Intel, Apple Silicon (Metal), CPU cores, RAM, disk 2. **VRAM estimation** — Weights + KV cache + activation + framework overhead (~500MB) 3. **Compatibility** — Full GPU / Partial Offload / CPU-only; compute capability and OS checks 4. **Speed** — tok/s from GPU memory bandwidth, quantization, backend, fit type, and MoE active parameters 5. **Scoring** — Benchmark (with confidence dampening), size, quantization penalty, fit type, speed, popularity, source trust (official vs repackager) 6. **Backend filter** — Apple Silicon and CPU-only restrict to GGUF for stability; Linux+NVIDIA allows AWQ/GPTQ ### Project structure ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Development ```bash git clone https://github.com/Andyyyy64/whichllm.git cd whichllm uv sync --dev uv run whichllm uv run pytest ``` ## Contributing Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. ## Support If whichllm helped you find a model or avoid a bad hardware guess, sponsoring is appreciated. It helps keep the project maintained: hardware reports, packaging, test fixtures, benchmark updates, and support for more machines. whichllm will stay open-source either way. Issues and PRs are always welcome. Useful? A GitHub star helps other people find it, and I'd genuinely like to know what it picked for your rig. Drop it in [Issues](https://github.com/Andyyyy64/whichllm/issues). ## Star History [](https://star-history.dera.page/#Andyyyy64/whichllm&Date) ## Requirements - Python 3.11+ - NVIDIA GPU detection via `nvidia-ml-py` (included by default) - AMD / Apple Silicon detected automatically ## License MIT --- ### README.Ja (`docs/README.ja.md`) # whichllm **手元のハードウェアで実際に動くローカルLLMを探すCLIです。** whichllm は GPU / CPU / RAM / ディスクを検出し、HuggingFace 上のモデルを 取得して、実行できる候補をランキングします。単に「VRAMに入る最大モデル」を 選ぶのではなく、ベンチマーク、量子化、速度、実行形態、モデル世代をまとめて 評価します。 [English version](../README.md) ## インストール ### uv 一度だけ試す場合: ```bash uvx whichllm@latest ``` 継続して使う場合: ```bash uv tool install whichllm uv tool upgrade whichllm # 既存インストールを更新 ``` ### Homebrew ```bash brew install andyyyy64/whichllm/whichllm ``` ### pip ```bash pip install whichllm ``` ## 余裕を持って動く候補だけ見たい場合 whichllm のデフォルトは少し攻めた推薦です。RAMへのpartial offloadや、 VRAMぎりぎりの候補も、動きそうならランキングに入れます。 LM Studioなどで余裕を持って動かしたい場合は、まずこれを使ってください。 ```bash uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1GB ``` GPUのVRAMに全部載る候補だけに絞り、遅い推定速度の候補を外し、実行時の 余白も1GB残します。 それでもLM Studio側で少しはみ出す場合は、余白を増やします。 ```bash uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1.5GB ``` ### 開発用 ```bash git clone https://github.com/Andyyyy64/whichllm.git cd whichllm uv sync --dev uv run whichllm uv run pytest ``` ## まず使う ```bash # 自動検出しておすすめモデルを表示 whichllm # GPUをシミュレートする whichllm --gpu "RTX 4090" whichllm --gpu "Apple M3 Max" # 複数GPUをシミュレートする whichllm --gpu "2x RTX 4090" whichllm --gpu "RTX 4090" --gpu "RTX 3090" # GPUのVRAMに全部載る候補だけを見る whichllm --gpu-only whichllm --fit gpu # 速度の最低ラインを指定する whichllm --speed usable whichllm --speed fast whichllm --min-speed 4 # GitHubやSlackに貼りやすいMarkdown表で出力する whichllm --markdown # 実行時のメモリ余白やRAM使用量を指定する whichllm --vram-headroom 1.5GB whichllm --ram-budget available # CPUのみとして評価する whichllm --cpu-only # JSONで出力する whichllm --json ``` JSONの各モデルには `estimated_tok_per_sec` に加えて、`fit_type`、 `vram_required_bytes`、`vram_available_bytes`、`uses_multi_gpu`、 `multi_gpu_effective_vram_bytes`、`speed_confidence`、 `speed_range_tok_per_sec`、`speed_notes`、`benchmark_source`、 `benchmark_confidence` が入ります。 速度は実測値ではなく、ハードウェア情報とモデル情報からの推定です。 通常の表には必要メモリ、推定生成速度、Fit種別、Published が表示されます。 Downloads まで見たい場合は `--details` を使います。 GitHub issue、README、Slack、Discord へ貼る場合は `--markdown` / `-m` でMarkdown表として出力できます。 ## 主なコマンド ```bash # 推薦ランキング whichllm --top 20 whichllm --quant Q4_K_M whichllm --min-speed 30 whichllm --speed usable whichllm --speed fast whichllm --markdown whichllm --profile coding whichllm --context-length 64k whichllm --details whichllm --gpu-only # ベンチ根拠の厳しさ whichllm --evidence strict whichllm --evidence base whichllm --direct # モデルから必要GPUを逆算 whichllm plan "llama 3 70b" whichllm plan "Qwen2.5-72B" --quant Q8_0 whichllm plan "mistral 7b" --context-length 32768 # 今のマシンと購入候補GPUを比較 whichllm upgrade "RTX 4090" "RTX 5090" "H100" # モデルをダウンロードしてチャット whichllm run "qwen 2.5 1.5b gguf" whichllm run # 実行用Pythonコードを表示 whichllm snippet "qwen 7b" # ハードウェア情報だけ表示 whichllm hardware ``` ## スコアの見方 各モデルには 0 から 100 のスコアが付きます。中心になるのはベンチマークと モデルサイズですが、実行時に遅すぎる候補や、CPUオフロードが大きい候補は 下がります。 | 要素 | 役割 | | --- | --- | | ベンチマーク | LiveBench、Artificial Analysis、Aider、Vision、Arena、Open LLM Leaderboard を統合 | | モデルサイズ | 知識量の近似。MoEは総パラメータを使う | | 量子化 | Q4 / Q5 / Q6 / Q8 などの品質低下を反映 | | 実行形態 | Full GPU、Partial Offload、CPU-only を区別 | | 速度 | tok/s が実用ラインを下回ると減点。表示時は推定の信頼度と幅も出す | | 根拠の強さ | direct、base_model、variant、line_interp、self_reported を区別 | | 世代補正 | 古い凍結ベンチだけで新世代を上回らないよう調整 | スコア横のマーカー: - `~`: 直接ベンチではなく、系列や派生から推定したスコア - `!sr`: アップローダー自己申告の評価値だけに基づくスコア - `?`: 利用できるベンチマーク根拠がないスコア 速度表示: - 赤: `4 tok/s` 未満の遅い生成速度 - 黄: `4-10 tok/s` のぎりぎり使える生成速度 - 緑: `10-30 tok/s` の実用的な生成速度 - 明るい緑: `30 tok/s` 以上の高速なローカル生成速度 - `~`: 速度推定の幅がある通常の推定値 - `?`: backend や runtime の影響が大きい低信頼の推定値 ## 仕組み 1. ハードウェアを検出します。NVIDIA、AMD、Intel、Apple Silicon、CPU、RAM、 ディスク空き容量を見ます。 2. HuggingFace APIからモデルを取得します。人気モデル、GGUF、最近更新された GGUF、trending、重要な frontier モデルを組み合わせます。 3. ベンチマークを読み込みます。現在系の LiveBench / Artificial Analysis / Aider / Vision と、凍結系の Arena / Open LLM Leaderboard を分けて扱います。 4. `base_model` とモデル名からファミリーを作り、同じモデルの派生やGGUFを束ねます。 5. 候補ごとに VRAM、互換性、速度、速度推定の信頼度、スコアを計算します。 6. ファミリーごとに最も良い候補を残して表示します。 通常は full GPU、partial offload、CPU-only の候補をまとめて見ます。GPUの VRAMに全部載るモデルだけを見たい場合は `--gpu-only` か `--fit gpu` を使います。遅い候補を最初から除外したい場合は `--speed usable`、`--speed fast` を使います。 それぞれ `10 tok/s`、`30 tok/s` が最低ラインです。 もっと低いラインを指定したい場合は `--min-speed 4` のように数値で指定します。 キャッシュは通常 `~/.cache/whichllm/` に保存されます。`XDG_CACHE_HOME` が 絶対パスで設定されている場合は、その配下の `whichllm/` を使います。 - `models.json`: 6時間 - `benchmark.json`: 24時間 ## プロジェクト構成 ```text src/whichllm/ ├── cli.py # Typer CLI: main, plan, upgrade, run, snippet, hardware ├── constants.py # 互換用のregistry再export ├── data/ # GPU、量子化、framework、lineageのregistry ├── hardware/ # ハードウェア検出とGPUシミュレーション ├── models/ # HuggingFace取得、ベンチ、キャッシュ、グルーピング ├── engine/ # VRAM、互換性、速度、ランキング └── output/ # Rich表示、JSON、plan/upgrade表示 ``` ## 詳細ドキュメント - [CLIリファレンス](cli.md) - [仕組み](how-it-works.md) - [スコアリング](scoring.md) - [ハードウェア検出とシミュレーション](hardware.md) - [run と snippet](run-snippet.md) - [トラブルシュート](troubleshooting.md) ## 動作環境 - Python 3.11+ - NVIDIA GPU検出は `nvidia-ml-py` と `nvidia-smi` fallback - AMD GPU検出は Linux / ROCm / sysfs / lspci と Windows fallback - Intel GPU検出は Linux / sysfs / lspci と Windows fallback - Strix Halo、Ryzen AI MAX、Radeon 890M 系は shared memory APU として扱う - Apple Silicon検出は macOS / `system_profiler` ## ライセンス MIT --- ### Cli (`docs/cli.md`) # CLI reference This page describes the commands exposed by `whichllm`. It is based on the Typer entrypoint in `src/whichllm/cli.py`. ## Main command ```bash whichllm [OPTIONS] ``` Detects the current machine, loads model and benchmark data, ranks compatible models, and prints a table. Common options: | Option | Meaning | | --- | --- | | `--top`, `-n` | Number of ranked models to show. Default: `10` | | `--context-length`, `-c` | Context length used for KV cache estimation. Accepts integers or `k` shorthand such as `64k`. Default: `4096` | | `--quant`, `-q` | Keep only a quantization type such as `Q4_K_M` | | `--min-speed` | Keep only models above an exact tok/s estimate | | `--speed` | Named speed floor: `any`, `usable` (`10 tok/s`), or `fast` (`30 tok/s`) | | `--fit` | Runtime fit filter: `any`, `gpu`, or `full-gpu` | | `--gpu-only` | Alias for `--fit full-gpu`; excludes partial offload and CPU-only candidates | | `--profile` | Ranking profile: `general`, `coding`, `vision`, `math`, `any` | | `--evidence` | Benchmark evidence filter: `strict`, `base`, `any` | | `--direct` | Alias for `--evidence strict` | | `--status` | Compatibility option. Runtime columns are now shown by default | | `--details` | Show download metadata instead of runtime columns | | `--min-params` | Minimum model knowledge capacity in billions of parameters | | `--json` | Print machine-readable JSON | | `--markdown`, `-m` | Print a pasteable GitHub-Flavored Markdown table | | `--refresh` | Ignore caches and fetch models/benchmarks again | | `--cpu-only` | Ignore GPUs and rank for CPU-only use | | `--gpu` | Simulate GPU(s) by name. Accepts repeated flags, comma-separated values, and count shorthand | | `--vram` | Override simulated GPU VRAM or detected GPU usable VRAM in GB | | `--bandwidth`, `--ram-bandwidth` | Override GPU/RAM bandwidth in GB/s | | `--gpu-index` | Detected GPU index to override when multiple GPUs are present | | `--vram-headroom` | Reserve per-GPU memory for runtime overhead. Default: `auto`. Accepts `none`, byte values like `1.5GB`, or percentages like `10%` | | `--ram-budget` | Cap RAM available for partial offload. Accepts `available`, byte values like `8GB`, or percentages like `50%` | | `--lm-studio-path` | Scan an additional LM Studio model library. Repeat the option for legacy or multiple custom locations | | `--version` | Print the installed package version | Environment variables: | Variable | Meaning | | --- | --- | | `HF_ENDPOINT` | Hugging Face endpoint root used for whichllm's own model metadata API calls. Example: `https://huggingface.co` or a compatible mirror root | `--fit any` is the default. It can include full-GPU, partial-offload, and CPU-only candidates when they are runnable. `--fit gpu`, `--fit full-gpu`, and `--gpu-only` keep only rows whose `fit_type` is `full_gpu`. The default table shows memory required, estimated generation speed, fit type, and published date. Use `--details` when you want download counts instead. Speed colors are absolute usability hints: red is under `4 tok/s`, yellow is `4-10 tok/s`, green is `10-30 tok/s`, and bright green is `30+ tok/s`. The `~` and `?` markers still refer to estimate confidence, not speed quality. `--vram-headroom auto` subtracts a small budget from each GPU before fit checks, so near-edge recommendations are less likely to overflow in tools such as LM Studio. Use `--vram-headroom none` to restore the raw detected VRAM. `--ram-budget available` caps offload planning to current available RAM. For detected iGPU or unified-memory systems, use `--vram` and `--bandwidth` / `--ram-bandwidth` to override the automatically detected usable memory and bandwidth. If multiple GPUs are detected, add `--gpu-index` with the GPU number from `whichllm hardware`. whichllm scans `~/.lmstudio/models` for GGUF files. A recommendation is marked `Installed` only when its resolved Hugging Face repository and exact artifact path match a readable local file. Every part of a split GGUF must be present. Libraries are only read; whichllm does not move, modify, or delete LM Studio files. Use `--lm-studio-path` for legacy or custom model libraries; an explicit path that cannot be read is reported as an error. Examples: ```bash whichllm whichllm --gpu "RTX 4090" whichllm --gpu "RTX 5060 Ti" --vram 16 whichllm --vram 8 --ram-bandwidth 68 whichllm --gpu-index 1 --vram 8 --bandwidth 68 whichllm --gpu "2x RTX 4090" whichllm --gpu "RTX 4090" --gpu "RTX 3090" whichllm --gpu "RTX 4090, RTX 3090" whichllm --profile coding --top 5 whichllm --context-length 64k whichllm --gpu-only whichllm --fit gpu whichllm --speed usable whichllm --speed fast whichllm --min-speed 4 whichllm --markdown whichllm --vram-headroom 1.5GB whichllm --ram-budget available whichllm --lm-studio-path /mnt/models/lm-studio whichllm --details whichllm --evidence strict whichllm --json | jq '.models[0]' ``` `--markdown` is mutually exclusive with `--json`. It prints a plain Markdown table without the Rich hardware panel, colors, or box-drawing characters. Ranking JSON model rows include: | Field | Meaning | | --- | --- | | `fit_type` | Runtime fit classification: `full_gpu`, `partial_offload`, or `cpu_only` | | `vram_required_bytes` | Estimated runtime memory requirement for the candidate | | `vram_available_bytes` | GPU memory budget used for the fit check | | `uses_multi_gpu` | Whether the fit check used more than one GPU | | `multi_gpu_effective_vram_bytes` | Conservative effective VRAM budget for multi-GPU fits, when applicable | | `estimated_tok_per_sec` | Point estimate used by ranking | | `speed_confidence` | `high`, `medium`, or `low` | | `speed_range_tok_per_sec` | Estimated lower/upper tok/s range, when available | | `speed_notes` | Short reasons for the confidence level | | `benchmark_status` | Display marker category for benchmark evidence | | `benchmark_source` | How benchmark evidence was matched: `direct`, `variant`, `base_model`, `line_interp`, `self_reported`, or `none` | | `benchmark_confidence` | Confidence in the benchmark match, `0.0`–`1.0` | | `local_match` | Whether a matching GGUF was found in an LM Studio library | | `local_path` | Absolute path to the matching local GGUF, or `null` | The top-level `hardware` object also includes `usable_vram_bytes` per GPU, `ram_budget_bytes`, and `budget_notes` when memory budgets are active. ## `hardware` ```bash whichllm hardware [OPTIONS] ``` Prints detected hardware without ranking models. The same simulation flags are available here: ```bash whichllm hardware whichllm hardware --cpu-only whichllm hardware --gpu "Apple M3 Max" whichllm hardware --gpu "RTX 3060" --vram 12 whichllm hardware --vram 8 --bandwidth 68 whichllm hardware --gpu "4x RTX 4090" ``` ## `plan` ```bash whichllm plan MODEL_NAME [OPTIONS] ``` Searches for a model by HuggingFace repo ID or fuzzy terms, then estimates the VRAM required for several quantization levels and common GPUs. Options: | Option | Meaning | | --- | --- | | `--context-length`, `-c` | Context length for the memory estimate. Accepts integers or `k` shorthand such as `128k`. Default: `4096` | | `--quant`, `-q` | Target quantization. Default: `Q4_K_M` | | `--json` | Print the plan as JSON | | `--refresh` | Ignore model cache and fetch again | Examples: ```bash whichllm plan "llama 3 70b" whichllm plan "Qwen2.5-72B" --quant Q8_0 whichllm plan "mistral 7b" --context-length 32768 whichllm plan "mistral 7b" --context-length 32k ``` ## `upgrade` ```bash whichllm upgrade TARGET_GPUS... [OPTIONS] ``` Compares the current machine against one or more simulated GPUs. The CPU, RAM, disk, and OS come from the current machine; only the GPU changes. Options: | Option | Meaning | | --- | --- | | `--context-length`, `-c` | Context length used for ranking. Accepts integers or `k` shorthand such as `64k`. Default: `8192` | | `--top`, `-n` | Best-N models to compare per GPU. Default: `3` | | `--profile` | Ranking profile. Default: `general` | | `--cpu-only` | Use CPU-only as the current baseline | | `--json` | Print comparison JSON | | `--refresh` | Ignore caches and fetch again | Examples: ```bash whichllm upgrade "RTX 4090" "RTX 5090" "H100" whichllm upgrade "Apple M4 Max" --top 5 whichllm upgrade "RX 7900 XTX" --profile coding whichllm upgrade "RTX 4090" --context-length 128k ``` ## `run` ```bash whichllm run [MODEL_NAME] [OPTIONS] ``` Creates a temporary Python script, launches it through `uv run --no-project`, installs the needed inference packages into that isolated run, and starts an interactive chat. If `MODEL_NAME` is omitted, whichllm ranks models for the current hardware and uses the top result. Options: | Option | Meaning | | --- | --- | | `--context-length`, `-c` | Context length for the generated chat script. Accepts integers or `k` shorthand such as `64k` | | `--quant`, `-q` | Preferred GGUF quantization | | `--refresh` | Ignore model cache and fetch again | | `--cpu-only` | Force CPU-only execution in the generated script | Examples: ```bash whichllm run whichllm run "qwen 2.5 1.5b gguf" whichllm run "phi 3 mini gguf" --cpu-only whichllm run "mistral 7b gguf" --context-length 64k ``` `run` requires `uv` in `PATH`. ## `snippet` ```bash whichllm snippet [MODEL_NAME] [OPTIONS] ``` Prints a ready-to-run Python snippet for the selected model. GGUF models use `llama-cpp-python`; non-GGUF models use `transformers`. Options: | Option | Meaning | | --- | --- | | `--quant`, `-q` | Preferred GGUF quantization | | `--refresh` | Ignore model cache and fetch again | Examples: ```bash whichllm snippet "qwen 7b" whichllm snippet "llama 3 8b gguf" --quant Q5_K_M ``` ## Evidence filters `--evidence` controls which benchmark matches are allowed into the ranking. | Mode | Allows | | --- | --- | | `strict` | Exact independent benchmark matches only | | `base` | Exact, variant, and `cardData.base_model` matches | | `any` | All evidence levels, including line interpolation and self-reported values | `--direct` is kept as a shorter alias for `--evidence strict`. ## Profiles The ranker detects specialization from repository names. | Profile | Behavior | | --- | --- | | `general` | Excludes coding, vision, and math-specialized names | | `coding` | Keeps coding-specialized names | | `vision` | Keeps vision or multimodal names and includes VLM candidates | | `math` | Keeps math-specialized names plus general reasoning models (excludes coding and vision names) | | `any` | Keeps all recognized model types and includes VLM candidates | --- ### Hardware (`docs/hardware.md`) # Hardware detection and simulation whichllm detects the current machine and can also simulate hardware for purchase planning. The source of truth is the `hardware/` package plus curated registry data in `data/gpu.py`. `constants.py` remains as a compatibility export layer for older imports. ## Detected data The ranker receives a `HardwareInfo` object with: - GPU list - CPU name - physical CPU cores - AVX2 and AVX-512 support - total RAM - free disk space - OS name Each GPU is represented as `GPUInfo`: - name - vendor - VRAM bytes - usable VRAM bytes, when a runtime headroom is active - NVIDIA compute capability, when known - CUDA or ROCm version, when known - memory bandwidth estimate - whether the GPU uses shared memory ## NVIDIA NVIDIA detection tries `nvidia-ml-py` first. If NVML is unavailable, fails to initialize, or returns no devices, whichllm falls back to: ```bash nvidia-smi --query-gpu=name,memory.total,clocks.max.memory --format=csv,noheader,nounits ``` If a driver rejects `clocks.max.memory`, whichllm retries the older `name,memory.total` query. For known cards, curated data and strict `dbgpu` lookups provide: - memory bandwidth - compute capability The max memory clock is used when a marketing name covers multiple memory types. For example, GTX 1650 GDDR5 and GDDR6 cards share the same broad driver name, so whichllm uses the reported memory clock when available and falls back to the conservative bandwidth when it is not. DGX Spark / NVIDIA GB10 uses unified system memory. When the driver reports `memory.total` as unavailable, whichllm treats GB10 as shared memory and uses system RAM for fit checks. Compute capability is used to warn when a card is below the minimum expected by common local inference tools. ## AMD On Linux, AMD detection tries `rocm-smi` first: - product name - VRAM - ROCm driver version If `rocm-smi` is unavailable, it falls back to `lspci` and then `/sys/class/drm`. On Windows, whichllm uses `Win32_VideoController` as a fallback for AMD GPUs. When possible, it also reads the 64-bit dedicated-memory value from the `Control\Video` registry path because `AdapterRAM` is a 32-bit field and can cap larger cards around 4 GB. AMD shared-memory APUs are treated differently from discrete GPUs. Names such as Strix Halo, Ryzen AI MAX, Radeon 8050S, Radeon 8060S, Radeon 890M, and Radeon 780M are modeled as shared-memory systems. If the reported VRAM is just a small aperture, whichllm uses the system memory pool for fit checks instead of treating it as a tiny discrete GPU. ## Intel Intel integrated GPUs are detected on Linux through `lspci` or sysfs, and on Windows through `Win32_VideoController`. They do not normally report dedicated VRAM, so whichllm records them with `0` dedicated VRAM and labels them as shared memory. Discrete Intel Arc cards are kept as dedicated-memory GPUs when the device name and memory report look like a discrete adapter. The Intel backend factor is lower than NVIDIA, AMD, and Apple because local LLM GPU inference support is less mature. ## Apple Silicon On macOS, whichllm uses: ```bash system_profiler SPHardwareDataType -json ``` Apple Silicon uses unified memory, so the detected chip memory is treated as available GPU memory. Memory bandwidth is looked up by chip family when known. Partial offload on Apple Silicon is not penalized like discrete PCIe offload. Weights still live in unified memory, so the speed penalty is milder. ## CPU and memory CPU detection reads: - `/proc/cpuinfo` on Linux - `sysctl` on macOS - `wmic` on Windows, then PowerShell / CIM when `wmic` is unavailable or only returns a header Physical core count comes from `psutil`, with a Linux `/proc/cpuinfo` fallback. RAM comes from `psutil.virtual_memory()`. Disk free space is checked under the user's home directory by default. ## GPU simulation Use `--gpu` to simulate a GPU: ```bash whichllm --gpu "RTX 4090" whichllm hardware --gpu "Apple M3 Max" whichllm upgrade "RTX 4090" "RTX 5090" "H100" ``` Simulation uses the `dbgpu` package for a TechPowerUp-backed GPU database. whichllm adds extra handling for common aliases and Apple Silicon chips because those are not covered by dbgpu. Use `--vram` when a GPU name is ambiguous, unknown, or has multiple variants: ```bash whichllm --gpu "RTX 5060 Ti" --vram 16 whichllm hardware --gpu "Unknown GPU" --vram 24 ``` For detected integrated or unified-memory GPUs, use `--vram` and `--bandwidth` / `--ram-bandwidth` to override the automatically detected usable capacity and memory bandwidth: ```bash whichllm --vram 8 --ram-bandwidth 68 whichllm hardware --vram 8 --bandwidth 68 ``` If multiple GPUs are detected, pass `--gpu-index` to choose the GPU shown by `whichllm hardware`: ```bash whichllm --gpu-index 1 --vram 8 --ram-bandwidth 68 ``` By default, whichllm applies a small automatic VRAM headroom before fit checks. This avoids recommending models that only fit on paper but overflow in runtimes that need extra graph buffers or loader overhead. Tune it with: ```bash whichllm --vram-headroom 1.5GB whichllm --vram-headroom 10% whichllm --vram-headroom none ``` Multi-GPU simulation accepts repeated flags, comma-separated values, and count shorthand: ```bash whichllm --gpu "2x RTX 4090" whichllm --gpu "RTX 4090" --gpu "RTX 3090" whichllm --gpu "RTX 4090, RTX 3090" ``` `--vram` is only supported for a single simulated GPU. For multi-GPU simulation, use known GPU names so whichllm can resolve each card's VRAM from the GPU database. ## Fit types Compatibility checks classify a candidate into one of three fit types: | Fit | Meaning | | --- | --- | | `full_gpu` | Required memory fits in available GPU memory | | `partial_offload` | GPU plus usable system RAM can hold the model | | `cpu_only` | Usable system RAM can hold the model without GPU | If neither GPU memory nor usable RAM can hold the model, the candidate is not ranked. whichllm keeps a bounded system-RAM reserve for the OS and other processes. Use `--ram-budget available` to cap partial-offload planning to the current available RAM reported by the OS, or pass a fixed budget such as `--ram-budget 8GB`. ## Multiple GPUs For fit checks, whichllm uses a conservative multi-GPU budget rather than pretending all VRAM is one perfect device. It starts from raw total VRAM, applies a small per-GPU overhead, and then applies a utilization factor. Homogeneous sets receive a less severe reduction than heterogeneous sets. If a dedicated GPU is present, low-aperture shared-memory integrated GPUs are not added to the fit pool. This avoids treating unrelated system RAM and dedicated VRAM as one full-GPU target. For speed estimates, whichllm uses the largest detected GPU as the representative device and marks multi-GPU speed as low-confidence. This avoids claiming ideal scaling when real performance depends on backend split mode, PCIe/NVLink bandwidth, NCCL/RCCL support, batch size, and model architecture. This is a practical fit approximation. It does not model every tensor-parallel or pipeline-parallel runtime configuration. The fit check compares a combined memory estimate with the effective GPU budget; it does not assign weights and KV cache separately to individual GPUs. LM Studio can use a different placement, so its per-GPU memory usage can differ even when the model runs within the estimated speed range. To compare a real run, record the model and quantization, context length, runtime version, GPU layer split, per-GPU memory use, KV-cache placement, and generation speed. See [#104](https://github.com/Andyyyy64/whichllm/issues/104) for the reported multi-GPU case. ## Disk checks The compatibility check also compares estimated model weight size with free disk space. If the model cannot be downloaded, it is marked unrunnable. ## Known limitations - GPU bandwidth is a lookup or database estimate, not a live benchmark. - Speed estimates are planning numbers. The default table and JSON fields such as `speed_confidence` and `speed_range_tok_per_sec` show uncertainty. - Driver, runtime, batch size, prompt length, and thermal limits can change real performance. - Multi-GPU runtime behavior depends on the inference backend and is only approximated. - Apple and shared-memory APU behavior is modeled as unified-memory style, but real results still depend on OS pressure and memory bandwidth. --- ### How It Works (`docs/how-it-works.md`) # How it works whichllm has one main job: start with the user's hardware, collect candidate models, estimate what can run, and rank the results. The implementation is intentionally split into small packages: ```text src/whichllm/ ├── cli.py ├── constants.py ├── data/ ├── hardware/ ├── models/ ├── engine/ └── output/ ``` ## Request flow The default `whichllm` command follows this path: 1. Validate CLI flags. 2. Detect hardware. 3. Load model cache or fetch from HuggingFace. 4. Load benchmark cache or fetch benchmark sources. 5. Group related model repos into families. 6. Flatten families back into rankable candidates. 7. Rank every candidate variant. 8. Backfill missing published dates for top results. 9. Print a Rich table or JSON. The `plan`, `upgrade`, `run`, `snippet`, and `hardware` subcommands reuse parts of the same pipeline. ## Hardware detection `hardware/detector.py` orchestrates detection. Each detector is fail-safe and returns an empty result on failure. | Module | Role | | --- | --- | | `hardware/nvidia.py` | Uses `nvidia-ml-py`; falls back to `nvidia-smi`, including optional memory-clock data | | `hardware/amd.py` | Uses `rocm-smi`; falls back to `lspci` and `/sys/class/drm` | | `hardware/intel.py` | Detects Linux Intel iGPUs through `lspci` or sysfs | | `hardware/windows.py` | Detects Windows AMD and Intel fallback GPUs through WMI and registry memory fields | | `hardware/apple.py` | Uses `system_profiler` on macOS | | `hardware/cpu.py` | Reads CPU name, physical cores, AVX2, and AVX-512 | | `hardware/memory.py` | Reads RAM and disk free space | | `hardware/gpu_simulator.py` | Builds synthetic GPUs for `--gpu` | The result is a `HardwareInfo` dataclass. GPUs are represented by `GPUInfo`. ## Model fetching `models/fetcher.py` reads the HuggingFace API and turns responses into `ModelInfo` objects. The fetcher combines several queries: 1. Popular `text-generation` models sorted by downloads. 2. Popular GGUF repos. 3. Recently modified GGUF repos. 4. Trending text-generation repos, with and without the GGUF filter. 5. A curated list of frontier and hard-to-find model IDs. 6. Vision candidates from `image-text-to-text` when the active profile needs them. For each repository, the parser extracts: - repo ID and display name - parameter count - active parameter count for known or config-detected MoE models - architecture and context length - license, downloads, likes, published date - GGUF variants and file sizes - `cardData.base_model` - conservative HuggingFace `evalResults` values Parameter counts can come from safetensors metadata, GGUF metadata, config estimation, name hints, or a curated fallback table for important models with missing metadata. ## Caches Both caches normally live under `~/.cache/whichllm/`. If `XDG_CACHE_HOME` is set to an absolute path, whichllm uses `$XDG_CACHE_HOME/whichllm/` instead. | File | TTL | Contents | | --- | --- | --- | | `models.json` | 6 hours | Serialized `ModelInfo` data | | `benchmark.json` | 24 hours | Combined benchmark score map | `--refresh` bypasses the relevant cache and writes a new one after fetching. ## Benchmark sources `models/benchmark.py` builds one score map from multiple sources. Scores are normalized to a 0-100 scale before ranking. whichllm separates sources into two tiers: | Tier | Sources | Treatment | | --- | --- | --- | | Current | LiveBench, Artificial Analysis, Aider, Vision | Overrides frozen scores for the same model | | Frozen | Open LLM Leaderboard v2, Chatbot Arena ELO | Kept for older coverage, capped and recency-demoted | Current sources can use live scrapes when reachable and curated snapshots when the upstream page shape changes. The snapshot month is printed below rankings. Within Artificial Analysis, mapped live scores replace stored AA scores even when lower or zero. The stored snapshot supplies models absent from the live response. This does not change how scores from different benchmark sources are combined. Use `--refresh` to bypass the 24-hour benchmark cache. Frozen-only scores are demoted by model lineage. This prevents an older model with a stale leaderboard score from outranking a newer generation simply because the newer model was never added to that frozen leaderboard. ## Benchmark evidence A model can receive benchmark evidence through several paths: | Evidence | Meaning | | --- | --- | | `direct` | Exact independent benchmark match | | `variant` | Suffix-stripped or `-Instruct` variant match | | `base_model` | Match through HuggingFace `cardData.base_model` | | `line_interp` | Size-aware interpolation within the same model line | | `self_reported` | Uploader-provided `evalResults` only | | `none` | No usable evidence | Inheritance is rejected when the actual model size differs too much from the reference. This catches small draft heads, MTP heads, and unrelated forks that would otherwise borrow a larger base model's score. ## Family grouping `models/grouper.py` groups related repos by: 1. `cardData.base_model`, when the relationship is explicitly `quantized`. 2. Normalized repository names. The normalizer removes packaging and quantization suffixes such as `-GGUF`, `-AWQ`, `-GPTQ`, and `-FP16`. It preserves organization namespaces, minor versions, checkpoint dates, and instruction/chat suffixes. Models that reference a base without an explicit quantization relationship keep their own repository identity. Quantizations of those derived models stay with the derived checkpoint rather than its ancestor. `family_id` reflects these checkpoint distinctions and is recomputed during grouping, including for models loaded from an existing cache. Within a family, the ranker evaluates all members and variants but keeps only the best result for the final table. ## Candidate variants For each `ModelInfo`, `engine/ranker.py` builds candidate variants: - Existing GGUF files are evaluated by quantization. - Extreme low-bit GGUF variants are skipped unless explicitly requested with `--quant`. - Official safetensors-only repos can receive synthetic GGUF estimates for common community conversions such as `Q4_K_M`, `Q5_K_M`, `Q6_K`, and `Q8_0`. - Pre-quantized repos such as AWQ, GPTQ, FP8, and BF16 are not given synthetic GGUF variants. Apple Silicon and CPU-only rankings are restricted to GGUF candidates for runtime stability. Linux with NVIDIA can also rank non-GGUF AWQ/GPTQ/FP16/BF16 repos. ## Ranking For each candidate variant: 1. Estimate memory. 2. Check whether it can run. 3. Estimate tok/s and attach speed confidence/range metadata. 4. Resolve benchmark evidence. 5. Compute a quality score. 6. Keep the best variant for the model family. The final sorting key stays close to the displayed quality score, with a small direct-benchmark bonus and a CPU-only penalty. Full-GPU candidates are already favored inside the score through the runtime-fit and speed adjustments, so the sort key does not add a second full-GPU bonus. See [Scoring](scoring.md) for the score details. ## Output Output is split by surface: - `output/ranking.py` renders hardware panels and recommendation tables. - `output/json_output.py` renders ranking, `plan`, and `upgrade` JSON. - `output/plan.py` renders `plan` tables. - `output/upgrade.py` renders upgrade comparison tables. - `output/display.py` re-exports those functions for older imports. Normal ranking tables show memory required, estimated generation speed, fit type, and published date. `--details` switches to download-oriented metadata. Speed color is based on absolute usability, while `~` marks estimates with a range and `?` marks low-confidence, backend-sensitive estimates. --- ### Run Snippet (`docs/run-snippet.md`) # Run and snippet whichllm can do more than print recommendations: - `whichllm run` starts an interactive chat with a selected model. - `whichllm snippet` prints a Python script for manual use. Both commands use the same model loading helpers in `cli.py`. ## `whichllm run` ```bash whichllm run [MODEL_NAME] ``` If `MODEL_NAME` is provided, whichllm searches the fetched model list for an exact ID, suffix match, or term match. If `MODEL_NAME` is omitted, whichllm ranks models for the current hardware and uses the top result. Examples: ```bash whichllm run whichllm run "qwen 2.5 1.5b gguf" whichllm run "phi 3 mini gguf" --cpu-only whichllm run "llama 3 8b gguf" --quant Q5_K_M ``` ## How `run` executes `run` requires `uv` in `PATH`. At runtime, whichllm: 1. Loads models from cache or HuggingFace. 2. Selects a model and quantization. 3. Generates a temporary Python chat script. 4. Runs that script with `uv run --no-project`. 5. Adds the required dependencies with repeated `--with` flags. 6. Deletes the temporary script after the chat exits. This keeps the project environment clean. The temporary runtime dependencies are not added to `pyproject.toml`. ## Supported model paths ### GGUF GGUF models use: - `llama-cpp-python` - `huggingface-hub` The generated script downloads the selected GGUF file with `hf_hub_download` and loads it through `llama_cpp.Llama`. GPU behavior: - default: `n_gpu_layers=-1` - `--cpu-only`: `n_gpu_layers=0` ### AWQ AWQ repos are inferred from the model ID and use: - `transformers` - `torch` - `accelerate` - `autoawq` ### GPTQ GPTQ repos are inferred from the model ID and use: - `transformers` - `torch` - `accelerate` - `auto-gptq` ### FP16, BF16, FP8, INT8, BNB 4-bit Other non-GGUF repos use the Transformers path. The generated script uses `device_map="auto"` unless `--cpu-only` is set. ## Quantization selection For GGUF repos, whichllm chooses a variant by this preference order unless `--quant` is provided: ```text Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q3_K_M, Q3_K_L, Q8_0, ... ``` This order favors a practical balance of memory and quality. Very low-bit variants are available when explicitly requested but are not preferred by default. If the requested quantization is not available, `run` warns and falls back to the best available match. ## Chat behavior GGUF scripts call: ```python llm.create_chat_completion(messages=messages, stream=True) ``` Transformers scripts use: ```python tokenizer.apply_chat_template(...) model.generate(...) ``` The chat loop keeps the current conversation history in memory until the process exits. Type `exit`, `quit`, or `q` to stop. ## `whichllm snippet` ```bash whichllm snippet [MODEL_NAME] ``` `snippet` prints a short Python example instead of running it. Examples: ```bash whichllm snippet "qwen 7b" whichllm snippet "llama 3 8b gguf" --quant Q5_K_M ``` If no model is provided, `snippet` picks the most-downloaded GGUF model from the fetched model list. This is different from `run`, which auto-ranks for the current hardware when no model name is provided. ## Manual execution The snippet output includes a suggested `uv run --no-project` command with the needed `--with` dependencies. Example shape: ```bash uv run --no-project --with llama-cpp-python --with huggingface-hub script.py ``` ## Practical notes - First run can take time because dependencies and model weights need to download. - HuggingFace access rules still apply. Gated models may require local HuggingFace authentication. - `run` is a convenience path, not a full model manager. - `snippet` is better when you want to adapt loading code into your own project. - Generated Transformers scripts use `trust_remote_code=True`, matching common HuggingFace local inference patterns. Review model repos before running code from untrusted sources. --- ### Scoring (`docs/scoring.md`) # Scoring whichllm does not pick the largest model that fits. It ranks candidates by a composite score that tries to answer a more practical question: > Of the models that can run here, which one is likely to be the best usable > choice? The source of truth is `engine/ranker.py`. ## Inputs Each candidate score uses: - model metadata from HuggingFace - detected or simulated hardware - estimated VRAM/RAM fit - estimated tok/s - quantization type - benchmark evidence - downloads and likes - source organization - model lineage and generation The score is capped to `0..100`. ## Benchmark evidence Independent benchmark matches are not all treated equally. | Source | Weight | Meaning | | --- | ---: | --- | | `direct` | `0.62` | Exact independent benchmark match | | `base_model` | `0.55` | Match through `cardData.base_model` | | `variant` | `0.50` | Suffix-stripped variant match | | `line_interp` | `0.40` | Size-aware model-line interpolation | | `self_reported` | `0.30` | Uploader-provided HuggingFace eval only | | `none` | `0.00` | No benchmark evidence | `self_reported` evidence is intentionally weak. HuggingFace model cards can contain useful evaluation data, but it is not the same as an independent leaderboard. ## Size score Model size is used as a rough world-knowledge proxy: ```text size_score = 4.2 * log2(params_b) + 9 ``` The result is capped at `35`. For dense models, `params_b` is the parameter count. For MoE models, whichllm uses total parameters for quality because all experts contribute to stored knowledge. Active parameters are used later for speed. ## Quantization penalty Lower-bit quantization can make a larger model fit, but it also reduces quality. The score core is multiplied by `(1 - quant_penalty)`. Examples: | Quant | Penalty | | --- | ---: | | `Q8_0` | `0.01` | | `Q6_K` | `0.02` | | `Q5_K_M` | `0.03` | | `Q4_K_M` | `0.05` | | `Q3_K_M` | `0.08` | | `Q2_K` | `0.25` | | `IQ2_XXS` | `0.40` | | `Q1_0` | `0.55` | Extreme low-bit variants are excluded by default when better candidates exist. They can still be requested explicitly with `--quant`. ## Evidence confidence After benchmark and size are combined, weak evidence is dampened: | Evidence state | Multiplier | | --- | ---: | | Direct benchmark | `1.00` | | Inherited evidence | `0.78` | | Self-reported evidence | `0.55` | | No benchmark | `0.55` | For inherited benchmark evidence, the raw score is also scaled by confidence before entering the scoring function. Line interpolation therefore receives a double discount: once for its interpolation confidence and once for being inherited evidence. ## Runtime fit The candidate's runtime form matters: | Fit | Multiplier | | --- | ---: | | Full GPU | `1.00` | | Partial offload | `0.42`-`0.88`, based on spill ratio | | CPU-only | `0.50` | Light partial offload is penalized less than heavy offload. MoE models receive a milder penalty when the active parameter working set can plausibly stay on GPU while inactive experts spill to CPU RAM. The final family selection key does not add a separate full-GPU bonus. Runtime fit is already reflected in the quality score through the multiplier above and the speed adjustment below. CPU-only results receive a small extra sort penalty when mixed with GPU-backed candidates. ## Speed adjustment Speed is treated as a usability gate. It is not the main quality signal. Required speed depends on fit: | Fit | Required speed | | --- | ---: | | Full GPU | `8 tok/s` | | Partial offload | `4 tok/s` | | CPU-only | `1.5 tok/s` | Candidates below the required speed receive up to `-8` points. Candidates above it receive up to `+8` points. After ranking, if any candidate is at least `5 tok/s`, whichllm drops candidates below `1.5 tok/s`. This avoids recommending models that technically fit but are not practical to use. The reported speed is a point estimate, not a live benchmark. Ranking also exposes speed confidence: | Confidence | Range factor | Typical cases | | --- | ---: | --- | | `medium` | `0.60x`-`1.60x` | Normal GPU estimates, synthetic GGUF estimates, AMD shared-memory APU MoE estimates | | `low` | `0.35x`-`2.00x` | CPU-only, partial offload, unknown bandwidth, Apple Silicon MoE | | `high` | `0.85x`-`1.20x` | Reserved for future measured-speed data | Speed cells are colored by absolute usability: red is under `4 tok/s`, yellow is `4-10 tok/s`, green is `10-30 tok/s`, and bright green is `30+ tok/s`. `~` marks medium-confidence estimates with a range, and `?` marks low-confidence estimates. JSON exposes the same uncertainty data as `speed_confidence`, `speed_range_tok_per_sec`, and `speed_notes`. ## Source trust The source organization contributes a small adjustment: - official model organizations receive a small bonus - trusted GGUF converters can inherit that trust - known repackagers receive a small penalty The adjustment is intentionally small. It should break ties, not replace benchmark and fit signals. ## Popularity Downloads and likes act as tie-breakers. Their weight is lower when benchmark evidence is strong and higher when evidence is weak. Popularity has no effect for direct benchmark matches. ## Generation lineage Some benchmark sources are frozen. A model released after a frozen leaderboard cannot appear there, while older models can keep strong but stale scores. whichllm uses family-specific lineage maps to avoid that inversion. Newer generations can receive a small bonus; older generations can receive a small penalty. This is applied carefully so direct benchmark evidence still matters. Examples of tracked lineages include: - Qwen - Llama - DeepSeek - Gemma - Phi - Mistral - GLM - Kimi - Granite - OLMo - T5 (incl. Flan-T5, mT5, ByT5, T5Gemma) ## Benchmark markers The table score can include a marker: | Marker | Meaning | | --- | --- | | none | Direct independent benchmark evidence | | `~` | Estimated or inherited benchmark evidence | | `!sr` | Self-reported HuggingFace eval only | | `?` | No benchmark evidence | Top-pick confidence is computed from the score gap, benchmark status, and fit type. Partial-offload and CPU-only top picks are reported with lower confidence than full-GPU direct-benchmark winners. ## Why a smaller model can win A smaller model can outrank a larger one when it has: - stronger current benchmark evidence - a newer generation signal - better quantization quality - full-GPU fit instead of partial offload - higher estimated speed - a more trustworthy source That is intentional. whichllm ranks likely usable quality, not parameter count alone. --- ### Troubleshooting (`docs/troubleshooting.md`) # Troubleshooting This page lists common issues and the first checks to make. ## Leaderboard rate limits Without pyarrow, whichllm reads the archived Open LLM Leaderboard in pages of 100 rows. It retries HTTP 429 responses before stopping. If an earlier page succeeded, it keeps the scores collected so far and writes a warning to stderr with the failed offset and retained score count. A failure on the first page still fails that source; the other benchmark sources can continue. Partial results use the same 24-hour benchmark cache as a complete fetch. The cache does not record which pages were missing, and later cache reads do not repeat the warning. Run `whichllm --refresh` to fetch the benchmark sources again. ## No GPU detected Run: ```bash whichllm hardware ``` If an NVIDIA GPU is missing: - check that the driver is installed - check `nvidia-smi` - check that `nvidia-ml-py` can load NVML whichllm falls back to `nvidia-smi`, but it still needs the NVIDIA driver tools to be working. If an AMD GPU is missing: - on Linux, check `rocm-smi`, `lspci`, and `/sys/class/drm` - on Windows, check that PowerShell can read `Win32_VideoController` - for Ryzen AI / Radeon integrated graphics, check whether `whichllm hardware` shows shared memory instead of a tiny 512 MB or 4 GB adapter If an Intel iGPU is missing: - Linux detection uses `lspci` or `/sys/class/drm` - Windows detection uses `Win32_VideoController` - many Intel iGPUs do not expose dedicated VRAM, so they may be shown as shared memory graphics ## Simulate hardware instead If detection is unavailable or you are planning a purchase, use `--gpu`: ```bash whichllm --gpu "RTX 4090" whichllm hardware --gpu "Apple M3 Max" whichllm --gpu "RTX 5060 Ti" --vram 16 whichllm --gpu "2x RTX 4090" whichllm --gpu "RTX 4090" --gpu "RTX 3090" ``` Use `--vram` when the GPU name has multiple memory variants or is not in the database. For detected iGPU or unified-memory systems, override the usable GPU memory and bandwidth directly: ```bash whichllm hardware --vram 8 --ram-bandwidth 68 whichllm --vram 8 --bandwidth 68 ``` If `whichllm hardware` lists multiple GPUs, add `--gpu-index` with the GPU number from that output. `--vram` only applies to one simulated or detected GPU. For multi-GPU simulation, use known GPU names and omit `--vram`. ## `--cpu-only` conflicts with `--gpu` These flags are mutually exclusive: ```bash whichllm --cpu-only --gpu "RTX 4090" ``` Choose one: ```bash whichllm --cpu-only whichllm --gpu "RTX 4090" ``` ## `--vram` / `--bandwidth` needs a GPU These overrides need either a detected GPU or a simulated GPU: ```bash whichllm --vram 8 --ram-bandwidth 68 whichllm --gpu "RTX 3060" --vram 12 ``` If no GPU is detected, use `--gpu` to simulate one or check the detection steps above. ## No compatible models found Try: ```bash whichllm whichllm --cpu-only whichllm --refresh ``` Common causes: - the selected `--quant` is too restrictive - `--gpu-only` or `--fit full-gpu` filters out partial-offload and CPU-only candidates - `--speed usable`, `--speed fast`, or `--min-speed` filters out slower candidates - `--min-speed` is too high - `--evidence strict` filters out all candidates - the requested context length is too large - available RAM is too low after reserving space for the OS - disk free space is too low for the model weights For very small machines, remove optional filters first: ```bash whichllm --top 20 ``` ## Recommendations use RAM or CPU offload, but I only want VRAM By default, whichllm includes any runnable candidate: full-GPU, partial-offload, and CPU-only. This is useful for finding what can run at all, but it can be too loose when you want only models that fit entirely in GPU VRAM. Use: ```bash whichllm --gpu-only whichllm --fit gpu whichllm --fit full-gpu ``` If no rows are shown, this machine has no ranked candidates that fit fully in GPU memory under the current filters. Remove `--gpu-only`, lower the context length, or try a smaller quantization. ## A model fits, but it is too slow The default ranking table shows estimated generation speed. Slow rows are red, marginal rows are yellow, usable rows are green, and fast rows are bright green. The `~` and `?` markers are confidence markers for the estimate. Filter slow rows with: ```bash whichllm --speed usable # >=10 tok/s whichllm --speed fast # >=30 tok/s whichllm --min-speed 4 # exact floor, if you want a lower threshold ``` For an exact threshold: ```bash whichllm --min-speed 10 ``` ## LM Studio or another runtime says the model barely does not fit whichllm estimates model memory, but real runtimes can need extra room for loader overhead, graph buffers, KV cache choices, and OS pressure. By default, whichllm reserves a small automatic VRAM headroom before fit checks. Tune it with: ```bash whichllm --vram-headroom 1.5GB whichllm --vram-headroom 10% whichllm --vram-headroom none ``` Use `none` when you want the old raw-VRAM behavior. ## RAM offload depends on what else is running Partial offload uses system RAM. If Docker, Elasticsearch, a browser, or another workload is already using a large amount of memory, cap the offload budget: ```bash whichllm --ram-budget available whichllm --ram-budget 8GB whichllm --ram-budget 50% ``` `available` reads the current available RAM from the OS at startup. Fixed values are useful when you know how much memory you want to leave for other processes. ## Results look stale whichllm caches model data for 6 hours and benchmark data for 24 hours. Force a refresh: ```bash whichllm --refresh whichllm plan "qwen 7b" --refresh ``` The caches live under: ```text ~/.cache/whichllm/ ``` If `XDG_CACHE_HOME` is set to an absolute path, the caches live under: ```text $XDG_CACHE_HOME/whichllm/ ``` ## `uvx` fails with `realpath: command not found` Some older macOS versions do not include a `realpath` command. If the `uvx` launcher fails before whichllm starts, with output like: ```text realpath: command not found /Users/.../python: No such file or directory ``` run whichllm through Python's module entry point instead: ```bash uvx --from whichllm python -m whichllm ``` Pass normal whichllm arguments after the module name: ```bash uvx --from whichllm python -m whichllm --gpu "RTX 4090" ``` ## The top pick has `~`, `!sr`, or `?` These markers describe benchmark evidence: | Marker | Meaning | | --- | --- | | `~` | Inherited or interpolated benchmark evidence | | `!sr` | Uploader-reported benchmark only | | `?` | No benchmark evidence | Use stricter evidence when you want only independently matched benchmark data: ```bash whichllm --evidence strict whichllm --direct ``` Use `--evidence base` when base-model matches are acceptable but interpolation and self-reported values are not. ## The largest model did not win That is expected. whichllm scores: - benchmark quality - model size - quantization loss - full GPU vs partial offload vs CPU-only - estimated speed - evidence confidence - source trust - generation lineage A smaller current-generation model with strong direct evidence can beat a larger model that only barely fits or relies on stale benchmark data. ## Estimated speed differs from real speed Speed is an estimate based on: - model weight size - MoE active parameters - GPU memory bandwidth - quantization efficiency - backend factor - partial-offload penalty Real performance depends on the inference runtime, driver, prompt length, batching, thermal limits, and background memory pressure. The default ranking table shows the speed estimate and its confidence marker. Use `--details` only when you want download counts instead. Speed colors and markers: - red: slow generation speed, under 4 tok/s - yellow: marginal generation speed, 4-10 tok/s - green: usable generation speed, 10-30 tok/s - bright green: fast local generation speed, 30+ tok/s - `~`: estimated speed range is available - `?`: low-confidence estimate; runtime/backend differences can be large JSON includes the same information as `speed_confidence`, `speed_range_tok_per_sec`, and `speed_notes`. ## Apple Silicon partial offload looks different Apple Silicon uses unified memory. Partial offload does not cross a discrete PCIe boundary, so whichllm applies a milder speed penalty than it does for discrete GPUs. The same is true for recognized AMD shared-memory APUs such as Strix Halo, Ryzen AI MAX, and Ryzen AI / Radeon 890M-class integrated graphics. DGX Spark / NVIDIA GB10 is handled the same way when NVIDIA reports GPU memory as unavailable. On Windows, `Win32_VideoController.AdapterRAM` can cap around 4 GB. whichllm uses the 64-bit registry memory value when it is available, and treats known shared-memory APUs as unified-memory style devices instead of tiny discrete GPUs. ## `run` says `uv is required` Install `uv` first: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Then retry: ```bash whichllm run ``` ## `run` cannot download a model Possible causes: - the model is gated on HuggingFace - local HuggingFace authentication is missing - the selected GGUF filename no longer exists - network access failed - disk space is too low Try a known public GGUF model first: ```bash whichllm run "qwen 2.5 1.5b gguf" ``` ## Hugging Face API access fails or needs a mirror whichllm uses the Hugging Face API to fetch model metadata. If direct access to `huggingface.co` fails in your network, set `HF_ENDPOINT` to a compatible endpoint root: ```powershell $env:HF_ENDPOINT = "https://huggingface.co" whichllm --refresh ``` ```bash HF_ENDPOINT="https://huggingface.co" whichllm --refresh ``` Do not include `/api` in `HF_ENDPOINT`; whichllm adds that path internally. ## How much disk space does `run` need? Normal ranking commands do not download model weights. They cache Hugging Face model metadata and benchmark metadata under the whichllm cache. `whichllm run` downloads the selected GGUF file through `huggingface_hub`. The required disk space is roughly the selected GGUF file size plus normal Hugging Face cache overhead. By default, Hugging Face stores downloaded files under: ```text ~/.cache/huggingface/hub ``` You can move that cache by setting `HF_HOME` or `HF_HUB_CACHE`. Cleanup is handled by the Hugging Face cache tools: ```bash hf cache scan hf cache delete ``` whichllm does not currently delete model files automatically after a run. ## Ollama names do not match HuggingFace IDs JSON output returns HuggingFace repo IDs: ```bash whichllm --top 1 --json | jq -r '.models[0].model_id' ``` Ollama model names often use a different naming scheme. Map the HuggingFace ID to your local Ollama model name before calling `ollama run`. ## Debugging a specific model Use `plan` to inspect memory requirements: ```bash whichllm plan "Qwen2.5-72B" --quant Q4_K_M whichllm plan "Qwen2.5-72B" --quant Q8_0 --context-length 32768 ``` Use plain output when filing issues: ```bash whichllm --gpu "RTX 4090" --json whichllm --gpu "RTX 4090" --markdown whichllm hardware ``` Include: - OS - GPU name and VRAM - CPU and RAM - command used - expected result - actual result --- --- METRICS --- - Files Extracted: 8 - Estimated Token Budget: ~17103 tokens - Canonical Reference: https://codewiki.google/github.com/Andyyyy64/whichllm