whichllm Documentation

File: README.md Copy Chapter

whichllm

Find the best local LLM that actually runs on your hardware.

Auto-detects your GPU/CPU/RAM and ranks the top models from HuggingFace that fit your system.

日本語版はこちら

Quick start Copy Chapter

Run the recommendation command once, with no project setup.

uvx whichllm@latest

Simulate a GPU before you buy hardware.

uvx whichllm@latest --gpu "RTX 4090"

Install it when you use it often.

uv tool install whichllm
uv tool upgrade whichllm  # update an existing install

Other install paths.

brew install andyyyy64/whichllm/whichllm
pip install whichllm

Want a safer pick? Copy Chapter

By default, whichllm is ambitious. It ranks the best model that looks runnable
on your machine, including partial RAM offload and near-edge VRAM fits when
they seem usable.

If you want a more comfortable LM Studio-style recommendation, start with:

uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1GB

This keeps only models that fit fully in GPU VRAM, filters out slow estimates,
and leaves extra VRAM for runtime overhead.

If LM Studio still says the model is slightly too large, increase the headroom:

uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1.5GB

Common workflows Copy Chapter

After install, run whichllm directly. For one-off runs, replace whichllm
with uvx whichllm@latest.

# Best models for this machine
whichllm

# Pretend you have a specific GPU
whichllm --gpu "RTX 4090"

# Override detected iGPU/unified-memory limits
whichllm --vram 8 --ram-bandwidth 68

# Only show models that fit fully in GPU VRAM
whichllm --gpu-only
whichllm --fit gpu

# Simulate a multi-GPU workstation
whichllm --gpu "2x RTX 4090"

# Hide models that are technically runnable but too slow
whichllm --speed usable
whichllm --speed fast

# Pasteable GitHub / Slack / Discord output
whichllm --markdown

# Compare upgrade candidates
whichllm upgrade "RTX 4090" "RTX 5090" "H100"

# Find the GPU needed for a model
whichllm plan "llama 3 70b"

# Start a chat with a model
whichllm run "qwen 2.5 1.5b gguf"

# Print copy-paste Python
whichllm snippet "qwen 7b"

# Return JSON for scripts
whichllm --top 1 --json

See it Copy Chapter

$ whichllm --gpu "RTX 4090"

#1  Qwen/Qwen3.6-27B     27.8B  Q5_K_M   score 92.8    27 t/s
#2  Qwen/Qwen3-32B       32.0B  Q4_K_M   score 83.0    31 t/s
#3  Qwen/Qwen3-30B-A3B   30.0B  Q5_K_M   score 82.7   102 t/s

The 32B model fits your card fine — whichllm still ranks the 27B #1,
because it scores higher on real benchmarks and is a newer generation.
A size-only "what fits?" tool would hand you the bigger one. That gap is
the whole point of whichllm. (Note #3: a MoE model at 102 t/s — speed is
ranked on active params, quality on total.)

What can I run? Copy Chapter

Real top picks (snapshot 2026-05 — your results track live HuggingFace
data, this is not a static list):

whichllm --gpu "<your card>" simulates any of these before you buy.
By default, rankings include full-GPU, partial-offload, and CPU-only
candidates when they are usable. Use --gpu-only or --fit full-gpu when
you only want models that fit entirely in GPU VRAM.
The default table shows memory, estimated generation speed, fit type, and
published date. Speed is colored by practical usability: under 4 tok/s is red,
4-10 is yellow, 10-30 is green, and 30+ is bright green. ~ / ? still mark
estimate confidence.

Why whichllm? Copy Chapter

Fitting a model into your VRAM is the easy part. The hard part is knowing
which of the models that fit is actually the best — and that is what
whichllm is built to get right.

• Evidence-based ranking, not a size heuristic — The top pick is
  chosen from merged real benchmarks (LiveBench, Artificial Analysis,
  Aider, multimodal/vision, Chatbot Arena ELO, Open LLM Leaderboard) —
  never "the biggest model that happens to fit."
• Recency-aware — Stale leaderboards are demoted along each model's
  lineage, so a 2024 model can't outrank a current-generation one on an
  outdated score. The benchmark snapshot date is printed under every
  ranking, so a stale recommendation is self-evident instead of silently
  trusted.
• Evidence-graded and guarded — Every score is tagged
  direct / variant / base / interpolated / self-reported and
  discounted by confidence. Fabricated uploader claims and cross-family
  inheritance (a small fork borrowing its much larger base's score) are
  actively rejected.
• Architecture-aware estimates — VRAM = weights + GQA KV cache +
  activation + overhead; speed is bandwidth-bound with per-quant
  efficiency, per-backend factors, MoE active-vs-total split, and
  unified-memory vs discrete-PCIe partial-offload modeling.
• One command, scriptable — whichllm prints the answer; add
  --json | jq for pipelines. No TUI, no keybindings to memorize.
• Live data — Models fetched directly from the HuggingFace API, with
  curated frozen fallbacks for offline or rate-limited use.

Features Copy Chapter

• Auto-detect hardware — NVIDIA, AMD, Intel, Apple Silicon, CPU-only
• Smart ranking — Scores models by VRAM fit, speed, and benchmark quality
• One-command chat — whichllm run downloads and starts a chat session instantly
• Code snippets — whichllm snippet prints ready-to-run Python for any model
• Live data — Fetches models directly from HuggingFace (cached for performance)
• Benchmark-aware — Integrates real eval scores with confidence-based dampening
• Task profiles — Filter by general, coding, vision, or math use cases
• GPU simulation — Test with any GPU: whichllm --gpu "RTX 4090"
• Multi-GPU simulation — Repeat --gpu, use commas, or write 2x RTX 4090
• Full-GPU filter — --gpu-only / --fit full-gpu hides offload candidates
• Speed-aware filtering — --speed usable|fast hides slow rows by threshold
• Markdown output — --markdown / -m prints pasteable GFM tables
• Runtime memory budgets — --vram-headroom and --ram-budget avoid edge fits
• LM Studio detection — Marks exact, complete GGUF artifacts already installed
• Hardware planning — Reverse lookup: whichllm plan "llama 3 70b"
• Upgrade planning — Compare your current machine with candidate GPUs
• JSON output — Pipe-friendly: whichllm --json

Run & Snippet Copy Chapter

Try any model with a single command. No manual installs needed — whichllm
creates an isolated environment via uv, installs dependencies, downloads the
model, and starts an interactive chat.

# Chat with a model (auto-picks the best GGUF variant)
whichllm run "qwen 2.5 1.5b gguf"

# Auto-pick the best model for your hardware and chat
whichllm run

# CPU-only mode
whichllm run "phi 3 mini gguf" --cpu-only

Works with all model formats:

• GGUF — via llama-cpp-python (lightweight, fast)
• AWQ / GPTQ — via transformers + autoawq / auto-gptq
• FP16 / BF16 — via transformers

Get a copy-paste Python snippet instead:

whichllm snippet "qwen 7b"

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Qwen/Qwen2.5-7B-Instruct-GGUF",
    filename="qwen2.5-7b-instruct-q4_k_m.gguf",
    n_ctx=4096,
    n_gpu_layers=-1,
    verbose=False,
)

output = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}],
)
print(output["choices"][0]["message"]["content"])

Usage Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

Markdown output is intended for GitHub issues, READMEs, Slack, Discord, and
blog posts:

whichllm --markdown
whichllm -m --top 5 --gpu "RTX 4090"

JSON model rows include fit_type, vram_required_bytes,
vram_available_bytes, uses_multi_gpu, multi_gpu_effective_vram_bytes,
estimated_tok_per_sec, speed_confidence, speed_range_tok_per_sec,
speed_notes, benchmark_source, and benchmark_confidence. The speed range
is a planning range, not a live benchmark.

Integrations Copy Chapter

Ollama Copy Section

Use JSON output to feed scripts that map HuggingFace IDs to your local Ollama
model names:

# Pick the top HuggingFace model ID
whichllm --top 1 --json | jq -r '.models[0].model_id'

# Find the best coding model ID
whichllm --profile coding --top 1 --json | jq -r '.models[0].model_id'

Ollama model names do not always match HuggingFace repo IDs, so a small mapping
step is usually needed before ollama run.

Shell alias Copy Section

Add to your .bashrc / .zshrc:

alias bestllm='whichllm --top 1 --json | jq -r ".models[0].model_id"'
# Usage: ollama run $(bestllm)

Scoring Copy Chapter

Each model gets a 0-100 score. Benchmark quality and size form the core;
evidence confidence and runtime fit then scale it, with speed, source
trust, and popularity as adjustments.

Score markers:

• ~ (yellow) — No direct benchmark; score inherited/interpolated from the model family
• !sr (bright yellow) — Uploader-reported benchmark only, not independently verified
• ? (red) — No benchmark data available

Speed display:

• red — Slow generation speed (<4 tok/s)
• yellow — Marginal generation speed (4-10 tok/s)
• green — Usable generation speed (10-30 tok/s)
• bright green — Fast local generation speed (>=30 tok/s)
• ~ (yellow) — Estimated tok/s range is available
• ? (red) — Low-confidence speed estimate; backend/runtime sensitivity is high

Documentation Copy Chapter

• CLI reference
• How it works
• Scoring
• Hardware detection and simulation
• Run and snippet
• Troubleshooting

How it works Copy Chapter

Data pipeline Copy Section

1. Model fetching — Fetches popular models from HuggingFace API:

   • Text-generation (downloads + recently updated)
   • GGUF-filtered (separate query for coverage)
   • Vision models (image-text-to-text) when --profile vision or any

2. Benchmark sources — Current tier (LiveBench, Artificial Analysis
   Index, Aider) merged live when reachable, plus a curated multimodal /
   vision index; frozen tier (Open LLM Leaderboard v2, Chatbot Arena
   ELO). Tiers have separate caps and lineage-aware recency demotion so
   stale leaderboards stop over-rewarding older generations.

3. Benchmark evidence — Five resolution levels, increasingly discounted:

   • direct — Exact model ID match
   • variant — Suffix-stripped or -Instruct variant
   • base_model — Base model from cardData
   • line_interp — Size-aware interpolation within model family
   • self_reported — Uploader-claimed eval (heavily discounted)

   Inheritance is rejected when a model's params diverge more than 2× from
   its family's dominant member, catching draft / MTP / abliterated forks
   that share a family_id with a much larger base.

4. Cache — normally ~/.cache/whichllm/, or $XDG_CACHE_HOME/whichllm/
   when XDG_CACHE_HOME is set to an absolute path:

   • models.json — 6h TTL
   • benchmark.json — 24h TTL

Ranking engine Copy Section

1. Hardware detection — NVIDIA (nvidia-ml-py), AMD (ROCm/dbgpu), Intel, Apple Silicon (Metal), CPU cores, RAM, disk
2. VRAM estimation — Weights + KV cache + activation + framework overhead (~500MB)
3. Compatibility — Full GPU / Partial Offload / CPU-only; compute capability and OS checks
4. Speed — tok/s from GPU memory bandwidth, quantization, backend, fit type, and MoE active parameters
5. Scoring — Benchmark (with confidence dampening), size, quantization penalty, fit type, speed, popularity, source trust (official vs repackager)
6. Backend filter — Apple Silicon and CPU-only restrict to GGUF for stability; Linux+NVIDIA allows AWQ/GPTQ

Project structure Copy Section

src/whichllm/
├── cli.py              # Typer CLI: main, plan, run, snippet, hardware
├── constants.py        # Backward-compatible exports for registry data
├── data/               # GPU, quantization, framework, and lineage registries
├── hardware/
│   ├── detector.py     # Orchestrates GPU/CPU/RAM detection
│   ├── nvidia.py       # NVIDIA GPU via nvidia-ml-py
│   ├── amd.py          # AMD GPU (Linux)
│   ├── apple.py        # Apple Silicon (Metal)
│   ├── cpu.py          # CPU name, cores, AVX support
│   ├── memory.py       # RAM and disk free
│   ├── gpu_simulator.py # --gpu flag: synthetic GPU from name
│   └── types.py        # GPUInfo, HardwareInfo
├── models/
│   ├── fetcher.py      # HuggingFace API, model parsing, evalResults
│   ├── benchmark.py    # Arena ELO, Leaderboard (parquet/rows API)
│   ├── grouper.py      # Family grouping by base_model and name
│   ├── cache.py        # JSON cache with TTL
│   └── types.py        # ModelInfo, GGUFVariant, ModelFamily
├── engine/
│   ├── vram.py         # VRAM = weights + KV cache + activation + overhead
│   ├── compatibility.py# Fit type, disk check, compute/OS warnings
│   ├── performance.py  # tok/s from bandwidth
│   ├── quantization.py # Bytes per weight, quality penalty, non-GGUF inference
│   ├── ranker.py       # Scoring, evidence filter, profile/match
│   └── types.py        # CompatibilityResult
└── output/
    ├── ranking.py      # Rich hardware and recommendation tables
    ├── json_output.py  # Ranking, plan, and upgrade JSON
    ├── plan.py         # plan command display
    ├── upgrade.py      # upgrade comparison display
    └── display.py      # Compatibility re-export shim

Development Copy Chapter

git clone https://github.com/Andyyyy64/whichllm.git
cd whichllm
uv sync --dev
uv run whichllm
uv run pytest

Contributing Copy Chapter

Contributions are welcome! See CONTRIBUTING.md for guidelines.

Support Copy Chapter

If whichllm helped you find a model or avoid a bad hardware guess,
sponsoring is appreciated. It helps keep the project maintained: hardware
reports, packaging, test fixtures, benchmark updates, and support for more
machines.

whichllm will stay open-source either way. Issues and PRs are always welcome.

Useful? A GitHub star helps other people find it, and I'd genuinely like to
know what it picked for your rig. Drop it in Issues.

Star History Copy Chapter

Requirements Copy Chapter

• Python 3.11+
• NVIDIA GPU detection via nvidia-ml-py (included by default)
• AMD / Apple Silicon detected automatically

License Copy Chapter

MIT

File: docs/cli.md Copy Chapter

CLI reference

This page describes the commands exposed by whichllm. It is based on the
Typer entrypoint in src/whichllm/cli.py.

Main command Copy Chapter

whichllm [OPTIONS]

Detects the current machine, loads model and benchmark data, ranks compatible
models, and prints a table.

Common options:

Environment variables:

--fit any is the default. It can include full-GPU, partial-offload, and
CPU-only candidates when they are runnable. --fit gpu, --fit full-gpu, and
--gpu-only keep only rows whose fit_type is full_gpu.

The default table shows memory required, estimated generation speed, fit type,
and published date. Use --details when you want download counts instead.
Speed colors are absolute usability hints: red is under 4 tok/s, yellow is
4-10 tok/s, green is 10-30 tok/s, and bright green is 30+ tok/s. The ~
and ? markers still refer to estimate confidence, not speed quality.

--vram-headroom auto subtracts a small budget from each GPU before fit
checks, so near-edge recommendations are less likely to overflow in tools such
as LM Studio. Use --vram-headroom none to restore the raw detected VRAM.
--ram-budget available caps offload planning to current available RAM.
For detected iGPU or unified-memory systems, use --vram and
--bandwidth / --ram-bandwidth to override the automatically detected
usable memory and bandwidth. If multiple GPUs are detected, add --gpu-index
with the GPU number from whichllm hardware.

whichllm scans ~/.lmstudio/models for GGUF files. A recommendation is marked
Installed only when its resolved Hugging Face repository and exact artifact
path match a readable local file. Every part of a split GGUF must be present.
Libraries are only read; whichllm does not move, modify, or delete LM Studio
files. Use --lm-studio-path for legacy or custom model libraries; an explicit
path that cannot be read is reported as an error.

Examples:

whichllm
whichllm --gpu "RTX 4090"
whichllm --gpu "RTX 5060 Ti" --vram 16
whichllm --vram 8 --ram-bandwidth 68
whichllm --gpu-index 1 --vram 8 --bandwidth 68
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
whichllm --gpu "RTX 4090, RTX 3090"
whichllm --profile coding --top 5
whichllm --context-length 64k
whichllm --gpu-only
whichllm --fit gpu
whichllm --speed usable
whichllm --speed fast
whichllm --min-speed 4
whichllm --markdown
whichllm --vram-headroom 1.5GB
whichllm --ram-budget available
whichllm --lm-studio-path /mnt/models/lm-studio
whichllm --details
whichllm --evidence strict
whichllm --json | jq '.models[0]'

--markdown is mutually exclusive with --json. It prints a plain Markdown
table without the Rich hardware panel, colors, or box-drawing characters.

Ranking JSON model rows include:

The top-level hardware object also includes usable_vram_bytes per GPU,
ram_budget_bytes, and budget_notes when memory budgets are active.

`hardware` Copy Chapter

whichllm hardware [OPTIONS]

Prints detected hardware without ranking models. The same simulation flags are
available here:

whichllm hardware
whichllm hardware --cpu-only
whichllm hardware --gpu "Apple M3 Max"
whichllm hardware --gpu "RTX 3060" --vram 12
whichllm hardware --vram 8 --bandwidth 68
whichllm hardware --gpu "4x RTX 4090"

`plan` Copy Chapter

whichllm plan MODEL_NAME [OPTIONS]

Searches for a model by HuggingFace repo ID or fuzzy terms, then estimates the
VRAM required for several quantization levels and common GPUs.

Options:

Examples:

whichllm plan "llama 3 70b"
whichllm plan "Qwen2.5-72B" --quant Q8_0
whichllm plan "mistral 7b" --context-length 32768
whichllm plan "mistral 7b" --context-length 32k

`upgrade` Copy Chapter

whichllm upgrade TARGET_GPUS... [OPTIONS]

Compares the current machine against one or more simulated GPUs. The CPU, RAM,
disk, and OS come from the current machine; only the GPU changes.

Options:

Examples:

whichllm upgrade "RTX 4090" "RTX 5090" "H100"
whichllm upgrade "Apple M4 Max" --top 5
whichllm upgrade "RX 7900 XTX" --profile coding
whichllm upgrade "RTX 4090" --context-length 128k

`run` Copy Chapter

whichllm run [MODEL_NAME] [OPTIONS]

Creates a temporary Python script, launches it through uv run --no-project,
installs the needed inference packages into that isolated run, and starts an
interactive chat.

If MODEL_NAME is omitted, whichllm ranks models for the current hardware and
uses the top result.

Options:

Examples:

whichllm run
whichllm run "qwen 2.5 1.5b gguf"
whichllm run "phi 3 mini gguf" --cpu-only
whichllm run "mistral 7b gguf" --context-length 64k

run requires uv in PATH.

`snippet` Copy Chapter

whichllm snippet [MODEL_NAME] [OPTIONS]

Prints a ready-to-run Python snippet for the selected model. GGUF models use
llama-cpp-python; non-GGUF models use transformers.

Options:

Examples:

whichllm snippet "qwen 7b"
whichllm snippet "llama 3 8b gguf" --quant Q5_K_M

Evidence filters Copy Chapter

--evidence controls which benchmark matches are allowed into the ranking.

--direct is kept as a shorter alias for --evidence strict.

Profiles Copy Chapter

The ranker detects specialization from repository names.

File: docs/hardware.md Copy Chapter

Hardware detection and simulation

whichllm detects the current machine and can also simulate hardware for
purchase planning.

The source of truth is the hardware/ package plus curated registry data in
data/gpu.py. constants.py remains as a compatibility export layer for older
imports.

Detected data Copy Chapter

The ranker receives a HardwareInfo object with:

• GPU list
• CPU name
• physical CPU cores
• AVX2 and AVX-512 support
• total RAM
• free disk space
• OS name

Each GPU is represented as GPUInfo:

• name
• vendor
• VRAM bytes
• usable VRAM bytes, when a runtime headroom is active
• NVIDIA compute capability, when known
• CUDA or ROCm version, when known
• memory bandwidth estimate
• whether the GPU uses shared memory

NVIDIA Copy Chapter

NVIDIA detection tries nvidia-ml-py first. If NVML is unavailable, fails to
initialize, or returns no devices, whichllm falls back to:

nvidia-smi --query-gpu=name,memory.total,clocks.max.memory --format=csv,noheader,nounits

If a driver rejects clocks.max.memory, whichllm retries the older
name,memory.total query.

For known cards, curated data and strict dbgpu lookups provide:

• memory bandwidth
• compute capability

The max memory clock is used when a marketing name covers multiple memory
types. For example, GTX 1650 GDDR5 and GDDR6 cards share the same broad driver
name, so whichllm uses the reported memory clock when available and falls back
to the conservative bandwidth when it is not.

DGX Spark / NVIDIA GB10 uses unified system memory. When the driver reports
memory.total as unavailable, whichllm treats GB10 as shared memory and uses
system RAM for fit checks.

Compute capability is used to warn when a card is below the minimum expected by
common local inference tools.

AMD Copy Chapter

On Linux, AMD detection tries rocm-smi first:

• product name
• VRAM
• ROCm driver version

If rocm-smi is unavailable, it falls back to lspci and then
/sys/class/drm.

On Windows, whichllm uses Win32_VideoController as a fallback for AMD GPUs.
When possible, it also reads the 64-bit dedicated-memory value from the
Control\Video registry path because AdapterRAM is a 32-bit field and can
cap larger cards around 4 GB.

AMD shared-memory APUs are treated differently from discrete GPUs. Names such
as Strix Halo, Ryzen AI MAX, Radeon 8050S, Radeon 8060S, Radeon 890M, and
Radeon 780M are modeled as shared-memory systems. If the reported VRAM is just
a small aperture, whichllm uses the system memory pool for fit checks instead
of treating it as a tiny discrete GPU.

Intel Copy Chapter

Intel integrated GPUs are detected on Linux through lspci or sysfs, and on
Windows through Win32_VideoController. They do not normally report dedicated
VRAM, so whichllm records them with 0 dedicated VRAM and labels them as
shared memory.

Discrete Intel Arc cards are kept as dedicated-memory GPUs when the device name
and memory report look like a discrete adapter.

The Intel backend factor is lower than NVIDIA, AMD, and Apple because local LLM
GPU inference support is less mature.

Apple Silicon Copy Chapter

On macOS, whichllm uses:

system_profiler SPHardwareDataType -json

Apple Silicon uses unified memory, so the detected chip memory is treated as
available GPU memory. Memory bandwidth is looked up by chip family when known.

Partial offload on Apple Silicon is not penalized like discrete PCIe offload.
Weights still live in unified memory, so the speed penalty is milder.

CPU and memory Copy Chapter

CPU detection reads:

• /proc/cpuinfo on Linux
• sysctl on macOS
• wmic on Windows, then PowerShell / CIM when wmic is unavailable or only
  returns a header

Physical core count comes from psutil, with a Linux /proc/cpuinfo fallback.

RAM comes from psutil.virtual_memory(). Disk free space is checked under the
user's home directory by default.

GPU simulation Copy Chapter

Use --gpu to simulate a GPU:

whichllm --gpu "RTX 4090"
whichllm hardware --gpu "Apple M3 Max"
whichllm upgrade "RTX 4090" "RTX 5090" "H100"

Simulation uses the dbgpu package for a TechPowerUp-backed GPU database.
whichllm adds extra handling for common aliases and Apple Silicon chips because
those are not covered by dbgpu.

Use --vram when a GPU name is ambiguous, unknown, or has multiple variants:

whichllm --gpu "RTX 5060 Ti" --vram 16
whichllm hardware --gpu "Unknown GPU" --vram 24

For detected integrated or unified-memory GPUs, use --vram and
--bandwidth / --ram-bandwidth to override the automatically detected usable
capacity and memory bandwidth:

whichllm --vram 8 --ram-bandwidth 68
whichllm hardware --vram 8 --bandwidth 68

If multiple GPUs are detected, pass --gpu-index to choose the GPU shown by
whichllm hardware:

whichllm --gpu-index 1 --vram 8 --ram-bandwidth 68

By default, whichllm applies a small automatic VRAM headroom before fit checks.
This avoids recommending models that only fit on paper but overflow in runtimes
that need extra graph buffers or loader overhead. Tune it with:

whichllm --vram-headroom 1.5GB
whichllm --vram-headroom 10%
whichllm --vram-headroom none

Multi-GPU simulation accepts repeated flags, comma-separated values, and count
shorthand:

whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
whichllm --gpu "RTX 4090, RTX 3090"

--vram is only supported for a single simulated GPU. For multi-GPU
simulation, use known GPU names so whichllm can resolve each card's VRAM from
the GPU database.

Fit types Copy Chapter

Compatibility checks classify a candidate into one of three fit types:

If neither GPU memory nor usable RAM can hold the model, the candidate is not
ranked.

whichllm keeps a bounded system-RAM reserve for the OS and other processes.
Use --ram-budget available to cap partial-offload planning to the current
available RAM reported by the OS, or pass a fixed budget such as
--ram-budget 8GB.

Multiple GPUs Copy Chapter

For fit checks, whichllm uses a conservative multi-GPU budget rather than
pretending all VRAM is one perfect device. It starts from raw total VRAM, applies
a small per-GPU overhead, and then applies a utilization factor. Homogeneous
sets receive a less severe reduction than heterogeneous sets.

If a dedicated GPU is present, low-aperture shared-memory integrated GPUs are
not added to the fit pool. This avoids treating unrelated system RAM and
dedicated VRAM as one full-GPU target.

For speed estimates, whichllm uses the largest detected GPU as the
representative device and marks multi-GPU speed as low-confidence. This avoids
claiming ideal scaling when real performance depends on backend split mode,
PCIe/NVLink bandwidth, NCCL/RCCL support, batch size, and model architecture.

This is a practical fit approximation. It does not model every tensor-parallel
or pipeline-parallel runtime configuration.

The fit check compares a combined memory estimate with the effective GPU budget;
it does not assign weights and KV cache separately to individual GPUs. LM Studio
can use a different placement, so its per-GPU memory usage can differ even when
the model runs within the estimated speed range. To compare a real run, record
the model and quantization, context length, runtime version, GPU layer split,
per-GPU memory use, KV-cache placement, and generation speed. See #104
for the reported multi-GPU case.

Disk checks Copy Chapter

The compatibility check also compares estimated model weight size with free
disk space. If the model cannot be downloaded, it is marked unrunnable.

Known limitations Copy Chapter

• GPU bandwidth is a lookup or database estimate, not a live benchmark.
• Speed estimates are planning numbers. The default table and JSON fields such
  as speed_confidence and speed_range_tok_per_sec show uncertainty.
• Driver, runtime, batch size, prompt length, and thermal limits can change real
  performance.
• Multi-GPU runtime behavior depends on the inference backend and is only
  approximated.
• Apple and shared-memory APU behavior is modeled as unified-memory style, but
  real results still depend on OS pressure and memory bandwidth.

File: docs/how-it-works.md Copy Chapter

How it works

whichllm has one main job: start with the user's hardware, collect candidate
models, estimate what can run, and rank the results.

The implementation is intentionally split into small packages:

src/whichllm/
├── cli.py
├── constants.py
├── data/
├── hardware/
├── models/
├── engine/
└── output/

Request flow Copy Chapter

The default whichllm command follows this path:

1. Validate CLI flags.
2. Detect hardware.
3. Load model cache or fetch from HuggingFace.
4. Load benchmark cache or fetch benchmark sources.
5. Group related model repos into families.
6. Flatten families back into rankable candidates.
7. Rank every candidate variant.
8. Backfill missing published dates for top results.
9. Print a Rich table or JSON.

The plan, upgrade, run, snippet, and hardware subcommands reuse parts
of the same pipeline.

Hardware detection Copy Chapter

hardware/detector.py orchestrates detection. Each detector is fail-safe and
returns an empty result on failure.

The result is a HardwareInfo dataclass. GPUs are represented by GPUInfo.

Model fetching Copy Chapter

models/fetcher.py reads the HuggingFace API and turns responses into
ModelInfo objects.

The fetcher combines several queries:

1. Popular text-generation models sorted by downloads.
2. Popular GGUF repos.
3. Recently modified GGUF repos.
4. Trending text-generation repos, with and without the GGUF filter.
5. A curated list of frontier and hard-to-find model IDs.
6. Vision candidates from image-text-to-text when the active profile needs
   them.

For each repository, the parser extracts:

• repo ID and display name
• parameter count
• active parameter count for known or config-detected MoE models
• architecture and context length
• license, downloads, likes, published date
• GGUF variants and file sizes
• cardData.base_model
• conservative HuggingFace evalResults values

Parameter counts can come from safetensors metadata, GGUF metadata, config
estimation, name hints, or a curated fallback table for important models with
missing metadata.

Caches Copy Chapter

Both caches normally live under ~/.cache/whichllm/. If XDG_CACHE_HOME is
set to an absolute path, whichllm uses $XDG_CACHE_HOME/whichllm/ instead.

--refresh bypasses the relevant cache and writes a new one after fetching.

Benchmark sources Copy Chapter

models/benchmark.py builds one score map from multiple sources. Scores are
normalized to a 0-100 scale before ranking.

whichllm separates sources into two tiers:

Current sources can use live scrapes when reachable and curated snapshots when
the upstream page shape changes. The snapshot month is printed below rankings.

Within Artificial Analysis, mapped live scores replace stored AA scores even
when lower or zero. The stored snapshot supplies models absent from the live
response. This does not change how scores from different benchmark sources
are combined. Use --refresh to bypass the 24-hour benchmark cache.

Frozen-only scores are demoted by model lineage. This prevents an older model
with a stale leaderboard score from outranking a newer generation simply because
the newer model was never added to that frozen leaderboard.

Benchmark evidence Copy Chapter

A model can receive benchmark evidence through several paths:

Inheritance is rejected when the actual model size differs too much from the
reference. This catches small draft heads, MTP heads, and unrelated forks that
would otherwise borrow a larger base model's score.

Family grouping Copy Chapter

models/grouper.py groups related repos by:

1. cardData.base_model, when the relationship is explicitly quantized.
2. Normalized repository names.

The normalizer removes packaging and quantization suffixes such as -GGUF,
-AWQ, -GPTQ, and -FP16. It preserves organization namespaces, minor versions, checkpoint dates,
and instruction/chat suffixes. Models that reference a base without an explicit
quantization relationship keep their own repository identity. Quantizations of
those derived models stay with the derived checkpoint rather than its ancestor.

family_id reflects these checkpoint distinctions and is recomputed during
grouping, including for models loaded from an existing cache.

Within a family, the ranker evaluates all members and variants but keeps only
the best result for the final table.

Candidate variants Copy Chapter

For each ModelInfo, engine/ranker.py builds candidate variants:

• Existing GGUF files are evaluated by quantization.
• Extreme low-bit GGUF variants are skipped unless explicitly requested with
  --quant.
• Official safetensors-only repos can receive synthetic GGUF estimates for
  common community conversions such as Q4_K_M, Q5_K_M, Q6_K, and Q8_0.
• Pre-quantized repos such as AWQ, GPTQ, FP8, and BF16 are not given synthetic
  GGUF variants.

Apple Silicon and CPU-only rankings are restricted to GGUF candidates for
runtime stability. Linux with NVIDIA can also rank non-GGUF AWQ/GPTQ/FP16/BF16
repos.

Ranking Copy Chapter

For each candidate variant:

1. Estimate memory.
2. Check whether it can run.
3. Estimate tok/s and attach speed confidence/range metadata.
4. Resolve benchmark evidence.
5. Compute a quality score.
6. Keep the best variant for the model family.

The final sorting key stays close to the displayed quality score, with a small
direct-benchmark bonus and a CPU-only penalty. Full-GPU candidates are already
favored inside the score through the runtime-fit and speed adjustments, so the
sort key does not add a second full-GPU bonus.

See Scoring for the score details.

Output Copy Chapter

Output is split by surface:

• output/ranking.py renders hardware panels and recommendation tables.
• output/json_output.py renders ranking, plan, and upgrade JSON.
• output/plan.py renders plan tables.
• output/upgrade.py renders upgrade comparison tables.
• output/display.py re-exports those functions for older imports.

Normal ranking tables show memory required, estimated generation speed, fit
type, and published date. --details switches to download-oriented metadata.
Speed color is based on absolute usability, while ~ marks estimates with a
range and ? marks low-confidence, backend-sensitive estimates.

File: docs/README.ja.md Copy Chapter

whichllm

手元のハードウェアで実際に動くローカルLLMを探すCLIです。

whichllm は GPU / CPU / RAM / ディスクを検出し、HuggingFace 上のモデルを
取得して、実行できる候補をランキングします。単に「VRAMに入る最大モデル」を
選ぶのではなく、ベンチマーク、量子化、速度、実行形態、モデル世代をまとめて
評価します。

English version

インストール Copy Chapter

uv Copy Section

一度だけ試す場合:

uvx whichllm@latest

継続して使う場合:

uv tool install whichllm
uv tool upgrade whichllm  # 既存インストールを更新

Homebrew Copy Section

brew install andyyyy64/whichllm/whichllm

pip Copy Section

pip install whichllm

余裕を持って動く候補だけ見たい場合 Copy Chapter

whichllm のデフォルトは少し攻めた推薦です。RAMへのpartial offloadや、
VRAMぎりぎりの候補も、動きそうならランキングに入れます。

LM Studioなどで余裕を持って動かしたい場合は、まずこれを使ってください。

uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1GB

GPUのVRAMに全部載る候補だけに絞り、遅い推定速度の候補を外し、実行時の
余白も1GB残します。

それでもLM Studio側で少しはみ出す場合は、余白を増やします。

uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1.5GB

開発用 Copy Section

git clone https://github.com/Andyyyy64/whichllm.git
cd whichllm
uv sync --dev
uv run whichllm
uv run pytest

まず使う Copy Chapter

# 自動検出しておすすめモデルを表示
whichllm

# GPUをシミュレートする
whichllm --gpu "RTX 4090"
whichllm --gpu "Apple M3 Max"

# 複数GPUをシミュレートする
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"

# GPUのVRAMに全部載る候補だけを見る
whichllm --gpu-only
whichllm --fit gpu

# 速度の最低ラインを指定する
whichllm --speed usable
whichllm --speed fast
whichllm --min-speed 4

# GitHubやSlackに貼りやすいMarkdown表で出力する
whichllm --markdown

# 実行時のメモリ余白やRAM使用量を指定する
whichllm --vram-headroom 1.5GB
whichllm --ram-budget available

# CPUのみとして評価する
whichllm --cpu-only

# JSONで出力する
whichllm --json

JSONの各モデルには estimated_tok_per_sec に加えて、fit_type、
vram_required_bytes、vram_available_bytes、uses_multi_gpu、
multi_gpu_effective_vram_bytes、speed_confidence、
speed_range_tok_per_sec、speed_notes、benchmark_source、
benchmark_confidence が入ります。
速度は実測値ではなく、ハードウェア情報とモデル情報からの推定です。
通常の表には必要メモリ、推定生成速度、Fit種別、Published が表示されます。
Downloads まで見たい場合は --details を使います。
GitHub issue、README、Slack、Discord へ貼る場合は --markdown / -m
でMarkdown表として出力できます。

主なコマンド Copy Chapter

# 推薦ランキング
whichllm --top 20
whichllm --quant Q4_K_M
whichllm --min-speed 30
whichllm --speed usable
whichllm --speed fast
whichllm --markdown
whichllm --profile coding
whichllm --context-length 64k
whichllm --details
whichllm --gpu-only

# ベンチ根拠の厳しさ
whichllm --evidence strict
whichllm --evidence base
whichllm --direct

# モデルから必要GPUを逆算
whichllm plan "llama 3 70b"
whichllm plan "Qwen2.5-72B" --quant Q8_0
whichllm plan "mistral 7b" --context-length 32768

# 今のマシンと購入候補GPUを比較
whichllm upgrade "RTX 4090" "RTX 5090" "H100"

# モデルをダウンロードしてチャット
whichllm run "qwen 2.5 1.5b gguf"
whichllm run

# 実行用Pythonコードを表示
whichllm snippet "qwen 7b"

# ハードウェア情報だけ表示
whichllm hardware

スコアの見方 Copy Chapter

各モデルには 0 から 100 のスコアが付きます。中心になるのはベンチマークと
モデルサイズですが、実行時に遅すぎる候補や、CPUオフロードが大きい候補は
下がります。

スコア横のマーカー:

• ~: 直接ベンチではなく、系列や派生から推定したスコア
• !sr: アップローダー自己申告の評価値だけに基づくスコア
• ?: 利用できるベンチマーク根拠がないスコア

速度表示:

• 赤: 4 tok/s 未満の遅い生成速度
• 黄: 4-10 tok/s のぎりぎり使える生成速度
• 緑: 10-30 tok/s の実用的な生成速度
• 明るい緑: 30 tok/s 以上の高速なローカル生成速度
• ~: 速度推定の幅がある通常の推定値
• ?: backend や runtime の影響が大きい低信頼の推定値

仕組み Copy Chapter

1. ハードウェアを検出します。NVIDIA、AMD、Intel、Apple Silicon、CPU、RAM、
   ディスク空き容量を見ます。
2. HuggingFace APIからモデルを取得します。人気モデル、GGUF、最近更新された
   GGUF、trending、重要な frontier モデルを組み合わせます。
3. ベンチマークを読み込みます。現在系の LiveBench / Artificial Analysis /
   Aider / Vision と、凍結系の Arena / Open LLM Leaderboard を分けて扱います。
4. base_model とモデル名からファミリーを作り、同じモデルの派生やGGUFを束ねます。
5. 候補ごとに VRAM、互換性、速度、速度推定の信頼度、スコアを計算します。
6. ファミリーごとに最も良い候補を残して表示します。

通常は full GPU、partial offload、CPU-only の候補をまとめて見ます。GPUの
VRAMに全部載るモデルだけを見たい場合は --gpu-only か
--fit gpu を使います。遅い候補を最初から除外したい場合は
--speed usable、--speed fast を使います。
それぞれ 10 tok/s、30 tok/s が最低ラインです。
もっと低いラインを指定したい場合は --min-speed 4 のように数値で指定します。

キャッシュは通常 ~/.cache/whichllm/ に保存されます。XDG_CACHE_HOME が
絶対パスで設定されている場合は、その配下の whichllm/ を使います。

• models.json: 6時間
• benchmark.json: 24時間

プロジェクト構成 Copy Chapter

src/whichllm/
├── cli.py              # Typer CLI: main, plan, upgrade, run, snippet, hardware
├── constants.py        # 互換用のregistry再export
├── data/               # GPU、量子化、framework、lineageのregistry
├── hardware/           # ハードウェア検出とGPUシミュレーション
├── models/             # HuggingFace取得、ベンチ、キャッシュ、グルーピング
├── engine/             # VRAM、互換性、速度、ランキング
└── output/             # Rich表示、JSON、plan/upgrade表示

詳細ドキュメント Copy Chapter

• CLIリファレンス
• 仕組み
• スコアリング
• ハードウェア検出とシミュレーション
• run と snippet
• トラブルシュート

動作環境 Copy Chapter

• Python 3.11+
• NVIDIA GPU検出は nvidia-ml-py と nvidia-smi fallback
• AMD GPU検出は Linux / ROCm / sysfs / lspci と Windows fallback
• Intel GPU検出は Linux / sysfs / lspci と Windows fallback
• Strix Halo、Ryzen AI MAX、Radeon 890M 系は shared memory APU として扱う
• Apple Silicon検出は macOS / system_profiler

ライセンス Copy Chapter

MIT

File: docs/run-snippet.md Copy Chapter

Run and snippet

whichllm can do more than print recommendations:

• whichllm run starts an interactive chat with a selected model.
• whichllm snippet prints a Python script for manual use.

Both commands use the same model loading helpers in cli.py.

`whichllm run` Copy Chapter

whichllm run [MODEL_NAME]

If MODEL_NAME is provided, whichllm searches the fetched model list for an
exact ID, suffix match, or term match.

If MODEL_NAME is omitted, whichllm ranks models for the current hardware and
uses the top result.

Examples:

whichllm run
whichllm run "qwen 2.5 1.5b gguf"
whichllm run "phi 3 mini gguf" --cpu-only
whichllm run "llama 3 8b gguf" --quant Q5_K_M

How `run` executes Copy Chapter

run requires uv in PATH.

At runtime, whichllm:

1. Loads models from cache or HuggingFace.
2. Selects a model and quantization.
3. Generates a temporary Python chat script.
4. Runs that script with uv run --no-project.
5. Adds the required dependencies with repeated --with flags.
6. Deletes the temporary script after the chat exits.

This keeps the project environment clean. The temporary runtime dependencies
are not added to pyproject.toml.

Supported model paths Copy Chapter

GGUF Copy Section

GGUF models use:

• llama-cpp-python
• huggingface-hub

The generated script downloads the selected GGUF file with hf_hub_download
and loads it through llama_cpp.Llama.

GPU behavior:

• default: n_gpu_layers=-1
• --cpu-only: n_gpu_layers=0

AWQ Copy Section

AWQ repos are inferred from the model ID and use:

• transformers
• torch
• accelerate
• autoawq

GPTQ Copy Section

GPTQ repos are inferred from the model ID and use:

• transformers
• torch
• accelerate
• auto-gptq

FP16, BF16, FP8, INT8, BNB 4-bit Copy Section

Other non-GGUF repos use the Transformers path. The generated script uses
device_map="auto" unless --cpu-only is set.

Quantization selection Copy Chapter

For GGUF repos, whichllm chooses a variant by this preference order unless
--quant is provided:

Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q3_K_M, Q3_K_L, Q8_0, ...

This order favors a practical balance of memory and quality. Very low-bit
variants are available when explicitly requested but are not preferred by
default.

If the requested quantization is not available, run warns and falls back to
the best available match.

Chat behavior Copy Chapter

GGUF scripts call:

llm.create_chat_completion(messages=messages, stream=True)

Transformers scripts use:

tokenizer.apply_chat_template(...)
model.generate(...)

The chat loop keeps the current conversation history in memory until the
process exits. Type exit, quit, or q to stop.

`whichllm snippet` Copy Chapter

whichllm snippet [MODEL_NAME]

snippet prints a short Python example instead of running it.

Examples:

whichllm snippet "qwen 7b"
whichllm snippet "llama 3 8b gguf" --quant Q5_K_M

If no model is provided, snippet picks the most-downloaded GGUF model from
the fetched model list. This is different from run, which auto-ranks for the
current hardware when no model name is provided.

Manual execution Copy Chapter

The snippet output includes a suggested uv run --no-project command with the
needed --with dependencies.

Example shape:

uv run --no-project --with llama-cpp-python --with huggingface-hub script.py

Practical notes Copy Chapter

• First run can take time because dependencies and model weights need to
  download.
• HuggingFace access rules still apply. Gated models may require local
  HuggingFace authentication.
• run is a convenience path, not a full model manager.
• snippet is better when you want to adapt loading code into your own project.
• Generated Transformers scripts use trust_remote_code=True, matching common
  HuggingFace local inference patterns. Review model repos before running code
  from untrusted sources.

File: docs/scoring.md Copy Chapter

Scoring

whichllm does not pick the largest model that fits. It ranks candidates by a
composite score that tries to answer a more practical question:

Of the models that can run here, which one is likely to be the best usable
choice?

The source of truth is engine/ranker.py.

Inputs Copy Chapter

Each candidate score uses:

• model metadata from HuggingFace
• detected or simulated hardware
• estimated VRAM/RAM fit
• estimated tok/s
• quantization type
• benchmark evidence
• downloads and likes
• source organization
• model lineage and generation

The score is capped to 0..100.

Benchmark evidence Copy Chapter

Independent benchmark matches are not all treated equally.

self_reported evidence is intentionally weak. HuggingFace model cards can
contain useful evaluation data, but it is not the same as an independent
leaderboard.

Size score Copy Chapter

Model size is used as a rough world-knowledge proxy:

size_score = 4.2 * log2(params_b) + 9

The result is capped at 35.

For dense models, params_b is the parameter count. For MoE models, whichllm
uses total parameters for quality because all experts contribute to stored
knowledge. Active parameters are used later for speed.

Quantization penalty Copy Chapter

Lower-bit quantization can make a larger model fit, but it also reduces quality.
The score core is multiplied by (1 - quant_penalty).

Examples:

Extreme low-bit variants are excluded by default when better candidates exist.
They can still be requested explicitly with --quant.

Evidence confidence Copy Chapter

After benchmark and size are combined, weak evidence is dampened:

For inherited benchmark evidence, the raw score is also scaled by confidence
before entering the scoring function. Line interpolation therefore receives a
double discount: once for its interpolation confidence and once for being
inherited evidence.

Runtime fit Copy Chapter

The candidate's runtime form matters:

Light partial offload is penalized less than heavy offload. MoE models receive
a milder penalty when the active parameter working set can plausibly stay on
GPU while inactive experts spill to CPU RAM.

The final family selection key does not add a separate full-GPU bonus. Runtime
fit is already reflected in the quality score through the multiplier above and
the speed adjustment below. CPU-only results receive a small extra sort penalty
when mixed with GPU-backed candidates.

Speed adjustment Copy Chapter

Speed is treated as a usability gate. It is not the main quality signal.

Required speed depends on fit:

Candidates below the required speed receive up to -8 points. Candidates above
it receive up to +8 points.

After ranking, if any candidate is at least 5 tok/s, whichllm drops candidates
below 1.5 tok/s. This avoids recommending models that technically fit but are
not practical to use.

The reported speed is a point estimate, not a live benchmark. Ranking also
exposes speed confidence:

Speed cells are colored by absolute usability: red is under 4 tok/s, yellow
is 4-10 tok/s, green is 10-30 tok/s, and bright green is 30+ tok/s. ~
marks medium-confidence estimates with a range, and ? marks low-confidence
estimates. JSON exposes the same uncertainty data as speed_confidence,
speed_range_tok_per_sec, and speed_notes.

Source trust Copy Chapter

The source organization contributes a small adjustment:

• official model organizations receive a small bonus
• trusted GGUF converters can inherit that trust
• known repackagers receive a small penalty

The adjustment is intentionally small. It should break ties, not replace
benchmark and fit signals.

Popularity Copy Chapter

Downloads and likes act as tie-breakers. Their weight is lower when benchmark
evidence is strong and higher when evidence is weak.

Popularity has no effect for direct benchmark matches.

Generation lineage Copy Chapter

Some benchmark sources are frozen. A model released after a frozen leaderboard
cannot appear there, while older models can keep strong but stale scores.

whichllm uses family-specific lineage maps to avoid that inversion. Newer
generations can receive a small bonus; older generations can receive a small
penalty. This is applied carefully so direct benchmark evidence still matters.

Examples of tracked lineages include:

• Qwen
• Llama
• DeepSeek
• Gemma
• Phi
• Mistral
• GLM
• Kimi
• Granite
• OLMo
• T5 (incl. Flan-T5, mT5, ByT5, T5Gemma)

Benchmark markers Copy Chapter

The table score can include a marker:

Top-pick confidence is computed from the score gap, benchmark status, and fit
type. Partial-offload and CPU-only top picks are reported with lower confidence
than full-GPU direct-benchmark winners.

Why a smaller model can win Copy Chapter

A smaller model can outrank a larger one when it has:

• stronger current benchmark evidence
• a newer generation signal
• better quantization quality
• full-GPU fit instead of partial offload
• higher estimated speed
• a more trustworthy source

That is intentional. whichllm ranks likely usable quality, not parameter count
alone.

File: docs/troubleshooting.md Copy Chapter

Troubleshooting

This page lists common issues and the first checks to make.

Leaderboard rate limits Copy Chapter

Without pyarrow, whichllm reads the archived Open LLM Leaderboard in pages of
100 rows. It retries HTTP 429 responses before stopping. If an earlier page
succeeded, it keeps the scores collected so far and writes a warning to stderr
with the failed offset and retained score count. A failure on the first page
still fails that source; the other benchmark sources can continue.

Partial results use the same 24-hour benchmark cache as a complete fetch. The
cache does not record which pages were missing, and later cache reads do not
repeat the warning. Run whichllm --refresh to fetch the benchmark sources again.

No GPU detected Copy Chapter

Run:

whichllm hardware

If an NVIDIA GPU is missing:

• check that the driver is installed
• check nvidia-smi
• check that nvidia-ml-py can load NVML

whichllm falls back to nvidia-smi, but it still needs the NVIDIA driver tools
to be working.

If an AMD GPU is missing:

• on Linux, check rocm-smi, lspci, and /sys/class/drm
• on Windows, check that PowerShell can read Win32_VideoController
• for Ryzen AI / Radeon integrated graphics, check whether whichllm hardware
  shows shared memory instead of a tiny 512 MB or 4 GB adapter

If an Intel iGPU is missing:

• Linux detection uses lspci or /sys/class/drm
• Windows detection uses Win32_VideoController
• many Intel iGPUs do not expose dedicated VRAM, so they may be shown as shared
  memory graphics

Simulate hardware instead Copy Chapter

If detection is unavailable or you are planning a purchase, use --gpu:

whichllm --gpu "RTX 4090"
whichllm hardware --gpu "Apple M3 Max"
whichllm --gpu "RTX 5060 Ti" --vram 16
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"

Use --vram when the GPU name has multiple memory variants or is not in the
database.

For detected iGPU or unified-memory systems, override the usable GPU memory and
bandwidth directly:

whichllm hardware --vram 8 --ram-bandwidth 68
whichllm --vram 8 --bandwidth 68

If whichllm hardware lists multiple GPUs, add --gpu-index with the GPU
number from that output.

--vram only applies to one simulated or detected GPU. For multi-GPU
simulation, use known GPU names and omit --vram.

`--cpu-only` conflicts with `--gpu` Copy Chapter

These flags are mutually exclusive:

whichllm --cpu-only --gpu "RTX 4090"

Choose one:

whichllm --cpu-only
whichllm --gpu "RTX 4090"

`--vram` / `--bandwidth` needs a GPU Copy Chapter

These overrides need either a detected GPU or a simulated GPU:

whichllm --vram 8 --ram-bandwidth 68
whichllm --gpu "RTX 3060" --vram 12

If no GPU is detected, use --gpu to simulate one or check the detection steps
above.

No compatible models found Copy Chapter

Try:

whichllm
whichllm --cpu-only
whichllm --refresh

Common causes:

• the selected --quant is too restrictive
• --gpu-only or --fit full-gpu filters out partial-offload and CPU-only candidates
• --speed usable, --speed fast, or --min-speed filters out slower candidates
• --min-speed is too high
• --evidence strict filters out all candidates
• the requested context length is too large
• available RAM is too low after reserving space for the OS
• disk free space is too low for the model weights

For very small machines, remove optional filters first:

whichllm --top 20

Recommendations use RAM or CPU offload, but I only want VRAM Copy Chapter

By default, whichllm includes any runnable candidate: full-GPU, partial-offload,
and CPU-only. This is useful for finding what can run at all, but it can be too
loose when you want only models that fit entirely in GPU VRAM.

Use:

whichllm --gpu-only
whichllm --fit gpu
whichllm --fit full-gpu

If no rows are shown, this machine has no ranked candidates that fit fully in
GPU memory under the current filters. Remove --gpu-only, lower the context
length, or try a smaller quantization.

A model fits, but it is too slow Copy Chapter

The default ranking table shows estimated generation speed. Slow rows are red,
marginal rows are yellow, usable rows are green, and fast rows are bright
green. The ~ and ? markers are confidence markers for the estimate.

Filter slow rows with:

whichllm --speed usable  # >=10 tok/s
whichllm --speed fast    # >=30 tok/s
whichllm --min-speed 4   # exact floor, if you want a lower threshold

For an exact threshold:

whichllm --min-speed 10

LM Studio or another runtime says the model barely does not fit Copy Chapter

whichllm estimates model memory, but real runtimes can need extra room for
loader overhead, graph buffers, KV cache choices, and OS pressure. By default,
whichllm reserves a small automatic VRAM headroom before fit checks.

Tune it with:

whichllm --vram-headroom 1.5GB
whichllm --vram-headroom 10%
whichllm --vram-headroom none

Use none when you want the old raw-VRAM behavior.

RAM offload depends on what else is running Copy Chapter

Partial offload uses system RAM. If Docker, Elasticsearch, a browser, or
another workload is already using a large amount of memory, cap the offload
budget:

whichllm --ram-budget available
whichllm --ram-budget 8GB
whichllm --ram-budget 50%

available reads the current available RAM from the OS at startup. Fixed
values are useful when you know how much memory you want to leave for other
processes.

Results look stale Copy Chapter

whichllm caches model data for 6 hours and benchmark data for 24 hours.

Force a refresh:

whichllm --refresh
whichllm plan "qwen 7b" --refresh

The caches live under:

~/.cache/whichllm/

If XDG_CACHE_HOME is set to an absolute path, the caches live under:

$XDG_CACHE_HOME/whichllm/

`uvx` fails with `realpath: command not found` Copy Chapter

Some older macOS versions do not include a realpath command. If the uvx
launcher fails before whichllm starts, with output like:

realpath: command not found
/Users/.../python: No such file or directory

run whichllm through Python's module entry point instead:

uvx --from whichllm python -m whichllm

Pass normal whichllm arguments after the module name:

uvx --from whichllm python -m whichllm --gpu "RTX 4090"

The top pick has `~`, `!sr`, or `?` Copy Chapter

These markers describe benchmark evidence:

Use stricter evidence when you want only independently matched benchmark data:

whichllm --evidence strict
whichllm --direct

Use --evidence base when base-model matches are acceptable but interpolation
and self-reported values are not.

The largest model did not win Copy Chapter

That is expected. whichllm scores:

• benchmark quality
• model size
• quantization loss
• full GPU vs partial offload vs CPU-only
• estimated speed
• evidence confidence
• source trust
• generation lineage

A smaller current-generation model with strong direct evidence can beat a
larger model that only barely fits or relies on stale benchmark data.

Estimated speed differs from real speed Copy Chapter

Speed is an estimate based on:

• model weight size
• MoE active parameters
• GPU memory bandwidth
• quantization efficiency
• backend factor
• partial-offload penalty

Real performance depends on the inference runtime, driver, prompt length,
batching, thermal limits, and background memory pressure.

The default ranking table shows the speed estimate and its confidence marker.
Use --details only when you want download counts instead.

Speed colors and markers:

• red: slow generation speed, under 4 tok/s
• yellow: marginal generation speed, 4-10 tok/s
• green: usable generation speed, 10-30 tok/s
• bright green: fast local generation speed, 30+ tok/s
• ~: estimated speed range is available
• ?: low-confidence estimate; runtime/backend differences can be large

JSON includes the same information as speed_confidence,
speed_range_tok_per_sec, and speed_notes.

Apple Silicon partial offload looks different Copy Chapter

Apple Silicon uses unified memory. Partial offload does not cross a discrete
PCIe boundary, so whichllm applies a milder speed penalty than it does for
discrete GPUs.

The same is true for recognized AMD shared-memory APUs such as Strix Halo,
Ryzen AI MAX, and Ryzen AI / Radeon 890M-class integrated graphics.
DGX Spark / NVIDIA GB10 is handled the same way when NVIDIA reports GPU memory
as unavailable.

On Windows, Win32_VideoController.AdapterRAM can cap around 4 GB. whichllm
uses the 64-bit registry memory value when it is available, and treats known
shared-memory APUs as unified-memory style devices instead of tiny discrete
GPUs.

`run` says `uv is required` Copy Chapter

Install uv first:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then retry:

whichllm run

`run` cannot download a model Copy Chapter

Possible causes:

• the model is gated on HuggingFace
• local HuggingFace authentication is missing
• the selected GGUF filename no longer exists
• network access failed
• disk space is too low

Try a known public GGUF model first:

whichllm run "qwen 2.5 1.5b gguf"

Hugging Face API access fails or needs a mirror Copy Chapter

whichllm uses the Hugging Face API to fetch model metadata. If direct access to
huggingface.co fails in your network, set HF_ENDPOINT to a compatible
endpoint root:

$env:HF_ENDPOINT = "https://huggingface.co"
whichllm --refresh

HF_ENDPOINT="https://huggingface.co" whichllm --refresh

Do not include /api in HF_ENDPOINT; whichllm adds that path internally.

How much disk space does `run` need? Copy Chapter

Normal ranking commands do not download model weights. They cache Hugging Face
model metadata and benchmark metadata under the whichllm cache.

whichllm run downloads the selected GGUF file through huggingface_hub. The
required disk space is roughly the selected GGUF file size plus normal Hugging
Face cache overhead.

By default, Hugging Face stores downloaded files under:

~/.cache/huggingface/hub

You can move that cache by setting HF_HOME or HF_HUB_CACHE.

Cleanup is handled by the Hugging Face cache tools:

hf cache scan
hf cache delete

whichllm does not currently delete model files automatically after a run.

Ollama names do not match HuggingFace IDs Copy Chapter

JSON output returns HuggingFace repo IDs:

whichllm --top 1 --json | jq -r '.models[0].model_id'

Ollama model names often use a different naming scheme. Map the HuggingFace ID
to your local Ollama model name before calling ollama run.

Debugging a specific model Copy Chapter

Use plan to inspect memory requirements:

whichllm plan "Qwen2.5-72B" --quant Q4_K_M
whichllm plan "Qwen2.5-72B" --quant Q8_0 --context-length 32768

Use plain output when filing issues:

whichllm --gpu "RTX 4090" --json
whichllm --gpu "RTX 4090" --markdown
whichllm hardware

Include:

• OS
• GPU name and VRAM
• CPU and RAM
• command used
• expected result
• actual result