# Technical Documentation: antirez/ds4
> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/antirez/ds4)
### Overview (`README.md`)
**DwarfStar** aims to be the best way to run a few excellent large
language models on consumer hardware (that is, hardware that people
can actually own). To reach this goal, we are building
a small native inference engine optimized first for
**DeepSeek V4 Flash** (including the experimental vision model),
**DeepSeek V4.1 Flash** (Metal, and text inference on CUDA),
and additionally **GLM 5.2 and 5.3**, **GLM 5.3 Flash** and
**DeepSeek V4 PRO**, and **Qwen3.8 Flash Next** (Metal and CUDA). The code is self-contained and
deliberately narrow, not a general GGUF runner: you need to use the
GGUF files the project produces, that are part of the project
itself.
We test things in integration: model loading, prompt rendering,
tool calls, KV state, the HTTP server, and the coding agent are built and tested together.
The repository also includes tools and data for GGUF, imatrix, quality, and speed.
## Supported hardware
* **Metal**, the primary target, on Macs with 64 GB (for Qwen Flash next) and 96 GB or more (for DeepSeek and GLM). Smaller machines can use SSD streaming. SSD streaming is also needed in order to run very large models such as full GLM 5.x (not Flash) on 128GB systems.
* **NVIDIA CUDA**, the DGX Spark is our main gaol. DwarfStar also supports multi-GPU systems that are not supported by other backends, for instance it can run DeepSeek v4 Flash on Ada Lovelace cards.
* **ROCm** on Strix Halo systems such as the Framework Desktop.
This project would not exist without **llama.cpp and GGML**, make sure to read
the acknowledgements section, a big thank you to Georgi Gerganov and all the
other contributors.
**Model support is intentionally opportunistic**. The project follows the best open
weights for useful local machine sizes, especially 128 GB laptops and 256/512 GB
workstations. A model may be removed when a better replacement arrives.
# So, what can I do with this software?
* You can run a very capable models in your consumer hardware, a MacBook, a DGX Spark, or a Strix Halo for example. Even if you have not enough RAM, with SSD streaming, you can run it at a decent speed.
* You can use multiple CUDA cards as a multi-user LLM server. Ada Lovelace, including L40S, is supported: newer models can run here even when their other inference implementations require newer GPUs. Our eight-L40S Flash setup has reached about 126 t/s aggregate generation with 16 sessions.
* Using two 128 GB Macs connected with RDMA, you can run 4-bit DeepSeek Flash or GLM 5.3 Flash with tensor parallelism. Larger GLM 5.2 quants need larger machines, such as Mac Studios.
* You can also use pipeline paralellism to glue together multiple systems to sum their RAM and run larger models.
## Motivations
* Capable open-weight models now fit on high-end personal machines.
* DeepSeek V4 Flash and PRO, GLM 5.2, tolerate aggressive routed-expert quantization.
* Compressed KV caches and fast local SSDs make long contexts practical.
* The idea of an inference system specialized for a few models.
# AI full disclosure
* This software is developed with **strong assistance from AI coding agents** and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without `llama.cpp` and GGML, largely written by hand.
## Acknowledgements to llama.cpp and GGML
`ds4.c` does not link against GGML, but it **exists thanks to the path opened by the
llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won
engineering knowledge developed there**.
We are thankful and indebted to [`llama.cpp`](https://github.com/ggml-org/llama.cpp)
and its contributors. Their implementation, kernels, tests, and design choices were
an essential reference while building this DeepSeek V4 specific inference path.
Some source-level pieces are retained or adapted here under the MIT license: GGUF
quant layouts and tables, CPU quant/dot logic, and certain kernels. For this
reason, and because we are genuinely grateful, we keep the GGML authors copyright
notice in our `LICENSE` file.
## Status
The software is currently very fast changing. Consider it beta quality.
Before each release, a big QA run is executed, however instabilities
and regressions are definitely possible.
# How to use this project?
I (Salvatore) believe that the way projects should be shipped and used changed because of AI. The main differences today are:
1. With AI, users can modify the software in significant ways with low efforts, costs, and even lacking deep domain knowledge about the task they want to accomplish. For instance, a DwarfStar user with a specific hardware setup can ask a coding agent to improve the inference speed of this software for the specific hardware setup, asking the model to reach the maximum prefill and generation speed without impacting correctness, and also asking to do a deep QA pass.
2. Similiarly, because of "1", software may be shipped in a different way than before. It must be more a working template for the biggest use cases, without trying to cover every possible setup. If DwarfStar showcases a few good implementations of tensor parallel execution, the code will work as a rail for implementing the same feature in specific conditions, for a new model, and so forth.
So, while this project attempts to be usable for the featured models and the most common hardware setups, I ask you, if you have access to coding agents, to consider using coding agents as an interface to discover the project, make modifications, create personalized setups. This way you can likely do more than what we ship, and certain things that are not documented or implemented, and that you require, are potentially very easy to achieve.
## Start Here
```sh
git clone https://github.com/antirez/ds4.git
cd ds4
```
Choose your build. The platform guides cover prerequisites, memory sizing,
and hardware-specific setups:
| Platform guide | Build |
| --- | --- |
| [Metal on Apple Silicon](docs/METAL.md) | `make` |
| [DGX Spark](docs/DGX_SPARK.md) | `make cuda-spark` |
| [Strix Halo / Framework Desktop](docs/STRIX_HALO.md) | `make strix-halo` |
| [One or more CUDA cards, including Ada/L40S](docs/CUDA_MULTI_GPU.md) | `make cuda-generic` |
For a first run on a 96 or 128 GB machine, download DeepSeek V4 Flash Q2:
```sh
./download_model.sh ds4f-q2
```
Downloads go in `gguf/`. Repeat the command to resume an interrupted download.
Leave memory for the context and runtime buffers as well as the model.
See [other models](docs/MODELS.md) or use [SSD streaming](docs/SSD_STREAMING.md)
on a smaller Mac.
## Everyday Use
Once built and with a model downloaded:
```sh
./ds4
./ds4 -p "Explain Redis streams in one paragraph."
./ds4-agent
./ds4-server --ctx 32768
```
The default model is `ds4flash.gguf`, a link updated by main-model downloads.
Pass `-m FILE` to choose explicitly. Commands normally run from the repository
root; use `--chdir /path/to/ds4` when launching elsewhere.
The server listens at `http://127.0.0.1:8000` by default; see [serving](docs/SERVER.md)
for API access and multiple sessions.
The interactive CLI keeps a multi-turn conversation. Use `/help`, `/read FILE`,
`/ctx N`, and `/quit`. Ctrl+C interrupts generation and returns to the prompt.
Run each binary with `--help` for its full options.
### Native coding agent
`ds4-agent` runs inference directly, without a separate HTTP server. It keeps
the token history and live model state together, shows prefill progress, and
uses the model's native tool format. DeepSeek and GLM have their own templates.
Use `/hints on` for occasional, brief explanations of the programming choices
behind the work, and `/hints off` to stop them. Changes take effect at the next
conversation boundary without rebuilding the cached context. New and resumed
sessions start with hints off.
Sessions are stored in `~/.ds4/kvcache`:
| Command | Action |
| --- | --- |
| `/save` | Save the current session |
| `/list` | List saved sessions |
| `/switch ` | Resume a session |
| `/del ` | Delete a saved session |
| `/strip ` | Keep text and title, removing the large KV payload |
Compatible local KV snapshots avoid rebuilding the prompt. Stripped sessions
and network TP restores require prefill. Sessions containing images cannot yet
be saved. Saved conversations and traces may contain private information.
For Pi, OpenCode, Codex CLI, or Claude Code, use `ds4-server` instead and follow
the [client setup guide](docs/CLIENTS.md).
### Models, images, and speculation
[Models and vision](docs/MODELS.md) lists the supported downloads and memory
requirements. DeepSeek Vision Experimental uses a different checkpoint from
Flash 0731; GLM 5.3 Flash and Qwen3.8 Flash Next add vision to the same text
model through a separate encoder.
DeepSeek V4.1 Flash text and vision run on Metal; text also runs on a DGX Spark.
Q2 runs with SSD streaming on one 128 GB Mac or Spark, or resident across two
Macs or two Sparks using RDMA. Q4 needs SSD streaming or a 512 GB Mac.
Engram tables remain on disk in every mode, so use a fast
local SSD. See the [model guide](docs/MODELS.md#deepseek-v41-flash) for downloads
and setup.
With the matching encoder passed as `--vision FILE`, use `/read image.png`
in the CLI or `view_image` in the native agent.
Qwen3.8's smaller Q2 release has **41.73 GiB** of main/MTP weights,
with imatrix IQ2_XXS gate/up experts and padded Q2_K down projections.
It is the starting option for 64 GB Macs.
The GGUF also contains 95.37 GiB of original BF16 n-grams, read directly
from disk rather than loaded into RAM. Keep it on a fast SSD. Start with 8K context:
```sh
./download_model.sh qwen38-q2
./ds4 --ctx 8192 --prefill-chunk 1024
```
The download fetches one 137.10 GiB file and updates `ds4flash.gguf`.
Add `--mtp` for speculative decoding. The larger
`qwen38-q4k` target is also available. Download the optional vision encoder
with `./download_model.sh qwen38-vision` and pass it with `--vision`.
See [Qwen setup](docs/QWEN38_FLASH_NEXT.md) for details.
Speculative decoding is opt-in. GLM and Qwen use `--mtp`; V4 Flash DSpark needs a matching
support GGUF. It can improve generation, but not every workload benefits.
Read [speculative decoding](docs/SPECULATIVE_DECODING.md) for setup and the
difference between default opportunistic sampling and `--mtp-exact-sampling`.
### Output and power
Thinking is enabled by default. Use `--nothink` or `/nothink` for direct
answers, and `--think` or `/think` to enable it again.
For V4.1, `ds4` and `ds4-agent` also accept
`--think-level 25` or `/think 25`: 1 to 100 sets the reasoning effort, and
0 disables thinking. `--think` selects 75, `--think-max` selects 100.
Changing the level in a conversation rebuilds its cached prefix.
The normal sampling defaults are temperature 1, top-p 1, and min-p 0.05;
`--temp 0` selects greedy output.
For DeepSeek V4, `--power N` trades throughput for lower sustained GPU load.
The default is 100. V4.1 and GLM currently require `--power 100`.
DeepSeek V4 Flash and GLM 5.3 Flash also support directional steering. Load a
vector with `--dir-steering-file FILE`; `/steer F` adjusts its scale for
subsequent tokens in a local CLI or agent session, without rebuilding the
existing KV cache. See [steering documentation](dir-steering/README.md).
`--prefix-file FILE` preloads complete `USER:` / `ASSISTANT:` pairs before
the live conversation. A turn marker must start a line, roles must alternate,
and the last turn must be `ASSISTANT:`.
## Capability Evaluation
`ds4-eval` runs embedded capability regression tests against a real GGUF.
These are DwarfStar integration checks, not official leaderboard scores.
```sh
./ds4-eval -m ds4flash.gguf --trace /tmp/ds4-eval.txt
./ds4-eval -m ds4flash.gguf --suite hard-smoke
./ds4-eval -m ds4flash.gguf --suite hard --retry-incomplete
```
The default suite is `core`; `--suite all` runs core and hard cases.
`--list-cases` lists tests without loading a model. `--plain` selects
non-interactive output, and `--regrade-trace FILE` scores an existing trace
without generating again. Sources and licenses are in [EVAL_DATA.md](EVAL_DATA.md).
For inference correctness and release checks, read [testing](docs/TESTING.md).
## Speed
This recorded DeepSeek V4 Flash Q2 sweep uses an M5 Max with 128 GB RAM,
2048-token continued-prefill intervals, and 128 greedy generation tokens per
frontier. It is a baseline, not a fresh benchmark of every commit.
See [performance and benchmarking](docs/PERFORMANCE.md) for the full numbers,
DGX Spark results, comparison conditions, and benchmark commands.
## Detailed Guides
- [Models and vision](docs/MODELS.md): Flash, PRO, GLM, Qwen, and matching encoders.
- [Qwen3.8 Flash Next](docs/QWEN38_FLASH_NEXT.md): model setup, MTP, vision, and validation.
- [SSD streaming](docs/SSD_STREAMING.md): run larger than RAM and size the cache.
- [Inference across machines](docs/DISTRIBUTED.md): two-Mac TP/RDMA and layer pipelines.
- [Speculative decoding](docs/SPECULATIVE_DECODING.md): DSpark, GLM and Qwen MTP, and sampling.
- [Serving](docs/SERVER.md): APIs, images, batching, and disk KV caches.
- [Coding agent clients](docs/CLIENTS.md): Pi, OpenCode, Codex CLI, and Claude Code.
- [Performance](docs/PERFORMANCE.md): reproducible measurements and recorded baselines.
- [Testing and development](docs/TESTING.md): regression tests, debugging, and model-building tools.
Read [CONTRIBUTING.md](CONTRIBUTING.md) before sending a pull request.
## Logo
The DwarfStar logo was designed by hand by Salvatore Sanfilippo, made more
graphical with AI, and manually reworked by Ben Gnomino, whose human touch made
it rock.
---
### Overview (`dir-steering/README.md`)
# Directional Steering
Directional steering is a runtime activation edit for DS4. A steering file is a
flat `f32` matrix with one normalized hidden-width direction per normal
transformer layer. During inference, ds4 can apply the edit after attention
outputs, FFN outputs, or both:
```text
y = y - scale * direction[layer] * dot(direction[layer], y)
```
Positive scale removes the represented direction. Negative scale amplifies it.
With no steering file or zero scales, ds4 follows the normal inference path.
The file shape depends on the model:
- DeepSeek V4 Flash: `43 x 4096`.
- GLM 5.3 Flash: `45 x 4096`. The separate MTP predictor layer is omitted.
- Qwen3.8 Flash Next: `48 x 2560`. FFN steering is applied to each
hyper-connection branch of the residual; dumps average those branches
at the last prompt token.
GLM 5.2 steering is not implemented.
## Runtime Options
```text
--dir-steering-file FILE load one f32 direction per normal model layer
--dir-steering-ffn F apply steering after FFN outputs; default is 1 when a file is provided
--dir-steering-attn F apply steering after attention outputs; default is 0
```
The FFN output is usually the best first target because it is late enough in
each layer to represent behavior, style, and topic signals. Attention steering
is available for experiments, but it can be more fragile.
## GLM 5.3 Example
Build a GLM 5.3 direction from paired target and control prompt lists:
```sh
python3 dir-steering/tools/build_direction.py \
--profile glm-5.3-flash \
--ds4 ./ds4 \
--model gguf/GLM-5.3-Flash-Q2.gguf \
--good-file /path/to/target-prompts.txt \
--bad-file /path/to/control-prompts.txt \
--out dir-steering/out/glm53-direction.json \
--component ffn_out \
--ctx 512
```
Generated `.f32` vectors are local artifacts and are not stored in the
repository. GLM 5.3 steering works with `--mtp`, `ds4-server`, native session
batching, and two-Mac tensor parallelism. For tensor parallelism, pass the same
steering file and scales to both the worker and coordinator.
## Verbosity Example
The bundled example builds a style direction from 100 paired prompts. Each pair
asks for the same information in two ways:
- `examples/succinct.txt`: terse target prompts.
- `examples/verbose.txt`: detailed contrast prompts.
Because the extracted direction is `succinct - verbose`, negative FFN scales
make answers shorter, while positive FFN scales tend to make answers longer and
more explanatory.
Build the vector:
```sh
python3 dir-steering/tools/build_direction.py \
--profile deepseek-v4-flash \
--ds4 ./ds4 \
--model ds4flash.gguf \
--good-file dir-steering/examples/succinct.txt \
--bad-file dir-steering/examples/verbose.txt \
--out dir-steering/out/verbosity.json \
--component ffn_out \
--ctx 512
```
This writes:
```text
dir-steering/out/verbosity.json
dir-steering/out/verbosity.f32
```
Try a terse run:
```sh
./ds4 -m ds4flash.gguf --nothink --temp 0 -n 160 \
--dir-steering-file dir-steering/out/verbosity.f32 \
--dir-steering-ffn -1 \
-p "Explain why databases use indexes."
```
Try a verbose run:
```sh
./ds4 -m ds4flash.gguf --nothink --temp 0 -n 220 \
--dir-steering-file dir-steering/out/verbosity.f32 \
--dir-steering-ffn 2 \
-p "Explain why databases use indexes."
```
The same vector can be used in either direction. The sign is the important part:
- negative scale amplifies the succinct target direction;
- positive scale suppresses that direction and usually gives the model more room
to elaborate.
## Evaluating Scales
Use the sweep helper to test several strengths on a fixed prompt set:
```sh
python3 dir-steering/tools/run_sweep.py \
--ds4 ./ds4 \
--model ds4flash.gguf \
--direction dir-steering/out/verbosity.f32 \
--prompts dir-steering/examples/eval_prompts.txt \
--scales "-1,-0.5,0,0.5,1,2" \
--tokens 180 \
--nothink
```
Start with FFN scales between `-1` and `2`. If the model becomes repetitive,
ignores the prompt, or starts losing factual content, the scale is too strong.
For this example, `-1` is a good first terse setting and `2` is a good first
verbose setting. Strong negative scales such as `-2` or `-3` can over-amplify
the terse direction and collapse into repetition on some prompts.
## Observed Effect
With the 100-pair vector built from the commands above, local greedy checks
showed the expected behavior:
- Prompt: `Explain why databases use indexes.`
- `--dir-steering-ffn -1`: 67 words, one compact paragraph.
- `--dir-steering-ffn 0`: 136 words, structured explanation.
- `--dir-steering-ffn 1`: 140 words, structured explanation with more detail.
On a prompt that the unsteered model already answered briefly, positive steering
made the expansion more visible:
- Prompt: `What does DNS do?`
- `--dir-steering-ffn 0`: 44 words.
- `--dir-steering-ffn 2`: 171 words, with sections and step-by-step detail.
## Building Other Directions
The extractor compares two prompt sets:
- `good-file`: target prompts for the direction you want to represent.
- `bad-file`: contrast prompts that should be separated from the target.
It captures DS4 activations from the same local GPU graph used for inference,
averages target minus contrast, normalizes one vector per layer, and writes both
metadata JSON and the runtime `.f32` file.
Concept removal:
1. Put concept-heavy prompts in `good-file`.
2. Put neutral prompts in `bad-file`.
3. Run with a positive FFN scale.
Concept amplification:
1. Put desired concept prompts in `good-file`.
2. Put neutral prompts in `bad-file`.
3. Run with a negative FFN scale.
Style control:
1. Put prompts for the target style in `good-file`.
2. Put contrasting style prompts in `bad-file`.
3. Use negative scale to amplify the target style, positive scale to reduce it.
The method is not a fine-tune. It is a low-rank runtime edit, so it works best
for coarse behavior, topic, or style directions that are consistently present in
the activation captures.
## Qwen3.8 Flash Next
Capture uses `--think` / `--nothink`
(not `--think-high`). Dumps track the prompt phase explicitly, including
one-token tails, and retain the last prompt token during ordinary and MTP
decode. `attn_out` captures the output projection of both GDN and full-attention
layers, giving one row for each of the 48 trunk layers:
```sh
python3 dir-steering/tools/build_direction.py \
--profile qwen3.8-flash-next \
--ds4 ./ds4 \
--model gguf/Qwen3.8-Flash-Next-Q4.gguf \
--good-file /path/to/target-prompts.txt \
--bad-file /path/to/control-prompts.txt \
--out dir-steering/out/qwen38-direction.json \
--component ffn_out \
--ctx 512
```
Qwen steering is Metal-only. `--mtp-model`, SSD streaming, and `--power`
remain unsupported for this graph. The bank contains only the 48 trunk layers;
the embedded MTP predictor remains unsteered. Its drafts are verified by the
steered target trunk, so `--mtp` remains supported.
---
### Overview (`gguf-tools/README.md`)
# DS4 GGUF Tools
This directory contains the offline tools used to build and evaluate DeepSeek
V4 Flash GGUF files for `ds4`.
The important pieces are:
- `deepseek4-quantize.c`: C HF-safetensors to GGUF quantizer.
- `quants.[ch]`: the deliberately small local quantization implementation used
by the quantizer. It implements the DS4 output formats we actually ship:
`q8_0`, `q8_K`, `q4_K`, `q2_K`, and `iq2_xxs`.
- `imatrix/`: dataset and instructions for collecting routed-MoE activation
importance with `ds4`.
- `quality-testing/`: prompts and scripts used to compare local GGUF variants
against official DeepSeek V4 Flash continuations.
## Qwen3.8 IQ2_XXS experiment
`qwen4_iq2.py` builds mixed IQ2_XXS/MXFP4 main weights with
embedded MTP. It quantizes the 48 trunk layers' gate/up experts directly from
the official BF16 checkpoint using the same pinned imatrix as the Q4_K pack.
It then copies every other tensor from an existing combined Q4_K `qwen4exp`
GGUF, including MXFP4 down projections, dense/control weights and MTP.
This isolates the gate/up precision change for a quality comparison.
The Python environment needs NumPy and `huggingface_hub`. Build the existing
native quantizer library with `make -C gguf-tools quants-shared`, then run:
```sh
python3 gguf-tools/qwen4_iq2.py quantize \
--source /path/to/official-bf16-checkpoint \
--imatrix /path/to/imatrix_unsloth.gguf_file \
--library gguf-tools/libds4quants.dylib \
--experts-dir /path/to/iq2-experts --threads 8
python3 gguf-tools/qwen4_iq2.py assemble \
--template /path/to/combined-q4k-mtp-pleext.gguf \
--experts-dir /path/to/iq2-experts \
--out /path/to/Qwen3.8-Flash-Next-IQ2XXSImatrix-MXFP4Down-MTP.gguf
```
The first stage can run beside the source checkpoint on another machine;
transfer the complete experts directory, including `experts.json`, for
assembly. Repeating the first command verifies completed tensors before
resuming. Assembly refuses existing outputs and verifies every written tensor
before publishing the completed GGUF and its JSON manifest.
The intermediate main model is approximately 50.3 GB. Finish it with the
[native n-gram packer](#native-qwen-n-grams) before inference. Quantization
changes model outputs; fitting weights on disk does not establish a 64 GB
runtime fit or acceptable reasoning quality. Compare with the same evaluator
settings using `ds4-eval --suite hard`.
### Experimental padded Q2_K down projections
Use `--projection down` on both stages to quantize the 48 trunk down
projections directly from BF16, with the pinned imatrix. Their 640 logical
inputs are zero-padded to 768 for three Q2_K blocks per row. Assemble against
the existing IQ2_XXS/MXFP4 combined model, using a new experts directory and
output path. All other tensors, including MTP, are copied unchanged:
```sh
uv run --with numpy --with huggingface_hub python gguf-tools/qwen4_iq2.py quantize \
--projection down --source /path/to/official-bf16-checkpoint \
--imatrix /path/to/imatrix_unsloth.gguf_file \
--library gguf-tools/libds4quants.dylib \
--experts-dir /path/to/q2-down-experts --threads 8
uv run --with numpy --with huggingface_hub python gguf-tools/qwen4_iq2.py assemble \
--projection down --template /path/to/IQ2XXSImatrix-MXFP4Down-MTP.gguf \
--experts-dir /path/to/q2-down-experts \
--out /path/to/IQ2XXSImatrix-Q2KDownPad768-MTP.gguf
```
This layout requires the padded-down Qwen runtime support. The GGUF stores
physical dimensions `[768, 2560, 512]`; the architecture's expert width remains
640. Metal decode and prefill use the padded weight stride while reading only
640 activation columns. The CPU reference supplies zero activation padding.
The shared expert and MTP retain their original dimensions and formats.
Saving 88 bytes per trunk down row reduces the main model by 5.15625 GiB;
quality and speed must be evaluated for this new recipe.
### Native Qwen n-grams
Finish main-only packs with the original BF16 n-gram shards, not the old
quantized sidecar:
```sh
python3 gguf-tools/qwen4_native_ngrams.py \
--model /path/to/IQ2XXSImatrix-Q2KDownPad768-MTP.gguf \
--source /path/to/Qwen3.8-Flash-Next \
--source-revision de4b8e4d43b917e7706784d8bb445c9af86a3540 \
--output gguf/Qwen3.8-Flash-Next-Q2.gguf
```
The source directory needs its index and the safetensors shards containing
`ple.ple_embedding` tensors. The packer checks the hash constants, copies all
main/MTP tensor bytes unchanged, and appends the original BF16 table at a
page-aligned offset. It verifies every copied payload before publishing the
file and its checksum report. Allow space for the complete new file plus
16 GiB spare. An incomplete output is never a runnable release artifact.
For the Q4 pack, use its main GGUF as `--model`. `--source` can also be a
previously verified native n-gram GGUF from the same pinned checkpoint.
This avoids keeping a second copy of the original safetensors. The final
Q2 and Q4 files include 95.37 GiB of disk-only n-grams; no table precision
is lost, and the main-model quantization is unchanged.
## Build
```sh
make -C gguf-tools
```
The quantizer is plain C and does not link GGML. GGUF metadata handling,
safetensors loading, FP4/FP8 dequantization, and the quantizers used by our Q2
and Q4 recipes live in this directory.
## Generate An Imatrix
First regenerate or inspect the calibration dataset:
```sh
python3 gguf-tools/imatrix/dataset/build_ds4_imatrix_dataset.py
```
Then collect activation statistics with the DS4 runtime:
```sh
./ds4 \
-m gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf \
--imatrix-dataset gguf-tools/imatrix/dataset/rendered_prompts.txt \
--imatrix-out gguf/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4.dat \
--ctx 32768
```
The imatrix file is useful immediately with this DS4 quantizer. Generic GGUF
tools need DS4-specific tensor-name mapping and per-expert slicing before they
can use it correctly. The accepted imatrix format is the legacy llama.cpp
binary `.dat` file emitted by `ds4 --imatrix-out`.
Generating this `.dat` file locally is possible, but slow: it runs the DS4
prefill graph over the full calibration corpus and reads routed-MoE activation
statistics back from the GPU. The latest published imatrix-generated GGUF files
are available in the antirez Hugging Face repository:
```text
https://huggingface.co/antirez/deepseek-v4-gguf/tree/main
```
## Generate Q2 And Q4 GGUFs
The template GGUF supplies metadata, tokenizer, tensor order, and logical
shapes. Tensor bytes are regenerated from the Hugging Face safetensors. Full
generation is intentionally offline and heavy: expect roughly 80-90 GB outputs
for the 2-bit template family and roughly 150-170 GB for the 4-bit routed-expert
family, plus enough free disk for the temporary output. Use `--dry-run` and
`--compare-tensor` before starting a full write, and use `--overwrite` only when
you really mean to replace an existing GGUF.
Q2 routed experts with imatrix:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash \
--template gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf \
--out gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
--imatrix gguf/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4.dat
```
Q4 routed experts with imatrix:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash \
--template gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf \
--out gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf \
--imatrix gguf/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4.dat
```
True Q8_K routed experts:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash \
--template gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf \
--out gguf/DeepSeek-V4-Flash-Q8KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf \
--experts q8_K \
--threads 8
```
You can override tensor families:
```sh
--experts iq2_xxs
--routed-w2 q2_k
--attention-proj q8_0
--shared q8_0
--output q8_0
```
Useful checks before writing a full model:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash \
--template MODEL.gguf \
--compare-tensor blk.0.attn_q_a.weight
```
`--compare-tensor` regenerates a single tensor and byte-compares it against the
template or `--compare-gguf`. `--threads N` controls routed-expert workers.
## Convert DeepSeek V4.1 Flash
V4.1 uses its own converter; a V4 template is not compatible. Install NumPy,
tokenizers and SymPy, then build the quantizer library:
```sh
make -C gguf-tools libds4quants.dylib
python3 gguf-tools/deepseek41_quantize.py \
--hf models/DeepSeek-V4.1-Flash \
--source-revision df42c109f1defefcbfcedbe7d905718a12266e40 \
--out gguf/DeepSeek-V4.1-Flash-IQ2_XXS-Q2_K-bootstrap.gguf --dry-run
```
Omit `--dry-run` to write the file. Add `--resume` after an interrupted conversion.
Use `libds4quants.so` on Linux. Gate/up experts use IQ2_XXS and down experts use
Q2_K; attention, shared experts and the output head use Q8_0. Engram rows retain
their original FP8 values and scales, packed together at the end of the GGUF for
disk lookups. Vision and DSpark weights are not included.
The first conversion uses weight-energy importance for IQ2_XXS. After runtime
calibration, add `--imatrix FILE` and choose a new output filename to regenerate
from the original safetensors. Do not requantize the first GGUF.
For Q4, add `--quant q4` and use `DeepSeek-V4.1-Flash-Q4.gguf` as the output.
This changes only the routed experts to Q4_K; the other tensor formats and
disk-only Engram layout stay the same. The same imatrix works for both recipes.
Check the finished artifact against the pinned source before running it:
```sh
python3 gguf-tools/deepseek41_validate_gguf.py \
--hf models/DeepSeek-V4.1-Flash \
--source-revision df42c109f1defefcbfcedbe7d905718a12266e40 \
--gguf gguf/DeepSeek-V4.1-Flash-IQ2_XXS-Q2_K-bootstrap.gguf --payload
```
For a calibrated file, pass the same `--imatrix FILE` used during conversion.
Pass `--quant q4` to the audit as well when checking a Q4 file.
The audit checks the complete layout, all non-expert tensors, sampled experts
and native Engram rows. It does not replace [inference quality tests](quality-testing/deepseek-v4.1-flash-20260910/README.md).
### Convert V4.1 DSpark
The V4.1 support converter needs NumPy, the official config and tensor index,
and only the shards containing `mtp.*` tensors (44, 45 and 46 of the pinned
checkpoint). No imatrix or main-model GGUF is needed:
```sh
python3 gguf-tools/deepseek41_dspark.py \
--hf models/DeepSeek-V4.1-Flash \
--source-revision df42c109f1defefcbfcedbe7d905718a12266e40 \
--out gguf/DeepSeek-V4.1-Flash-DSpark.gguf
```
The result is 8.04 GiB. Routed experts keep their native MXFP4 payloads;
dense weights expand losslessly to F16 or F32. Add `--dry-run` to inspect the
layout, or `--verify` to compare every tensor in an existing output against
the source. This file works only with the V4.1 Flash target.
## Convert A DSpark Support Checkpoint
The DSpark Flash checkpoint is published as Hugging Face safetensors and stores
the draft module under `mtp.0`, `mtp.1`, and `mtp.2`. Before writing a support
GGUF, inspect the official index and verify that every DSpark tensor name is
understood by the converter:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash-0731 \
--dspark-manifest > /tmp/dspark-manifest.tsv
```
The manifest reads only `model.safetensors.index.json`; it does not require the
large shard files to be present. The final summary should report three DSpark
stages and zero unknown DSpark tensors before attempting a full conversion.
To build the support GGUF used by `ds4 --dspark`, run the DSpark support mode. This
mode writes standalone DSpark metadata plus the packed `mtp.*` tensor payloads;
it does not require a base-model GGUF template:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash-0731 \
--dspark-support \
--out DeepSeek-V4-Flash-DSpark-support-0731.gguf
```
Add `--experts mxfp4` to preserve the checkpoint's routed expert weights
without requantization. Use a different output filename, such as
`DeepSeek-V4-Flash-DSpark-MXFP4-support-0731.gguf`. This takes about 10.1 GiB
instead of 5.6 GiB; the other tensors keep the same formats. A more accurate
drafter can improve acceptance, but does not guarantee faster generation.
`--dspark-support --dry-run` reads safetensors shard headers to derive exact
GGUF shapes and types, but it does not read tensor payloads. The DSpark metadata
defaults match the published Flash DSpark config: block size 5, target layers
40,41,42, Markov rank 256, and noise token 128799. Override them with
`--dspark-block-size`, `--dspark-target-layers`, `--dspark-markov-rank`, and
`--dspark-noise-token-id` if converting a different checkpoint.
Before a full write, regenerate one support tensor and record its checksum:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash-0731 \
--dspark-support \
--compare-tensor mtp.0.main_proj.weight
```
This reads only the payloads needed for that tensor. Add `--compare-gguf
DeepSeek-V4-Flash-DSpark-support-0731.gguf` to byte-compare against an existing
support GGUF.
## When No Imatrix Is Given
`iq2_xxs` requires an importance vector. If `--imatrix` is not provided and
the target type requires one, `deepseek4-quantize` computes a synthetic fallback
from the dequantized weight itself:
```text
importance[column] = sum(row[column]^2) over all rows
```
This is a weight-energy heuristic. It is not as good as measuring real DS4
activations, but it gives the quantizer a stable column weighting and was good
enough for the first working 2-bit GGUFs.
## Quality Testing
See `quality-testing/README.md`. The short version is:
```sh
python3 gguf-tools/quality-testing/collect_official.py
make -C gguf-tools quality-score
gguf-tools/quality-testing/score_official MODEL.gguf gguf-tools/quality-testing/data/manifest.tsv /tmp/model.tsv 4096
python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv
```
## Qwen3.8 Flash Next
`qwen4_exp_convert.py` wraps llama.cpp's `convert_hf_to_gguf.py` and writes
the `qwen4exp` schema DS4 loads, including the MTP block as `blk..nextn.*`.
It needs a llama.cpp master checkout (b96806d96 or newer, found through
`--llama-cpp` or `$LLAMA_CPP`) and a Python with torch, safetensors and
transformers, such as llama.cpp's own venv:
```sh
python gguf-tools/qwen4_exp_convert.py --src /path/to/Qwen3.8-Flash-Next \
--source-revision de4b8e4d43b917e7706784d8bb445c9af86a3540 \
--out Qwen3.8-Flash-Next-Q8.gguf --outtype q8_0
python gguf-tools/qwen4_exp_convert.py --src /path/to/Qwen3.8-Flash-Next \
--source-revision de4b8e4d43b917e7706784d8bb445c9af86a3540 \
--out Qwen3.8-Flash-Next-MXFP4.gguf --outtype q8_0 --experts mxfp4
```
Options: `--outtype q8_0|f32` (dense projections), `--experts
q8_0|mxfp4|q4_k|f32` (routed experts; `--experts-down` picks the 640-wide down
projection type when the gate/up type needs 256-wide rows), `--hc-type f16|f32|q8_0`
(hyper-connection mixers), `--indexer bf16|f16|q8_0|f32` (the QSA indexer
projections, kept at the released BF16 by default), `--no-mtp` and
`--dry-run`. Norms, conv kernels, `ssm_a`, dt biases and the routers stay F32.
N-grams always retain the original BF16 bytes. The converter finishes through
the native packer above; its temporary main-weight file is removed on success.
`gen_qwen4_unicode.py` regenerates `ds4_qwen4_unicode.inc` for the `qwen35`
pre-tokenizer from a current `regex` release.
`make test-qwen4-kernels` runs the Metal kernel tests and
`make test-qwen4-vision` checks the vision tower against the HF implementation
(`tests/qwen4_vision_ref.py`).
---
### Overview (`speed-bench/README.md`)
## Benchmarking
Here we collect prefill and generation speed obtained with different hardware.
Run `ds4-bench` as:
```
./ds4-bench \
-m ds4flash.gguf \
--prompt-file speed-bench/promessi_sposi.txt \
--ctx-start 2048 \
--ctx-max 65536 \
--step-incr 2048 \
--gen-tokens 128
```
Provide PR including your numbers if your hardware was not already tested.
Call the benchmark csv file something like `m3_max.csv` or alike, so that
it is clear what hardware was used for the benchmark.
To generate an SVG graph from a CSV file:
```
python3 speed-bench/plot_speed.py speed-bench/m3_max.csv --title "M3 Max t/s"
```
The script uses only the Python standard library. By default it writes a file
next to the CSV using the `_ts.svg` suffix, such as `speed-bench/m3_max_ts.svg`.
### Metal decode schedule A/B
Build the balanced, same-engine Metal decode comparison with:
```
make metal-decode-schedule-bench
./speed-bench/metal_decode_schedule_bench \
-m ds4flash.gguf \
--include-selection
```
The harness prefills two sessions and alternates both variant order and
variant-to-session assignment. It aborts unless every full-vocabulary logit
row is bit-identical and, with `--include-selection`, both variants select the
same non-EOS token. Use `--candidate-env NAME` to measure a rollback control,
or `--help` to compare explicit split schedules.
To compare the default pre-M5 ratio-4 compressor pack/transpose fusion with the
legacy decode path, including token selection, use:
```
./speed-bench/metal_decode_schedule_bench \
--candidate-env DS4_METAL_DISABLE_PRE_M5_COMPRESSOR_RATIO4_DECODE_PACK_FUSION \
--include-selection \
--tokens 1024
```
### Metal prefill variant A/B
Build the balanced prefill comparison. To compare the default resident pre-M5
MXFP4 pair tail-SIMDgroup cull against the original pair kernel, make the
rollback path the candidate:
```
make metal-prefill-variant-bench
./speed-bench/metal_prefill_variant_bench \
--candidate-env DS4_METAL_DISABLE_PRE_M5_MXFP4_MOE_MM_ID_PAIR_TAIL_SIMDGROUP_CULL
```
To isolate the default routed-down tail-SIMDgroup cull from the retained pair
default, use its down-specific rollback as the candidate:
```
./speed-bench/metal_prefill_variant_bench \
--candidate-env DS4_METAL_DISABLE_PRE_M5_MXFP4_MOE_MM_ID_DOWN_TAIL_SIMDGROUP_CULL
```
The harness uses one Metal engine and fresh sessions for every run. It warms
both variants with at least 32 tokens, alternates control/candidate order in
ABBA and BAAB blocks, poisons host logit buffers before copying, and aborts
unless every final full-vocabulary logit row is bit-identical. Defaults are an
8192-token prefix, an automatically sized 8193-token context, and two repeats;
use `--help` to override them.
Numeric runtime tuning controls can use `--candidate-value TEXT` (default `1`), and
`--prefill-chunk N` selects the same chunk size for both variants (default
4096). The control unsets the named variable. Use only controls read at
dispatch time: this harness keeps one engine alive and cannot compare settings
cached during initialization.
`DS4_QWEN4_TIMING=2` reports each prefill chunk's position, token count, and
stage times, including the attention mixer separately from GDN/attention.
These measurements include synchronization overhead and are for attribution;
use uninstrumented alternating runs for throughput claims.
`DS4_QWEN4_MOE_PROFILE=1` further splits tiled MoE into routing, expert-list
construction, routed/shared gate-up, routed/shared down, and reduction,
reporting one line per layer. Its synchronization also makes it diagnostic only.
Qwen routed MoE quantization specialization is enabled by default on M3 Ultra.
`DS4_QWEN4_MOE_MM_SPECIALIZE=0` restores the generic kernels; `=1` opts in on
other devices. To compare the final default against rollback, pass
`--candidate-env DS4_QWEN4_MOE_MM_SPECIALIZE --candidate-value 0` to the harness.
### Concurrency: many requests at once
`session_concurrency_bench` measures the engine without HTTP overhead.
```sh
make session-concurrency-bench
./speed-bench/session_concurrency_bench -m ds4flash.gguf \
--concurrency 1,2,4,8 --ctx 4096 --gen 128 --budget-gib 28 \
--csv /tmp/concurrency.csv
```
Streams use different corpus offsets to avoid identical expert routing.
The report includes aggregate and per-stream throughput, step latency and
prefill time. Set `--budget-gib` to bound session allocations; monitor actual
memory and swap too. `--mixed` adds a concurrent continued prefill, `--spec`
enables Qwen MTP, and `--verify` fails on a greedy-token mismatch against
serial decoding. Use the official-continuation scorer to assess quality when
different reduction orders change a close token choice.
For an interleaved comparison against a diagnostic rollback:
```sh
./speed-bench/session_concurrency_bench --ctx 4096 --concurrency 8 \
--candidate-env DS4_QWEN4_SESSION_BATCH --candidate-value 0 --repeat 3
```
The control unsets the variable; the candidate sets it in alternating ABBA
blocks. Use only controls read at dispatch time. Keep prompts, context, memory
pressure and thermal conditions comparable, and retain generated text when
measuring speculation. `concurrency_sweep.sh` runs each cell in a fresh process.
`serve_concurrency_bench.py` measures the HTTP server, including queueing,
prompt rendering, prefix reuse and streaming:
```sh
python3 speed-bench/serve_concurrency_bench.py \
--concurrency 8 --prompt-tokens 4096 --max-tokens 128 --requests 32 \
--ignore-eos --json /tmp/serve-8.json
```
Start the server separately with an appropriate model, context and
`--batched-session 8`. The report includes first-token and inter-token
latency, request latency and throughput. Prompts use fresh nonces;
`--shared-prefix` tests cache reuse instead.
---
### Overview (`gguf-tools/imatrix/README.md`)
# DS4 Imatrix Pipeline
This directory contains the calibration dataset and instructions used to build
activation importance matrices for DeepSeek V4 Flash and Pro GGUF
quantization.
The current imatrix target is the routed MoE path. Flash has 43 layers and
256 routed experts per layer. Pro has 61 layers and 384 routed experts per
layer. Both variants expose three routed expert tensors per layer:
- `blk.N.ffn_gate_exps.weight`
- `blk.N.ffn_up_exps.weight`
- `blk.N.ffn_down_exps.weight`
For gate/up tensors, the collector records the squared FFN-normalized input
activation. For down tensors, it records the squared routed SwiGLU row after
route weighting. The result tells the quantizer which input columns are used
more heavily by the actual DS4 inference graph.
## 1. Build The Calibration Dataset
The tracked dataset is in `gguf-tools/imatrix/dataset/`. Regenerate it from
the repository root with:
```sh
python3 gguf-tools/imatrix/dataset/build_ds4_imatrix_dataset.py
```
The important output is:
```text
gguf-tools/imatrix/dataset/rendered_prompts.txt
```
It contains DS4-rendered chat prompts, separated by visible
`DS4_IMATRIX_PROMPT` markers. The prompts include:
- C/Metal source-review prompts from this repository.
- Long-context snippets.
- Agent/tool-call prompts using DS4's DSML syntax.
- Language/prose rewriting, summarization, extraction, and translation prompts.
- `ds4-eval` GPQA Diamond, SuperGPQA, and AIME2025 benchmark prompts.
- Both thinking and non-thinking assistant prefixes.
The current tracked dataset has 4682 rendered prompts and roughly 2.91M tokens
by the coarse bytes/4 estimate. Check
`gguf-tools/imatrix/dataset/manifest.json` for the exact generated-file
summary.
## 2. Collect The Imatrix
Use the DS4 runtime itself to collect routed MoE activation statistics. The
collector uses the loaded GGUF metadata, so the same command shape works for
Flash and Pro.
Flash example:
```sh
./ds4 \
-m ../deepseek-v4-quants/gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf \
--imatrix-dataset gguf-tools/imatrix/dataset/rendered_prompts.txt \
--imatrix-out ../deepseek-v4-quants/imatrix/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4-1p5m.dat \
--ctx 32768
```
Pro example with a smaller calibration budget:
```sh
./ds4 \
-m ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct.gguf \
--imatrix-dataset gguf-tools/imatrix/dataset/rendered_prompts.txt \
--imatrix-out ../deepseek-v4-quants/imatrix/DeepSeek-V4-Pro-Instruct-routed-moe-ds4-small.dat \
--imatrix-max-prompts 16 \
--imatrix-max-tokens 32768 \
--ctx 32768
```
Useful smoke-test limits:
```sh
./ds4 \
-m MODEL.gguf \
--imatrix-dataset gguf-tools/imatrix/dataset/rendered_prompts.txt \
--imatrix-out /tmp/ds4-test.imatrix.dat \
--imatrix-max-prompts 1 \
--imatrix-max-tokens 4096
```
The collector is Metal-only because it hooks the layer-major Metal prefill graph.
It does not change inference math; it reads the already materialized MoE inputs
and accumulates `sum(x[column]^2)` per routed expert.
The output format is llama.cpp's legacy binary `.dat` imatrix format. DS4 packs
per-expert vectors into one entry per routed expert tensor:
```text
entry length = n_expert * n_columns
```
The quantizer slices the right expert's segment when quantizing each expert.
For Pro, that means each routed tensor entry contains 384 vectors instead of
Flash's 256.
## 3. Generate GGUF Files With The Imatrix
The local C quantization tool in `gguf-tools/` supports:
```text
--imatrix FILE
--imatrix-strict
```
Example Q4 routed-expert regeneration:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash \
--template ../deepseek-v4-quants/gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf \
--out ../deepseek-v4-quants/gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf \
--imatrix ../deepseek-v4-quants/imatrix/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4-1p5m.dat
```
Example Q2 regeneration:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Flash \
--template ../deepseek-v4-quants/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf \
--out ../deepseek-v4-quants/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
--imatrix ../deepseek-v4-quants/imatrix/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4-1p5m.dat
```
Example Pro Q2 regeneration from a Pro template:
```sh
gguf-tools/deepseek4-quantize \
--hf ../deepseek-v4-quants/hf/DeepSeek-V4-Pro \
--template ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct.gguf \
--out ../deepseek-v4-quants/gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix-small.gguf \
--imatrix ../deepseek-v4-quants/imatrix/DeepSeek-V4-Pro-Instruct-routed-moe-ds4-small.dat \
--threads 24
```
`deepseek4-quantize` reads the routed expert count from the template GGUF. Use
`--n-experts` only when working with an older template that lacks
`deepseek4.expert_count`.
For Q4, the imatrix does not change the runtime tensor type: routed experts
remain `Q4_K`. It changes how quantization error is weighted while choosing
scales and codes. For Q2, it replaces the previous synthetic weight-energy
fallback used for `IQ2_XXS` gate/up experts with real activation statistics.
## 4. Evaluate
Useful local tools:
```text
misc/quant_eval.c
gguf-tools/quality-testing/
```
`misc/quant_eval.c` compares local GGUF variants by greedy/top-logit behavior.
`gguf-tools/quality-testing/` can score local GGUFs against official DeepSeek
API continuations by target-token negative log likelihood.
The Q4 imatrix file uploaded to Hugging Face was tested on 100 official
DeepSeek V4 Flash continuations:
```text
old Q4 avg NLL: 0.177357819
Q4 imatrix avg NLL: 0.173895148
relative NLL change: -1.95%
case wins: 54 imatrix / 46 old
first-token matches: 83 imatrix / 81 old
avg greedy LCP: 12.21 imatrix / 11.94 old
```
## Compatibility
The `.dat` file is intentionally in llama.cpp's legacy imatrix format, so the
data is not conceptually tied to DS4. In practice, it is immediately useful
only with a quantizer that understands DS4's tensor names and packed per-expert
entries. The current `deepseek4-quantize` tooling does that.
Other GGUF creation tools can use the same imatrix if they implement the same
name mapping and per-expert slicing convention. Without that DS4-specific
mapping, a generic imatrix loader will see valid data but will not know how to
apply the packed routed-expert vectors correctly.
---
### Overview (`gguf-tools/mixed/README.md`)
# Mixed GGUF Splicing
This directory contains a small GGUF splicer for experiments where only some
routed-expert layers are upgraded from one quantization to another.
The tool does not dequantize or requantize weights. It reads two compatible
GGUF files, uses the first one as the base, and copies selected routed-expert
tensors from the second one. All other tensors remain byte-for-byte identical to
the base file.
## Last Six Layers Q4 Experiment
The best mixed Flash experiment so far keeps the fixed-imatrix 2-bit GGUF as the
base, then replaces only the routed experts in layers `37-42` with the same
tensors from the fixed-imatrix Q4 GGUF.
This produces a file around 91 GB instead of the full Q4 file around 153 GB. In
the local output-agreement checks, this last-six-layers file was statistically
closer to the full Q4 GGUF than the plain 2-bit GGUF. This is not the same as a
benchmark score, but it is a useful signal that the mixed file behaves more like
the stronger Q4 model while keeping most of the 2-bit size.
## Usage
```bash
python3 gguf-tools/mixed/splice_mixed_expert_layers_gguf.py \
--base /path/to/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf \
--donor /path/to/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-fixed.gguf \
--q4-layers 37-42 \
--out /path/to/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf
```
Use `--dry-run` first to check the selected tensors and expected size delta.
Use `--force` only when you really want to overwrite the output file.
Layer ranges can be changed without editing the script:
```bash
--q4-layers 0-2,40-42
```
---
### Overview (`gguf-tools/quality-testing/README.md`)
# Official-Continuation Quality Testing
This directory contains the prompts, tracked official fixtures, and scripts used
to compare local GGUF variants against hosted-model continuations.
The main metric is target-token negative log likelihood: collect an
official continuation, then ask each local GGUF how much probability it assigns
to that exact continuation token by token. This avoids judging quality from one
sampled answer.
## 1. Tracked Fixture Sets
Curated fixtures are kept in the repository so release QA can run without
calling hosted APIs:
- `data/glm52-openrouter-100`: 100 GLM 5.2 continuations collected through
OpenRouter `z-ai/glm-5.2` with `top_logprobs=20`.
- `data/glm53-flash-openrouter-zai-fp8-100`: 100 GLM 5.3 Flash continuations
from OpenRouter's pinned Z.AI FP8 endpoint. The endpoint does not expose
logprobs, so these are deterministic continuation fixtures.
- `data/glm53-flash-openrouter-zai-fp8-long`: eight code-context continuations
from the same FP8 endpoint, with 3K-7K prompts that exercise sparse attention.
Use the rendered prompts below to preserve the API's reasoning prefix.
- `data/flash`: 100 DeepSeek V4 Flash 0731 continuations collected from the
official DeepSeek API with `top_logprobs=20`.
- `data/pro`: 100 DeepSeek V4 PRO preview continuations collected from the
official DeepSeek API with `top_logprobs=20`.
- `data/pro-0813`: 100 DeepSeek V4 PRO 0813 continuations collected from the
official DeepSeek API with `top_logprobs=20`.
- `data/qwen38-flash-alibaba-100`: 100 non-thinking Qwen3.8 Flash
continuations from Alibaba through OpenRouter, with top-five logprobs.
- `data/qwen38-flash-alibaba-long`: 12 archive/code continuations from the
same endpoint, with prompts from 2K to 24K tokens.
- [deepseek-v4.1-flash-20260919-router](deepseek-v4.1-flash-20260919-router/README.md):
112 fresh official V4.1 Flash continuations, including 12 longer prompts up
to 23K tokens. Temperature-1 samples with top-20 logprobs, intended for
paired numerical comparisons. These are not V4 Flash or Vision-Exp vectors.
DeepSeek V4 Flash also has tracked official smoke vectors in
`tests/test-vectors/`. Those vectors drive `./ds4_test --logprob-vectors` and
include short prompts plus long-prompt attention cases.
The hosted APIs expose output-token logprobs and top-logprob alternatives, not
full vocabulary logits.
The scorer verifies each API token's bytes against the local token boundaries,
not just the number of tokens. If they differ, it still scores the continuation
text but skips API logprob comparisons for that case. Alternatives containing
Unicode replacement characters are excluded because the original token bytes
may have been lost by the provider.
## 2. Collect Official Continuations
The Flash 0731 fixture is historical. The official API now redirects the old
Flash names to V4.1; do not overwrite `data/flash` with new calls. Collect
current V4.1 references in a separate directory:
```sh
export DEEPSEEK_API_KEY=...
python3 gguf-tools/quality-testing/collect_official.py \
--model deepseek-flash \
--endpoint https://api.deepseek.com/chat/completions \
--prompts gguf-tools/quality-testing/prompts.jsonl \
--out /tmp/deepseek-v4.1-flash-new \
--count 100 \
--max-tokens 24 \
--top-logprobs 20 \
--thinking disabled \
--reasoning-effort omit
```
For GLM 5.2 through OpenRouter:
```sh
export OPENROUTER_API_KEY=...
python3 gguf-tools/quality-testing/collect_official.py \
--model z-ai/glm-5.2 \
--endpoint https://openrouter.ai/api/v1/chat/completions \
--api-key-env OPENROUTER_API_KEY \
--prompts gguf-tools/quality-testing/prompts.jsonl \
--out gguf-tools/quality-testing/data/glm52-openrouter \
--count 100 \
--max-tokens 24 \
--top-logprobs 20 \
--token-limit-field max_tokens \
--provider-order parasail/fp8 \
--require-parameters \
--thinking omit \
--reasoning-effort none
```
For GLM 5.3 Flash through the pinned Z.AI FP8 endpoint:
```sh
export OPENROUTER_API_KEY=...
python3 gguf-tools/quality-testing/collect_official.py \
--model z-ai/glm-5.3-flash \
--endpoint https://openrouter.ai/api/v1/chat/completions \
--api-key-env OPENROUTER_API_KEY \
--prompts gguf-tools/quality-testing/prompts.jsonl \
--out gguf-tools/quality-testing/data/glm53-flash-openrouter-zai-fp8-100 \
--count 100 \
--max-tokens 128 \
--top-logprobs 0 \
--token-limit-field max_tokens \
--provider-order z-ai/fp8 \
--thinking omit \
--reasoning-effort low
```
Use one output directory per checkpoint. For PRO 0813 through the official
DeepSeek API:
```sh
python3 gguf-tools/quality-testing/collect_official.py \
--model deepseek-v4-pro \
--prompts gguf-tools/quality-testing/prompts.jsonl \
--out gguf-tools/quality-testing/data/pro-0813 \
--count 100 \
--max-tokens 24 \
--top-logprobs 20
```
The script writes:
- `data//prompts/case_*.txt`
- `data//continuations/case_*.txt`
- `data//responses/case_*.json`
- `data//manifest.tsv`
The prompt list is tracked in `prompts.jsonl`. Curated fixture directories are
also tracked after review; ad-hoc API collection directories should stay
untracked until they are intentionally promoted into the release QA set.
Use `--system TEXT` when the reference needs an explicit system message.
`--system ''` sends an empty message, which is different from omitting it:
some providers insert a default system prompt when none is supplied. The
collector records this setting and rejects a changed system message on resume.
For a nonempty system message, render the matching local chat prompt and use
`--rendered-prompt`; the scorer's ordinary prompt mode adds no system message.
For Qwen, use the following settings in a new output directory:
```sh
python3 gguf-tools/quality-testing/collect_official.py \
--out /tmp/qwen-official \
--model qwen/qwen3.8-flash \
--endpoint https://openrouter.ai/api/v1/chat/completions \
--provider-order alibaba --require-parameters \
--require-response-provider Alibaba --require-response-model qwen/qwen3.8-flash \
--system '' --thinking omit --reasoning-effort none \
--count 100 --max-tokens 64 --top-logprobs 5
```
This reads `OPENROUTER_API_KEY` from the environment and disables provider
fallbacks. Omitting `--system ''` caused Alibaba to add 13 prompt tokens in
the September 2026 checks. The empty message matches Qwen's published
no-thinking template and the scorer's ordinary prompt mode.
Alibaba does not disclose this endpoint's precision. The
[official model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) describes
hosted Flash as based on Flash Next, not as a byte-identical checkpoint.
Agreement with this service is an external quality check, not proof of exact
native-weight inference.
## 3. Build The Local Scorer
```sh
make -C gguf-tools quality-score
```
The scorer links against the DS4 runtime, using Metal on macOS and CUDA on
Linux. On a DGX Spark, pass `CUDA_ARCH=sm_121` to the build command.
Build the optional llama.cpp control scorer with:
```sh
make -C gguf-tools quality-llama-score
```
## 4. Score GGUF Variants
```sh
gguf-tools/quality-testing/score_official \
../deepseek-v4-quants/gguf/OLD.gguf \
gguf-tools/quality-testing/data/pro/manifest.tsv \
/tmp/old.tsv \
4096
gguf-tools/quality-testing/score_official \
../deepseek-v4-quants/gguf/NEW.gguf \
gguf-tools/quality-testing/data/pro/manifest.tsv \
/tmp/new.tsv \
4096
```
Use `data/flash/manifest.tsv` for Flash GGUFs and
`data/glm52-openrouter-100/manifest.tsv` for GLM 5.2 GGUFs. Use
`data/glm53-flash-openrouter-zai-fp8-100/manifest.tsv` for GLM 5.3 Flash. Use
`data/pro/manifest.tsv` for the PRO preview checkpoint and
`data/pro-0813/manifest.tsv` for PRO 0813. The scorer and comparator do not
care which model produced the manifest; the manifest path selects the
continuation set.
Add `--continued-prefill N` to process each prompt in two calls, leaving its
last `N` tokens for the second call. This checks continued-prefill quality
against the same official answers. Every prompt must contain more than `N`
tokens; otherwise the scorer fails instead of silently skipping the test.
Add `--session-batch N` (2 to 8) to score an official continuation alongside
unrelated, independently advancing sessions. The scorer rotates their row
order on every step. Compare against a run without this option using the same
manifest, model and context; budget memory for all `N` sessions.
Check session isolation separately on a dedicated Metal host:
```sh
DS4_TEST_MODEL=MODEL.gguf DS4_TEST_BATCH_ISOLATION=1 \
DS4_TEST_SESSION_COUNT=4 DS4_TEST_DECODE_STEPS=32 \
MTL_DEBUG_LAYER=1 ./tests/test_metal_session_batch
```
This requires exact logits when companion prompts and row order change, then
checks a continued prefill and resumed serial decoding. It also checks that
invalid batches leave the target unchanged. It does not replace serial/batch
quality comparisons: different arithmetic can round differently without
sessions contaminating one another. Use `DS4_TEST_PROMPT_FILE` and
`DS4_TEST_CONTEXT_SIZE` to repeat at longer prefixes. For physical TP, the test
also accepts `DS4_TEST_TP_RDMA_DEVICE` and `DS4_TEST_TP_GID_INDEX` alongside
its coordinator/worker settings. Throughput measurements must run separately
without Metal API validation.
For Metal V4.1 prefill scheduling changes, also run:
```sh
make tests/test_deepseek41_prefill
./tests/test_deepseek41_prefill --dispatch
MTL_DEBUG_LAYER=1 ./tests/test_deepseek41_prefill MODEL.gguf speed-bench/promessi_sposi.txt
```
The model test uses SSD streaming and two 128K sessions. Run it alone on a
dedicated host with at least 128 GiB RAM. It alternates small and large appends
through 113K context, checking dispatch, progress, unchanged-prefix reuse,
saved state and subsequent decoding against a control without decoder deferral.
Keep the official continuation checks too: this scheduling test does not judge
quality across different floating-point operation orders. Measure speed without
Metal API validation, separately for initial and continued prefills.
Run the same mixed-prefix test over physical RDMA on two dedicated Metal hosts.
Start the worker with the same model and a 131072-token context, then the test
on the coordinator (replace the host and RDMA device names):
```sh
# Worker
MTL_DEBUG_LAYER=1 ./ds4 -m MODEL.gguf --ctx 131072 \
--tensor-parallel --role worker --coordinator COORDINATOR 19455 \
--transport rdma --rdma-device WORKER_DEVICE --rdma-gid-index 1
# Coordinator
MTL_DEBUG_LAYER=1 ./tests/test_deepseek41_prefill --tensor-parallel \
MODEL.gguf speed-bench/promessi_sposi.txt COORDINATOR 19455 COORDINATOR_DEVICE 1
```
This compares equal TP prefill partitions and queued versus synchronous decode,
including full logits and cache state. TP snapshots rebuild both ranks from
tokens, so restoration is compared with an equivalent fresh replay. Also run
the official scorer in TP mode for initial and continued prompts; scheduling
parity alone does not establish model quality.
Add `--quality` to disable DS4's speed-oriented numerical shortcuts. For an
independent llama.cpp comparison of a DeepSeek V4 GGUF, use the same manifest
and the token-identical DS4 prompt renderer:
```sh
gguf-tools/quality-testing/score_llama \
/path/to/model.gguf \
gguf-tools/quality-testing/data/flash/manifest.tsv \
/tmp/llama.tsv \
4096 \
deepseek-ds4
```
GLM's hosted endpoint requires reasoning. A no-thinking local prompt does not
match a reply generated after reasoning, even at temperature zero. For the long
GLM fixtures, render the official `Reasoning Effort: Low` template and include
the returned reasoning before scoring the answer:
```sh
python3 gguf-tools/quality-testing/render_glm_references.py \
gguf-tools/quality-testing/data/glm53-flash-openrouter-zai-fp8-long
gguf-tools/quality-testing/score_official /path/to/GLM-5.3-Flash-Q2.gguf \
gguf-tools/quality-testing/data/glm53-flash-openrouter-zai-fp8-long/manifest-rendered.tsv \
/tmp/glm-long.tsv 8192 --rendered-prompt
```
`--rendered-prompt` reads the prompt file with its existing chat markers; it
does not add another chat template. The renderer can also be applied to the
older 100-case GLM 5.3 Flash fixture. Keep rendered and legacy no-thinking
scores separate: their prefixes differ. Z.AI supplies no token logprobs, so
judge these runs by continuation NLL and greedy agreement, not logprob parity.
For a full-residency vs SSD-streaming comparison, score the same model twice and
add the streaming flags to one run:
```sh
gguf-tools/quality-testing/score_official \
/path/to/model.gguf \
gguf-tools/quality-testing/data/glm52-openrouter-100/manifest.tsv \
/tmp/streaming.tsv \
4096 \
--ssd-streaming
```
## 5. Compare
```sh
python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv
```
Output fields:
- `avg_nll`: average negative log likelihood; lower is better.
- `delta_new_minus_old`: negative means the new GGUF fits the official
continuation better.
- `case_wins_new_old_ties`: per-prompt NLL wins.
- `first_token_matches`: how often the local greedy first token matches the
official first token.
- `avg_greedy_lcp`: average greedy longest common prefix against the official
continuation. This measures exact agreement, not general answer quality;
one early mismatch discards all later agreement. Do not use it alone as a
quality gate, especially for sampled references. Compare paired NLL and
API probability agreement under matching execution settings as well.
- `api_target_mae`: when the manifest includes `response_file`, absolute
local-vs-API logprob delta for aligned official output tokens.
- `api_top_coverage`: fraction of API top-logprob alternatives that map exactly
to one local tokenizer token.
- `api_top1_rate`: how often the API top alternative equals the local greedy
token.
- `api_topn_recall`: fraction of mapped API top-N alternatives found in the
local top-N for the same position.
- `api_top_mae`: local-vs-API logprob MAE over mapped API top alternatives.
- `api_pair_rate`: pairwise ordering agreement among mapped API alternatives.
## 6. Validate regression artifacts
Two standard-library Python tools check saved artifacts without running inference. For matched `score_official` runs with API logprob coverage, require the complete intended manifest and finite, consistent counts and scores:
```sh
python3 gguf-tools/quality-testing/validate_scores.py \
--manifest gguf-tools/quality-testing/data/flash/manifest.tsv \
--baseline /tmp/old.tsv --candidate /tmp/new.tsv --strict-identical > /tmp/scores.json
```
Exit 0 with `--strict-identical` requires every per-case value to match; omit it to report valid differences without deciding their acceptability. Unlike `compare_scores.py`, validation rejects missing/duplicate cases, malformed fields and changed coverage denominators. This stricter mode requires the current four-column manifest and scorer TSV format with nonzero API coverage; use the existing comparator for continuation-only references without API logprobs.
Compare complete vocabulary dumps from `ds4-bench --dump-frontier-logits-dir`:
```sh
python3 gguf-tools/quality-testing/compare_frontier_logits.py \
/tmp/old-logits /tmp/new-logits --frontiers 2048 4096 --ctx 8192 \
--model /models/model.gguf --backend rocm --quality false --quant-bits 2 --vocab 129280 \
--output /tmp/frontiers.json
```
Supply the exact expected metadata: `ctx` is allocated capacity; each frontier is total context, and prefill is its increase from the previous frontier (initially zero). Each directory must contain exactly the requested dumps. Exit 0 requires complete finite float32 bit identity; exit 1 reports drift, invalid artifacts or an operational error. JSON distinguishes `identical`, `drift` and `invalid`, with full-vector hashes, differing counts and maximum absolute differences. The output path must be new. The parser follows the writer's nine-significant-digit float format and preserves signed zero.
Matching argmax is insufficient: a measured 8K–64K comparison retained all four argmax IDs while full vectors differed, with maximum absolute difference 7.45228. This is numerical drift, not by itself proof of worse generation quality. Record prompt/weight hashes, source/build identity, settings and successful run completion separately; these tools cannot recover that provenance from score tables or dumps, and do not replace model-quality checks.
Run the host-only positive and negative controls:
```sh
python3 -m unittest discover -s gguf-tools/quality-testing/tests -v
```
## 7. Qwen3.8 Spark Baseline
Measured on 2026-09-15, using one DGX Spark per quant. The short set has 100
cases and 5,696 target tokens; the long set has 12 cases and 766 target tokens,
with prefixes from 1,971 to 23,985 tokens. These are ordinary decoding and
prefill checks, without MTP or vision.
| Suite | GGUF | API top-token agreement | NLL, default | NLL, `--quality` |
| --- | --- | ---: | ---: | ---: |
| Short | Q2 | 88.90% | 0.352980 | 0.352915 |
| Short | Q4 | 92.08% | 0.290297 | 0.290538 |
| Long | Q2 | 93.99% | 0.154199 | 0.154106 |
| Long | Q4 | 97.26% | 0.124611 | 0.125546 |
Agreement is measured after feeding the same official continuation prefix,
not as a percentage of correct answers. The two short cases with damaged
emoji metadata contribute to NLL but not API agreement, which covers 5,568
short-set tokens. All 766 long-set tokens align. Every local prompt and target
token ID was also checked against Qwen's published tokenizer.
The default fast path shows no material loss relative to `--quality` in these
sets. The paired case-bootstrap intervals include zero for both quants and
both suites. Q4 is closer to the hosted reference than Q2 on average.
These measurements do not prove identical native weights or correctness at
every context length.
Leaving the last 1 or 256 prompt tokens for a separate continued prefill
changed aggregate NLL by less than 0.0008 for either quant. Independent CLI
answers at about 2K and 24K context also retrieved the correct archive owner
and revision, and gave the correct clamp outputs and boundary tests, for
both Q2 and Q4.
Raw per-case scores, weight/source hashes and comparison details are retained
in [short results](data/qwen38-flash-alibaba-100/cuda-spark-20260915/results.json)
and [long results](data/qwen38-flash-alibaba-long/cuda-spark-20260915/results.json).
---
### Overview (`speed-bench/qwen38-checkpoints/README.md`)
# Qwen checkpoint fixes: benchmark comparison
Measured on 2026-09-13 using an Apple M3 Ultra with 512 GiB RAM. The baseline
is PR head `91095b7`; the fixed runtime was tested at `d8ba365`, before these
charts were added to the squashed commit. Both builds used the same
`Qwen3.8-Flash-Next-IQ2XXSImatrix-Q2KDownPad768-MTP.gguf` model and
`Qwen3.8-Flash-Next-PLE-Q4_1.gguf` sidecar.
Both executables were warmed before measurement. Each case ran in baseline,
fixed, fixed, baseline order with no concurrent model workloads. Charts show
medians of two runs per build; [results.csv](results.csv) retains every run.
All throughput differences were below 0.7%. Generated text was byte-identical
across both builds and repeats in all six generation cases. These short local
runs do not establish statistical equivalence.
Each context run prefills the complete prefix from a fresh session, with
8,192-token chunks, then evaluates 128 teacher-forced decode tokens. The shared
prompt was a frozen copy of `ds4.c` from `91095b7`. Context allocation was the
prefix length plus 257 tokens. The 32K case includes intermediate chunks where
the fix now computes valid checkpoint logits.
Generation uses an 8,192-token context, temperature zero and thinking disabled.
MTP runs use `--mtp --mtp-timing`. The prompts and token limits are:
| Case | Limit | Prompt |
|---|---:|---|
| Hamlet | 120 | Write a three-sentence summary of the plot of Hamlet. |
| Fibonacci | 400 | List the first 30 Fibonacci numbers with indices. |
| Networking | 256 | Explain how a computer sends a web request over TCP and receives the response. Write four clear paragraphs for a programmer learning networking. |
Regenerate the SVG charts from the CSV:
```sh
uv run --with matplotlib python speed-bench/qwen38-checkpoints/plot.py
```
Add `--png-dir /path/to/output` to export PNG copies too. The figures use
zero-based axes and separate prefill and decode panels.
---
### Overview (`tests/test-vectors/README.md`)
# DeepSeek V4 Flash Test Vectors
These vectors were captured from the official DeepSeek V4 Flash API using
`deepseek-v4-flash`, greedy decoding, thinking disabled, and
`top_logprobs=20`. The hosted API does not expose full logits, so these files
store the best logprob slice the API provides.
Flash checkpoints are not interchangeable for this test. Each checkpoint has
its own directory:
- `flash-0731/`: current July 31 checkpoint and the default used by
`ds4_test`.
- `flash-pre-0731/`: historical vectors and local golden from the checkpoint
served before July 31.
A GGUF whose name contains `0731` must use `flash-0731`. The older undated
GGUF uses `flash-pre-0731`. A mismatch between checkpoint and fixture is an
invalid test, not a model-quality result.
Files:
- `flash-CHECKPOINT/prompts/*.txt`: exact user prompts.
- `flash-CHECKPOINT/official/*.official.json`: official API continuations and
top-logprobs.
- `flash-CHECKPOINT/official.vec`: compact C-test fixture generated from the
official JSON.
- `flash-CHECKPOINT/local-golden.vec`: local top-k/logit fixture captured from
the matching GGUF. It catches substantial backend drift that can keep the
same greedy token while damaging the logits distribution.
Regenerate official vectors:
```sh
DEEPSEEK_API_KEY=... ./tests/test-vectors/fetch_official_vectors.py \
--checkpoint 0731
```
The checkpoint label is required. By default the fetcher creates
`tests/test-vectors/flash-CHECKPOINT`; a future API checkpoint must go into a
new directory instead of replacing an existing fixture. Confirm which
checkpoint the API is serving before assigning the label. Running the fetcher
without `--only` also regenerates `official.vec`.
The C runner defaults to the 0731 official and local-golden fixtures:
```sh
./ds4_test --logprob-vectors
./ds4_test --local-golden-vectors
```
The historical checkpoint remains available explicitly:
```sh
DS4_TEST_MODEL=/path/to/pre-0731.gguf \
DS4_TEST_VECTOR_FILE=tests/test-vectors/flash-pre-0731/official.vec \
./ds4_test --logprob-vectors
DS4_TEST_MODEL=/path/to/pre-0731.gguf \
DS4_TEST_LOCAL_GOLDEN_FILE=tests/test-vectors/flash-pre-0731/local-golden.vec \
./ds4_test --local-golden-vectors
```
GLM 5.2 OpenRouter vectors are kept in a separate directory:
```sh
OPENROUTER_API_KEY=... ./tests/test-vectors/fetch_openrouter_glm_vectors.py
DS4_TEST_MODEL=models/GLM-5.2-UD-Q4_K_XL.gguf \
DS4_TEST_VECTOR_FILE=tests/test-vectors/glm-openrouter/official.vec \
./ds4_test --logprob-vectors
```
The same fetcher also writes `tests/test-vectors/glm-openrouter/manifest.tsv`
for `gguf-tools/quality-testing/score_official`. By default it routes to
OpenRouter `parasail/fp8` with strict parameter matching so top-logprob slices
are present in the fixture.
The Metal SSD-streaming cache-pressure repro for issue #384 is a focused
variant of the official-vector check. It forces a 16GiB routed-expert cache and
runs only the `short_code_completion` case that exposes wrong logits when
layer-batched decode reuses expert-cache buffers before the command buffer has
completed:
```sh
DS4_TEST_MODEL=gguf/DeepSeek-V4-Flash-0731-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
./ds4_test --metal-ssd-streaming-cache-pressure
```
The runner opens the normal non-quality path with accelerator-specific fast
routes disabled and pins `DS4_METAL_PREFILL_CHUNK=2048` for this strict
official-vector check.
`official.vec` is intentionally trivial to parse from C: each case points to a
prompt file and each expected token is hex-encoded by bytes. The official JSON
files remain in the tree so the compact fixture can be audited against the API
response.
To inspect a local top-logprob dump manually:
```sh
./ds4 --metal --nothink -sys "" --temp 0 -n 4 --ctx 16384 \
--prompt-file tests/test-vectors/flash-0731/prompts/long_code_audit.txt \
--dump-logprobs /tmp/long_code_audit.ds4.json \
--logprobs-top-k 20
```
---
### Overview (`gguf-tools/imatrix/dataset/README.md`)
# DS4 Imatrix Calibration Dataset
This directory contains DS4-rendered chat prompts for collecting activation
statistics before building new low-bit GGUF files.
Run:
```sh
python3 gguf-tools/imatrix/dataset/build_ds4_imatrix_dataset.py
```
Generated files:
- `prompts.jsonl`: structured records with messages and rendered prompt text.
- `rendered_prompts.txt`: all rendered prompts, separated by visible markers.
- `rendered_prompts_nothink.txt`: only prompts ending with ``.
- `rendered_prompts_think.txt`: only prompts ending with ``.
- `manifest.json`: counts, byte totals, and rough token estimate.
The renderer mirrors the server prompt shape:
```text
<｜begin▁of▁sentence｜>system<｜User｜>...<｜Assistant｜>
<｜begin▁of▁sentence｜>system<｜User｜>...<｜Assistant｜>
```
Some records include DSML tool schemas, sampled DSML tool calls, and tool-result
turns so the imatrix sees the same special-token patterns used by agent clients.
The corpus is provider-neutral and also includes language/prose rewriting,
summarization, copy-editing, extraction, multilingual translation, programming
prompts, Bash scripting, algorithm recall, `ds4-eval` benchmark-reasoning
prompts, long-context code synthesis, agent transcript replay, log diagnosis,
prose fact recovery, delayed-constraint and small needle tasks, Metal/C code
review tasks, and inference-specific debugging tasks.
For normal imatrix collection, use `rendered_prompts.txt` so calibration covers
both thinking and non-thinking modes. Split files are provided for ablations.
---
### Overview (`gguf-tools/quality-testing/data/README.md`)
# Official Quality Fixtures
This directory contains curated hosted-model continuation fixtures that are
safe to commit and use in release QA.
- `glm52-openrouter-100`: 100 GLM 5.2 OpenRouter continuations with API
top-logprob slices.
- `glm53-flash-openrouter-zai-fp8-100`: 100 deterministic GLM 5.3 Flash
continuations from OpenRouter's pinned Z.AI FP8 endpoint. That endpoint did
not return logprobs.
- `flash`: 100 DeepSeek V4 Flash 0731 continuations from the official DeepSeek
API, with API top-logprob slices.
- `pro`: 100 DeepSeek V4 PRO preview continuations with API top-logprob slices.
- `pro-0813`: 100 DeepSeek V4 PRO 0813 continuations with API top-logprob
slices.
- `qwen38-flash-alibaba-100`: 100 Qwen3.8 Flash continuations from Alibaba,
pinned through OpenRouter, with top-five logprobs and no thinking.
- `qwen38-flash-alibaba-long`: 12 continuations from the same endpoint, with
archive and code prompts from 2K to 24K tokens, crossing sparse attention.
Each fixture directory contains:
- `prompts/case_*.txt`: exact user prompts.
- `continuations/case_*.txt`: deterministic hosted-model continuations.
- `responses/case_*.json`: hosted responses, including logprob slices.
Some curated sets omit unrelated response IDs and billing fields.
- `manifest.tsv`: paths consumed by `score_official`.
DeepSeek V4 Flash smoke vectors are also tracked in `tests/test-vectors/` and
are run by `./ds4_test --logprob-vectors`.
Qwen's `collection.json` records the explicit empty system message, provider,
template hashes and token-alignment checks. All prompt lengths match the
published tokenizer. Short cases 058 and 069 contain damaged emoji bytes in
the API logprob metadata: keep their text likelihood, but exclude their API
agreement. `manifest-aligned.tsv` selects the cases with intact token bytes.
Alibaba does not disclose this endpoint's precision, and hosted Flash is
based on Flash Next rather than guaranteed to use identical checkpoint weights.
---
### Overview (`gguf-tools/quality-testing/deepseek-v4.1-flash-20260910-batch/README.md`)
# DeepSeek V4.1 Flash batched-prefill continuations
Twelve held-out prompts with approximately 384, 768, 1536 and 4096 tokens,
covering C bounds checking, network-server design and Italian prose. The
archive background varies in length; each size is tested with all three tasks.
These prompts exercise matrix-prefill and sparse attention, unlike the much
shorter prompts in the general set.
Collected from the official DeepSeek API on 2026-09-10 using `deepseek-flash`,
thinking disabled, temperature 1, up to 64 output tokens and top-20 logprobs.
All responses have fingerprint `aeb56401ca74e127821c4f9126dcb669`, matching the
other V4.1 sets. The reference weight revision is
`df42c109f1defefcbfcedbe7d905718a12266e40`.
These are sampled continuations, not greedy answers or full-vocabulary API
logits. Compare teacher-forced likelihood and API token probabilities. Retain
the separate sparse-boundary, snapshot, cancellation and coding-session tests.
```sh
./gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/deepseek-v4.1-flash-20260910-batch/manifest.tsv \
scores.tsv 6144 --ssd-streaming --ssd-streaming-cache-experts 64gb
```
---
### Overview (`gguf-tools/quality-testing/deepseek-v4.1-flash-20260910-general/README.md`)
# DeepSeek V4.1 Flash general continuations
The 100 prompts in `../prompts.jsonl`, collected from the official DeepSeek API
on 2026-09-10 using `deepseek-flash`. The provider's model table identified this
as V4.1 Flash. All responses have fingerprint
`aeb56401ca74e127821c4f9126dcb669`, matching the
[sparse-boundary set](../deepseek-v4.1-flash-20260910/README.md).
Reference weights: `deepseek-ai/DeepSeek-V4.1-Flash`, revision
`df42c109f1defefcbfcedbe7d905718a12266e40`.
Thinking is disabled, temperature is 1, and each response has at most 32 tokens
with top-20 logprobs. The set contains 2994 API output tokens. These are sampled
continuations, not greedy continuations or full-vocabulary logits. Use
teacher-forced likelihood and API top-token agreement, not greedy prefix length
alone. `collection.json` records the request settings.
```sh
./gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/deepseek-v4.1-flash-20260910-general/manifest.tsv \
scores.tsv 4096 --ssd-streaming --ssd-streaming-cache-experts 64gb
```
Use this set alongside the sparse-boundary cases when comparing quantizations.
It broadens the short-prompt comparison; it does not replace long-context,
session-state or actual coding-task tests. None of these prompts occurs verbatim
in the 8192-token corpus used to calibrate the first V4.1 mixed-Q2 candidate.
---
### Overview (`gguf-tools/quality-testing/deepseek-v4.1-flash-20260910/README.md`)
# DeepSeek V4.1 Flash continuations
Collected from the official DeepSeek API on 2026-09-10, using `deepseek-flash`.
The [provider's model table](https://api-docs.deepseek.com/quick_start/pricing)
identified this as DeepSeek-V4.1-Flash. These are not V4 Flash vectors.
Reference weights: `deepseek-ai/DeepSeek-V4.1-Flash`, revision
`df42c109f1defefcbfcedbe7d905718a12266e40`.
Thinking is disabled. Temperature is **1**, because this endpoint reports
post-temperature probabilities: at temperature zero the logprobs collapse to
`0` and `-9999`. Responses retain the returned top-20 logprobs and fingerprint.
These are sampled continuations, not greedy continuations or full logits.
Use teacher-forced NLL and API top-token agreement; a short greedy prefix alone
is not a quality failure for this set.
Cases 000-004 cover prose, C, arithmetic and Italian. Cases 005-016 retrieve a
warehouse number from an archive at these exact rendered prompt lengths:
127/128/129, 511/512/513, 1023/1024/1025 and 16383/16384/16385 tokens.
The expected retrieval answer is `074`. This crosses the sliding-window,
compressed-key selection and hierarchical candidate boundaries.
Run from the repository root on a supported Metal host:
```sh
./gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/deepseek-v4.1-flash-20260910/manifest.tsv \
scores.tsv 17408 --ssd-streaming --ssd-streaming-cache-experts 32gb
```
Add `--max-cases 5` for the short subset. The hosted implementation uses bounded
decoder replay, which the technical report describes as approximate; API scores
complement, rather than replace, inference-graph and session-state tests.
---
### Overview (`gguf-tools/quality-testing/deepseek-v4.1-flash-20260911-extended/README.md`)
# DeepSeek V4.1 Flash extended-context continuations
Six held-out prompts at approximately 64k and 96k tokens, covering C bounds
checking, network-server design and Italian prose. Use alongside the short,
sparse-boundary and 8k/16k/32k sets.
Collected September 11 from the official DeepSeek API using `deepseek-flash`,
thinking disabled, temperature 1, 64 output tokens and top-20 logprobs.
Fingerprint `aeb56401ca74e127821c4f9126dcb669` matches the earlier V4.1 sets
and weight revision `df42c109f1defefcbfcedbe7d905718a12266e40`.
These are sampled continuations, not greedy answers or full-vocabulary API
logits. Compare teacher-forced likelihood and API probabilities. Also test
continued prefills, restored sessions and real coding tasks separately.
```sh
./gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/deepseek-v4.1-flash-20260911-extended/manifest.tsv \
scores.tsv 102400 --ssd-streaming --ssd-streaming-cache-experts 64gb
```
---
### Overview (`gguf-tools/quality-testing/deepseek-v4.1-flash-20260911-long/README.md`)
# DeepSeek V4.1 Flash long-prefill continuations
Nine held-out prompts at approximately 8k, 16k and 32k tokens. Each length
ends with one of three tasks: C bounds checking, network-server design or
Italian prose. Use these alongside the shorter batch and sparse-boundary sets.
Collected on September 11 locally (September 10 at 23:11 UTC) from the
official DeepSeek API using `deepseek-flash`, thinking
disabled, temperature 1, 64 output tokens and top-20 logprobs. The fingerprint
is `aeb56401ca74e127821c4f9126dcb669`, matching the other V4.1 sets and weight
revision `df42c109f1defefcbfcedbe7d905718a12266e40`.
These are sampled continuations, not greedy answers or full-vocabulary API
logits. Compare teacher-forced likelihood and API probabilities. Also check
continued prefills, restored sessions and real coding tasks.
```sh
./gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/deepseek-v4.1-flash-20260911-long/manifest.tsv \
scores.tsv 34816 --ssd-streaming --ssd-streaming-cache-experts 64gb
```
---
### Overview (`gguf-tools/quality-testing/deepseek-v4.1-flash-20260919-router/README.md`)
# DeepSeek V4.1 Flash router comparison
112 held-out prompts collected from the official DeepSeek API on September
19, 2026: 100 short prompts and 12 longer archive, C audit and Italian tasks.
API prompt lengths range from 16 to 23,454 tokens. Responses contain 6,922
output tokens in total. The prompts were fixed before examining local scores;
they do not reuse the earlier 100-prompt Flash set.
Model: `deepseek-flash`, identified as DeepSeek-V4.1-Flash by the
[official model table](https://api-docs.deepseek.com/quick_start/pricing/).
At collection time the old V4 Flash and Vision-Exp API names also redirected
to V4.1. Do not use these references to judge those older checkpoints.
All responses report fingerprint `aeb56401ca74e127821c4f9126dcb669`, also
present in the earlier V4.1 fixtures.
Thinking is disabled; no system message is sent. Temperature is 1, the output
limit is 64, and top-20 logprobs are retained. `top_p` uses the provider's
default of 1. These are sampled continuations, not greedy answers or complete
vocabulary logits. Per-case settings, lengths, groups and prompt hashes are
in `collection.json`; raw responses are retained.
## Scoring
On a dedicated Metal machine with at least 128 GiB RAM:
```sh
./gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/deepseek-v4.1-flash-20260919-router/manifest.tsv \
/tmp/router-scores.tsv 34816 \
--ssd-streaming --ssd-streaming-cache-experts 48gb
```
`manifest-short.tsv` and `manifest-long.tsv` select the two groups. Compare
matched weights and settings, checking API/local token alignment before
interpreting probability metrics. Changing `--continued-prefill` or Metal
math mode is a separate numerical configuration, not an identical rerun.
Teacher-forced NLL measures the probability assigned to the reference tokens.
API top-token agreement compares each position after feeding the same prefix.
LCP is only an exact-prefix diagnostic here: the reference tokens were sampled.
`case_047` exposes a tokenizer discrepancy in the tested source: DwarfStar
encodes its continuation as 65 tokens, while both the API and DeepSeek's
published tokenizer give 64. The text contains curly quotation marks. Keep
the case as a reproducer, but exclude it from API probability comparisons;
the scorer does this automatically. Text NLL still includes it. The remaining
111 continuations align with the API, covering 6,858 positions. Do not treat
the discrepancy as a router effect or a damaged API response.
The router experiment also uses an isolated adaptation of Emilian Bold's
[overlap scorer](https://github.com/emilianbold/ds4/commit/f3e16a19d37f9c17d2f42869474742e9993c7098).
It sums `min(p_local, p_api)` over uniquely mapped API alternatives, without
renormalizing the truncated list. This is a lower bound on full distribution
overlap. The reported API probability mass bounds the missing contribution;
unmapped tokens are not treated as zero-probability API tokens. The comparison
used temporary shader overrides and a private scorer build, without changing
production source during the experiment.
## Router Comparison
Tested `aafc65b444f` against the two Metal shader changes from PR #1044,
using `DeepSeek-V4.1-Flash-IQ2_XXS-Q2_K-imatrix.gguf` on M5 Max. The full
112-prompt pair used a 48 GiB expert cache. Two additional pairs used
`manifest-sensitivity.tsv` (every fourth short prompt plus all long prompts,
37 cases), with a 64 GiB cache on the other M5 Max. Each pair kept its model,
host, context allocation and settings fixed.
| Setting | NLL, base / PR | Overlap lower bound, base / PR | API top-token agreement, base / PR | LCP, base / PR |
| --- | --- | --- | --- | --- |
| Default, 112 prompts | 0.396840 / 0.395353 | 0.907623 / 0.907639 | 90.347% / 90.245% | 5.563 / 6.071 |
| Continued prefill, last token separate | 0.301148 / 0.300111 | 0.924795 / 0.924768 | 91.876% / 91.697% | 9.568 / 9.811 |
| Safe Metal math | 0.306108 / 0.305066 | 0.924984 / 0.924797 | 91.697% / 91.921% | 9.270 / 8.270 |
The overall paired 95% prompt-bootstrap intervals include zero for NLL,
overlap and top-token agreement in all three settings. For the full pair,
the NLL difference is -0.001488, interval [-0.003328, +0.000199]; overlap
changes by +0.000015, interval [-0.000375, +0.000427]. This does not show a
clear regression, nor establish universal equivalence. No equivalence margin
was specified in advance.
LCP moves more sharply than the probability metrics. In safe math, just two
prompts account for its entire decline: case 096 changes from 0 to 13 and
case 102 from 64 to 14. That is not evidence of a one-token loss in general
answer quality.
Long-prompt overlap improves slightly in each setting; default long-prompt
NLL worsens by 0.000195 while the other two improve. These 12 prompts reuse
three task families, so their bootstrap intervals are only descriptive.
These results do not settle the earlier Vision-Exp checkpoint comparison.
Per-case score tables, probability bounds, paired intervals and source
provenance are in [results/comparison.json](results/comparison.json). The
full baseline was repeated at 48 and 64 GiB cache budgets with byte-identical
scores and per-position traces. All reported pairs completed; an interrupted
earlier candidate run was excluded. Trace arithmetic was independently
checked against the scorer totals.
---
### Overview (`tests/vision-fixtures/glm53/README.md`)
# GLM 5.3 vision fixtures
These small images test facts that a plausible generic description cannot
cover: exact OCR, diagram order, spatial relations, screenshot diagnostics,
photograph recognition, and rejection of an unrelated premise. The expected
facts live in `cases.json`.
Run a local `ds4-server` with the GLM 5.3 Flash text GGUF and vision sidecar,
then use:
```sh
python3 tests/run_glm53_vision_quality.py
```
To compare with the official Z.AI provider through OpenRouter:
```sh
DS4_VISION_ENDPOINT=https://openrouter.ai/api/v1/chat/completions \
DS4_VISION_OFFICIAL_ZAI=1 OPENROUTER_API_KEY=... \
python3 tests/run_glm53_vision_quality.py
```
For `ds4-agent`, copy only the PNG/JPEG files to a temporary directory before
testing. Keeping vector sources or expected-answer files beside the images lets
an agent bypass visual inspection with its text tools.
`earth.jpg` is NASA's public-domain Apollo 17 image AS17-148-22727, downloaded
from Wikimedia Commons:
https://commons.wikimedia.org/wiki/File:Earth_apollo17.jpg
---
### Overview (`tests/vision-fixtures/jpeg/README.md`)
# Progressive JPEG AC-refinement ZRL fixtures
These images exercise successive-approximation AC refinement (`Ss=1, Se=63,
Ah=2, Al=1`) with restart markers. The unpatched Iris decoder mishandles ZRL
(`run=15, size=0`) in that scan: it keeps consuming correction bits after the
16th currently-zero coefficient.
Generated with libjpeg-turbo 3.2.0 `cjpeg`. Re-run `./generate.sh` if you have
`cjpeg` on `PATH`.
| File | Size | Why it is here |
| --- | --- | --- |
| `prog_ac_refine_zrl_gray.jpg` | 24x16 grayscale | Unpatched decode succeeds but pixels differ from libjpeg-turbo. Patched decode is bit-exact with `djpeg`. |
| `prog_ac_refine_zrl_420.jpg` | 64x48 YCbCr 4:2:0 | Unpatched `jpeg_load` returns NULL. Patched decode is bit-exact with `djpeg`. |
`make test` covers both through `tests/test_image_decode`.
---
### Overview (`tests/vision-fixtures/qwen38/README.md`)
# Qwen3.8 vision parity fixtures
Two synthetic 640 x 480 images created locally for the CLI and encoder checks:
- `orbit.png`: ORBIT 4729 above a red square on white.
- `maple.png`: MAPLE 8153 above a blue circle on white.
Their dimensions require no resizing at the default 64–1024 token limits.
The ORBIT fixture exposed Q8-versus-original checkpoint drift below the 0.99
minimum per-token cosine threshold, while the same-weight Metal/HF comparison
passed. Keep this fixture when changing quantized reference handling.
The documented suite also uses the existing GLM fixtures for OCR, diagrams,
photography and resize coverage. These are numerical comparisons; the metric
does not grade generated image descriptions.
---
### Overview (`gguf-tools/quality-testing/data/pro-0813/README.md`)
# DeepSeek V4 Pro 0813 Fixture
This fixture contains 100 official DeepSeek API continuations for the August
2026 DeepSeek V4 Pro checkpoint. It was collected between
2026-08-12T23:06:53Z and 2026-08-12T23:09:33Z while the official API pricing
page identified the served model as `DeepSeek-V4-Pro-0813`.
The API request used the public model alias `deepseek-v4-pro`. Every response
reports this system fingerprint:
```text
fp_v4pro_20260812_prod0820_fp8_kvcache_20260402
```
The older preview fixture in `../pro` reports
`fp_9954b31ca7_prod0820_fp8_kvcache_20260402`. Keep the two fixtures separate
and score each GGUF against the fixture for its own checkpoint.
Collection parameters:
- 100 prompts from `gguf-tools/quality-testing/prompts.jsonl`;
- at most 24 output tokens per prompt;
- top-20 API log probabilities for every returned token;
- thinking disabled and reasoning effort omitted.
The exact command was:
```sh
DEEPSEEK_API_KEY=... \
python3 gguf-tools/quality-testing/collect_official.py \
--model deepseek-v4-pro \
--endpoint https://api.deepseek.com/chat/completions \
--prompts gguf-tools/quality-testing/prompts.jsonl \
--out gguf-tools/quality-testing/data/pro-0813 \
--count 100 \
--max-tokens 24 \
--top-logprobs 20 \
--thinking disabled \
--reasoning-effort omit
```
Ninety-one responses reached 24 tokens. Nine ended normally before the token
limit. The complete fixture contains 2,275 output tokens.
---
### CLIENTS (`docs/CLIENTS.md`)
# Coding Agent Clients
[README](../README.md) | [Server](SERVER.md)
Start the server first:
```sh
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
```
Set the client's context limit no higher than the server's. Output tokens also
consume that context; a client limit does not enlarge the server allocation.
The examples use Flash and localhost. Change the address for a trusted remote
server. The `dsv4-local` strings are placeholders, not server authentication.
## Pi
Add this provider to `~/.pi/agent/models.json`:
```json
{
"providers": {
"ds4": {
"baseUrl": "http://127.0.0.1:8000/v1",
"api": "openai-completions",
"apiKey": "dsv4-local",
"compat": {
"supportsStore": false,
"supportsDeveloperRole": false,
"supportsReasoningEffort": true,
"supportsUsageInStreaming": true,
"maxTokensField": "max_tokens",
"supportsStrictMode": false,
"thinkingFormat": "deepseek",
"requiresReasoningContentOnAssistantMessages": true
},
"models": [{
"id": "deepseek-v4-flash",
"name": "DwarfStar Flash",
"reasoning": true,
"thinkingLevelMap": {
"off": null, "minimal": "low", "low": "low",
"medium": "medium", "high": "high", "xhigh": "xhigh"
},
"input": ["text"],
"contextWindow": 100000,
"maxTokens": 16384,
"cost": {"input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0}
}]
}
}
}
```
Select `ds4/deepseek-v4-flash` in Pi. This is a text-only client configuration;
image use also requires the appropriate client input declaration and server
vision encoder.
## OpenCode
Merge the provider into `~/.config/opencode/opencode.json`:
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"ds4": {
"name": "DwarfStar",
"npm": "@ai-sdk/openai-compatible",
"options": {
"baseURL": "http://127.0.0.1:8000/v1",
"apiKey": "dsv4-local"
},
"models": {
"deepseek-v4-flash": {
"name": "DwarfStar Flash",
"limit": {"context": 100000, "output": 16384}
}
}
}
}
}
```
Select `ds4/deepseek-v4-flash` as the model.
## Codex CLI
Use the Responses API. Add a provider to the Codex configuration:
```toml
[model_providers.ds4]
name = "DwarfStar"
base_url = "http://127.0.0.1:8000/v1"
wire_api = "responses"
stream_idle_timeout_ms = 1000000
```
```sh
codex --model deepseek-v4-flash -c model_provider=ds4
```
## Claude Code
Use the Anthropic-compatible endpoint. A shell wrapper can select the local
model for the main agent and its secondary model roles:
```sh
#!/bin/sh
unset ANTHROPIC_API_KEY
export ANTHROPIC_BASE_URL="http://127.0.0.1:8000"
export ANTHROPIC_AUTH_TOKEN="dsv4-local"
export ANTHROPIC_MODEL="deepseek-v4-flash"
export ANTHROPIC_DEFAULT_SONNET_MODEL="deepseek-v4-flash"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="deepseek-v4-flash"
export ANTHROPIC_DEFAULT_OPUS_MODEL="deepseek-v4-flash"
export CLAUDE_CODE_SUBAGENT_MODEL="deepseek-v4-flash"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export CLAUDE_STREAM_IDLE_TIMEOUT_MS=600000
exec claude "$@"
```
Give the wrapper a different name from `claude`. Agent clients can send large
initial prompts, so the first prefill may take time. Disk caching helps reuse
compatible prefixes on later sessions.
---
### CUDA MULTI GPU (`docs/CUDA_MULTI_GPU.md`)
# NVIDIA CUDA GPUs
[README](../README.md) | [Getting started](../README.md#start-here)
**Ada Lovelace is supported, including 48 GB L40S cards.** This is an important
use case: a model whose other inference implementations require newer GPUs
can still run here when DwarfStar supports its GGUF layout. The Flash kernels
do not require Blackwell's native FP4 instructions; Ada uses an appropriate
CUDA path. Support remains model-specific, not a promise to run arbitrary GGUFs.
## Build
Install the NVIDIA driver and CUDA toolkit, including `nvcc` and cuBLAS.
Use the local GPU architecture, or select Ada explicitly:
```sh
make cuda-generic
# For an Ada build:
make cuda CUDA_ARCH=sm_89
```
For a GB10 machine use the [DGX Spark](DGX_SPARK.md) target instead.
## Normal layer placement
Without tensor parallelism, DwarfStar places complete layers on the selected
GPUs. `--gpu-vram auto` uses reported free VRAM, reserving space for the graph
and context. Explicit budgets are comma-separated GiB values, one per device.
```sh
./download_model.sh ds4f-q2
./ds4 --cuda --gpu-devices 0,1,2,3 --gpu-vram auto --ctx 32768
```
This is also the multi-GPU placement mode for GLM. Startup refuses a layout
that would require unsupported CPU execution; reduce context or choose a
smaller model if necessary.
## Flash tensor parallelism
`--cuda-tensor-parallel` pairs GPUs and divides routed-expert work inside
each pair. Pairs also own successive layer ranges. This is an in-process
configuration, not the network `--role coordinator` / `--role worker` mode.
The device order matters. List all layer-home GPUs first, then their partners.
For physical pairs `(0,1)`, `(2,3)`, `(4,5)`, `(6,7)`, use:
```text
0,2,4,6,1,3,5,7
```
Check your machine's topology with `nvidia-smi topo -m`; do not assume this
ordering gives the best pairs on another server. Each pair stores one half
of the routed experts per GPU. Dense attention, routers, and shared experts
are replicated within the pair; the output head is vocabulary-sharded.
For the tested eight-L40S setup:
```sh
./download_model.sh ds4f-q4
./ds4-agent --cuda --cuda-tensor-parallel \
--gpu-devices 0,2,4,6,1,3,5,7 --gpu-vram auto --ctx 100000
```
Q4 has native grouped routed kernels for multi-user throughput. Q2 needs less
memory, but unsupported grouped shapes use a slower, correct fallback.
Four 48 GB cards with Q2 and eight with Q4 are tested configurations. An even
device count alone does not guarantee memory fit.
## Serve multiple users
```sh
./ds4-server --cuda --cuda-tensor-parallel \
--gpu-devices 0,2,4,6,1,3,5,7 --gpu-vram auto \
--ctx 100000 --batched-session 16 --host 0.0.0.0
```
The supported eight-L40S Flash configuration has reached roughly 126 aggregate
generation tokens/s with 16 decode rows. This is total throughput, not the
speed of one user. The recorded conditions and regression requirements are in
[the QA guide](../QA_BEFORE_RELEASES.md).
CUDA TP defaults to 2048-token prefill chunks. Keep that default for the
16-session, 100k-context setup; larger chunks need more scratch memory.
Reduce session count or context if all KV states do not fit.
The repository's [server launcher](../run-nvidia-tp-server.sh) is an example
from the L40S deployment, not a portable default: it uses host-specific model
and cache paths and defaults to MXFP4. The commands above need no launcher or
environment tuning. See [serving](SERVER.md) for disk caches and API access.
---
### DGX SPARK (`docs/DGX_SPARK.md`)
# DGX Spark
[README](../README.md) | [Getting started](../README.md#start-here)
Use the NVIDIA driver and CUDA development
toolkit for the machine; the build needs `nvcc` and cuBLAS.
Check that `nvidia-smi` sees the GPU before building.
## Build and run
```sh
make cuda-spark
./download_model.sh ds4f-q2
./ds4 --cuda
```
The build targets GB10 (`sm_121a`) and enables the Blackwell-specific kernels.
Do not use `--cuda-tensor-parallel` on a single Spark.
Stop other inference services before loading a model so they do not compete
for memory. Restore any services you stopped when finished.
## DeepSeek V4.1 Flash
Q2 needs SSD streaming on one 128 GB Spark:
```sh
./download_model.sh ds41f-q2
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
--ssd-streaming --ctx 32768
```
The file is 341 GiB, including 189 GiB of disk-only Engram tables. Keep it on
a fast local SSD. The expert cache is sized automatically; leave memory for
other programs and the context. Use the same options with `ds4-agent` or
`ds4-server`.
Two Sparks can instead keep half the experts each, using
[network tensor parallelism](DISTRIBUTED.md#tensor-parallelism-between-two-sparks).
Both need the complete GGUF on their local SSD. Single-host V4.1
[DSpark](SPECULATIVE_DECODING.md#deepseek-v41-flash-dspark) also works with SSD
streaming; TP speculation and CUDA vision are not supported for V4.1.
The [model guide](MODELS.md#deepseek-v41-flash) covers
thinking levels and the separate Metal configurations.
September 13-14, 2026, Q2 SSD with automatic cache sizing: a 3,241-token
`/read README.md` prefill reached 93-96 t/s, up from 49-54 t/s, with 32K
allocated context. No extra flag is needed. A separate 64 GiB cache-budget
run with 64K allocated context reached 384 t/s for a 32K initial prefill and
88 t/s for a 3.2K append. These are SSD-streamed V4.1 results, not resident
V4 Flash numbers; speed depends on prompt length and the expert cache.
The automatic-cache 256-token reply test decoded about 5% slower, but
prefill plus the reply fell from about 87 to 59 seconds.
See the [QA record](../QA_BEFORE_RELEASES.md#cuda-ssd-streaming) for conditions
and longer-context measurements.
On two Sparks over the direct RoCE link, Q2 measured **21.8 t/s** for one
session on October 2, 2026: a 1K prompt, 64K allocated context and 2,048
teacher-forced decode tokens, without speculation.
| Work | Speed |
| --- | ---: |
| Fresh 32K prefill | 411 t/s |
| Add 8K after 32K | 314 t/s |
Both runs used 64K allocated context and 512 decode tokens per frontier;
decode at these longer contexts was about 18 t/s. Engram tables stayed on disk.
Decode batching starts at five ready sessions; smaller groups run in order.
The earlier eight-session reference is about **28 aggregate t/s**, not per
client. See the [network TP QA record](../QA_BEFORE_RELEASES.md#cuda-network-tensor-parallelism)
for the workloads and memory limits.
## Flash 0731 on two Sparks
[Network tensor parallelism](DISTRIBUTED.md#tensor-parallelism-between-two-sparks)
also supports Flash 0731 Q2 and MXFP4, including DSpark with the matching
support file. Each host holds about 45 GiB of Q2 weights or 77 GiB of MXFP4,
plus context and runtime buffers. Download `ds4f-mxfp4` on both for MXFP4.
October 2, 2026, Q2, direct 200 Gb/s RoCE, 64K allocated context: a 105-token
English prose prompt decoded 512 tokens at **26.0 t/s** without speculation.
A fresh 32K prefill reached **618 t/s**; appending 8K reached **610 t/s**.
Ordinary decode at those longer frontiers was about 19 t/s. These are single
measurements, not release medians.
October 4, 2026, matching 0731 DSpark support, default speculation settings,
temperature zero, 64K allocated context and 512 generated tokens. Both formats
use the drafter on both ranks over RDMA. These are medians of two runs per case
(four for MXFP4 prose), including the first token or block. Benchmark processes
ran on the performance cores (`taskset -c 5-9,15-19`):
| Weights | Prompt | DSpark |
| --- | --- | ---: |
| Q2 | English prose, 105 tokens | 44.0 t/s |
| Q2 | C hash table, 31 tokens | 83.3 t/s |
| MXFP4 | English prose, 105 tokens | 41.7 t/s |
| MXFP4 | C hash table, 31 tokens | 75.6 t/s |
On one Spark, the same Q2 prose and coding prompts reached 23.4 and 44.0 t/s
with DSpark. These are workload-specific measurements, not a promised speedup;
poor draft acceptance can still make it slower. The measured settings are
automatic once DSpark is enabled; speculation itself remains opt-in.
## Qwen3.8 Flash Next
Both Q2 and Q4 fit on one Spark. After building:
```sh
./download_model.sh qwen38-q2
./ds4 --mtp --ctx 32768
```
Q2 keeps 41.73 GiB of weights resident; Q4 keeps 69.74 GiB. Both GGUFs also
contain 95.37 GiB of BF16 n-grams, read directly from the SSD. Leave room
for context and runtime buffers. Vision, the native agent, server APIs and
text-session checkpoints work on CUDA; see [Qwen setup](QWEN38_FLASH_NEXT.md).
September 15, 2026, Q2 on one Spark: a 16K prefill reached 250 t/s, extending
it to 30K reached 243 t/s, and ordinary decode stayed around 17.3 t/s.
These are single runs with 8,192-token prefill chunks and 32 decode tokens.
A separate 256-token prose run measured 17.1 t/s ordinarily and 19.9 t/s with
MTP, at temperature zero and 8K allocated context. Speculation is opt-in;
its benefit depends on the prompt.
## GLM 5.3 Flash
Q2 is the resident target for one Spark:
```sh
./download_model.sh glm53-q2
./ds4 --cuda -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 16384
```
Q4 does not fit resident. GLM Spark-to-Spark tensor parallelism is not
implemented; the two-Mac RDMA instructions do not apply to GLM on CUDA.
## Vision and speculative decoding
DeepSeek Vision Experimental needs its matching text model and encoder:
```sh
./download_model.sh ds4f-vision-q2
./ds4 --cuda --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf
```
Use `/read image.png` in the CLI. GLM vision also works on this backend; see
[models and vision](MODELS.md#vision).
For Flash 0731, [DSpark](SPECULATIVE_DECODING.md) uses the separate 0731 support
file. Vision Experimental has a different drafter. GLM uses its built-in MTP
block with `--mtp`. None is enabled by default.
## Flash Q2 with DSpark
September 6, 2026, fully resident Flash 0731: median generation speed from
three runs of 256 tokens, 4K allocated context and a 512-token prefill chunk.
| Prompt | Temperature | Ordinary decode | DSpark |
| --- | ---: | ---: | ---: |
| C hash table | 0 | 19.72 t/s | 31.41 t/s |
| C hash table | 1 | 19.53 t/s | 29.98 t/s |
| Unpredictable prose | 1 | 19.53 t/s | 18.81 t/s |
Longer coding runs reached about 33 t/s. Poor draft acceptance can still make
DSpark slower. Temperature 1 uses the default opportunistic policy; these are
not exact-sampling results. DSpark does not accelerate prefill.
To reproduce the greedy coding case after downloading `ds4f-q2` and
`ds4f-dspark`:
```sh
./ds4 --cuda -m ds4flash.gguf \
--dspark --mtp-model gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
--ctx 4096 --prefill-chunk 512 --nothink --temp 0 --seed 12345 -n 256 \
-p 'Write a complete C hash table implementation with string keys, insert, find, delete, and a test main. Output only C code.'
```
For the temperature-1 rows, use `--temp 1 --top-p 0.95 --min-p 0.05`.
The [QA record](../QA_BEFORE_RELEASES.md#16-speed-regression) includes the previous
implementation, continued-context checks and quality comparisons.
## Larger models and serving
CUDA also has [SSD streaming](SSD_STREAMING.md) paths for larger weights.
Memory fit and speed depend on the model layout; start with Q2 for normal use.
V4.1 Q2 SSD serving batches up to eight decode rows together. An eight-session
test reached 11.0 aggregate t/s versus 8.4 with ordered execution; this is
total throughput, not speed per client. Other single-Spark model paths retain
their existing scheduling.
See [serving](SERVER.md) and the recorded [benchmarks](PERFORMANCE.md).
---
### DISTRIBUTED (`docs/DISTRIBUTED.md`)
# Inference Across Machines
[README](../README.md)
There are two modes:
| Mode | Split | Main use |
| --- | --- | --- |
| Tensor parallelism | Routed experts and per-layer work across two Macs or two Sparks | Resident inference with lower per-token work on each GPU |
| Pipeline parallelism | Complete layer ranges across several machines | Fit larger models and overlap long prefills |
These are separate from [tensor parallelism across CUDA cards](CUDA_MULTI_GPU.md).
Network protocols have no authentication or encryption. Use trusted machines
and a trusted network; run the same commit on every peer. Model paths and
artifacts must agree. Update all TP peers together when changing versions.
## Tensor parallelism between two Macs
This is a 50/50 split with exactly one worker. Do not pass `--layers`.
Routed experts are sharded; attention partitioning depends on the model. Both
GPUs work on the same token and exchange partial results. This can reduce generation
latency, but the gain depends on the model, link, and comparison setup.
Two 128 GB Macs are useful for V4 Flash Q4/MXFP4 or GLM 5.3 Flash Q4.
GLM 5.2 IQ2_XXS is another tested capacity setup. V4.1 Flash Q2 also runs
on two 128 GB Macs, with disk-only Engram tables.
A larger quant may need larger machines even though its tensor layout is supported.
### Link setup
Use a Thunderbolt cable. RDMA requires an active verbs device with an
IPv4-mapped GID; a working ping alone does not establish that.
```sh
rdma_ctl status
ibv_devinfo -v
```
Addresses must be on the cabled member interfaces, not only the Thunderbolt
bridge. For example, after checking which interfaces are active:
```sh
# Machine A, example member interface en1.
sudo ifconfig en1 inet 10.99.0.2/30 alias
# Machine B, example member interface en6.
sudo ifconfig en6 inet 10.99.0.1/30 alias
```
For the large tested shards on otherwise idle 128 GB Macs, the setup raised
the per-boot GPU wired-memory limit on both machines:
```sh
sudo sysctl iogpu.wired_limit_mb=120000
```
This is specific to that memory configuration. It grants a larger GPU budget;
it does not create more RAM. Check model, context, and system headroom before
raising a limit on your machine.
### Start the pair
Download the same model on both machines. For GLM 5.3 Flash Q4:
```sh
./download_model.sh glm53-q4
```
Start the worker first; it retries while the coordinator loads:
```sh
# Machine B.
./ds4 --tensor-parallel --role worker \
--coordinator 10.99.0.2 9911 --transport rdma --ctx 8192
# Machine A.
./ds4 --tensor-parallel --role coordinator \
--listen 10.99.0.2 9911 --transport rdma --ctx 8192
```
The verbs device and GID are selected automatically. If ambiguous, specify
`--rdma-device` and `--rdma-gid-index` from `ibv_devinfo`. Use `--transport tcp`
on both peers when RDMA is unavailable. Do not probe the waiting coordinator
with `curl` or `nc`: it may treat the connection as a worker handshake.
Keep workers running in a terminal or managed session and retain both logs.
Do not treat repeated handshake or RDMA timeouts as successful QA merely
because a retry works.
The coordinator can be `ds4`, `ds4-agent`, `ds4-server`, or `ds4-bench`;
workers run `ds4`. For models with vision support, pass the same `--vision FILE`
to both for image input.
For GLM MTP, enable `--mtp` on both. For DeepSeek DSpark, both need the
matching support model and DSpark options. V4.1 supports vision but not
speculative decoding.
TP disk-cache restore currently rebuilds the exact saved token prefix on both
ranks rather than restoring the coordinator alone. Expect prefill on restore.
See [speculation](SPECULATIVE_DECODING.md) and [serving](SERVER.md).
## Tensor parallelism between two Sparks
V4 Flash 0731 Q2/MXFP4 and V4.1 Flash Q2 text inference support one GPU per rank,
with a 50/50 expert split. Each Spark holds about 45 GiB of Flash 0731 Q2
weights, 77 GiB of Flash MXFP4, or 81 GiB for V4.1, plus context and runtime
buffers. V4.1 Engram tables stay on disk. Do not add `--ssd-streaming` or
`--cuda-tensor-parallel`: those select different memory/execution modes.
Build the same commit with `make cuda-spark` on both machines and download
`ds41f-q2` on both. RDMA needs the libibverbs development headers at build time,
its runtime library, and an active RoCEv2 link. Check `ibv_devinfo -v` and use
the addresses of the directly connected ports, not the management network.
For example, with the coordinator at `172.31.250.1` on the direct link:
```sh
# Worker.
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf --ctx 32768 \
--tensor-parallel --role worker --coordinator 172.31.250.1 9911 \
--transport rdma
# Coordinator.
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf --ctx 32768 \
--tensor-parallel --role coordinator --listen 172.31.250.1 9911 \
--transport rdma
```
The link address selects the matching verbs device and GID. If selection is
ambiguous, use `--rdma-device` and `--rdma-gid-index`. TCP is also available
with `--transport tcp` on both peers.
For Flash 0731, download `ds4f-q2` or `ds4f-mxfp4` on both hosts and use its
GGUF path in the same commands. For DSpark, download `ds4f-dspark` on both and
add `--dspark --mtp-model PATH` to both commands. With MXFP4 over RDMA, both
GPUs also share the drafter work. This needs about 2.5 GiB extra per rank,
beside the support model and context. If the worker has no drafter, or the
configuration cannot use the split, drafting stays on the coordinator.
Speed depends on the prompt.
The coordinator can be `ds4-agent`, `ds4-server` or `ds4-bench`. Match context
sizes and any `--prefill-chunk` overrides on both sides. For V4.1, five or more
ready sessions use native batched decode; smaller groups run in order.
Use `--batched-session 8` on the server and allow memory for all eight contexts.
Flash 0731 network sessions currently run in order, without native batching.
CUDA network TP does not support vision, V4.1 DSpark or other model/quant layouts.
## Pipeline parallelism
Each process maps only its assigned layers, retaining that slice of the KV
state. Layer ranges are inclusive. `N:output` includes the final layer and
output head. Activations travel from one stage to the next over TCP.
For Flash Q4 on two machines, download `ds4f-q4` on both, then start each side.
Replace the example address with your coordinator's reachable address:
```sh
# Machine A.
./ds4 --role coordinator --layers 0:19 --listen 10.99.0.2 9911
# Machine B.
./ds4 --role worker --layers 20:output --coordinator 10.99.0.2 9911
```
Normally give the output head to the final worker. With several workers,
choose non-overlapping ranges covering the entire model. Workers register
their ranges with the coordinator; intermediate workers forward activations
directly to the next stage.
Long prefill chunks can occupy different stages simultaneously. A single
generation stream cannot use that overlap: each token must finish the route
before the next one is sampled. Use pipeline mode primarily for capacity and
long-prefill throughput, not as a guaranteed decode speedup.
### Full PRO Q4
For two 512 GB Mac Studios, use the split artifacts:
```sh
# Machine A.
./download_model.sh pro-q4-layers00-30
./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf \
--role coordinator --layers 0:30 --listen 10.99.0.2 9911
# Machine B.
./download_model.sh pro-q4-layers31-output
./ds4 -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf \
--role worker --layers 31:output --coordinator 10.99.0.2 9911
```
These downloads do not change `ds4flash.gguf`. Startup is expensive because
each side must make its model slice resident.
### Tuning and recovery
Keep default chunk sizes first. `--dist-prefill-window N` controls the number
of chunks in flight; `--dist-prefill-chunk N` overrides the session-derived
chunk size. `--debug` shows route and per-hop timings.
Activations use 32-bit transport by default. `--dist-activation-bits 16` halves
the payload; `8` is more aggressive. These change numerical precision on the
wire, not weights or KV storage. Validate output when changing them.
A disconnected worker invalidates the route. In-flight work can fail; later
requests need a complete route before proceeding, and the coordinator can
replay the saved token prefix to rebuild worker state. Pipeline snapshots
serialize all layer slices into one payload and redistribute them when loaded.
For protocol details, see [ds4_distributed.c](../ds4_distributed.c)
and [ds4_tp.c](../ds4_tp.c).
---
### METAL (`docs/METAL.md`)
# Metal on Apple Silicon
[README](../README.md) | [Getting started](../README.md#start-here)
## Build and run
Install Apple's command-line developer tools if needed:
```sh
xcode-select --install
```
From the repository root:
```sh
make
./download_model.sh ds4f-q2
./ds4
```
The same build supports M3 and M5 Macs. Hardware-specific fast paths are
selected automatically; no environment variable is needed to enable them.
Leave other GPU and memory-heavy applications idle when comparing performance.
## Choose a model
| Memory | Starting point |
| --- | --- |
| 64 GB | Flash Q2 with `--ssd-streaming` |
| 96 GB | Flash Q2; leave room for the context and other applications |
| 128 GB | Flash Q2, or GLM 5.3 Flash Q2 with modest initial context |
| 256 GB | Flash Q4/MXFP4 or GLM 5.3 Flash Q4 |
| 512 GB | Larger models, including PRO Q2 |
Here, Flash means DeepSeek V4 Flash. V4.1 Flash has different memory
requirements; see its [model guide](MODELS.md#deepseek-v41-flash).
These are starting points, not guarantees that every context or session count
will fit. GLM 5.3 Flash Q2 is about 90 GiB before runtime allocations.
Stop other memory-heavy workloads before loading it resident.
```sh
./download_model.sh glm53-q2
./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 32768
```
For a model larger than RAM, start with automatic cache sizing:
```sh
./ds4 --ssd-streaming
```
See [SSD streaming](SSD_STREAMING.md) before increasing the expert cache.
Do not bypass the memory guard just to make an oversized resident model start.
## Two Macs
Two 128 GB Macs can run a larger model fully resident with a 50/50 routed-expert
split. Thunderbolt RDMA is the low-latency option; TCP is also supported.
Follow [tensor parallel setup](DISTRIBUTED.md#tensor-parallelism-between-two-macs).
For more than two machines, use [pipeline parallelism](DISTRIBUTED.md#pipeline-parallelism).
## Next steps
- [Vision and GLM models](MODELS.md)
- [DSpark and GLM MTP](SPECULATIVE_DECODING.md)
- [Batched serving](SERVER.md#multiple-sessions)
- [Benchmarking](PERFORMANCE.md)
For DeepSeek V4 Flash and PRO, `--power 70` trades throughput for lower
sustained GPU load. V4.1 and GLM currently require `--power 100`.
---
### MODELS (`docs/MODELS.md`)
# Models and Vision
[README](../README.md) | [Getting started](../README.md#start-here)
DwarfStar is not a general GGUF runner. Use the download targets below: other
GGUFs may have unsupported tensor layouts, metadata, or quantization mixes.
Run `./download_model.sh --help` for filenames and all available targets.
Main-model downloads update `ds4flash.gguf`. Encoders, draft models, packaged
FP8 weights, and split PRO pieces do not. Pass `-m FILE` to avoid depending on
which model was downloaded last.
Some downloads require the Hugging Face CLI; the script prints installation
instructions when needed. Authentication is optional for public weights;
your cached Hugging Face token or `HF_TOKEN` is used when present.
## DeepSeek V4
| Target | Use |
| --- | --- |
| `ds4f-q2` | Flash 0731, about 81 GiB; starting point for 96/128 GB systems |
| `ds4f-q2-q4` | Mostly Q2, with the last six routed-expert layers at Q4; needs more memory |
| `ds4f-q4` | Flash 0731 Q4; larger-memory or distributed systems |
| `ds4f-mxfp4` | Native MXFP4 routed experts; larger-memory or distributed systems |
| `pro-q2-imatrix` | PRO 0813; 512 GB resident target, or SSD streaming |
| `pro-q4-split` | Both PRO Q4 pieces for pipeline execution |
```sh
./download_model.sh ds4f-q2
./ds4
```
The Flash Q2 recipe spends most of its compression on routed experts:
IQ2_XXS gate/up and Q2_K down. Other components use higher precision, including
Q8 projections, shared experts and output, plus F16/F32 tensors. They are not
all untouched source weights. The imatrix guides the routed quantization.
The MXFP4 recipe preserves DeepSeek's released MXFP4 routed experts without
requantizing them. Metal and CUDA support it. Blackwell uses native FP4 matrix
instructions and FP4 activations for batched expert work; CUDA decode and
other CUDA architectures use Q8 activations. ROCm also has a resident MXFP4
path, including pipeline execution for models too large for one host.
To build weights rather than download them, see [GGUF tools](../gguf-tools/README.md).
## DeepSeek V4.1 Flash
V4.1 Flash text and vision inference work on Metal. CUDA supports text with
Q2 SSD streaming on one Spark or resident shards across two Sparks. ROCm
supports single-host text inference with SSD streaming. It needs its own
GGUF, tokenizer and inference graph; V4 Flash weights and DSpark support files
are not interchangeable with it.
| Target | File size | Main weights |
| --- | ---: | ---: |
| `ds41f-q2` | 341 GiB | 152 GiB |
| `ds41f-q4` | 483 GiB | 294 GiB |
Both use imatrix-calibrated routed experts and include 189 GiB of Engram
tables. Engram rows are read directly from the file as needed in every mode,
never loaded as a resident table. Keep the GGUF on a fast local SSD.
On one 128 GB Mac, use SSD streaming. Leave the expert cache budget automatic:
```sh
./download_model.sh ds41f-q2
./ds4 -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
--ssd-streaming --ctx 32768
```
Use `ds4-agent` or `ds4-server` with the same model and memory options.
On a DGX Spark, add `--cuda`; see the [Spark guide](DGX_SPARK.md#deepseek-v41-flash).
`--think-level 25` sets reasoning effort explicitly; the range is 1 to 100,
with 0 disabling thinking. `/think 25` changes it in the CLI or native agent.
`--think` selects 75 and `--think-max` selects 100.
For two 128 GB Macs or Sparks, follow the [TP/RDMA setup](DISTRIBUTED.md), passing this
GGUF with `-m` on both ranks and omitting `--ssd-streaming`. Each rank holds
about 81 GiB of main weights, plus context and runtime buffers. Both machines
need the complete GGUF on disk. A 256 GB or larger Mac can instead hold all
main weights; full residency has been tested on an M3 Ultra with 512 GB.
Q4 needs SSD streaming on smaller Macs, or a 512 GB Mac for full residency.
It does not fit resident TP across two 128 GB Macs. Download it with
`./download_model.sh ds41f-q4` and select `gguf/DeepSeek-V4.1-Flash-Q4.gguf`.
The download comes in two parts; the script joins and verifies them automatically.
Allow another 37 GiB of free disk space while joining. Rerun the command to resume an
interrupted download or join.
Large SSD prefills process layers in wide batches. Metal overlaps computation
with the next layer's reads; CUDA stages experts into its bounded device cache.
Short appends keep using the expert cache.
Resident and TP inference also batch continued prefills automatically.
For concurrent serving, see [session batching](SERVER.md#multiple-sessions).
Each slot needs its own context memory; start with `--ctx 4096` before
increasing both context and slot count. CUDA Q2 SSD mode batches up to eight
decode rows; CUDA network TP currently serves sessions in order.
[DSpark](SPECULATIVE_DECODING.md#deepseek-v41-flash-dspark) uses a
separate V4.1 support GGUF and currently requires one host. Pipeline execution
is not implemented for V4.1; vision requires Metal.
Scalar, batched and tensor-parallel execution are not numerically identical.
Q4 batched prefill shows a small probability-score loss on the short official
continuation test, with unchanged overall top-token agreement. See the
[QA results](../QA_BEFORE_RELEASES.md#17-deepseek-v41-flash) for details
and remaining differences.
For images, download the matching encoder and add it to the same command:
```sh
./download_model.sh ds41f-vision
./ds4-agent -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
--ssd-streaming --vision gguf/DeepSeek-V4.1-Flash-Vision.gguf
```
Vision works with SSD streaming, full residency and two-Mac TP. Pass the encoder
on both TP ranks. Use `/read image.png` in `ds4`, `view_image` in `ds4-agent`,
or the [server image API](SERVER.md#images). V4 Flash vision encoders do not
work with V4.1. See [conversion](../gguf-tools/README.md#convert-deepseek-v41-flash)
to build the GGUFs from safetensors.
## Qwen3.8 Flash Next
`./download_model.sh qwen38-q2` downloads one **137.10 GiB** GGUF: 41.73 GiB
of main/MTP weights and 95.37 GiB of original BF16 n-grams kept on disk.
Its gate/up experts use IQ2_XXS; down experts use Q2_K with 640 logical inputs
padded to 768 in the weight file. It replaces the larger MXFP4-down IQ2 release.
For 64 GB Macs, start at 8K context with a 1,024-token prefill chunk; runtime
allocations add to the main weights, but the n-gram table is not mapped or
preloaded. Keep the GGUF on a fast local SSD.
The larger `qwen38-q4k` target uses 165.11 GiB on disk and 69.74 GiB for
resident weights, before runtime buffers.
This model runs on Metal and single-GPU CUDA, including DGX Spark.
The script links `ds4flash.gguf` to the combined GGUF:
```sh
./ds4 --ctx 8192 --prefill-chunk 1024
```
Add `--mtp` for speculation; no second file is needed.
See [Qwen setup](QWEN38_FLASH_NEXT.md)
for memory, conversion, vision, and sampling details.
Vision uses a separate encoder. `./download_model.sh qwen38-vision` downloads
llama.cpp's Q8_0 mmproj from
[ggml-org/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF);
pass it at runtime with `--vision`.
## GLM 5.3 Flash
| Target | Approximate file size | Use |
| --- | ---: | --- |
| `glm53-q2` | 90 GiB | One 128 GB Mac or DGX Spark; ROCm also supported |
| `glm53-q4` | 178 GiB | Larger Mac, two 128 GB Macs, or SSD streaming |
| `glm53-fp8` | 305 GiB | Packaged native weights only; inference not implemented |
```sh
./download_model.sh glm53-q2
./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 32768
```
GLM 5.3 Flash has recurrent KDA layers, sparse DSA attention, hyper-connections,
and a built-in MTP block. The Q2 file uses imatrix-guided IQ2_XXS gate/up and
Q2_K down experts. Q4 is the higher-precision alternative.
Q2 is close enough to a 128 GB machine's memory budget that other workloads
and context size matter. Follow the [Metal](METAL.md), [Spark](DGX_SPARK.md),
or [Strix Halo](STRIX_HALO.md) starting configuration for your host.
Ordinary decode is the default. Enable the embedded draft block with `--mtp`:
```sh
./ds4-agent -m gguf/GLM-5.3-Flash-Q2.gguf --mtp --ctx 50000
```
No second model file is needed. See [sampling behavior](SPECULATIVE_DECODING.md)
before choosing between the default opportunistic mode and exact sampling.
## Full GLM 5.3 and GLM 5.2
Full GLM 5.3 Q2 is about 197 GiB. Use a sufficiently large machine or streaming:
```sh
./download_model.sh glm53-full-q2
./ds4 --ssd-streaming
```
GLM 5.2 downloads are `glm-antirez-iq2xxs`, `glm-antirez-q2`,
`glm-antirez-q4`, and the 11-shard `glm-unsloth-q4`. They are much larger
than GLM 5.3 Flash; choose memory capacity before choosing the quantization.
GLM runs on Metal, CUDA and ROCm. The routed paths include IQ2_XXS, Q2_K, and
Q4_K, with additional mixed layouts supported by the tested GGUFs. Two-Mac
ownership-aware TP accepts IQ2_XXS, Q2_K, and Q4_K gate/up layouts; this does
not mean every Q4 model fits two 128 GB machines. Use a tested artifact, not
an arbitrary combination of supported tensor types.
GLM uses graph-selected prefill chunks and does not accept `--prefill-chunk`
or an external `--mtp-model`. It currently requires `--power 100`.
Directional steering is supported for GLM 5.3, not GLM 5.2.
## Vision
PNG and JPEG input works in the CLI, native agent, and HTTP server on Metal,
single-GPU CUDA, and ROCm. The encoder must match the model.
V4.1 Flash vision is currently Metal-only; its setup is [above](#deepseek-v41-flash).
### DeepSeek Flash Vision Experimental
Vision Experimental is a different language checkpoint from Flash 0731.
The main download includes its encoder:
```sh
./download_model.sh ds4f-vision-q2
./ds4 --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf
```
Larger targets are `ds4f-vision-q2-q4` and `ds4f-vision-mxfp4`.
`ds4f-vision-encoder` downloads just the encoder when the language GGUF is
already present. Vision Experimental has its own [DSpark drafter](SPECULATIVE_DECODING.md).
### GLM 5.3 Flash
The text GGUF stays the same. Download and add the encoder explicitly:
```sh
./download_model.sh glm53-vision
./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf \
--vision gguf/GLM-5.3-Flash-Vision-Encoder.gguf
```
Use `/read image.png` in `ds4`, or start `ds4-agent` with the same `--vision`
argument to enable `view_image`. Agent sessions containing images cannot yet
be saved with `/save`.
For two-Mac TP, pass the same encoder on both ranks. The coordinator encodes
the image and sends the projected visual tokens to the worker.
For HTTP image formats and limits, see [serving](SERVER.md#images).
### Qwen3.8 Flash Next
The text GGUF stays the same. Download and add the encoder explicitly:
```sh
./download_model.sh qwen38-vision
./ds4 --mtp \
--vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf
```
The encoder is llama.cpp's Q8_0 mmproj conversion of the model's Qwen3-VL
tower. Use `/read image.png` in `ds4` or `image_url` parts over HTTP; see
[Qwen setup](QWEN38_FLASH_NEXT.md) for image limits and resize behavior.
---
### PERFORMANCE (`docs/PERFORMANCE.md`)
# Performance and Benchmarking
[README](../README.md)
Compare the same checkpoint, quantization, context, and sampling settings.
Record the commit and whether weights were resident, streamed, or distributed.
Keep other GPU workloads idle and repeat in alternating order: one favorable
run is not a speed result.
## Context sweeps
`ds4-bench` measures prefill and generation at successive context frontiers:
```sh
./ds4-bench -m ds4flash.gguf \
--prompt-file speed-bench/promessi_sposi.txt \
--ctx-start 2048 --ctx-max 65536 --step-incr 2048 --gen-tokens 128
```
Each prefill number measures the newly added interval. Generation uses a fixed
greedy, non-EOS probe. The benchmark normally restores a memory snapshot after
each probe; network TP/pipeline runs and snapshots beyond its memory limit use
prefix replay instead. Do not interpret replay time as continued-prefill speed.
Use `--step-mul F` for exponential context spacing. Output is CSV, including
prefill throughput, generation throughput, and snapshot size when available.
The prompt is the cleaned public-domain *I Promessi Sposi* text in
[speed-bench](../speed-bench/README.md).
Prefill defaults are configuration-specific: ordinary DeepSeek uses 4096-token
chunks, CUDA TP uses 2048, and long PRO prompts may use 8192. GLM chooses its
own chunks and rejects `--prefill-chunk`. The strict DeepSeek API-vector test
pins 2048; do not generalize that setting to every benchmark.
## Recorded Flash Q2 baseline
These existing sweeps use 2048-token intervals and 128 generation tokens per
frontier. They are recorded baselines, not measurements of every subsequent
commit. Full data: [M5 Max](../speed-bench/m5_max.csv) and
[DGX Spark](../speed-bench/gb10.csv).
| Machine | Context | Prefill | Generation |
| --- | ---: | ---: | ---: |
| M5 Max, 128 GB | 2048 | 790.18 t/s | 39.35 t/s |
| M5 Max, 128 GB | 16384 | 572.53 t/s | 36.14 t/s |
| M5 Max, 128 GB | 32768 | 557.04 t/s | 34.36 t/s |
| M5 Max, 128 GB | 65536 | 398.50 t/s | 27.64 t/s |
| DGX Spark, 128 GB | 2048 | 825.76 t/s | 18.05 t/s |
| DGX Spark, 128 GB | 16384 | 872.44 t/s | 15.10 t/s |
| DGX Spark, 128 GB | 32768 | 855.94 t/s | 14.43 t/s |
| DGX Spark, 128 GB | 65536 | 822.98 t/s | 13.84 t/s |
Historical PRO Q2 measurements on the M3 Ultra are retained in this chart:
## What to compare next
- For SSD streaming, record the effective cache and distinguish cold startup
from a warm cache.
- For multiple sessions, report both individual latency and aggregate
throughput. Ordered fallback is not native batching.
- For DSpark/MTP, compare plain decode too. A faster drafter path can still be
slower than ordinary decoding on an unpredictable prompt.
- For TP, keep quantization and prompt equal; comparing resident Q4 on two
machines with streamed Q4 on one measures capacity benefits as well as parallelism.
Recent focused TP and DSpark comparisons, including their limitations, live in
[QA_BEFORE_RELEASES.md](../QA_BEFORE_RELEASES.md). Use its speed-regression
procedure rather than accumulating one-off timings in the main README.
---
### QWEN38 FLASH NEXT (`docs/QWEN38_FLASH_NEXT.md`)
# Qwen3.8 Flash Next
[Back to README](../README.md)
Qwen3.8-Flash-Next uses the `qwen4exp` GGUF architecture and a dedicated
Metal/CUDA graph with gated delta-net, gated GQA, block-sparse attention,
hyper-connections, n-gram embeddings, MoE, and MTP.
## Download and run
```sh
make
./download_model.sh qwen38-q2
./ds4 --ctx 8192 --prefill-chunk 1024
```
The download installs one GGUF containing the main model, MTP and original
BF16 n-grams, and updates `ds4flash.gguf`:
| Target | File size | Resident weights |
| --- | ---: | ---: |
| `qwen38-q2` | 137.10 GiB | 41.73 GiB |
| `qwen38-q4k` | 165.11 GiB | 69.74 GiB |
The 95.37 GiB n-gram table stays on disk. The runtime reads selected rows
directly from the GGUF, without mapping or preloading the table. Keep the
file on a fast local SSD. No sidecar or n-gram option is needed.
The Q2 model uses IQ2_XXS gate/up experts and Q2_K down projections padded
from 640 to 768 columns, calibrated with an imatrix. Q4 uses calibrated
Q4_K gate/up and MXFP4 down experts. Context and runtime buffers still need
RAM beyond the resident weights; start Q2 with the context and chunk above
on a 64 GB machine. Q4 needs a larger machine.
On a DGX Spark, build with `make cuda-spark` instead of `make`. Both Q2 and
Q4 fit resident on an otherwise idle 128 GB Spark; the n-grams stay on SSD.
The same commands support text, vision, MTP, steering and text-session
checkpoints in `ds4`, `ds4-agent` and `ds4-server`. No CUDA-specific model
conversion or feature flags are needed.
Leave `--prefill-chunk` unset on the Spark to use the faster 8192-token
default. The 1024-token example above saves memory on smaller Macs.
Use the same model options with `ds4-agent` or `ds4-server`.
Add `--mtp` for speculative decoding using the built-in MTP weights.
`ds4-bench` benchmarks ordinary decoding; it does not accept `--mtp`.
`--nothink` disables thinking. The server exposes
`qwen3.8-flash-next`, `qwen3.8-flash-next-chat`, and
`qwen3.8-flash-next-reasoner` aliases. Tool calls use the native
`` format.
On Metal, `ds4-server --batched-session N` decodes the slots together. Work that
only reads weights runs once for the whole batch, while the delta-net
recurrence, the attention caches and the n-gram convolution stay per session.
The slots also share one prefill workspace, so an extra slot costs its caches
rather than another few GiB of transients. As with the other natively batched
models, grouping changes the order of floating point reductions, so a batched
reply can differ from the same prompt decoded alone.
Add `--mtp` to use batched speculative decoding as well. The server accepts
matching greedy drafts by default, including at nonzero temperature. With
`--mtp-exact-sampling`, sampled requests use ordinary batched decoding;
temperature-zero requests can still speculate. Default greedy acceptance also
means a seed need not reproduce a reply under a different batching schedule.
Speculative batches above 16 sessions, images and steering use the ordered
fallback. CUDA currently
decodes sessions in order.
Disk KV checkpoints include recurrent state. Rewinding to an earlier position
replays the retained prefix on the next evaluation. The native context is
262144 tokens; `DS4_QWEN4_YARN_FACTOR=2` or `=4` enables static YaRN for
longer contexts, with a possible quality cost on shorter prompts.
## DGX Spark performance
Measured on a single Spark with resident weights and disk-only n-grams:
| Model | First 1024 tokens | Next 7168 tokens | Decode at 8192 tokens |
| --- | ---: | ---: | ---: |
| Q2 | 516 t/s | 745 t/s | 22.6 t/s |
| Q4 | 513 t/s | 755 t/s | 21.2 t/s |
These are averages of two runs using `speed-bench/promessi_sposi.txt`,
8192-token prefill chunks and 128 teacher-forced decode tokens, without MTP.
Loading the model is excluded. Q2 also reached about 771 t/s on a fresh
32K-token prefix with `--prefill-chunk 32768`. Small continuations have lower
throughput: adding 32 tokens after an 8K Q4 prefix took about 273 ms.
MTP speed depends on how often drafts are accepted; the table measures
ordinary decoding.
## Conversion
See [GGUF conversion](../gguf-tools/README.md) for conversion from the HF
checkpoint and calibrated low-bit expert quantization.
`gguf-tools/qwen4_pack_to_qwen4exp.py` migrates old DS4 fast packs to the
active `qwen4exp` schema. Finish older main-only conversions with
`gguf-tools/qwen4_native_ngrams.py`, using the original HF n-gram shards.
Expanding an old quantized table to BF16 does not restore its lost precision.
## Vision
Images go through the model's Qwen3-VL vision tower. Download the encoder
with `./download_model.sh qwen38-vision`, then use
`--vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf`. The CLI accepts
`/read image.png`; the server accepts `image_url` parts. Each image is resized to
multiples of 32 pixels within 64 to 1024 tokens (`DS4_QWEN4_IMAGE_MAX_TOKENS`
raises the cap), encoded on the GPU, and takes the model's 3D rope positions;
live KV reuse keys on the image fingerprints. `make test-qwen4-vision`
compares the tower with the Hugging Face implementation using the same GGUF
weights, dequantized to float32. It separately reports GGUF-versus-original
checkpoint quality and GPU-versus-original agreement. Both comparisons use
the unchanged minimum per-token cosine threshold of 0.99; implementation
parity gates the default exit status. Add `--require-quality` to also fail on
GGUF-versus-original quality loss. A passing implementation check alone does
not mean the quantized encoder matches the original checkpoint.
For a small suite including OCR, diagrams, a photograph, and resizing:
```sh
make tests/test_qwen4_vision
uv run --with numpy --with torch --with torchvision --with pillow \
--with safetensors --with transformers --with gguf tests/qwen4_vision_ref.py \
--snapshot /path/to/HF-checkpoint \
--mmproj gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf \
--image tests/vision-fixtures/qwen38/orbit.png \
--image tests/vision-fixtures/qwen38/maple.png \
--image tests/vision-fixtures/glm53/diagram.png \
--image tests/vision-fixtures/glm53/text.png \
--image tests/vision-fixtures/glm53/earth.jpg \
--image tests/vision-fixtures/glm53/screenshot.png \
--json-report /tmp/qwen38-vision-parity.json
```
Transformers must include `qwen4_exp`; the reference runs on CPU. Repeat
`--image` to add cases. The original single-image `--out` embedding dump is
still supported. Metric checks without model weights run with
`uv run --with numpy python -m unittest discover -s tests -p test_qwen4_vision_ref.py`.
JPEG decoder agreement with Pillow, including progressive scans, subsampling,
and image edges, is checked separately with
`uv run --with numpy --with pillow python -m unittest discover -s tests -p test_jpeg_decode.py`.
To check CLI `/read` image turns and text follow-ups with ordinary and MTP
decode, use two different PNG or JPEG images with the model-backed regression:
```sh
make ds4
uv run tests/test_qwen4_cli_vision.py --model /path/to/main-with-mtp.gguf \
--vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf \
--image /path/to/first.png --image /path/to/second.png
```
The test checks that all four turns complete in each mode, including errors
that the interactive CLI can report without a nonzero process exit. It saves
the responses and diagnostics for inspection; it does not grade image content.
The Metal and CUDA graphs accept Q8_0, Q4_0, F16, BF16 and F32
dense weights, Q8_0/MXFP4/Q4_0/Q4_K/Q2_K/IQ2_XXS experts, F16/F32/Q8_0
hyper-connection mixers and the original BF16 n-gram table.
Tensor parallelism, pipeline execution and SSD expert streaming are not
implemented for this model yet.
ROCm is not supported. CPU code is a correctness reference, not a general
inference backend.
## Validation
```sh
make test-qwen4-kernels test-qwen4-q2 test-qwen4-prefill-reuse test-q8-prefill-variants
make test-frontends
make test-qwen4-ngrams
make tests/test_qwen4_ngram_state
make -B -C gguf-tools quants-shared
python3 -m unittest discover -s gguf-tools/tests -p test_qwen4_pack.py
python3 -m unittest discover -s gguf-tools/tests -p test_qwen4_native_ngrams.py
```
On CUDA, use `make test-qwen4-cuda` for the kernel tests. They compare the
active kernels with independent CPU references, without model weights.
Vision and end-to-end checks additionally require the checkpoints above.
Run `tests/test_qwen4_ngram_state MODEL.gguf` on either backend to check
failed disk reads during prefill, decode and MTP, then exact recovery.
Official Alibaba continuations are tracked for 100 short prompts and 12
archive/code prompts from 2K to 24K tokens. Build the quality scorer, then run
from the repository root:
```sh
gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/data/qwen38-flash-alibaba-100/manifest.tsv /tmp/qwen-short.tsv 4096
gguf-tools/quality-testing/score_official MODEL.gguf \
gguf-tools/quality-testing/data/qwen38-flash-alibaba-long/manifest.tsv /tmp/qwen-long.tsv 32768
```
Repeat with `--quality` and, for the long set, `--continued-prefill 1` and
`--continued-prefill 256`. These fixtures match the no-thinking template;
no rendered-prompt flag is needed. See [quality testing](../gguf-tools/quality-testing/README.md)
for collection settings, measurements and the hosted-checkpoint limitations.
[Checkpoint-fix benchmark charts and measurements](../speed-bench/qwen38-checkpoints/README.md)
compare prefill, ordinary decode, and MTP decode against the preceding PR head.
Qwen prefill checkpoints include matching logits at each completed chunk,
including cancellation frontiers. To test save/restore and continuation:
```sh
DS4_TEST_MODEL=/path/to/main-with-mtp.gguf \
DS4_TEST_GLM_MTP=1 DS4_TEST_MTP_EXACT=1 \
./ds4_test --qwen4-prefill-checkpoints --qwen4-restore-reuse \
--session-snapshot --session-rewind --session-rewind-resample
python3 tests/test_qwen4_checkpoint_replay.py \
--model /path/to/main-with-mtp.gguf
python3 tests/test_qwen4_mtp_limits.py \
--model /path/to/main-with-mtp.gguf
python3 tests/test_qwen4_logit_dump.py \
--model /path/to/main-with-mtp.gguf
```
The server regression checks continued disk saves, cache-budget eviction,
and text-answer history with and without tool schemas. It covers clients
that omit reasoning and clients that echo `reasoning_content`, including
server restart after a continuation. For tools-enabled histories, the first
answer keeps the exact disk key; after a client continues without reasoning,
the next checkpoint uses the visible history key for restart reuse.
The MTP tests cover small prefill buffers, rewinds, reused sessions, and
truncated checkpoints. The logit-dump test requires NumPy and compares all-row
prefill with teacher-forced decode across a chunk boundary. For official
continuation scoring, use `--rendered-prompt`
when a fixture already contains the complete model chat template.
`tests/test_qwen4_prefill MODEL PROMPT 8192` checks mixed prefill sizes,
progress callbacks and exact replay through the sparse-attention boundary.
It also reports differences against a fresh prefill followed by individual
decodes. Those schedules can round differently and exchange nearly tied
experts; their full logits need not match. Use model-quality checks as well,
not only state replay:
```sh
python3 tests/test_server_story.py --url http://127.0.0.1:8000 \
--model qwen3.8-flash-next --output /tmp/qwen-story
```
Start the server with at least `--ctx 49152`. This checks all sixteen facts
in a 31K-token story, then a correction turn that must reuse the prefix.
Qwen directional steering and activation capture support all 48 trunk layers;
see [directional steering](../dir-steering/README.md). Model-backed regressions:
```sh
python3 tests/test_qwen4_steering.py --model /path/to/main-with-mtp.gguf
./ds4_agent_test --full-context-save /path/to/main-with-mtp.gguf
make test-web-recovery
```
The agent can save an already-full text transcript without a KV payload.
Restart with a larger `--ctx`, then `/switch` to the saved session and
`/compact` to recover room. Sessions containing images still use the existing
save restriction. Automatic compaction reserves summary space and continues
interrupted generation through the existing agent compaction path.
---
### SERVER (`docs/SERVER.md`)
# Serving Models
[README](../README.md) | [Client setup](CLIENTS.md)
## Start locally
```sh
./ds4-server --ctx 32768
```
The default address is `http://127.0.0.1:8000`. Use `--host 0.0.0.0` to listen
on other interfaces. Restrict access to trusted clients; for an Internet-facing
deployment, put authentication and TLS in front of the server.
`--cors` enables browser cross-origin headers. It does not change the listening
address or provide access control.
The selected build chooses the backend. Pass `-m FILE` for an explicit model.
Use `--chdir /path/to/ds4` when starting outside the project directory so
relative runtime files such as Metal kernels can be found.
## APIs
| Endpoint | Use |
| --- | --- |
| `GET /v1/models` | Loaded model information |
| `POST /v1/chat/completions` | OpenAI-style chat |
| `POST /v1/responses` | Responses-style requests and continuations |
| `POST /v1/completions` | Text completions |
| `POST /v1/messages` | Anthropic-style messages |
The Flash and PRO names accepted by the model endpoints are compatibility
aliases, not separate loaded models. The GGUF passed at startup selects the model.
```sh
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"Explain Redis streams."}],"stream":true}'
```
Chat, Responses, and Anthropic support tools and SSE streaming. Reasoning is
returned separately from visible text in each API's native form. Standard
sampling and output-budget fields are supported; explicit request parameters
take precedence over defaults.
The default sampling settings are temperature 1, top-p 1, and min-p 0.05.
For DeepSeek, thinking is on by default. `reasoning_effort=max` selects Think
Max only with sufficient context; otherwise it falls back to normal thinking.
`xhigh` maps to normal thinking, not Think Max. Use `think:false`, a disabled
thinking object, or a non-thinking model alias for direct answers.
## Multiple sessions
```sh
./ds4-server --ctx 4096 --batched-session 4
```
Without `--batched-session`, there is one resident session. With it, the server
preallocates independent KV states and queues requests when all slots are busy.
Choose context and slot count together: a context that fits once may not fit
four times. Idle slots can be cached before reuse; active requests are not evicted.
Where the model supports it, the slots share one prefill workspace instead of
each keeping its own, so an extra slot costs only its caches. That matters most
for Qwen3.8 Flash Next, whose transients are sized by the prefill chunk rather
than by the context: at the default chunk they run to several GiB per session.
The startup line reports both figures.
| Backend/model | Decode execution |
| --- | --- |
| Metal, resident Flash | Native shared-expert/QKV batching where supported |
| Metal, resident V4.1 Flash | Native decoding for 2-8 sessions |
| Metal RDMA TP, V4.1 Flash | Native decoding for 3-8 sessions; ordered fallback for two |
| Metal SSD streaming, V4.1 Flash | Ordered fallback |
| Metal, GLM 5.2 | Ordered fallback |
| Metal, GLM 5.3 | Native batching through 2051 visible tokens; ordered fallback afterward |
| Metal, Qwen3.8 Flash Next | Native batching of the shared work; recurrent state, caches and PLE history stay per session |
| CUDA, supported multi-GPU Flash TP layout | Native grouped decode and mixed prefill/decode |
| Single-GPU CUDA, including Spark | Ordered fallback |
Fallback executes the rows separately. It provides concurrency and scheduling
fairness, not the aggregate speedup of native batching. Native grouping may
change floating-point reduction order slightly. V4.1 sessions containing images
use the ordered fallback.
Long prefills yield to active decoders in bounded intervals, normally 128
tokens. `--mixed-prefill-quantum N` changes that interval for testing.
Session-batched serving uses ordinary target decoding, except Qwen3.8 on
Metal, where `--mtp` also batches speculative decoding. Its
`--mtp-exact-sampling` mode uses ordinary batches for nonzero-temperature
requests. Other models do not use MTP/DSpark while session batching is active.
For the eight-L40S example, see [CUDA GPUs](CUDA_MULTI_GPU.md#serve-multiple-users).
## Images
Start with the matching language GGUF and `--vision FILE`; see
[model-specific instructions](MODELS.md#vision).
OpenAI chat and Responses accept inline PNG/JPEG data URIs. Anthropic accepts
base64 image sources. Remote URLs and server-side file paths are rejected.
Image blocks preserve their order in the request. The limit is 16 images and
a 64 MiB HTTP body.
## Disk KV cache
Disk caching saves useful prefixes across slot reuse and server restarts:
```sh
./ds4-server --ctx 100000 \
--kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
```
Clients can resend complete conversation histories. The server first tries
the live token prefix, then compatible rendered-text prefixes from disk, and
prefills the new suffix. With multiple slots, each slot has its own live state;
disk is the additional persistence layer, not the only way to retain sessions.
TP cache loading rebuilds the saved token prefix on both ranks. It is not an
instantaneous restoration of both GPUs. Pipeline loading redistributes the
saved layer state over its route.
Defaults are intended to avoid saving fragile token boundaries. For unusual
workloads, the controls are `--kv-cache-min-tokens`,
`--kv-cache-cold-max-tokens`, `--kv-cache-continued-interval-tokens`,
`--kv-cache-boundary-trim-tokens`, and `--kv-cache-boundary-align-tokens`.
Check `./ds4-server --help` for their defaults.
Quantization variants may share compatible prefixes. Add
`--kv-cache-reject-different-quant` for same-quant reuse only.
Cache files contain prompt text and model state: treat the directory as
private. It is disposable; stop the server before clearing it.
## Tool history and debugging
For DeepSeek, the server preserves sampled DSML tool blocks and assigns
unguessable tool IDs. Replaying those IDs avoids retokenizing a differently
formatted JSON history. The bounded replay map can be stored in cache files.
When exact replay is unavailable, canonical rendering may require rebuilding
part of the prefix.
`--tool-memory-max-ids` bounds this map.
`--disable-exact-dsml-tool-replay` disables it for diagnostic comparisons.
Use `--trace /tmp/ds4-trace.txt` to record prompt rendering, cache decisions,
generated text, and tool-parser events. Traces can contain sensitive content.
Cache formats are implementation details. The current header and extension
definitions are in [ds4_kvstore.h](../ds4_kvstore.h) and
[ds4_kvstore.c](../ds4_kvstore.c); model-specific payload handling is in
[ds4.c](../ds4.c).
---
### SPECULATIVE DECODING (`docs/SPECULATIVE_DECODING.md`)
# Speculative Decoding
[README](../README.md)
Speculation proposes future tokens with a smaller draft block, then checks
them with the main model. An accepted prefix advances generation by several
tokens in one verification pass. It does not accelerate prefill.
It is opt-in. Gains depend on the prompt, model, backend, and context length;
poor acceptance can make it slower. Measure your workload rather than assuming
that a draft model always helps.
## DeepSeek Flash: DSpark
DSpark is a separate support GGUF, not a standalone language model. It proposes
up to five future tokens. Match its checkpoint to the main model:
| Main checkpoint | Download | Support file |
| --- | --- | --- |
| Flash 0731 | `ds4f-dspark` | `gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf` |
| Flash Vision Experimental | `ds4f-vision-dspark` | `gguf/DeepSeek-V4-Flash-Vision-Exp-DSpark-support.gguf` |
For the 0731 Q2 model:
```sh
./download_model.sh ds4f-q2
./download_model.sh ds4f-dspark
./ds4 --dspark --mtp-model gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf
```
For Vision Experimental, substitute its matching main model and support file.
Do not mix the two checkpoints. DSpark is not supported for PRO.
The same flags work in `ds4-agent` and non-batched `ds4-server` requests.
The support file adds about 5.6 GiB of weights plus runtime state. The main
model must be resident; external DSpark is not supported with SSD streaming.
DSpark replaces the legacy one-stage MTP drafter for that run; the two are
not stacked.
Resident M5 paths batch supported verifier expert rows, including two-Mac TP.
On DGX Spark, resident Flash 0731 Q2 and two-Spark MXFP4 also batch the seed
with longer drafts and use small-batch Q8 and expert kernels. No extra flags
are needed.
Single-Spark Q2 uses a confidence-pruning threshold of 0.4. Two-Spark
Flash 0731 Q2 and MXFP4 leave confidence pruning off and split the drafter
over RDMA when both peers load its support file. `--dspark-confidence`
overrides the threshold. Exact sampling keeps its default threshold of 0.8.
The scheduler can back off when drafting is unproductive. Defaults select the
fast paths; diagnostic environment variables are not needed for normal use.
Recorded comparisons are in [the QA guide](../QA_BEFORE_RELEASES.md).
For the tested Strix Halo coding configuration, use `--dspark --dspark-confidence 0.7` with the default five-token draft cap and scheduler. Client sampling is temperature `1.0`, `top_p=0.95`, `min_p=0`, and `top_k=0`; high reasoning was also checked on coding and tool-use requests. This uses opportunistic sampling as described below; exact-mode throughput is not qualified by these measurements. `--mtp-draft` controls legacy autoregressive MTP, not the DSpark draft width.
## DeepSeek V4.1 Flash: DSpark
V4.1 has its own 8.04 GiB support GGUF. It retains the native MXFP4 experts
and uses a different draft graph from V4 Flash. Do not interchange the files.
See [conversion](../gguf-tools/README.md#convert-v41-dspark) to build it.
```sh
./ds4 -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
--ssd-streaming --dspark \
--mtp-model gguf/DeepSeek-V4.1-Flash-DSpark.gguf
```
The same options work with `ds4-agent` and `ds4-server`. Metal, CUDA and ROCm
support single-host execution, with either resident target weights or SSD
streaming. The drafter stays resident, so leave another 8 GiB for it.
Two-host tensor parallelism works on Metal and CUDA: add the same `--dspark
--mtp-model FILE` options to both the coordinator and worker. Each peer keeps
the drafter resident; the target and its verification run across both GPUs.
Up to five drafts are checked together. The sampling choices below apply;
`--mtp-exact-sampling` uses sampled proposals and the full acceptance/correction
rule for V4.1. Restored sessions warm the draft window with 128 ordinary tokens
before speculation resumes. Measure speed on your prompts: acceptance and SSD
cache size both matter. V4.1 DSpark can still be slower than ordinary decoding,
including in TP mode.
## GLM: built-in MTP
GLM's draft block is already in its main GGUF:
```sh
./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf --mtp
```
`--mtp-timing` also enables it and prints acceptance and timing counters.
The current GLM cycle commits up to two tokens. No external support file is
needed, and ordinary decode remains the default.
## Qwen3.8: built-in MTP
Both Qwen downloads include MTP and native BF16 n-grams in the main GGUF:
```sh
./download_model.sh qwen38-q4k
./ds4 --mtp
```
Ordinary decode uses the same file with `--mtp` omitted. For non-zero
temperature, add `--mtp-exact-sampling` to preserve the target sampling
distribution. See [Qwen setup](QWEN38_FLASH_NEXT.md) for Metal and CUDA.
The cycle drafts one token ahead by default and engages a **second, chained
draft** (one extra nextn-layer step conditioned on the predictor's own
stream, verified in a 3-row pass) while recent first-draft acceptance is
perfect, disengaging after repeated second-draft rejections.
`DS4_QWEN4_MTP_DEPTH=2` or `=3` fixes the depth;
`0` (default) is the adaptive policy.
## Sampling and reproducibility
At temperature zero, accepted drafts must match the target's greedy
continuation. At non-zero temperature, the default mode is opportunistic:
ordinary tokens use the requested sampling settings, but matching greedy
drafts are accepted directly. Sampling resumes when the proposed suffix does
not match. This is deliberately more deterministic than ordinary sampling.
Use `--mtp-exact-sampling` to preserve the ordinary target sampling
distribution. Greedy proposals are accepted with their target probability;
sampled proposals use the target-to-draft probability ratio. Rejections sample
from the remaining target probability mass.
When a verified block crosses a tool sampling-mode boundary (for example
entering tool-call syntax during server decoding), the server rewinds to the
block start and re-evaluates the boundary token so the next sample uses the
new mode. Under exact sampling that rewind restores a pre-verify snapshot of
the recurrent state instead of resetting the graph, so long retained
contexts are not replayed at every boundary.
Accepted tokens keep the state produced by the batched verifier. Floating-point
reduction order can differ from one-token decode, so long greedy continuations
need not be byte-identical. For DeepSeek comparisons against the ordinary
target-only path, use `--quality` or `--dspark-strict`; these disable the
speculative acceptance path. They do not promise identical output across
different hardware or execution configurations.
Session-batched serving uses ordinary target decoding instead of combining
DSpark/MTP with the session batch. See [serving](SERVER.md#multiple-sessions).
---
### SSD STREAMING (`docs/SSD_STREAMING.md`)
# Models Larger Than RAM
[README](../README.md)
Use resident inference when the model and runtime fit: it is faster.
SSD streaming keeps a bounded cache of routed experts and reads missing
experts from the GGUF. It trades speed for capacity; it does not remove the
memory needed for other weights, activations, scratch, and the context.
Metal supports streaming for DeepSeek and GLM. CUDA has streaming paths too,
and ROCm supports GLM 5.2/5.3 streaming. Do not infer support for every model
and tensor layout from the existence of the flag.
## Start with the automatic budget
```sh
./ds4 -m ds4flash.gguf --ssd-streaming
```
The startup report shows the effective cache, resident layers where applicable,
and memory requirements. Prefer the automatic budget for a first run. Use a
local SSD and leave memory for the OS and other applications.
Examples:
```sh
# Flash Q2 on a smaller Mac.
./download_model.sh ds4f-q2
./ds4 --ssd-streaming --ctx 32768 --nothink
# GLM 5.3 Flash Q4 on a 128 GB Mac.
./download_model.sh glm53-q4
./ds4 --ssd-streaming --ctx 4096
# PRO Q2 on a 128 GB Mac: usable for inspection, but slow.
./download_model.sh pro-q2-imatrix
./ds4 --ssd-streaming --ctx 32768 --nothink
```
Generation is usually more sensitive to cache misses than prefill. A large
model that starts successfully can still be too slow for interactive work.
Use a short generation before committing to a long task.
## Adjusting the cache
To leave more room for context or other sessions:
```sh
./ds4 --ssd-streaming --ssd-streaming-cache-experts 32GB
```
A byte budget is a target, not a guaranteed allocation. DwarfStar reserves
routed-prefill headroom and fits the cache to the remaining model, graph,
context, and backend budget. The effective value may be smaller than requested.
Non-routed weights and KV state are additional to that expert-cache budget.
A plain number, such as `--ssd-streaming-cache-experts 4000`, requests dynamic
expert slots rather than a byte budget. It is also subject to memory limits.
By default, GLM spends the cache budget on selected experts across all layers.
`--ssd-streaming-full-layers N` reserves full routed prefix layers instead.
Metal also keeps the non-routed weights resident when it can; they are needed
by every token, and paging them back in delays generation after a prompt.
An oversized expert cache can displace those weights and slow decoding.
More cache helps only while the rest of the working set still fits.
More context and more server sessions consume more memory. Metal GLM budgeting
includes planned and already resident sessions; a request can still be refused
when its mandatory allocations cannot fit. Disabling the memory guard is not
a remedy for insufficient RAM.
Leave expert preloading enabled for normal use. `--ssd-streaming-cold` and
`--ssd-streaming-preload-experts N` are mainly useful for controlled measurements.
See [benchmarking](PERFORMANCE.md) and the [release QA guide](../QA_BEFORE_RELEASES.md).
## M5 Max measurements
On a 128 GB M5 Max, with automatic cache sizing and no speculative decoding:
| Model | Initial prefill | Continued prefill | Generation after each | Runs |
| --- | ---: | ---: | ---: | --- |
| GLM 5.3 Flash Q4_K, 177.77 GiB | 121 t/s | 104 t/s | 11.9 / 14.9 t/s | Three-run median |
| DeepSeek Flash Vision Exp MXFP4, 145.26 GiB | 300 t/s | 263 t/s | 11.9 / 19.3 t/s | Single run |
September 6, 2026. GLM used a 2K prompt and a 1K append; DeepSeek used 8K
and a 4K append. Both generated 128 tokens at each frontier. DeepSeek's latest
run uses an 86.2 GiB expert cache and updated eviction priorities; it is not
directly comparable to the earlier 64-token measurement. Generation includes
the first-token wait. Cache reuse depends on the prompt, so these are
workload references, not a speed
guarantee for every model larger than RAM.
Full GLM 5.3 IQ2_XXS (196.58 GiB) also benefits from caching short tool-result
prompts. With an 8K context and a 61.35 GiB effective expert cache, a 16-token
append fell from 30.8 to 2.9 seconds; generation afterward rose from 4.09 to
5.11 t/s (three-run medians, 64 generated tokens). All compared logits and
generated text were identical. This is a short-append improvement, not an
equivalent gain in initial prefill or every generation workload.
---
### STRIX HALO (`docs/STRIX_HALO.md`)
# AMD Strix Halo
[README](../README.md) | [Getting started](../README.md#start-here)
The reference system is a 128 GB Strix Halo with Radeon 8060S (`gfx1151`),
such as the Framework Desktop. The ROCm build uses the standard binary names
and selects the ROCm backend by default.
## Prerequisites
For a container setup, see the maintained
[ROCm toolbox](https://github.com/kyuz0/strix-halo-ds4-toolbox/blob/main/toolboxes/Dockerfile.rocm-10.0).
It can also be managed with
[AI Toolbox Cockpit](https://github.com/kyuz0/ai-toolbox-cockpit).
For a native Ubuntu build you need HIP, hipBLAS, hipBLASLt, rocBLAS, rocWMMA,
and hipCUB development files. The Ubuntu 26.04 setup used these packages:
```sh
sudo apt-get update
sudo apt-get install -y hipcc rocminfo rocm-smi \
libamdhip64-dev libhipblas-dev libhipblaslt-dev librocblas-dev \
librocwmma-dev libhipcub-dev
sudo usermod -aG render,video "$USER"
```
Log out and back in after changing groups. `rocminfo` must report `gfx1151`
and be able to open `/dev/kfd` before DwarfStar can run.
Some packaged rocWMMA headers omit `rocwmma/internal/`. If compilation fails
there, install the complete headers matching your ROCm installation, or use
the container. Do not mix header versions as a general workaround.
## GPU-visible memory
Check the memory pool reported by `rocminfo`. Some 128 GB configurations expose
only about 62 GB to the GPU, which is insufficient for resident Flash Q2 plus
runtime buffers. Firmware and kernel GTT/TTM settings control this limit.
The native reference setup used these memory parameters:
```text
amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856
```
They are a system-specific starting point, not an allocation budget for
DwarfStar. Preserve existing boot options and consult your kernel's settings
before changing them. Keep RAM available for the OS even when the GPU can
address most of it. Do not disable the IOMMU merely to copy another host's
configuration; doing so changes device isolation.
## Build and run Flash
```sh
make strix-halo
./download_model.sh ds4f-q2
./ds4 --rocm
```
`make rocm` is an alias. Use the current 0731 Q2 download for a first run;
larger mixed and Q4 models have substantially higher memory requirements.
Flash's ROCm resident and pipeline paths should not be confused with the GLM
SSD-streaming path.
## DeepSeek V4.1 Flash
Use Q2 with SSD streaming on a 128 GB machine:
```sh
./download_model.sh ds41f-q2
./ds4 --rocm -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
--ssd-streaming --ctx 4096
```
The 341 GiB GGUF includes disk-only Engram tables. Keep it on a fast local
SSD and start with automatic cache sizing. The matching
[V4.1 DSpark](SPECULATIVE_DECODING.md#deepseek-v41-flash-dspark) support file
is optional. V4.1 vision and network TP are not supported on ROCm.
Large prefills are not yet tuned on this backend and can be slow.
## GLM 5.3 Flash
The reference Q2 setup uses SSD streaming to leave room for its graph and KV
state. Begin with automatic cache sizing and a small context:
```sh
./download_model.sh glm53-q2
./ds4 --rocm -m gguf/GLM-5.3-Flash-Q2.gguf \
--ssd-streaming --ctx 4096
```
GLM 5.2 also supports ROCm streaming. Full-model GLM 5.2 inference requires it;
distributed layer slices can be resident. See [SSD streaming](SSD_STREAMING.md)
before adjusting the cache budget.
Both GLM 5.3 Flash and DeepSeek Flash Vision Experimental support images on
ROCm. Add the matching encoder with `--vision FILE`, as described in
[models and vision](MODELS.md#vision).
For a model-free routed-kernel check, use `make test-mxfp4-rocm`.
Full-model validation is described in [testing](TESTING.md).
---
### TESTING (`docs/TESTING.md`)
# Testing and Development
[README](../README.md)
Read [CONTRIBUTING.md](../CONTRIBUTING.md) before proposing an inference change.
The [release QA guide](../QA_BEFORE_RELEASES.md) defines the hardware matrix,
checkpoint-matched quality tests, long-context agent tasks, and speed checks.
It also records which checks were not completed. A smoke test is not full QA.
## Focused checks
Model-free checks include:
```sh
make ds4_test ds4_agent_test test-session-state
./ds4_test --server
./ds4_agent_test
```
On Metal, small GPU tensor tests are available without loading a full GGUF:
```sh
make tests/test_session_state_gpu tests/test_glm53_kda tests/test_mxfp4_metal
./tests/test_session_state_gpu
./tests/test_glm53_kda
./tests/test_mxfp4_metal
```
`make test` also includes model-backed tests. Select the right GGUF and ensure
that it fits before running it; do not accidentally load a large model on a
single device during multi-GPU QA. ROCm has `make test-rocm`.
Official-vector tests must use continuations from the same checkpoint as the
GGUF. Flash 0731 and Vision Experimental are not interchangeable fixtures.
See [test vectors](../tests/test-vectors/README.md) and
[quality scoring](../gguf-tools/quality-testing/README.md).
## Investigating output
```sh
./ds4 --dump-tokens -p "..."
./ds4 --dump-logprobs /tmp/out.json --logprobs-top-k 20 --temp 0 -p "..."
./ds4 --dump-logits /tmp/logits.json --nothink --prompt-file prompt.txt
./ds4-server --trace /tmp/ds4-trace.txt
```
Token dumps catch template differences without inference. Logits and
continuations help distinguish sampling changes from graph errors. Server
traces include cache and tool-parser decisions. Keep traces private when they
contain real conversations.
## Tools and source references
- [GGUF conversion and quantization](../gguf-tools/README.md)
- [Imatrix collection](../gguf-tools/imatrix/README.md) and
[calibration corpus](../gguf-tools/imatrix/dataset/README.md)
- [Steering vectors](../dir-steering/README.md)
- [Benchmark scripts and charts](../speed-bench/README.md)
- [Evaluation data and licenses](../EVAL_DATA.md)
- [Session payload implementation](../ds4.c) and
[cache header definitions](../ds4_kvstore.h)
- [Pipeline protocol](../ds4_distributed.c) and [TP protocol](../ds4_tp.c)
For changes to state handling, cover rewind/replay, save/load, images, and
multiple sessions as well as a fresh prompt. For distributed changes, exercise
both ranks and failures; a local command-parser test is not physical TP QA.
---
--- METRICS ---
- Files Extracted: 34
- Estimated Token Budget: ~43214 tokens
- Canonical Reference: https://codewiki.google/github.com/antirez/ds4