Step-3.5-Flash Documentation

Technical Documentation: stepfun-ai/Step-3.5-Flash

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

1. Introduction Copy Chapter

Step 3.5 Flash (visit website) is our most capable open-source foundation model, engineered to deliver frontier reasoning and agentic capabilities with exceptional efficiency. Built on a sparse Mixture of Experts (MoE) architecture, it selectively activates only 11B of its 196B parameters per token. This "intelligence density" allows it to rival the reasoning depth of top-tier proprietary models, while maintaining the agility required for real-time interaction.

Contents Copy Section

• Key Capabilities
• Performance
• Architecture Details
• Quick Start
• Local Deployment
• Using Step 3.5 Flash on Agent Platforms
• Cookbooks
• Known Issues and Future Directions
• Co-Developing the Future
• License

2. Key Capabilities Copy Chapter

• Deep Reasoning at Speed: While chatbots are built for reading, agents must reason fast. Powered by 3-way Multi-Token Prediction (MTP-3), Step 3.5 Flash achieves a generation throughput of 100–300 tok/s in typical usage (peaking at 350 tok/s for single-stream coding tasks). This allows for complex, multi-step reasoning chains with immediate responsiveness.

• A Robust Engine for Coding & Agents: Step 3.5 Flash is purpose-built for agentic tasks, integrating a scalable RL framework that drives consistent self-improvement. It achieves 74.4% on SWE-bench Verified and 51.0% on Terminal-Bench 2.0, proving its ability to handle sophisticated, long-horizon tasks with unwavering stability.

• Efficient Long Context: The model supports a cost-efficient 256K context window by employing a 3:1 Sliding Window Attention (SWA) ratio—integrating three SWA layers for every full-attention layer. This hybrid approach ensures consistent performance across massive datasets or long codebases while significantly reducing the computational overhead typical of standard long-context models.

• Accessible Local Deployment: Optimized for accessibility, Step 3.5 Flash brings elite-level intelligence to local environments. It runs securely on high-end consumer hardware (e.g., Mac Studio M4 Max, NVIDIA DGX Spark), ensuring data privacy without sacrificing performance.

3. Performance Copy Chapter

Step 3.5 Flash delivers performance parity with leading closed-source systems while remaining open and efficient.

Performance of Step 3.5 Flash measured across Reasoning, Coding, and Agentic Capabilities. Open-source models (left) are sorted by their total parameter count, while top-tier proprietary models are shown on the right. xbench-DeepSearch scores are sourced from official publications for consistency. The shadowed bars represent the enhanced performance of Step 3.5 Flash using Parallel Thinking.

Detailed Benchmarks Copy Section

Notes Copy Chapter

• "—" indicates the score is not publicly available or not tested.
• "*" indicates the original score was inaccessible or lower than our reproduced, so we report the evaluation under the same test conditions as Step 3.5 Flash to ensure fair comparability.
• BrowseComp (with Context Manager): when the effective context length exceeds a predefined threshold, the agent resets the context and restarts the agent loop. (By contrast, Kimi K2.5 and DeepSeek-V3.2 used a discard-all strategy.)
• In decoding cost section, decoding is estimated using a similar but more accurate approach than arxiv.org/abs/2507.19427.

4. Architecture Details Copy Chapter

Step 3.5 Flash is built on a Sparse Mixture-of-Experts (MoE) transformer architecture, optimized for high throughput and low VRAM usage during inference.

4.1 Technical Specifications Copy Section

4.2 Mixture of Experts (MoE) Routing Copy Section

Unlike traditional dense models, Step 3.5 Flash uses a fine-grained routing strategy to maximize efficiency:

• Fine-Grained Experts: 288 routed experts per layer + 1 shared expert (always active).
• Sparse Activation: Only the Top-8 experts are selected per token.
• Result: The model retains the "memory" of a 196B parameter model but executes with the speed of an 11B model.

4.3 Multi-Token Prediction (MTP) Copy Section

To improve inference speed, we utilize a specialized MTP Head consisting of a sliding-window attention mechanism and a dense Feed-Forward Network (FFN). This module predicts 4 tokens simultaneously in a single forward pass, significantly accelerating inference without degrading quality.

5. Quick Start Copy Chapter

You can get started with Step 3.5 Flash in minutes using Cloud API via our supported providers.

5.1 OpenRouter Copy Section

OpenRouter provides uniform access to Step 3.5 Flash with both free and paid tiers.

Models:

• Free Tier: stepfun/step-3.5-flash:free
• Standard Tier: stepfun/step-3.5-flash

Configuration:

• Base URL: https://openrouter.ai/api/v1
• API Key: Sign up at OpenRouter to get your key.

5.2 StepFun Platform Copy Section

StepFun offers official API endpoints for both International and Chinese users.

Note: China platform requires +86 phone number verification.

5.3 OpenClaw (Recommended) Copy Section

For a full agentic experience, we recommend using OpenClaw.
👉 View the OpenClaw Cookbook to get started in minutes.

5.4 Implementation Example Copy Section

Install the standard OpenAI SDK (compatible with both platforms):

pip install --upgrade "openai>=1.0"

Python Example:

from openai import OpenAI

# Configuration for OpenRouter
# base_url = "https://openrouter.ai/api/v1"
# api_key = "sk-or-..." 
# model = "stepfun/step-3.5-flash"

# Configuration for StepFun (International)
base_url = "https://api.stepfun.ai/v1" 
# For China: base_url = "https://api.stepfun.com/v1"
api_key = "your-stepfun-api-key"
model = "step-3.5-flash"

client = OpenAI(api_key=api_key, base_url=base_url)

completion = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello! Introduce yourself."}
    ]
)

print(completion.choices[0].message.content)

6. Local Deployment Copy Chapter

Step 3.5 Flash is optimized for local inference and supports industry-standard backends including vLLM, SGLang, Hugging Face Transformers and llama.cpp.

6.1 vLLM Copy Section

We recommend using the latest nightly build of vLLM.

1. Install vLLM.

# via Docker
docker pull vllm/vllm-openai:nightly

# or via pip (nightly wheels)
pip install -U vllm --pre \
  --index-url https://pypi.org/simple \
  --extra-index-url https://wheels.vllm.ai/nightly

Note: Bug fixes for tool parser and reasoning parser, as well as support for the v1/messages interface, are being merged into vLLM. In the meantime, you can use the vllm/vllm-openai:v0.15.1-x86_64 image with step3.5_vllm_v0.15.1.patch for deployment. The patch file step3.5_vllm_v0.15.1.patch is provided in this repository; download or copy it and ensure it is accessible from the directory where you run git apply (or use its absolute path).

# via Docker
# refer to "step3.5_vllm_v0.15.1.Dockerfile"

# or via pip 
pip install -U vllm==0.15.1
cd /path/to/lib/python3.12/site-packages  # Replace with the parent dir of vLLM installation path.
git apply /path/to/step3.5_vllm_v0.15.1.patch  # Replace with the actual path to the patch file (e.g., from the repository root).

2. Launch the server.

Note: Full MTP3 support is not yet available in vLLM. We are actively working on a Pull Request to integrate this feature, which is expected to significantly enhance decoding performance.

• For fp8 model

vllm serve <MODEL_PATH_OR_HF_ID> \
  --served-model-name step3p5-flash \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --disable-cascade-attn \
  --reasoning-parser step3p5 \
  --enable-auto-tool-choice \
  --tool-call-parser step3p5 \
  --hf-overrides '{"num_nextn_predict_layers": 1}' \
  --speculative_config '{"method": "step3p5_mtp", "num_speculative_tokens": 1}' \
  --trust-remote-code \
  --quantization fp8

• For bf16 model

vllm serve <MODEL_PATH_OR_HF_ID> \
  --served-model-name step3p5-flash \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --disable-cascade-attn \
  --reasoning-parser step3p5 \
  --enable-auto-tool-choice \
  --tool-call-parser step3p5 \
  --hf-overrides '{"num_nextn_predict_layers": 1}' \
  --speculative_config '{"method": "step3p5_mtp", "num_speculative_tokens": 1}' \
  --trust-remote-code

6.2 SGLang Copy Section

1. Install SGLang.

# via Docker
docker pull lmsysorg/sglang:dev-pr-18084
# or from source (pip)
pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git"

2. Launch the server.

• For bf16 model

SGLANG_ENABLE_SPEC_V2=1
sglang serve \
  --model-path <MODEL_PATH_OR_HF_ID> \
  --served-model-name step3p5-flash \
  --tp-size 8 \
  --tool-call-parser step3p5 \
  --reasoning-parser step3p5 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-multi-layer-eagle \
  --host 0.0.0.0 \
  --port 8000

• For fp8 model

SGLANG_ENABLE_SPEC_V2=1
sglang serve \
  --model-path <MODEL_PATH_OR_HF_ID> \
  --served-model-name step3p5-flash \
  --tp-size 8 \
  --ep-size 8 \
  --tool-call-parser step3p5 \
  --reasoning-parser step3p5 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-multi-layer-eagle \
  --host 0.0.0.0 \
  --port 8000

6.3 Transformers (Debug / Verification) Copy Section

Use this snippet for quick functional verification. For high-throughput serving, use vLLM or SGLang.

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_PATH = "<MODEL_PATH_OR_HF_ID>"

# 1. Setup
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)

# 2. Prepare Input
messages = [{"role": "user", "content": "Explain the significance of the number 42."}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

# 3. Generate
generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
output_text = tokenizer.decode(generated_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)

print(output_text)

6.4 llama.cpp Copy Section

System Requirements

• GGUF Model Weights(int4): 111.5 GB
• Runtime Overhead: ~7 GB
• Minimum VRAM: 120 GB (e.g., Mac studio, DGX-Spark, AMD Ryzen AI Max+ 395)
• Recommended: 128GB unified memory

Notes

• llama.cpp build must include PR #19283
• If you encounter tool calling issues on llama.cpp mainline, try building with: PR #18675
• Community GGUF quantizations: Step-3.5-Flash Quant Collection

Steps

1. Use official llama.cpp

the folder Step-3.5-Flash/tree/main/llama.cpp is obsolete

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

2. Build llama.cpp on Mac:

cmake -S . -B build-macos \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=ON \
  -DGGML_ACCELERATE=ON \
  -DLLAMA_BUILD_EXAMPLES=ON \
  -DLLAMA_BUILD_COMMON=ON \
  -DGGML_LTO=ON
cmake --build build-macos -j8

3. Build llama.cpp on DGX-Spark:

cmake -S . -B build-cuda \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_GRAPHS=ON \
  -DLLAMA_CURL=OFF \
  -DLLAMA_BUILD_EXAMPLES=ON \
  -DLLAMA_BUILD_COMMON=ON
cmake --build build-cuda -j8

4. Build llama.cpp on AMD Windows:

cmake -S . -B build-vulkan \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=OFF \
  -DGGML_OPENMP=ON \
  -DGGML_VULKAN=ON
cmake --build build-vulkan -j8

5. Run with llama-cli

./llama-cli -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -fa on --temp 1.0 -p "What's your name?"

6. Test performance with llama-batched-bench (Benchmark details)

./llama-batched-bench -m step3.5_flash_Q4_K_S.gguf -c 32768 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1

7. Using Step 3.5 Flash on Agent Platforms Copy Chapter

Note: As mentioned in the Quick Start, you can access Step 3.5 Flash via OpenRouter or the StepFun Platform. Choose the base URL and API key corresponding to your preferred provider when configuring these agents.

7.1 OpenClaw (Recommended) Copy Section

OpenClaw is a powerful agentic platform that works seamlessly with Step 3.5 Flash.
Step 3.5 Flash is a perfect fit for OpenClaw due to its high speed and strong agentic capabilities (maintaining deep reasoning and consistency during execution).

Quick Setup:

1. Install: curl -fsSL https://openclaw.ai/install.sh | bash
2. Onboard: Run openclaw onboard.
3. Configure: In WebUI (Config -> Models), add a new provider:
   • Type: openai-completions
   • Base URL:
     • StepFun: https://api.stepfun.ai/v1 (International) or https://api.stepfun.com/v1 (China)
     • OpenRouter: https://openrouter.ai/api/v1
   • Model ID: step-3.5-flash (or stepfun/step-3.5-flash for OpenRouter)

For a full walkthrough, see our OpenClaw Cookbook.

7.2 Claude Code Copy Section

You can configure Claude Code to use Step 3.5 Flash by modifying your settings.

Update ~/.claude/settings.json:

{
  "env": {
    "ANTHROPIC_AUTH_TOKEN": "YOUR_STEPFUN_API_KEY",
    "ANTHROPIC_BASE_URL": "https://api.stepfun.ai/"  // Use "https://api.stepfun.com/" for China endpoint
  },
  "model": "step-3.5-flash"
}

Note: For more advanced routing or OpenAI-style usage, we recommend claude-code-router.

7.3 OpenAI Codex (CLI) Copy Section

To use Step 3.5 Flash with the @openai/codex CLI, update your ~/.codex/config.toml:

model="step-3.5-flash"
model_provider = "stepfun-chat"
preferred_auth_method = "apikey"

[model_providers.stepfun-chat]
name = "OpenAI using response"
base_url = "https://api.stepfun.ai/v1"
env_key = "OPENAI_API_KEY"
wire_api = "chat"

7.4 Step-DeepResearch Copy Section

Step 3.5 Flash powers the reasoning core of Step-DeepResearch.
To use it, simply set MODEL_NAME to Step-3.5-Flash in your .env file.

See the Step-DeepResearch Repository for full setup instructions.

8. Cookbooks Copy Chapter

Explore our Cookbooks directory for practical examples and integration guides, including:

• Hybrid Local Agent: Build a local, privacy-first agentic sandbox.
• OpenClaw Integration: Deploy and integrate with OpenClaw.
• Roo Code Integration: Use Step 3.5 Flash as the backend for Roo Code in VS Code.

Visit the Cookbooks README for more details and contribution guidelines.

9. Known Issues and Future Directions Copy Chapter

1. Token Efficiency. Step 3.5 Flash achieves frontier-level agentic intelligence but currently relies on longer generation trajectories than Gemini 3.0 Pro to reach comparable quality.
2. Efficient Universal Mastery. We aim to unify generalist versatility with deep domain expertise. To achieve this efficiently, we are advancing variants of on-policy distillation, allowing the model to internalize expert behaviors with higher sample efficiency.
3. RL for More Agentic Tasks. While Step 3.5 Flash demonstrates competitive performance on academic agentic benchmarks, the next frontier of agentic AI necessitates the application of RL to intricate, expert-level tasks found in professional work, engineering, and research.
4. Operational Scope and Constraints. Step 3.5 Flash is tailored for coding and work-centric tasks, but may experience reduced stability during distribution shifts. This typically occurs in highly specialized domains or long-horizon, multi-turn dialogues, where the model may exhibit repetitive reasoning, mixed-language outputs, or inconsistencies in time and identity awareness.

10. Co-Developing the Future Copy Chapter

We view our roadmap as a living document, evolving continuously based on real-world usage and developer feedback.
As we work to shape the future of AGI by expanding broad model capabilities, we want to ensure we are solving the right problems. We invite you to be part of this continuous feedback loop—your insights directly influence our priorities.

• Join the Conversation: Our Discord community is the primary hub for brainstorming future architectures, proposing capabilities, and getting early access updates 🚀
• Report Friction: Encountering limitations? You can open an issue on GitHub or flag it directly in our Discord support channels.

License Copy Chapter

This project is open-sourced under the Apache 2.0 License.

Star History Copy Chapter

Llama.Cpp Docs Backend Hexagon README (`llama.cpp/docs/backend/hexagon/README.md`) Copy Section

Snapdragon-based Android devices

How to Build Copy Chapter

The easiest way to build llama.cpp for a Snapdragon-based Android device is using the toolchain Docker image (see github.com/snapdragon-toolchain).
This image includes Android NDK, OpenCL SDK, Hexagon SDK, CMake, etc.

This method works on Linux, macOS, and Windows. macOS and Windows users should install Docker Desktop.

~/src/llama.cpp$ docker run -it -u $(id -u):$(id -g) --volume $(pwd):/workspace --platform linux/amd64 ghcr.io/snapdragon-toolchain/arm64-android:v0.3
[d]/> cd /workspace

The rest of the Android build process assumes that you're running inside the toolchain container.
Let's build llama.cpp with CPU, OpenCL, and Hexagon backends via CMake presets:

[d]/workspace> cp docs/backend/hexagon/CMakeUserPresets.json .

[d]/workspace> cmake --preset arm64-android-snapdragon-release -B build-snapdragon
Preset CMake variables:
  ANDROID_ABI="arm64-v8a"
  ...
  CMAKE_TOOLCHAIN_FILE="/opt/android-ndk-r28b/build/cmake/android.toolchain.cmake"
  GGML_HEXAGON="ON"
  GGML_OPENCL="ON"
  GGML_OPENMP="OFF"
  HEXAGON_SDK_ROOT="/opt/hexagon/6.4.0.2"
...
-- Including OpenCL backend
-- Including Hexagon backend
...
-- Build files have been written to: /workspace/build-snapdragon

[d]/workspace> cmake --build build-snapdragon
...
[144/356] Performing build step for 'htp-v73'
[1/16] Generating htp_iface_skel.c, htp_iface_stub.c, htp_iface.h
[2/16] Building C object CMakeFiles/ggml-htp-v73.dir/hvx-sigmoid.c.obj
[3/16] Building C object CMakeFiles/ggml-htp-v73.dir/htp-dma.c.obj
[4/16] Building C object CMakeFiles/ggml-htp-v73.dir/worker-pool.c.obj
...
-- Installing: /workspace/build-snapdragon/ggml/src/ggml-hexagon/libggml-htp-v73.so
-- Installing: /workspace/build-snapdragon/ggml/src/ggml-hexagon/libggml-htp-v75.so
...

To generate an installable "package" simply use cmake --install:

[d]/workspace> cmake --install build-snapdragon --prefix pkg-adb/llama.cpp
-- Install configuration: "Release"
-- Installing: /workspace/pkg-adb/llama.cpp/lib/libggml-cpu.so
-- Installing: /workspace/pkg-adb/llama.cpp/lib/libggml-opencl.so
-- Installing: /workspace/pkg-adb/llama.cpp/lib/libggml-hexagon.so
-- Installing: /workspace/pkg-adb/llama.cpp/lib/libggml-htp-v73.so
-- Installing: /workspace/pkg-adb/llama.cpp/lib/libggml-htp-v75.so
-- Installing: /workspace/pkg-adb/llama.cpp/lib/libggml-htp-v79.so
-- Installing: /workspace/pkg-adb/llama.cpp/lib/libggml-htp-v81.so
-- Installing: /workspace/pkg-adb/llama.cpp/lib/libggml.so
...
-- Installing: /workspace/pkg-adb/llama.cpp/bin/llama-bench
-- Installing: /workspace/pkg-adb/llama.cpp/bin/llama-cli
...

How to Install Copy Chapter

For this step, your device needs to be configured for on-device development.
Please see https://developer.android.com/studio/debug/dev-options for details.

Once ADB is enabled, use adb push to install pkg-snapdragon on the device.
Note that the toolchain Docker image doesn't have ADB and doesn't set up the ADB bridge. Please use native ADB on the host.

~/src/llama.cpp$ adb push pkg-adb/llama.cpp /data/local/tmp/
pkg-adb/llama.cpp/bin/: 67 files pushed, 0 skipped. 190.2 MB/s (919095042 bytes in 4.607s)
pkg-adb/llama.cpp/include/: 19 files pushed, 0 skipped. 20.5 MB/s (255173 bytes in 0.012s)
pkg-adb/llama.cpp/lib/: 16 files pushed, 0 skipped. 144.4 MB/s (43801382 bytes in 0.289s)
102 files pushed, 0 skipped. 186.9 MB/s (963151597 bytes in 4.914s)

At this point, you should also install some models:

~/src/llama.cpp$ wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_0.gguf
...
2025-10-11 12:04:52 (10.7 MB/s) - ‘Llama-3.2-1B-Instruct-Q4_0.gguf’ saved [773025920/773025920]

~/src/llama.cpp$ adb push Llama-3.2-1B-Instruct-Q4_0.gguf /data/local/tmp/gguf
Llama-3.2-1B-Instruct-Q4_0.gguf: 1 file pushed, 0 skipped. 38.3 MB/s (773025920 bytes in 19.250s)

How to Run Copy Chapter

The easiest way to run llama.cpp cli tools is using provided wrapper scripts that properly set up all required environment variables.

llama.cpp supports three backends on Snapdragon-based devices: CPU, Adreno GPU (GPUOpenCL), and Hexagon NPU (HTP0-4).
You can select which backend to run the model on using the D= variable, which maps to the --device option.

Hexagon NPU behaves as a "GPU" device when it comes to -ngl and other offload-related options.

Here are some examples of running various llama.cpp tools via ADB.

Simple question for Llama-3.2-1B

/* Detailed source-code truncated for AI context efficiency. */

Summary request for OLMoE-1B-7B. This is a large model that requires two HTP sessions/devices

/* Detailed source-code truncated for AI context efficiency. */

Op test for MUL_MAT

~/src/llama.cpp$ HB=0 ./scripts/snapdragon/adb/run-tool.sh test-backend-ops -b HTP0 -o MUL_MAT
...
Backend 2/3: HTP0
Device description: Hexagon
Device memory: 2048 MB (2048 MB free)
MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK
MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK
MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK

~/src/llama.cpp-hexagon$ M=Llama-3.2-1B-Instruct-Q4_0.gguf ./scripts/snapdragon/adb/run-bench.sh -p 128 -n 64
...
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v79
ggml-hex: allocating new session: HTP0
ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v79.so?htp_iface_skel_handle_invoke&_modver=1.0&_dom=cdsp&_session=0 handle 0xb400007d4b231090
| model          |       size | params | backend    | ngl | threads | n_batch | mmap |  test |           t/s |
| ---------------| ---------: | -----: | ---------- | --: | ------: | ------: | ---: | ----: | ------------: |
| llama 1B Q4_0  | 729.75 MiB | 1.24 B | HTP        |  99 |       4 |     128 |    0 | pp128 | 169.42 ± 1.75 |
| llama 1B Q4_0  | 729.75 MiB | 1.24 B | HTP        |  99 |       4 |     128 |    0 |  tg64 |  51.54 ± 1.13 |

build: 6a8cf8914 (6733)

Environment variables Copy Chapter

• GGML_HEXAGON_NDEV=1
  Controls the number of devices/sessions to allocate. The default is 1.
  Most quantized models under 4B fit into a single session; an 8B model needs two, and a 20B model needs four.

• GGML_HEXAGON_NHVX=0
  Controls the number of HVX hardware threads to use. The default is all (actual number varies depending on the hardware version).

• GGML_HEXAGON_HOSTBUF=1
  Controls whether the Hexagon backend allocates host buffers. By default, all buffers except for REPACK are host buffers.
  This option is required for testing Ops that require REPACK buffers (MUL_MAT and MUL_MAT_ID).

• GGML_HEXAGON_VERBOSE=1
  Enables verbose logging of Ops from the backend. Example output:

  text Copy

  ggml-hex: HTP0 graph-compute n_nodes 2
  ggml-hex: HTP0 matmul : blk.27.ffn_up.weight x ffn_norm-27 -> ffn_up-27 : 3072:8192 x 3072:1 -> 8192:1 : q4_0 x f32 -> f32 : HTP0 x HTP0 -> HTP0 : flags 0x1
  ggml-hex: HTP0 matmul : blk.27.ffn_gate.weight x ffn_norm-27 -> ffn_gate-27 : 3072:8192 x 3072:1 -> 8192:1 : q4_0 x f32 -> f32 : HTP0 x HTP0 -> HTP0 : flags 0x3
  ggml-hex: HTP0 graph-compute n_nodes 1
  ggml-hex: HTP0 matmul : blk.27.ffn_down.weight x ffn_gate_par-27 -> ffn_out-27 : 8192:3072 x 8192:1 -> 3072:1 : q4_0 x f32 -> f32 : HTP0 x HTP0 -> HTP0 : flags 0x0
  ggml-hex: HTP0 get-tensor result_output : data 0x7592487000 offset 0 size 513024

• GGML_HEXAGON_PROFILE=1
  Generates a host-side profile for the ggml-hexagon Ops.

• GGML_HEXAGON_OPMASK=0x0
  Allows enabling specific stages of the processing pipeline:

  • 0x1 Enable Op Queue (i.e., queuing Ops into NPU)
  • 0x2 Enable Dynamic Quantizer (if needed for the Op)
  • 0x4 Enable Op Compute (MUL_MAT, etc.)

  Examples:

  text Copy

  `GGML_HEXAGON_OPMASK=0x1 llama-completion ...` - Ops are enqueued but NPU-side processing is stubbed out
  `GGML_HEXAGON_OPMASK=0x3 llama-completion ...` - NPU performs dynamic quantization and skips the rest
  `GGML_HEXAGON_OPMASK=0x7 llama-completion ...` - Full queuing and processing of Ops (default)

Llama.Cpp Docs Android (`llama.cpp/docs/android.md`) Copy Section

Android

Build GUI binding using Android Studio Copy Chapter

Import the examples/llama.android directory into Android Studio, then perform a Gradle sync and build the project.

This Android binding supports hardware acceleration up to SME2 for Arm and AMX for x86-64 CPUs on Android and ChromeOS devices.
It automatically detects the host's hardware to load compatible kernels. As a result, it runs seamlessly on both the latest premium devices and older devices that may lack modern CPU features or have limited RAM, without requiring any manual configuration.

A minimal Android app frontend is included to showcase the binding’s core functionalities:

1. Parse GGUF metadata via GgufMetadataReader from either a ContentResolver provided Uri from shared storage, or a local File from your app's private storage.
2. Obtain a InferenceEngine instance through the AiChat facade and load your selected model via its app-private file path.
3. Send a raw user prompt for automatic template formatting, prefill, and batch decoding. Then collect the generated tokens in a Kotlin Flow.

For a production-ready experience that leverages advanced features such as system prompts and benchmarks, plus friendly UI features such as model management and Arm feature visualizer, check out Arm AI Chat on Google Play.
This project is made possible through a collaborative effort by Arm's CT-ML, CE-ML and STE groups:

Build CLI on Android using Termux Copy Chapter

Termux is an Android terminal emulator and Linux environment app (no root required). As of writing, Termux is available experimentally in the Google Play Store; otherwise, it may be obtained directly from the project repo or on F-Droid.

With Termux, you can install and run llama.cpp as if the environment were Linux. Once in the Termux shell:

$ apt update && apt upgrade -y
$ apt install git cmake

Then, follow the build instructions, specifically for CMake.

Once the binaries are built, download your model of choice (e.g., from Hugging Face). It's recommended to place it in the ~/ directory for best performance:

$ curl -L {model-url} -o ~/{model}.gguf

Then, if you are not already in the repo directory, cd into llama.cpp and:

$ ./build/bin/llama-cli -m ~/{model}.gguf -c {context-size} -p "{your-prompt}"

Here, we show llama-cli, but any of the executables under examples should work, in theory. Be sure to set context-size to a reasonable number (say, 4096) to start with; otherwise, memory could spike and kill your terminal.

To see what it might look like visually, here's an old demo of an interactive session running on a Pixel 5 phone:

https://user-images.githubusercontent.com/271616/225014776-1d567049-ad71-4ef2-b050-55b0b3b9274c.mp4

Cross-compile CLI using Android NDK Copy Chapter

It's possible to build llama.cpp for Android on your host system via CMake and the Android NDK. If you are interested in this path, ensure you already have an environment prepared to cross-compile programs for Android (i.e., install the Android SDK). Note that, unlike desktop environments, the Android environment ships with a limited set of native libraries, and so only those libraries are available to CMake when building with the Android NDK (see: https://developer.android.com/ndk/guides/stable_apis.)

Once you're ready and have cloned llama.cpp, invoke the following in the project directory:

$ cmake \
  -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=android-28 \
  -DCMAKE_C_FLAGS="-march=armv8.7a" \
  -DCMAKE_CXX_FLAGS="-march=armv8.7a" \
  -DGGML_OPENMP=OFF \
  -DGGML_LLAMAFILE=OFF \
  -B build-android

Notes:

• While later versions of Android NDK ship with OpenMP, it must still be installed by CMake as a dependency, which is not supported at this time
• llamafile does not appear to support Android devices (see: https://github.com/Mozilla-Ocho/llamafile/issues/325)

The above command should configure llama.cpp with the most performant options for modern devices. Even if your device is not running armv8.7a, llama.cpp includes runtime checks for available CPU features it can use.

Feel free to adjust the Android ABI for your target. Once the project is configured:

$ cmake --build build-android --config Release -j{n}
$ cmake --install build-android --prefix {install-dir} --config Release

After installing, go ahead and download the model of your choice to your host system. Then:

$ adb shell "mkdir /data/local/tmp/llama.cpp"
$ adb push {install-dir} /data/local/tmp/llama.cpp/
$ adb push {model}.gguf /data/local/tmp/llama.cpp/
$ adb shell

In the adb shell:

$ cd /data/local/tmp/llama.cpp
$ LD_LIBRARY_PATH=lib ./bin/llama-simple -m {model}.gguf -c {context-size} -p "{your-prompt}"

That's it!

Be aware that Android will not find the library path lib on its own, so we must specify LD_LIBRARY_PATH in order to run the installed executables. Android does support RPATH in later API levels, so this could change in the future. Refer to the previous section for information about context-size (very important!) and running other examples.

Llama.Cpp Docs Build Riscv64 Spacemit (`llama.cpp/docs/build-riscv64-spacemit.md`) Copy Section

[!IMPORTANT]
This build documentation is specific only to RISC-V SpacemiT SOCs.

Build llama.cpp locally (for riscv64) Copy Chapter

1. Prepare Toolchain For RISCV

wget https://archive.spacemit.com/toolchain/spacemit-toolchain-linux-glibc-x86_64-v1.1.2.tar.xz

2. Build
   Below is the build script: it requires utilizing RISC-V vector instructions for acceleration. Ensure the GGML_CPU_RISCV64_SPACEMIT compilation option is enabled. The currently supported optimization version is RISCV64_SPACEMIT_IME1, corresponding to the RISCV64_SPACEMIT_IME_SPEC compilation option. Compiler configurations are defined in the riscv64-spacemit-linux-gnu-gcc.cmake file. Please ensure you have installed the RISC-V compiler and set the environment variable via export RISCV_ROOT_PATH={your_compiler_path}.

cmake -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_CPU_RISCV64_SPACEMIT=ON \
    -DLLAMA_CURL=OFF \
    -DGGML_RVV=ON \
    -DGGML_RV_ZFH=ON \
    -DGGML_RV_ZICBOP=ON \
    -DGGML_RV_ZIHINTPAUSE=ON \
    -DRISCV64_SPACEMIT_IME_SPEC=RISCV64_SPACEMIT_IME1 \
    -DCMAKE_TOOLCHAIN_FILE=${PWD}/cmake/riscv64-spacemit-linux-gnu-gcc.cmake \
    -DCMAKE_INSTALL_PREFIX=build/installed

cmake --build build --parallel $(nproc) --config Release

pushd build
make install
popd

Simulation Copy Chapter

You can use QEMU to perform emulation on non-RISC-V architectures.

1. Download QEMU

wget https://archive.spacemit.com/spacemit-ai/qemu/jdsk-qemu-v0.0.14.tar.gz

2. Run Simulation
   After build your llama.cpp, you can run the executable file via QEMU for simulation, for example:

export QEMU_ROOT_PATH={your QEMU file path}
export RISCV_ROOT_PATH_IME1={your RISC-V compiler path}

${QEMU_ROOT_PATH}/bin/qemu-riscv64 -L ${RISCV_ROOT_PATH_IME1}/sysroot -cpu max,vlen=256,elen=64,vext_spec=v1.0 ${PWD}/build/bin/llama-cli -m ${PWD}/models/Qwen2.5-0.5B-Instruct-Q4_0.gguf -t 1

Performance Copy Chapter

Quantization Support For Matrix

model name      : Spacemit(R) X60
isa             : rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnapot_svpbmt
mmu             : sv39
uarch           : spacemit,x60
mvendorid       : 0x710
marchid         : 0x8000000058000001

Q4_0

Q4_1

Q4_K

Llama.Cpp Docs Build S390x (`llama.cpp/docs/build-s390x.md`) Copy Section

[!IMPORTANT]
This build documentation is specific only to IBM Z & LinuxONE mainframes (s390x). You can find the build documentation for other architectures: build.md.

Build llama.cpp locally (for s390x)

The main product of this project is the llama library. Its C-style interface can be found in include/llama.h.

The project also includes many example programs and tools using the llama library. The examples range from simple, minimal code snippets to sophisticated sub-projects such as an OpenAI-compatible HTTP server.

To get the code:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

CPU Build with BLAS Copy Chapter

Building llama.cpp with BLAS support is highly recommended as it has shown to provide performance improvements. Make sure to have OpenBLAS installed in your environment.

cmake -S . -B build             \
    -DCMAKE_BUILD_TYPE=Release  \
    -DGGML_BLAS=ON              \
    -DGGML_BLAS_VENDOR=OpenBLAS

cmake --build build --config Release -j $(nproc)

Notes:

• For faster repeated compilation, install ccache

• By default, VXE/VXE2 is enabled. To disable it (not recommended):

  bash Copy

  cmake -S . -B build             \
      -DCMAKE_BUILD_TYPE=Release  \
      -DGGML_BLAS=ON              \
      -DGGML_BLAS_VENDOR=OpenBLAS \
      -DGGML_VXE=OFF
  
  cmake --build build --config Release -j $(nproc)

• For debug builds:

  bash Copy

  cmake -S . -B build             \
      -DCMAKE_BUILD_TYPE=Debug    \
      -DGGML_BLAS=ON              \
      -DGGML_BLAS_VENDOR=OpenBLAS
  cmake --build build --config Debug -j $(nproc)

• For static builds, add -DBUILD_SHARED_LIBS=OFF:

  bash Copy

  cmake -S . -B build             \
      -DCMAKE_BUILD_TYPE=Release  \
      -DGGML_BLAS=ON              \
      -DGGML_BLAS_VENDOR=OpenBLAS \
      -DBUILD_SHARED_LIBS=OFF
  
  cmake --build build --config Release -j $(nproc)

IBM zDNN Accelerator Copy Chapter

This provides acceleration using the IBM zAIU co-processor located in the Telum I and Telum II processors. Make sure to have the IBM zDNN library installed.

Compile from source from IBM

You may find the official build instructions here: Building and Installing zDNN

Compilation Copy Section

cmake -S . -B build             \
    -DCMAKE_BUILD_TYPE=Release  \
    -DGGML_ZDNN=ON
cmake --build build --config Release -j$(nproc)

Getting GGUF Models Copy Chapter

All models need to be converted to Big-Endian. You can achieve this in three cases:

1. Use pre-converted models verified for use on IBM Z & LinuxONE (easiest)

   You can find popular models pre-converted and verified at s390x Verified Models or s390x Runnable Models.

   These models have already been converted from safetensors to GGUF Big-Endian and their respective tokenizers verified to run correctly on IBM z15 and later system.

2. Convert safetensors model to GGUF Big-Endian directly (recommended)

   The model you are trying to convert must be in safetensors file format (for example IBM Granite 3.3 2B). Make sure you have downloaded the model repository for this case.

   Ensure that you have installed the required packages in advance

   bash Copy

   pip3 install -r requirements.txt

   Convert the safetensors model to GGUF

   bash Copy

   python3 convert_hf_to_gguf.py \
       --outfile model-name-be.f16.gguf \
       --outtype f16 \
       --bigendian \
       model-directory/

   For example,

   bash Copy

   python3 convert_hf_to_gguf.py \
       --outfile granite-3.3-2b-instruct-be.f16.gguf \
       --outtype f16 \
       --bigendian \
       granite-3.3-2b-instruct/

3. Convert existing GGUF Little-Endian model to Big-Endian

   The model you are trying to convert must be in gguf file format (for example IBM Granite 3.3 2B GGUF). Make sure you have downloaded the model file for this case.

   bash Copy

   python3 gguf-py/gguf/scripts/gguf_convert_endian.py model-name.f16.gguf BIG

   For example,

   bash Copy

   python3 gguf-py/gguf/scripts/gguf_convert_endian.py granite-3.3-2b-instruct-le.f16.gguf BIG
   mv granite-3.3-2b-instruct-le.f16.gguf granite-3.3-2b-instruct-be.f16.gguf

   Notes:

   • The GGUF endian conversion script may not support all data types at the moment and may fail for some models/quantizations. When that happens, please try manually converting the safetensors model to GGUF Big-Endian via Step 2.

IBM Accelerators Copy Chapter

1. SIMD Acceleration Copy Section

Only available in IBM z15/LinuxONE 3 or later system with the -DGGML_VXE=ON (turned on by default) compile flag. No hardware acceleration is possible with llama.cpp with older systems, such as IBM z14/arch12. In such systems, the APIs can still run but will use a scalar implementation.

2. zDNN Accelerator (WIP) Copy Section

Only available in IBM z17/LinuxONE 5 or later system with the -DGGML_ZDNN=ON compile flag. No hardware acceleration is possible with llama.cpp with older systems, such as IBM z15/arch13. In such systems, the APIs will default back to CPU routines.

3. Spyre Accelerator Copy Section

Only available with IBM z17 / LinuxONE 5 or later system. No support currently available.

Performance Tuning Copy Chapter

1. Virtualization Setup Copy Section

It is strongly recommended to use only LPAR (Type-1) virtualization to get the most performance.

Note: Type-2 virtualization is not supported at the moment, while you can get it running, the performance will not be the best.

2. IFL (Core) Count Copy Section

It is recommended to allocate a minimum of 8 shared IFLs assigned to the LPAR. Increasing the IFL count past 8 shared IFLs will only improve Prompt Processing performance but not Token Generation.

Note: IFL count does not equate to vCPU count.

3. SMT vs NOSMT (Simultaneous Multithreading) Copy Section

It is strongly recommended to disable SMT via the kernel boot parameters as it negatively affects performance. Please refer to your Linux distribution's guide on disabling SMT via kernel boot parameters.

4. BLAS vs NOBLAS Copy Section

IBM VXE/VXE2 SIMD acceleration depends on the BLAS implementation. It is strongly recommended to use BLAS.

Frequently Asked Questions (FAQ) Copy Chapter

1. I'm getting the following error message while trying to load a model: gguf_init_from_file_impl: failed to load model: this GGUF file version 50331648 is extremely large, is there a mismatch between the host and model endianness?

   Answer: Please ensure that the model you have downloaded/converted is GGUFv3 Big-Endian. These models are usually denoted with the -be suffix, i.e., granite-3.3-2b-instruct-be.F16.gguf.

   You may refer to the Getting GGUF Models section to manually convert a safetensors model to GGUF Big Endian.

2. I'm getting extremely poor performance when running inference on a model

   Answer: Please refer to the Appendix B: SIMD Support Matrix to check if your model quantization is supported by SIMD acceleration.

3. I'm building on IBM z17 and getting the following error messages: invalid switch -march=z17

   Answer: Please ensure that your GCC compiler is of minimum GCC 15.1.0 version, and have binutils updated to the latest version. If this does not fix the problem, kindly open an issue.

4. Failing to install the sentencepiece package using GCC 15+

   Answer: The sentencepiece team are aware of this as seen in this issue.

   As a temporary workaround, please run the installation command with the following environment variables.

   bash Copy

   export CXXFLAGS="-include cstdint"

   For example,

   bash Copy

   CXXFLAGS="-include cstdint" pip3 install -r requirements.txt

Getting Help on IBM Z & LinuxONE Copy Chapter

1. Bugs, Feature Requests

   Please file an issue in llama.cpp and ensure that the title contains "s390x".

2. Other Questions

   Please reach out directly to [email protected].

Appendix A: Hardware Support Matrix Copy Chapter

• ✅ - supported and verified to run as intended
• 🚫 - unsupported, we are unlikely able to provide support

Appendix B: SIMD Support Matrix Copy Chapter

• ✅ - acceleration available
• 🚫 - acceleration unavailable, will still run using scalar implementation
• ❓ - acceleration unknown, please contribute if you can test it yourself

Last Updated by Aaron Teo ([email protected]) on Sep 7, 2025.

Llama.Cpp Docs Build (`llama.cpp/docs/build.md`) Copy Section

Build llama.cpp locally

The main product of this project is the llama library. Its C-style interface can be found in include/llama.h.

The project also includes many example programs and tools using the llama library. The examples range from simple, minimal code snippets to sophisticated sub-projects such as an OpenAI-compatible HTTP server.

To get the Code:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

The following sections describe how to build with different backends and options.

CPU Build Copy Chapter

Build llama.cpp using CMake:

cmake -B build
cmake --build build --config Release

Notes:

• For faster compilation, add the -j argument to run multiple jobs in parallel, or use a generator that does this automatically such as Ninja. For example, cmake --build build --config Release -j 8 will run 8 jobs in parallel.

• For faster repeated compilation, install ccache

• For debug builds, there are two cases:

  1. Single-config generators (e.g. default = Unix Makefiles; note that they just ignore the --config flag):

     bash Copy

     cmake -B build -DCMAKE_BUILD_TYPE=Debug
     cmake --build build

  2. Multi-config generators (-G param set to Visual Studio, XCode...):

     bash Copy

     cmake -B build -G "Xcode"
     cmake --build build --config Debug

  For more details and a list of supported generators, see the CMake documentation.

• For static builds, add -DBUILD_SHARED_LIBS=OFF:

  text Copy

  cmake -B build -DBUILD_SHARED_LIBS=OFF
  cmake --build build --config Release

• Building for Windows (x86, x64 and arm64) with MSVC or clang as compilers:

  • Install Visual Studio 2022, e.g. via the Community Edition. In the installer, select at least the following options (this also automatically installs the required additional tools like CMake,...):
  • Tab Workload: Desktop-development with C++
  • Tab Components (select quickly via search): C++-CMake Tools for Windows, Git for Windows, C++-Clang Compiler for Windows, MS-Build Support for LLVM-Toolset (clang)
  • Please remember to always use a Developer Command Prompt / PowerShell for VS2022 for git, build, test
  • For Windows on ARM (arm64, WoA) build with:
  bash Copy

  cmake --preset arm64-windows-llvm-release -D GGML_OPENMP=OFF
  cmake --build build-arm64-windows-llvm-release

  For building with ninja generator and clang compiler as default:
  -set path:set LIB=C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22621.0\um\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\lib\x64\uwp;C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22621.0\ucrt\x64
  bash cmake --preset x64-windows-llvm-release cmake --build build-x64-windows-llvm-release

• Curl usage is enabled by default and can be turned off with -DLLAMA_CURL=OFF. Otherwise you need to install development libraries for libcurl.

  • Debian / Ubuntu: sudo apt-get install libcurl4-openssl-dev # (or libcurl4-gnutls-dev if you prefer GnuTLS)
  • Fedora / RHEL / Rocky / Alma: sudo dnf install libcurl-devel
  • Arch / Manjaro: sudo pacman -S curl # includes libcurl headers

BLAS Build Copy Chapter

Building the program with BLAS support may lead to some performance improvements in prompt processing using batch sizes higher than 32 (the default is 512). Using BLAS doesn't affect the generation performance. There are currently several different BLAS implementations available for build and use:

Accelerate Framework Copy Section

This is only available on Mac PCs and it's enabled by default. You can just build using the normal instructions.

OpenBLAS Copy Section

This provides BLAS acceleration using only the CPU. Make sure to have OpenBLAS installed on your machine.

• Using CMake on Linux:

  bash Copy

  cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
  cmake --build build --config Release

BLIS Copy Section

Check BLIS.md for more information.

Intel oneMKL Copy Section

Building through oneAPI compilers will make avx_vnni instruction set available for intel processors that do not support avx512 and avx512_vnni. Please note that this build config does not support Intel GPU. For Intel GPU support, please refer to llama.cpp for SYCL.

• Using manual oneAPI installation:
  By default, GGML_BLAS_VENDOR is set to Generic, so if you already sourced intel environment script and assign -DGGML_BLAS=ON in cmake, the mkl version of Blas will automatically been selected. Otherwise please install oneAPI and follow the below steps:

  bash Copy

  source /opt/intel/oneapi/setvars.sh # You can skip this step if  in oneapi-basekit docker image, only required for manual installation
  cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64lp -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_NATIVE=ON
  cmake --build build --config Release

• Using oneAPI docker image:
  If you do not want to source the environment vars and install oneAPI manually, you can also build the code using intel docker container: oneAPI-basekit. Then, you can use the commands given above.

Check Optimizing and Running LLaMA2 on Intel® CPU for more information.

Other BLAS libraries Copy Section

Any other BLAS library can be used by setting the GGML_BLAS_VENDOR option. See the CMake documentation for a list of supported vendors.

Metal Build Copy Chapter

On MacOS, Metal is enabled by default. Using Metal makes the computation run on the GPU.
To disable the Metal build at compile time use the -DGGML_METAL=OFF cmake option.

When built with Metal support, you can explicitly disable GPU inference with the --n-gpu-layers 0 command-line argument.

SYCL Copy Chapter

SYCL is a higher-level programming model to improve programming productivity on various hardware accelerators.

llama.cpp based on SYCL is used to support Intel GPU (Data Center Max series, Flex series, Arc series, Built-in GPU and iGPU).

For detailed info, please refer to llama.cpp for SYCL.

CUDA Copy Chapter

This provides GPU acceleration using an NVIDIA GPU. Make sure to have the CUDA toolkit installed.

Download directly from NVIDIA

You may find the official downloads here: NVIDIA developer site.

Compile and run inside a Fedora Toolbox Container

We also have a guide for setting up CUDA toolkit in a Fedora toolbox container.

Recommended for:

• Necessary for users of Atomic Desktops for Fedora; such as: Silverblue and Kinoite.
  • (there are no supported CUDA packages for these systems)
• Necessary for users that have a host that is not a: Supported Nvidia CUDA Release Platform.
  • (for example, you may have Fedora 42 Beta as your your host operating system)
• Convenient For those running Fedora Workstation or Fedora KDE Plasma Desktop, and want to keep their host system clean.
• Optionally toolbox packages are available: Arch Linux, Red Hat Enterprise Linux >= 8.5, or Ubuntu

Compilation Copy Section

Make sure to read the notes about the CPU build for general instructions for e.g. speeding up the compilation.

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

Non-Native Builds Copy Section

By default llama.cpp will be built for the hardware that is connected to the system at that time.
For a build covering all CUDA GPUs, disable GGML_NATIVE:

cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF

The resulting binary should run on all CUDA GPUs with optimal performance, though some just-in-time compilation may be required.

Override Compute Capability Specifications Copy Section

If nvcc cannot detect your gpu, you may get compile warnings such as:

nvcc warning : Cannot find valid GPU for '-arch=native', default arch is used

One option is to do a non-native build as described above.
However, this will result in a large binary that takes a long time to compile.
Alternatively it is also possible to explicitly specify CUDA architectures.
This may also make sense for a non-native build, for that one should look at the logic in ggml/src/ggml-cuda/CMakeLists.txt as a starting point.

To override the default CUDA architectures:

1. Take note of the `Compute Capability` of your NVIDIA devices: ["CUDA: Your GPU Compute > Capability"](https://developer.nvidia.com/cuda-gpus).

GeForce RTX 4090      8.9
GeForce RTX 3080 Ti   8.6
GeForce RTX 3070      8.6

2. Manually list each varying `Compute Capability` in the `CMAKE_CUDA_ARCHITECTURES` list.

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86;89"

Overriding the CUDA Version Copy Section

If you have multiple CUDA installations on your system and want to compile llama.cpp for a specific one, e.g. for CUDA 11.7 installed under /opt/cuda-11.7:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER=/opt/cuda-11.7/bin/nvcc -DCMAKE_INSTALL_RPATH="/opt/cuda-11.7/lib64;\$ORIGIN" -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON

Fixing Compatibility Issues with Old CUDA and New glibc

If you try to use an old CUDA version (e.g. v11.7) with a new glibc version you can get errors like this:

/usr/include/bits/mathcalls.h(83): error: exception specification is
  incompatible with that of previous function "cospi"


  /opt/cuda-11.7/bin/../targets/x86_64-linux/include/crt/math_functions.h(5545):
  here

It seems the least bad solution is to patch the CUDA installation to declare the correct signatures.
Replace the following lines in /path/to/your/cuda/installation/targets/x86_64-linux/include/crt/math_functions.h:

// original lines
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ double                 cospi(double x);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ float                  cospif(float x);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ double                 sinpi(double x);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ float                  sinpif(float x);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ double                 rsqrt(double x);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ float                  rsqrtf(float x);

// edited lines
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ double                 cospi(double x) noexcept (true);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ float                  cospif(float x) noexcept (true);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ double                 sinpi(double x) noexcept (true);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ float                  sinpif(float x) noexcept (true);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ double                 rsqrt(double x) noexcept (true);
extern __DEVICE_FUNCTIONS_DECL__ __device_builtin__ float                  rsqrtf(float x) noexcept (true);

Runtime CUDA environmental variables Copy Section

You may set the cuda environmental variables at runtime.

# Use `CUDA_VISIBLE_DEVICES` to hide the first compute device.
CUDA_VISIBLE_DEVICES="-0" ./build/bin/llama-server --model /srv/models/llama.gguf

Unified Memory Copy Section

The environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 can be used to enable unified memory in Linux. This allows swapping to system RAM instead of crashing when the GPU VRAM is exhausted. In Windows this setting is available in the NVIDIA control panel as System Memory Fallback.

Performance Tuning Copy Section

The following compilation options are also available to tweak performance:

MUSA Copy Chapter

This provides GPU acceleration using a Moore Threads GPU. Make sure to have the MUSA SDK installed.

Download directly from Moore Threads

You may find the official downloads here: Moore Threads developer site.

Compilation Copy Section

cmake -B build -DGGML_MUSA=ON
cmake --build build --config Release

Override Compute Capability Specifications

By default, all supported compute capabilities are enabled. To customize this behavior, you can specify the MUSA_ARCHITECTURES option in the CMake command:

cmake -B build -DGGML_MUSA=ON -DMUSA_ARCHITECTURES="21"
cmake --build build --config Release

This configuration enables only compute capability 2.1 (MTT S80) during compilation, which can help reduce compilation time.

Compilation options

Most of the compilation options available for CUDA should also be available for MUSA, though they haven't been thoroughly tested yet.

• For static builds, add -DBUILD_SHARED_LIBS=OFF and -DCMAKE_POSITION_INDEPENDENT_CODE=ON:
  text Copy

  cmake -B build -DGGML_MUSA=ON \
    -DBUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON
  cmake --build build --config Release

Runtime MUSA environmental variables Copy Section

You may set the musa environmental variables at runtime.

# Use `MUSA_VISIBLE_DEVICES` to hide the first compute device.
MUSA_VISIBLE_DEVICES="-0" ./build/bin/llama-server --model /srv/models/llama.gguf

Unified Memory Copy Section

The environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 can be used to enable unified memory in Linux. This allows swapping to system RAM instead of crashing when the GPU VRAM is exhausted.

HIP Copy Chapter

This provides GPU acceleration on HIP-supported AMD GPUs.
Make sure to have ROCm installed.
You can download it from your Linux distro's package manager or from here: ROCm Quick Start (Linux).

• Using CMake for Linux (assuming a gfx1030-compatible AMD GPU):

  bash Copy

  HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
      cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release \
      && cmake --build build --config Release -- -j 16

  Note: GPU_TARGETS is optional, omitting it will build the code for all GPUs in the current system.

  To enhance flash attention performance on RDNA3+ or CDNA architectures, you can utilize the rocWMMA library by enabling the -DGGML_HIP_ROCWMMA_FATTN=ON option. This requires rocWMMA headers to be installed on the build system.

  The rocWMMA library is included by default when installing the ROCm SDK using the rocm meta package provided by AMD. Alternatively, if you are not using the meta package, you can install the library using the rocwmma-dev or rocwmma-devel package, depending on your system's package manager.

  As an alternative, you can manually install the library by cloning it from the official GitHub repository, checkout the corresponding version tag (e.g. rocm-6.2.4) and set -DCMAKE_CXX_FLAGS="-I/library/include/" in CMake. This also works under Windows despite not officially supported by AMD.

  Note that if you get the following error:

  text Copy

  clang: error: cannot find ROCm device library; provide its path via '--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library

  Try searching for a directory under HIP_PATH that contains the file
  oclc_abi_version_400.bc. Then, add the following to the start of the
  command: HIP_DEVICE_LIB_PATH=<directory-you-just-found>, so something
  like:

  bash Copy

  HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -p)" \
  HIP_DEVICE_LIB_PATH=<directory-you-just-found> \
      cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release \
      && cmake --build build -- -j 16

• Using CMake for Windows (using x64 Native Tools Command Prompt for VS, and assuming a gfx1100-compatible AMD GPU):

  bash Copy

  set PATH=%HIP_PATH%\bin;%PATH%
  cmake -S . -B build -G Ninja -DGPU_TARGETS=gfx1100 -DGGML_HIP=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Release
  cmake --build build

  If necessary, adapt GPU_TARGETS to the GPU arch you want to compile for. The above example uses gfx1100 that corresponds to Radeon RX 7900XTX/XT/GRE. You can find a list of targets here
  Find your gpu version string by matching the most significant version information from rocminfo | grep gfx | head -1 | awk '{print $2}' with the list of processors, e.g. gfx1035 maps to gfx1030.

The environment variable HIP_VISIBLE_DEVICES can be used to specify which GPU(s) will be used.
If your GPU is not officially supported you can use the environment variable [HSA_OVERRIDE_GFX_VERSION] set to a similar GPU, for example 10.3.0 on RDNA2 (e.g. gfx1030, gfx1031, or gfx1035) or 11.0.0 on RDNA3.

Unified Memory Copy Section

On Linux it is possible to use unified memory architecture (UMA) to share main memory between the CPU and integrated GPU by setting environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1. However, this hurts performance for non-integrated GPUs (but enables working with integrated GPUs).

Vulkan Copy Chapter

For Windows Users: Copy Section

w64devkit

Download and extract w64devkit.

Download and install the Vulkan SDK with the default settings.

Launch w64devkit.exe and run the following commands to copy Vulkan dependencies:

SDK_VERSION=1.3.283.0
cp /VulkanSDK/$SDK_VERSION/Bin/glslc.exe $W64DEVKIT_HOME/bin/
cp /VulkanSDK/$SDK_VERSION/Lib/vulkan-1.lib $W64DEVKIT_HOME/x86_64-w64-mingw32/lib/
cp -r /VulkanSDK/$SDK_VERSION/Include/* $W64DEVKIT_HOME/x86_64-w64-mingw32/include/
cat > $W64DEVKIT_HOME/x86_64-w64-mingw32/lib/pkgconfig/vulkan.pc <<EOF
Name: Vulkan-Loader
Description: Vulkan Loader
Version: $SDK_VERSION
Libs: -lvulkan-1
EOF

Switch into the llama.cpp directory and build using CMake.

cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

Git Bash MINGW64

Download and install Git-SCM with the default settings

Download and install Visual Studio Community Edition and make sure you select C++

Download and install CMake with the default settings

Download and install the Vulkan SDK with the default settings.

Go into your llama.cpp directory and right click, select Open Git Bash Here and then run the following commands

cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

Now you can load the model in conversation mode using Vulkan

build/bin/Release/llama-cli -m "[PATH TO MODEL]" -ngl 100 -c 16384 -t 10 -n -2 -cnv

MSYS2

Install MSYS2 and then run the following commands in a UCRT terminal to install dependencies.

pacman -S git \
    mingw-w64-ucrt-x86_64-gcc \
    mingw-w64-ucrt-x86_64-cmake \
    mingw-w64-ucrt-x86_64-vulkan-devel \
    mingw-w64-ucrt-x86_64-shaderc

Switch into the llama.cpp directory and build using CMake.

cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

For Docker users: Copy Section

You don't need to install the Vulkan SDK. It will be installed inside the container.

# Build the image
docker build -t llama-cpp-vulkan --target light -f .devops/vulkan.Dockerfile .

# Then, use it:
docker run -it --rm -v "$(pwd):/app:Z" --device /dev/dri/renderD128:/dev/dri/renderD128 --device /dev/dri/card1:/dev/dri/card1 llama-cpp-vulkan -m "/app/models/YOUR_MODEL_FILE" -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33

For Linux users: Copy Section

Using the LunarG Vulkan SDK

First, follow the official LunarG instructions for the installation and setup of the Vulkan SDK in the Getting Started with the Linux Tarball Vulkan SDK guide.

[!IMPORTANT]
After completing the first step, ensure that you have used the source command on the setup_env.sh file inside of the Vulkan SDK in your current terminal session. Otherwise, the build won't work. Additionally, if you close out of your terminal, you must perform this step again if you intend to perform a build. However, there are ways to make this persistent. Refer to the Vulkan SDK guide linked in the first step for more information about any of this.

Using system packages

On Debian / Ubuntu, you can install the required dependencies using:

sudo apt-get install libvulkan-dev glslc

Common steps

Second, after verifying that you have followed all of the SDK installation/setup steps, use this command to make sure before proceeding:

vulkaninfo

Then, assuming you have cd into your llama.cpp folder and there are no errors with running vulkaninfo, you can proceed to build llama.cpp using the CMake commands below:

cmake -B build -DGGML_VULKAN=1
cmake --build build --config Release

Finally, after finishing your build, you should be able to do something like this:

# Test the output binary
# "-ngl 99" should offload all of the layers to GPU for most (if not all) models.
./build/bin/llama-cli -m "PATH_TO_MODEL" -p "Hi you how are you" -ngl 99

# You should see in the output, ggml_vulkan detected your GPU. For example:
# ggml_vulkan: Using Intel(R) Graphics (ADL GT2) | uma: 1 | fp16: 1 | warp size: 32

CANN Copy Chapter

This provides NPU acceleration using the AI cores of your Ascend NPU. And CANN is a hierarchical APIs to help you to quickly build AI applications and service based on Ascend NPU.

For more information about Ascend NPU in Ascend Community.

Make sure to have the CANN toolkit installed. You can download it from here: CANN Toolkit

Go to llama.cpp directory and build using CMake.

cmake -B build -DGGML_CANN=on -DCMAKE_BUILD_TYPE=release
cmake --build build --config release

You can test with:

./build/bin/llama-cli -m PATH_TO_MODEL -p "Building a website can be done in 10 steps:" -ngl 32

If the following info is output on screen, you are using llama.cpp with the CANN backend:

llm_load_tensors:       CANN model buffer size = 13313.00 MiB
llama_new_context_with_model:       CANN compute buffer size =  1260.81 MiB

For detailed info, such as model/device supports, CANN install, please refer to llama.cpp for CANN.

ZenDNN Copy Chapter

ZenDNN provides optimized deep learning primitives for AMD EPYC™ CPUs. It accelerates matrix multiplication operations for inference workloads.

Compilation Copy Section

• Using CMake on Linux (automatic build):

  bash Copy

  cmake -B build -DGGML_ZENDNN=ON
  cmake --build build --config Release

  The first build will automatically download and build ZenDNN, which may take 5-10 minutes. Subsequent builds will be much faster.

• Using CMake with custom ZenDNN installation:

  bash Copy

  cmake -B build -DGGML_ZENDNN=ON -DZENDNN_ROOT=/path/to/zendnn/install
  cmake --build build --config Release

Testing Copy Section

You can test with:

./build/bin/llama-cli -m PATH_TO_MODEL -p "Building a website can be done in 10 steps:" -n 50

For detailed information about hardware support, setup instructions, and performance optimization, refer to llama.cpp for ZenDNN.

Arm® KleidiAI™ Copy Chapter

KleidiAI is a library of optimized microkernels for AI workloads, specifically designed for Arm CPUs. These microkernels enhance performance and can be enabled for use by the CPU backend.

To enable KleidiAI, go to the llama.cpp directory and build using CMake

cmake -B build -DGGML_CPU_KLEIDIAI=ON
cmake --build build --config Release

You can verify that KleidiAI is being used by running

./build/bin/llama-cli -m PATH_TO_MODEL -p "What is a car?"

If KleidiAI is enabled, the ouput will contain a line similar to:

load_tensors: CPU_KLEIDIAI model buffer size =  3474.00 MiB

KleidiAI's microkernels implement optimized tensor operations using Arm CPU features such as dotprod, int8mm and SME. llama.cpp selects the most efficient kernel based on runtime CPU feature detection. However, on platforms that support SME, you must manually enable SME microkernels by setting the environment variable GGML_KLEIDIAI_SME=1.

Depending on your build target, other higher priority backends may be enabled by default. To ensure the CPU backend is used, you must disable the higher priority backends either at compile time, e.g. -DGGML_METAL=OFF, or during run-time using the command line option --device none.

OpenCL Copy Chapter

This provides GPU acceleration through OpenCL on recent Adreno GPU.
More information about OpenCL backend can be found in OPENCL.md for more information.

Android Copy Section

Assume NDK is available in $ANDROID_NDK. First, install OpenCL headers and ICD loader library if not available,

mkdir -p ~/dev/llm
cd ~/dev/llm

git clone https://github.com/KhronosGroup/OpenCL-Headers && \
cd OpenCL-Headers && \
cp -r CL $ANDROID_NDK/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include

cd ~/dev/llm

git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && \
cd OpenCL-ICD-Loader && \
mkdir build_ndk && cd build_ndk && \
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
  -DOPENCL_ICD_LOADER_HEADERS_DIR=$ANDROID_NDK/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=24 \
  -DANDROID_STL=c++_shared && \
ninja && \
cp libOpenCL.so $ANDROID_NDK/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android

Then build llama.cpp with OpenCL enabled,

cd ~/dev/llm

git clone https://github.com/ggml-org/llama.cpp && \
cd llama.cpp && \
mkdir build-android && cd build-android

cmake .. -G Ninja \
  -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=android-28 \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_OPENCL=ON

ninja

Windows Arm64 Copy Section

First, install OpenCL headers and ICD loader library if not available,

mkdir -p ~/dev/llm

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers
mkdir build && cd build
cmake .. -G Ninja `
  -DBUILD_TESTING=OFF `
  -DOPENCL_HEADERS_BUILD_TESTING=OFF `
  -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF `
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader
mkdir build && cd build
cmake .. -G Ninja `
  -DCMAKE_BUILD_TYPE=Release `
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" `
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

Then build llama.cpp with OpenCL enabled,

cmake .. -G Ninja `
  -DCMAKE_TOOLCHAIN_FILE="$HOME/dev/llm/llama.cpp/cmake/arm64-windows-llvm.cmake" `
  -DCMAKE_BUILD_TYPE=Release `
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" `
  -DBUILD_SHARED_LIBS=OFF `
  -DGGML_OPENCL=ON
ninja

Android Copy Chapter

To read documentation for how to build on Android, click here

WebGPU [In Progress] Copy Chapter

The WebGPU backend relies on Dawn. Follow the instructions here to install Dawn locally so that llama.cpp can find it using CMake. The currrent implementation is up-to-date with Dawn commit bed1a61.

In the llama.cpp directory, build with CMake:

cmake -B build -DGGML_WEBGPU=ON
cmake --build build --config Release

Browser Support Copy Section

WebGPU allows cross-platform access to the GPU from supported browsers. We utilize Emscripten to compile ggml's WebGPU backend to WebAssembly. Emscripten does not officially support WebGPU bindings yet, but Dawn currently maintains its own WebGPU bindings called emdawnwebgpu.

Follow the instructions here to download or build the emdawnwebgpu package (Note that it might be safer to build the emdawbwebgpu package locally, so that it stays in sync with the version of Dawn you have installed above). When building using CMake, the path to the emdawnwebgpu port file needs to be set with the flag EMDAWNWEBGPU_DIR.

IBM Z & LinuxONE Copy Chapter

To read documentation for how to build on IBM Z & LinuxONE, click here

Notes about GPU-accelerated backends Copy Chapter

The GPU may still be used to accelerate some parts of the computation even when using the -ngl 0 option. You can fully disable GPU acceleration by using --device none.

In most cases, it is possible to build and use multiple backends at the same time. For example, you can build llama.cpp with both CUDA and Vulkan support by using the -DGGML_CUDA=ON -DGGML_VULKAN=ON options with CMake. At runtime, you can specify which backend devices to use with the --device option. To see a list of available devices, use the --list-devices option.

Backends can be built as dynamic libraries that can be loaded dynamically at runtime. This allows you to use the same llama.cpp binary on different machines with different GPUs. To enable this feature, use the GGML_BACKEND_DL option when building.

Llama.Cpp Docs Docker (`llama.cpp/docs/docker.md`) Copy Section

Docker

Prerequisites Copy Chapter

• Docker must be installed and running on your system.
• Create a folder to store big models & intermediate files (ex. /llama/models)

Images Copy Chapter

We have three Docker images available for this project:

1. ghcr.io/ggml-org/llama.cpp:full: This image includes both the llama-cli and llama-completion executables and the tools to convert LLaMA models into ggml and convert into 4-bit quantization. (platforms: linux/amd64, linux/arm64, linux/s390x)
2. ghcr.io/ggml-org/llama.cpp:light: This image only includes the llama-cli and llama-completion executables. (platforms: linux/amd64, linux/arm64, linux/s390x)
3. ghcr.io/ggml-org/llama.cpp:server: This image only includes the llama-server executable. (platforms: linux/amd64, linux/arm64, linux/s390x)

Additionally, there the following images, similar to the above:

• ghcr.io/ggml-org/llama.cpp:full-cuda: Same as full but compiled with CUDA support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:light-cuda: Same as light but compiled with CUDA support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:server-cuda: Same as server but compiled with CUDA support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:full-rocm: Same as full but compiled with ROCm support. (platforms: linux/amd64, linux/arm64)
• ghcr.io/ggml-org/llama.cpp:light-rocm: Same as light but compiled with ROCm support. (platforms: linux/amd64, linux/arm64)
• ghcr.io/ggml-org/llama.cpp:server-rocm: Same as server but compiled with ROCm support. (platforms: linux/amd64, linux/arm64)
• ghcr.io/ggml-org/llama.cpp:full-musa: Same as full but compiled with MUSA support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:light-musa: Same as light but compiled with MUSA support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:server-musa: Same as server but compiled with MUSA support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:full-intel: Same as full but compiled with SYCL support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:light-intel: Same as light but compiled with SYCL support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:server-intel: Same as server but compiled with SYCL support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:full-vulkan: Same as full but compiled with Vulkan support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:light-vulkan: Same as light but compiled with Vulkan support. (platforms: linux/amd64)
• ghcr.io/ggml-org/llama.cpp:server-vulkan: Same as server but compiled with Vulkan support. (platforms: linux/amd64)

The GPU enabled images are not currently tested by CI beyond being built. They are not built with any variation from the ones in the Dockerfiles defined in .devops/ and the GitHub Action defined in .github/workflows/docker.yml. If you need different settings (for example, a different CUDA, ROCm or MUSA library, you'll need to build the images locally for now).

Usage Copy Chapter

The easiest way to download the models, convert them to ggml and optimize them is with the --all-in-one command which includes the full docker image.

Replace /path/to/models below with the actual path where you downloaded the models.

docker run -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:full --all-in-one "/models/" 7B

On completion, you are ready to play!

docker run -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:full --run -m /models/7B/ggml-model-q4_0.gguf
docker run -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:full --run-legacy -m /models/32B/ggml-model-q8_0.gguf -no-cnv -p "Building a mobile app can be done in 15 steps:" -n 512

or with a light image:

docker run -v /path/to/models:/models --entrypoint /app/llama-cli ghcr.io/ggml-org/llama.cpp:light -m /models/7B/ggml-model-q4_0.gguf
docker run -v /path/to/models:/models --entrypoint /app/llama-completion ghcr.io/ggml-org/llama.cpp:light -m /models/32B/ggml-model-q8_0.gguf -no-cnv -p "Building a mobile app can be done in 15 steps:" -n 512

or with a server image:

docker run -v /path/to/models:/models -p 8080:8080 ghcr.io/ggml-org/llama.cpp:server -m /models/7B/ggml-model-q4_0.gguf --port 8080 --host 0.0.0.0 -n 512

In the above examples, --entrypoint /app/llama-cli is specified for clarity, but you can safely omit it since it's the default entrypoint in the container.

Docker With CUDA Copy Chapter

Assuming one has the nvidia-container-toolkit properly installed on Linux, or is using a GPU enabled cloud, cuBLAS should be accessible inside the container.

Building Docker locally Copy Chapter

docker build -t local/llama.cpp:full-cuda --target full -f .devops/cuda.Dockerfile .
docker build -t local/llama.cpp:light-cuda --target light -f .devops/cuda.Dockerfile .
docker build -t local/llama.cpp:server-cuda --target server -f .devops/cuda.Dockerfile .

You may want to pass in some different ARGS, depending on the CUDA environment supported by your container host, as well as the GPU architecture.

The defaults are:

• CUDA_VERSION set to 12.4.0
• CUDA_DOCKER_ARCH set to the cmake build default, which includes all the supported architectures

The resulting images, are essentially the same as the non-CUDA images:

1. local/llama.cpp:full-cuda: This image includes both the llama-cli and llama-completion executables and the tools to convert LLaMA models into ggml and convert into 4-bit quantization.
2. local/llama.cpp:light-cuda: This image only includes the llama-cli and llama-completion executables.
3. local/llama.cpp:server-cuda: This image only includes the llama-server executable.

Usage Copy Chapter

After building locally, Usage is similar to the non-CUDA examples, but you'll need to add the --gpus flag. You will also want to use the --n-gpu-layers flag.

docker run --gpus all -v /path/to/models:/models local/llama.cpp:full-cuda --run -m /models/7B/ggml-model-q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 512 --n-gpu-layers 1
docker run --gpus all -v /path/to/models:/models local/llama.cpp:light-cuda -m /models/7B/ggml-model-q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 512 --n-gpu-layers 1
docker run --gpus all -v /path/to/models:/models local/llama.cpp:server-cuda -m /models/7B/ggml-model-q4_0.gguf --port 8080 --host 0.0.0.0 -n 512 --n-gpu-layers 1

Docker With MUSA Copy Chapter

Assuming one has the mt-container-toolkit properly installed on Linux, muBLAS should be accessible inside the container.

Building Docker locally Copy Chapter

docker build -t local/llama.cpp:full-musa --target full -f .devops/musa.Dockerfile .
docker build -t local/llama.cpp:light-musa --target light -f .devops/musa.Dockerfile .
docker build -t local/llama.cpp:server-musa --target server -f .devops/musa.Dockerfile .

You may want to pass in some different ARGS, depending on the MUSA environment supported by your container host, as well as the GPU architecture.

The defaults are:

• MUSA_VERSION set to rc4.3.0

The resulting images, are essentially the same as the non-MUSA images:

1. local/llama.cpp:full-musa: This image includes both the llama-cli and llama-completion executables and the tools to convert LLaMA models into ggml and convert into 4-bit quantization.
2. local/llama.cpp:light-musa: This image only includes the llama-cli and llama-completion executables.
3. local/llama.cpp:server-musa: This image only includes the llama-server executable.

Usage Copy Chapter

After building locally, Usage is similar to the non-MUSA examples, but you'll need to set mthreads as default Docker runtime. This can be done by executing (cd /usr/bin/musa && sudo ./docker setup $PWD) and verifying the changes by executing docker info | grep mthreads on the host machine. You will also want to use the --n-gpu-layers flag.

docker run -v /path/to/models:/models local/llama.cpp:full-musa --run -m /models/7B/ggml-model-q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 512 --n-gpu-layers 1
docker run -v /path/to/models:/models local/llama.cpp:light-musa -m /models/7B/ggml-model-q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 512 --n-gpu-layers 1
docker run -v /path/to/models:/models local/llama.cpp:server-musa -m /models/7B/ggml-model-q4_0.gguf --port 8080 --host 0.0.0.0 -n 512 --n-gpu-layers 1

Llama.Cpp Docs Function Calling (`llama.cpp/docs/function-calling.md`) Copy Section

Function Calling

chat.h (https://github.com/ggml-org/llama.cpp/pull/9639) adds support for OpenAI-style function calling and is used in:

• llama-server when started w/ --jinja flag

Universal support w/ Native & Generic handlers Copy Chapter

Function calling is supported for all models (see https://github.com/ggml-org/llama.cpp/pull/9639):

• Native tool call formats supported:

  • Llama 3.1 / 3.3 (including builtin tools support - tool names for wolfram_alpha, web_search / brave_search, code_interpreter), Llama 3.2
  • Functionary v3.1 / v3.2
  • Hermes 2/3, Qwen 2.5
  • Qwen 2.5 Coder
  • Mistral Nemo
  • Firefunction v2
  • Command R7B
  • DeepSeek R1 (WIP / seems reluctant to call any tools?)

• Generic tool call is supported when the template isn't recognized by native format handlers (you'll see Chat format: Generic in the logs).

  • Use --chat-template-file to override the template when appropriate (see examples below)
  • Generic support may consume more tokens and be less efficient than a model's native format.

• Multiple/parallel tool calling is supported on some models but disabled by default, enable it by passing "parallel_tool_calls": true in the completion endpoint payload.

Show some common templates and which format handler they use

This table can be generated with:

./build/bin/test-chat ../minja/build/tests/*.jinja 2>/dev/null

Usage - need tool-aware Jinja template

First, start a server with any model, but make sure it has a tools-enabled template: you can verify this by inspecting the chat_template or chat_template_tool_use properties in http://localhost:8080/props).

Here are some models known to work (w/ chat template override when needed):

/* Detailed source-code truncated for AI context efficiency. */

To get the official template from original HuggingFace repos, you can use scripts/get_chat_template.py (see examples invocations in models/templates/README.md)

[!TIP]
If there is no official tool_use Jinja template, you may want to set --chat-template chatml to use a default that works with many models (YMMV!), or write your own (e.g. we provide a custom llama-cpp-deepseek-r1.jinja for DeepSeek R1 distills)

[!CAUTION]
Beware of extreme KV quantizations (e.g. -ctk q4_0), they can substantially degrade the model's tool calling performance.

Test in CLI (or with any library / software that can use OpenAI-compatible API backends):

/* Detailed source-code truncated for AI context efficiency. */

Show output

{
"choices": [
    {
    "finish_reason": "tool",
    "index": 0,
    "message": {
        "content": null,
        "tool_calls": [
        {
            "name": "python",
            "arguments": "{\"code\":\" \\nprint(\\\"Hello, World!\\\")\"}"
        }
        ],
        "role": "assistant"
    }
    }
],
"created": 1727287211,
"model": "gpt-3.5-turbo",
"object": "chat.completion",
"usage": {
    "completion_tokens": 16,
    "prompt_tokens": 44,
    "total_tokens": 60
},
"id": "chatcmpl-Htbgh9feMmGM0LEH2hmQvwsCxq3c6Ni8"
}

Llama.Cpp Docs Install (`llama.cpp/docs/install.md`) Copy Section

Install pre-built version of llama.cpp

Winget (Windows) Copy Chapter

winget install llama.cpp

The package is automatically updated with new llama.cpp releases. More info: https://github.com/ggml-org/llama.cpp/issues/8188

Homebrew (Mac and Linux) Copy Chapter

brew install llama.cpp

The formula is automatically updated with new llama.cpp releases. More info: https://github.com/ggml-org/llama.cpp/discussions/7668

MacPorts (Mac) Copy Chapter

sudo port install llama.cpp

See also: https://ports.macports.org/port/llama.cpp/details/

Nix (Mac and Linux) Copy Chapter

nix profile install nixpkgs#llama-cpp

For flake enabled installs.

Or

nix-env --file '<nixpkgs>' --install --attr llama-cpp

For non-flake enabled installs.

This expression is automatically updated within the nixpkgs repo.

Llama.Cpp Docs Llguidance (`llama.cpp/docs/llguidance.md`) Copy Section

LLGuidance Support in llama.cpp

LLGuidance is a library for constrained decoding (also called constrained sampling or structured outputs) for Large Language Models (LLMs). Initially developed as the backend for the Guidance library, it can also be used independently.

LLGuidance supports JSON Schemas and arbitrary context-free grammars (CFGs) written in a variant of Lark syntax. It is very fast and has excellent JSON Schema coverage but requires the Rust compiler, which complicates the llama.cpp build process.

Building Copy Chapter

To enable LLGuidance support, build llama.cpp with the LLAMA_LLGUIDANCE option:

cmake -B build -DLLAMA_LLGUIDANCE=ON
make -C build -j

For Windows use cmake --build build --config Release instead of make.

This requires the Rust compiler and the cargo tool to be installed.

Interface Copy Chapter

There are no new command-line arguments or modifications to common_params. When enabled, grammars starting with %llguidance are passed to LLGuidance instead of the current llama.cpp grammars. Additionally, JSON Schema requests (e.g., using the -j argument in llama-cli) are also passed to LLGuidance.

For your existing GBNF grammars, you can use gbnf_to_lark.py script to convert them to LLGuidance Lark-like format.

Performance Copy Chapter

Computing a "token mask" (i.e., the set of allowed tokens) for a llama3 tokenizer with 128k tokens takes, on average, 50μs of single-core CPU time for the JSON Schema Bench. The p99 time is 0.5ms, and the p100 time is 20ms. These results are due to the lexer/parser split and several optimizations.

JSON Schema Copy Chapter

LLGuidance adheres closely to the JSON Schema specification. For example:

• additionalProperties defaults to true, unlike current grammars, though you can set "additionalProperties": false if needed.
• any whitespace is allowed.
• The definition order in the "properties": {} object is maintained, regardless of whether properties are required (current grammars always puts required properties first).

Unsupported schemas result in an error message—no keywords are silently ignored.

Why Not Reuse GBNF Format? Copy Chapter

GBNF lacks the concept of a lexer.

Most programming languages, including JSON, use a two-step process: a lexer (built with regular expressions) converts a byte stream into lexemes, which are then processed by a CFG parser. This approach is faster because lexers are cheaper to evaluate, and there is ~10x fewer lexemes than bytes.
LLM tokens often align with lexemes, so the parser is engaged in under 0.5% of tokens, with the lexer handling the rest.

However, the user has to provide the distinction between lexemes and CFG symbols. In Lark, lexeme names are uppercase, while CFG symbols are lowercase.
The gbnf_to_lark.py script can often take care of this automatically.
See LLGuidance syntax docs for more details.

Error Handling Copy Chapter

Errors are currently printed to stderr, and generation continues. Improved error handling may be added in the future.

Llama.Cpp Docs Multimodal (`llama.cpp/docs/multimodal.md`) Copy Section

Multimodal

llama.cpp supports multimodal input via libmtmd. Currently, there are 2 tools support this feature:

• llama-mtmd-cli
• llama-server via OpenAI-compatible /chat/completions API

Currently, we support image and audio input. Audio is highly experimental and may have reduced quality.

To enable it, you can use one of the 2 methods below:

• Use -hf option with a supported model (see a list of pre-quantized model below)
  • To load a model using -hf while disabling multimodal, use --no-mmproj
  • To load a model using -hf while using a custom mmproj file, use --mmproj local_file.gguf
• Use -m model.gguf option with --mmproj file.gguf to specify text and multimodal projector respectively

By default, multimodal projector will be offloaded to GPU. To disable this, add --no-mmproj-offload

For example:

# simple usage with CLI
llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF

# simple usage with server
llama-server -hf ggml-org/gemma-3-4b-it-GGUF

# using local file
llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf

# no GPU offload
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --no-mmproj-offload

Pre-quantized models Copy Chapter

These are ready-to-use models, most of them come with Q4_K_M quantization by default. They can be found at the Hugging Face page of the ggml-org: https://huggingface.co/collections/ggml-org/multimodal-ggufs-68244e01ff1f39e5bebeeedc

Replaces the (tool_name) with the name of binary you want to use. For example, llama-mtmd-cli or llama-server

NOTE: some models may require large context window, for example: -c 8192

Vision models:

# Gemma 3
(tool_name) -hf ggml-org/gemma-3-4b-it-GGUF
(tool_name) -hf ggml-org/gemma-3-12b-it-GGUF
(tool_name) -hf ggml-org/gemma-3-27b-it-GGUF

# SmolVLM
(tool_name) -hf ggml-org/SmolVLM-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM-256M-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM-500M-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM2-2.2B-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM2-256M-Video-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM2-500M-Video-Instruct-GGUF

# Pixtral 12B
(tool_name) -hf ggml-org/pixtral-12b-GGUF

# Qwen 2 VL
(tool_name) -hf ggml-org/Qwen2-VL-2B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen2-VL-7B-Instruct-GGUF

# Qwen 2.5 VL
(tool_name) -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen2.5-VL-7B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen2.5-VL-32B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen2.5-VL-72B-Instruct-GGUF

# Mistral Small 3.1 24B (IQ2_M quantization)
(tool_name) -hf ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF

# InternVL 2.5 and 3
(tool_name) -hf ggml-org/InternVL2_5-1B-GGUF
(tool_name) -hf ggml-org/InternVL2_5-4B-GGUF
(tool_name) -hf ggml-org/InternVL3-1B-Instruct-GGUF
(tool_name) -hf ggml-org/InternVL3-2B-Instruct-GGUF
(tool_name) -hf ggml-org/InternVL3-8B-Instruct-GGUF
(tool_name) -hf ggml-org/InternVL3-14B-Instruct-GGUF

# Llama 4 Scout
(tool_name) -hf ggml-org/Llama-4-Scout-17B-16E-Instruct-GGUF

# Moondream2 20250414 version
(tool_name) -hf ggml-org/moondream2-20250414-GGUF

Audio models:

# Ultravox 0.5
(tool_name) -hf ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF
(tool_name) -hf ggml-org/ultravox-v0_5-llama-3_1-8b-GGUF

# Qwen2-Audio and SeaLLM-Audio
# note: no pre-quantized GGUF this model, as they have very poor result
# ref: https://github.com/ggml-org/llama.cpp/pull/13760

# Mistral's Voxtral
(tool_name) -hf ggml-org/Voxtral-Mini-3B-2507-GGUF

Mixed modalities:

# Qwen2.5 Omni
# Capabilities: audio input, vision input
(tool_name) -hf ggml-org/Qwen2.5-Omni-3B-GGUF
(tool_name) -hf ggml-org/Qwen2.5-Omni-7B-GGUF

Finding more models: Copy Chapter

GGUF models on Huggingface with vision capabilities can be found here: https://huggingface.co/models?pipeline_tag=image-text-to-text&sort=trending&search=gguf

Llama.Cpp Docs Ops (`llama.cpp/docs/ops.md`) Copy Section

GGML Operations

List of GGML operations and backend support status.

How to add a backend to this table: Copy Chapter

1. Run test-backend-ops support --output csv with your backend name and redirect output to a csv file in docs/ops/ (e.g., docs/ops/CUDA.csv)
2. Regenerate /docs/ops.md via ./scripts/create_ops_docs.py

Legend:

• ✅ Fully supported by this backend
• 🟡 Partially supported by this backend
• ❌ Not supported by this backend

Llama.Cpp Docs Step3.5 Flash (`llama.cpp/docs/step3.5-flash.md`) Copy Section

Step3.5-Flash Copy Chapter

We support deploying int4 quantized models on devices with 128GB unified memory, enabling a GPT-OSS-120B–comparable model experience on personal workstations while significantly reducing inference cost and latency.

The currently validated platforms include:

• Mac Studio M4 Max
• NVIDIA DGX-Spark
• AMD Ryzen AI Max+ 395 (Windows)

Prepare Model and Code Copy Section

Download the Q4_K_S quantized GGUF weight file Step3.5-Flash, which can be used directly for on-device inference.

Use llama.cpp:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Using Step3.5-Flash on Mac Copy Section

Build llama.cpp with CMake:

cmake -S . -B build-macos \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=ON \
  -DGGML_ACCELERATE=ON \
  -DLLAMA_BUILD_EXAMPLES=ON \
  -DLLAMA_BUILD_COMMON=ON \
  -DGGML_LTO=ON
cmake --build build-macos -j8

Basic inference command:

./llama-cli -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -fa on --temp 1.0 -p "What's your name?"

Performance benchmark command:

./llama-batched-bench -m .step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

Note: Due to memory limitations on Apple silicon, testing long context requires increasing the available memory to 120GB using the following command. With -ub 1024, up to a 256K context can be run:

sudo sysctl iogpu.wired_limit_mb=122880

Alternatively, for contexts not exceeding 76K, you can increase runtime parameters to enable KV cache quantization, at the cost of some precision:

-ctk q8_0 -ctv q8_0

The performance benchmark results are as follows:

./llama-batched-bench -m step3.5_flash_Q4_K_S.gguf -c 262150 -b 2048 -ub 1024 -npp 0,2048,8192,16384,32768,65536,262144 -ntg 128 -npl 1

Using Step3.5-Flash on DGX-Spark Copy Section

Build llama.cpp with CMake:

cmake -S . -B build-cuda \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_GRAPHS=ON \
  -DLLAMA_CURL=OFF \
  -DLLAMA_BUILD_EXAMPLES=ON \
  -DLLAMA_BUILD_COMMON=ON
cmake --build build-cuda -j8

Basic inference command:

./llama-cli -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -fa on --temp 1.0 -p "What's your name?"

Performance benchmark command:

./llama-batched-bench -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

When testing long context (e.g. 256K), OOM may occur. The build flag -DGGML_CUDA_FORCE_MMQ=ON, environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1, and runtime parameters -ctk q8_0 -ctv q8_0 can help mitigate memory issues. You can also use the following command to clear memory cache:

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

The performance benchmark results are as follows:

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ./llama-batched-bench -m step3.5_flash_Q4_K_S.gguf -c 262150 -b 2048 -ub 1024 -npp 0,2048,8192,16384,32768,65536,262144 -ntg 128 -npl 1 -ctk q8_0 -ctv q8_0

Using Step3.5-Flash on AMD Ryzen AI Max+ 395 Copy Section

Build llama.cpp with CMake (Vulkan-SDK must be installed in advance):

cmake -S . -B build-vulkan \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=OFF \
  -DGGML_OPENMP=ON \
  -DGGML_VULKAN=ON
cmake --build build-vulkan -j8

Basic inference command:

llama-cli.exe -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -fa on --temp 1.0 -p "What's your name?"

Performance benchmark command:

llama-batched-bench.exe -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

The performance benchmark results are as follows:

llama-batched-bench.exe -m step3p5_flash_Q4_K_S.gguf -c 65540 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768,65536 -ntg 128 -npl 1 --mmap -t 32

TODO:

• Further improve quantization quality using higher-quality calibration datasets
• Increase maximum context length and inference speed on AMD platforms
• Add MTP support

We welcome questions and suggestions—let’s work together to continuously improve the private deployment experience of large language models.

Llama.Cpp Docs Step3.5 Flash Zh (`llama.cpp/docs/step3.5-flash_zh.md`) Copy Section

Step3.5-Flash Copy Chapter

我们支持将 int4 量化后的模型部署到有 128GB 统一内存的设备上，通过个人工作站获得对标 gpt-oss-120B 的模型体验并显著降低推理成本与延迟。

当前已验证的平台包括：

• Mac Studio M4 Max
• NVIDIA DGX-Spark
• AMD Ryzen AI Max+ 395（window系统）

准备模型和代码 Copy Section

下载 Q4_K_S 量化版本的 GGUF 权重文件 Step3.5-Flash ，可直接用于端侧设备推理.

Use llama.cpp:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

在 Mac 上使用 Step3.5-Flash Copy Section

使用 CMake 编译 llama.cpp:

cmake -S . -B build-macos \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=ON \
  -DGGML_ACCELERATE=ON \
  -DLLAMA_BUILD_EXAMPLES=ON \
  -DLLAMA_BUILD_COMMON=ON \
  -DGGML_LTO=ON
cmake --build build-macos -j8

基础推理命令：

./llama-cli -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -fa on --temp 1.0 -p "What's your name?"

性能测试命令：

./llama-batched-bench -m .step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

注意：由于 Apple silicon 有内存限制，如果想测试长上下文，需要通过以下命令来将可用内存提高到120GB，在设置 -ub 1024 的情况下最高可运行 256k 上下文

sudo sysctl iogpu.wired_limit_mb=122880

或者在不超过 76K 上下文的情况下增加运行时参数进行 kv cache 量化，但可能会牺牲部分精度

-ctk q8_0 -ctv q8_0

性能测试的 BenchMark 如下：

./llama-batched-bench -m step3.5_flash_Q4_K_S.gguf -c 262150 -b 2048 -ub 1024 -npp 0,2048,8192,16384,32768,65536,262144 -ntg 128 -npl 1

在 DGX-Spark 上使用 Step3.5-Flash Copy Section

使用 CMake 编译 llama.cpp:

cmake -S . -B build-cuda \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_GRAPHS=ON \
  -DLLAMA_CURL=OFF \
  -DLLAMA_BUILD_EXAMPLES=ON \
  -DLLAMA_BUILD_COMMON=ON
cmake --build build-cuda -j8

基础推理命令：

./llama-cli -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -fa on --temp 1.0 -p "What's your name?"

性能测试命令：

./llama-batched-bench -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

如果想测试长上下文（例如 256K）可能会出现OOM，编译参数 -DGGML_CUDA_FORCE_MMQ=ON 、环境变量 GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 以及运行时参数 -ctk q8_0 -ctv q8_0 可以用于缓解内存问题。同时可以用以下命令 清理内存cache

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

性能测试的 BenchMark 如下：

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ./llama-batched-bench -m step3.5_flash_Q4_K_S.gguf -c 262150 -b 2048 -ub 1024 -npp 0,2048,8192,16384,32768,65536,262144 -ntg 128 -npl 1 -ctk q8_0 -ctv q8_0

在 AMD Ryzen AI Max+ 395 上使用 Step3.5-Flash Copy Section

使用 CMake 编译 llama.cpp（需要预先安装Vulkan-SDK）:

cmake -S . -B build-vulkan \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=OFF \
  -DGGML_OPENMP=ON \
  -DGGML_VULKAN=ON
cmake --build build-vulkan -j8

基础推理命令：

llama-cli.exe -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -fa on --temp 1.0 -p "What's your name?"

性能测试命令：

llama-batched-bench.exe -m step3.5_flash_Q4_K_S.gguf -c 16384 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

性能测试的 BenchMark 如下：

llama-batched-bench.exe -m step3p5_flash_Q4_K_S.gguf -c 65540 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768,65536 -ntg 128 -npl 1 --mmap -t 32

TODO:

• 进一步提高量化质量，使用更高质量的量化校准集
• 提升AMD机型的最大上下文和推理速度
• 支持MTP

欢迎向我们提出问题与建议，一起让大语言模型私有化部署的体验越来越好！

Llama.Cpp Docs Backend BLIS (`llama.cpp/docs/backend/BLIS.md`) Copy Section

BLIS Installation Manual Copy Chapter

BLIS is a portable software framework for high-performance BLAS-like dense linear algebra libraries. It has received awards and recognition, including the 2023 James H. Wilkinson Prize for Numerical Software and the 2020 SIAM Activity Group on Supercomputing Best Paper Prize. BLIS provides a new BLAS-like API and a compatibility layer for traditional BLAS routine calls. It offers features such as object-based API, typed API, BLAS and CBLAS compatibility layers.

Project URL: https://github.com/flame/blis

Prepare: Copy Section

Compile BLIS:

git clone https://github.com/flame/blis
cd blis
./configure --enable-cblas -t openmp,pthreads auto
# will install to /usr/local/ by default.
make -j

Install BLIS:

sudo make install

We recommend using openmp since it's easier to modify the cores being used.

llama.cpp compilation Copy Section

CMake:

mkdir build
cd build
cmake -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=FLAME ..
make -j

llama.cpp execution Copy Section

According to the BLIS documentation, we could set the following
environment variables to modify the behavior of openmp:

export GOMP_CPU_AFFINITY="0-19"
export BLIS_NUM_THREADS=14

And then run the binaries as normal.

Intel specific issue Copy Section

Some might get the error message saying that libimf.so cannot be found.
Please follow this stackoverflow page.

Reference: Copy Section

1. https://github.com/flame/blis#getting-started
2. https://github.com/flame/blis/blob/master/docs/Multithreading.md

Llama.Cpp Docs Backend CANN (`llama.cpp/docs/backend/CANN.md`) Copy Section

llama.cpp for CANN

• Background
• News
• OS
• Hardware
• Model Supports
• DataType Supports
• Docker
• Linux
• Environment variable setup
• TODO

Background Copy Chapter

Ascend NPU is a range of AI processors using Neural Processing Unit. It will efficiently handle matrix-matrix multiplication, dot-product and scalars.

CANN (Compute Architecture for Neural Networks) is a heterogeneous computing architecture for AI scenarios, providing support for multiple AI frameworks on the top and serving AI processors and programming at the bottom. It plays a crucial role in bridging the gap between upper and lower layers, and is a key platform for improving the computing efficiency of Ascend AI processors. Meanwhile, it offers a highly efficient and easy-to-use programming interface for diverse application scenarios, allowing users to rapidly build AI applications and services based on the Ascend platform.

Llama.cpp + CANN

The llama.cpp CANN backend is designed to support Ascend NPU. It utilize the ability of AscendC and ACLNN which are intergrated to CANN Toolkit and kernels to using Ascend NPU directly.

News Copy Chapter

• 2024.11
  • Support F16 and F32 data type model for Ascend 310P NPU.
• 2024.8
  • Support Q4_0 and Q8_0 data type for Ascend NPU.
• 2024.7
  • Create CANN backend for Ascend NPU.

OS Copy Chapter

Hardware Copy Chapter

Ascend NPU Copy Section

Verified devices

Notes:

• If you have trouble with Ascend NPU device, please create a issue with [CANN] prefix/tag.
• If you run successfully with your Ascend NPU device, please help update the upper table.

Model Supports Copy Chapter

Multimodal

DataType Supports Copy Chapter

Docker Copy Chapter

Build Images Copy Section

You can get a image with llama.cpp in one command.

docker build -t llama-cpp-cann -f .devops/llama-cli-cann.Dockerfile .

Run container Copy Section

# Find all cards.
npu-smi info

# Select the cards that you want to use, make sure these cards are not used by someone.
# Following using cards of device0.
docker run --name llamacpp --device /dev/davinci0  --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info -v /PATH_TO_YOUR_MODELS/:/app/models -it llama-cpp-cann -m /app/models/MODEL_PATH -ngl 32 -p "Building a website can be done in 10 simple steps:"

Notes:

• You may need to install Ascend Driver and firmware on the host machine (Please refer to the Linux configuration for details).

Linux Copy Chapter

I. Setup Environment Copy Section

1. Install Ascend Driver and firmware

   sh Copy

   # create driver running user.
   sudo groupadd -g HwHiAiUser
   sudo useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash
   sudo usermod -aG HwHiAiUser $USER
   
   # download driver from https://www.hiascend.com/hardware/firmware-drivers/community according to your system
   # and install driver.
   sudo sh Ascend-hdk-910b-npu-driver_x.x.x_linux-{arch}.run --full --install-for-all

   Once installed, run npu-smi info to check whether driver is installed successfully.

   text Copy

/* Detailed source-code truncated for AI context efficiency. */

2. **Install Ascend Firmware**
    ```sh
    # download driver from https://www.hiascend.com/hardware/firmware-drivers/community according to your system
    # and install driver.
    sudo sh Ascend-hdk-910b-npu-firmware_x.x.x.x.X.run --full
    ```
    If the following messaage appers, firmware is installed successfully.
    ```sh
    Firmware package installed successfully!
    ```


3. **Install CANN toolkit and kernels**

    CANN toolkit and kernels can be obtained from the official [CANN Toolkit](https://www.hiascend.com/zh/developer/download/community/result?module=cann) page.

    Please download the corresponding version that satified your system. The minimum version required is 8.0.RC2.alpha002 and here is the install command.
    ```sh
    pip3 install attrs numpy decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions
    sh Ascend-cann-toolkit_8.0.RC2.alpha002_linux-aarch64.run --install
    sh Ascend-cann-kernels-910b_8.0.RC2.alpha002_linux.run --install
    ```

    Set Ascend Variables:
    ```sh
    echo "source ~/Ascend/ascend-toolkit/set_env.sh" >> ~/.bashrc
    source ~/.bashrc
    ```

Upon a successful installation, CANN is enabled for the available ascend devices.

### II. Build llama.cpp

```sh
cmake -B build -DGGML_CANN=on -DCMAKE_BUILD_TYPE=release
cmake --build build --config release

III. Run the inference Copy Section

1. Retrieve and prepare model

   You can refer to the general Prepare and Quantize guide for model prepration.

   Notes:

   • CANN backend only supports FP16/Q4_0/Q8_0 models currently.

2. Launch inference

   There are two device selection modes:

   • Single device: Use one device target specified by the user.
   • Multiple devices: Automatically choose the devices with the same backend.

   Device selection	Parameter
   Single device	--split-mode none --main-gpu DEVICE_ID
   Multiple devices	--split-mode layer (default)

   Examples:

   • Use device 0:
   sh Copy

   ./build/bin/llama-cli -m path_to_model -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 -sm none -mg 0

   • Use multiple devices:
   sh Copy

   ./build/bin/llama-cli -m path_to_model -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 -sm layer

**GitHub contribution**: Copy Section

Please add the [CANN] prefix/tag in issues/PRs titles to help the CANN-team check/address them without delay.

Updates Copy Chapter

Basic Flash Attention Support Copy Section

The basic FA kernel with aclnnops has been added in aclnn_ops.cpp.
Currently, the FA only supports the cases with FP16 KV tensors and NO logit softcap.
Since the aclnn interface for flash attention cannot support the logit softcap, we will only update the quantized version in the future.

Authors from Peking University: Bizhao Shi ([email protected]), Yuxin Yang ([email protected]), Ruiyang Ma ([email protected]), and Guojie Luo ([email protected]).

We would like to thank Tuo Dai, Shanni Li, and all of the project maintainers from Huawei Technologies Co., Ltd for their help during the code development and pull request.

Environment variable setup Copy Chapter

GGML_CANN_MEM_POOL Copy Section

Specifies the memory pool management strategy, Default is vmm.

• vmm: Utilizes a virtual memory manager pool. If hardware support for VMM is unavailable, falls back to the legacy (leg) memory pool.

• prio: Employs a priority queue-based memory pool management.

• leg: Uses a fixed-size buffer pool.

GGML_CANN_DISABLE_BUF_POOL_CLEAN Copy Section

Controls automatic cleanup of the memory pool. This option is only effective when using the prio or leg memory pool strategies.

GGML_CANN_WEIGHT_NZ Copy Section

Converting the matmul weight format from ND to NZ to improve performance. Enabled by default.

GGML_CANN_ACL_GRAPH Copy Section

Operators are executed using ACL graph execution, rather than in op-by-op (eager) mode. Enabled by default. This option is only effective if USE_ACL_GRAPH was enabled at compilation time. To enable it, recompile using:

cmake -B build -DGGML_CANN=on -DCMAKE_BUILD_TYPE=release -DUSE_ACL_GRAPH=ON
cmake --build build --config release

GGML_CANN_GRAPH_CACHE_CAPACITY Copy Section

Maximum number of compiled CANN graphs kept in the LRU cache, default is 12. When the number of cached graphs exceeds this capacity, the least recently used graph will be evicted.

GGML_CANN_PREFILL_USE_GRAPH Copy Section

Enable ACL graph execution during the prefill stage, default is false. This option is only effective when FA is enabled.

GGML_CANN_OPERATOR_FUSION Copy Section

Enable operator fusion during computation, default is false. This option fuses compatible operators (e.g., ADD + RMS_NORM) to reduce overhead and improve performance.

Llama.Cpp Docs Backend CUDA FEDORA (`llama.cpp/docs/backend/CUDA-FEDORA.md`) Copy Section

Setting Up CUDA on Fedora

In this guide we setup Nvidia CUDA in a toolbox container. This guide is applicable for:

• Fedora Workstation
• Atomic Desktops for Fedora
• Fedora Spins
• Other Distributions, including Red Hat Enterprise Linux >= 8.5, Arch Linux, and Ubuntu.

Table of Contents Copy Chapter

• Prerequisites
• Using the Fedora 41 CUDA Repository
• Creating a Fedora Toolbox Environment
• Installing Essential Development Tools
• Adding the CUDA Repository
• Installing Nvidia Driver Libraries
• Installing the CUDA Meta-Package
• Configuring the Environment
• Verifying the Installation
• Conclusion
• Troubleshooting
• Additional Notes
• References

Prerequisites Copy Chapter

• Toolbox Installed on the Host System Fedora Silverblue and Fedora Workstation both have toolbox by default, other distributions may need to install the toolbox package.
• NVIDIA Drivers and Graphics Card installed on Host System (recommended) To run CUDA program, such as llama.cpp, the host should be setup to access your NVIDIA hardware. Fedora Hosts can use the RPM Fusion Repository.
• Internet connectivity to download packages.

Using the Fedora 41 CUDA Repository Copy Section

The latest release is 41.

• Fedora 41 CUDA Repository

Note: We recommend using a toolbox environment to prevent system conflicts.

Creating a Fedora Toolbox Environment Copy Chapter

This guide focuses on Fedora hosts, but with small adjustments, it can work for other hosts. Using the Fedora Toolbox allows us to install the necessary packages without affecting the host system.

Note: Toolbox is available for other systems, and even without Toolbox, it is possible to use Podman or Docker.

1. Create a Fedora 41 Toolbox:

   bash Copy

   toolbox create --image registry.fedoraproject.org/fedora-toolbox:41 --container fedora-toolbox-41-cuda

2. Enter the Toolbox:

   bash Copy

   toolbox enter --container fedora-toolbox-41-cuda

   Inside the toolbox, you have root privileges and can install packages without affecting the host system.

Installing Essential Development Tools Copy Chapter

1. Synchronize the DNF Package Manager:

   bash Copy

   sudo dnf distro-sync

2. Install Vim the default text editor (Optional):

   bash Copy

   sudo dnf install vim-default-editor --allowerasing

   The --allowerasing flag will allow the removal of the conflicting nano-default-editor package.

3. Install Development Tools and Libraries:

   bash Copy

   sudo dnf install @c-development @development-tools cmake

   This installs essential packages for compiling software, including gcc, make, and other development headers.

Adding the CUDA Repository Copy Chapter

Add the NVIDIA CUDA repository to your DNF configuration:

sudo dnf config-manager addrepo --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora41/x86_64/cuda-fedora41.repo

After adding the repository, synchronize the package manager again:

sudo dnf distro-sync

Installing Nvidia Driver Libraries Copy Chapter

First, we need to detect if the host is supplying the NVIDIA driver libraries into the toolbox:

ls -la /usr/lib64/libcuda.so.1

If *`libcuda.so.1`* is missing: Copy Section

ls: cannot access '/usr/lib64/libcuda.so.1': No such file or directory

Explanation:
The host dose not supply the CUDA drivers, install them now:

Install the Nvidia Driver Libraries on Guest:

sudo dnf install nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced

If *`libcuda.so.1`* exists: Copy Section

lrwxrwxrwx. 1 root root 21 Mar 24 11:26 /usr/lib64/libcuda.so.1 -> libcuda.so.570.133.07

Explanation:
The host is supply the CUDA drivers, we need to update the guest RPM Database accordingly:

Update the Toolbox RPM Database to include the Host-Supplied Libraries:

Note: we do not actually install the libraries, we just update the DB so that the guest system knows they are supplied by the host.

1. Download `nvidia-` parts that are supplied by the host RPM's (with dependencies)

sudo dnf download --destdir=/tmp/nvidia-driver-libs --resolve --arch x86_64 nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced

2. Update the RPM database to assume the installation of these packages.

sudo rpm --install --verbose --hash --justdb /tmp/nvidia-driver-libs/*

Note:

• The --justdb option only updates the RPM database, without touching the filesystem elsewhere.

Check that the RPM Database has been correctly updated:

Note: This is the same command as in the "Install the Nvidia Driver Libraries on Guest" for if libcuda.so.1 was missing.

sudo dnf install nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced

(this time it will not install anything, as the database things that these packages are already installed)

Updating and loading repositories:
Repositories loaded.
Package "nvidia-driver-cuda-3:570.124.06-1.fc41.x86_64" is already installed.
Package "nvidia-driver-libs-3:570.124.06-1.fc41.x86_64" is already installed.
Package "nvidia-driver-cuda-libs-3:570.124.06-1.fc41.x86_64" is already installed.
Package "nvidia-persistenced-3:570.124.06-1.fc41.x86_64" is already installed.

Nothing to do.

Installing the CUDA Meta-Package Copy Chapter

Now that the driver libraries are installed, proceed to install CUDA:

sudo dnf install cuda

This installs the CUDA toolkit and associated packages.

Configuring the Environment Copy Chapter

To use CUDA, add its binary directory to your system's PATH.

1. Create a Profile Script:

   bash Copy

   sudo sh -c 'echo "export PATH=\$PATH:/usr/local/cuda/bin" >> /etc/profile.d/cuda.sh'

   Explanation:

   • We add to /etc/profile.d/ as the /etc/ folder is unique to this particular container, and is not shared with other containers or the host system.
   • The backslash \ before $PATH ensures the variable is correctly written into the script.

2. Make the Script Executable:

   bash Copy

   sudo chmod +x /etc/profile.d/cuda.sh

3. Source the Script to Update Your Environment:

   bash Copy

   source /etc/profile.d/cuda.sh

   Note: This command updates your current shell session with the new PATH. The /etc/profile.d/cuda.sh script ensures that the CUDA binaries are available in your PATH for all future sessions.

Verifying the Installation Copy Chapter

To confirm that CUDA is correctly installed and configured, check the version of the NVIDIA CUDA Compiler (nvcc):

nvcc --version

You should see output similar to:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

This output confirms that the CUDA compiler is accessible and indicates the installed version.

Conclusion Copy Chapter

You have successfully set up CUDA on Fedora within a toolbox environment using the Fedora 41 CUDA repository. By manually updating the RPM db and configuring the environment, you can develop CUDA applications without affecting your host system.

Troubleshooting Copy Chapter

• Installation Failures:

  • If you encounter errors during installation, carefully read the error messages. They often indicate conflicting files or missing dependencies.
  • You may use the --excludepath option with rpm to exclude conflicting files during manual RPM installations.

• Rebooting the Container:

  • Sometimes there may be a bug in the NVIDIA driver host passthrough (such as missing a shared library). Rebooting the container may solve this issue:
  bash Copy

  # on the host system
  podman container restart --all

• Environment Variables Not Set:

  • If nvcc is not found after installation, ensure that /usr/local/cuda/bin is in your PATH.
  • Run echo $PATH to check if the path is included.
  • Re-source the profile script or open a new terminal session.

Additional Notes Copy Chapter

• Updating CUDA in the Future:

  • Keep an eye on the official NVIDIA repositories for updates to your Fedora version.
  • When an updated repository becomes available, adjust your dnf configuration accordingly.

• Building llama.cpp:

  • With CUDA installed, you can follow these build instructions for llama.cpp to compile it with CUDA support.
  • Ensure that any CUDA-specific build flags or paths are correctly set in your build configuration.

• Using the Toolbox Environment:

  • The toolbox environment is isolated from your host system, which helps prevent conflicts.
  • Remember that system files and configurations inside the toolbox are separate from the host. By default the home directory of the user is shared between the host and the toolbox.

Disclaimer: Manually installing and modifying system packages can lead to instability of the container. The above steps are provided as a guideline and may need adjustments based on your specific system configuration. Always back up important data before making significant system changes, especially as your home folder is writable and shared with he toolbox.

Acknowledgments: Special thanks to the Fedora community and NVIDIA documentation for providing resources that assisted in creating this guide.

References Copy Chapter

• Fedora Toolbox Documentation
• NVIDIA CUDA Installation Guide
• Podman Documentation

Llama.Cpp Docs Backend OPENCL (`llama.cpp/docs/backend/OPENCL.md`) Copy Section

llama.cpp for OpenCL

• Background
• OS
• Hardware
• DataType Supports
• Model Preparation
• CMake Options
• Android
• Windows 11 Arm64
• Known Issue
• TODO

Background Copy Chapter

OpenCL (Open Computing Language) is an open, royalty-free standard for cross-platform, parallel programming of diverse accelerators found in supercomputers, cloud servers, personal computers, mobile devices and embedded platforms. OpenCL specifies a programming language (based on C99) for programming these devices and application programming interfaces (APIs) to control the platform and execute programs on the compute devices. Similar to CUDA, OpenCL has been widely used to program GPUs and is supported by most GPU vendors.

Llama.cpp + OpenCL Copy Section

The llama.cpp OpenCL backend is designed to enable llama.cpp on Qualcomm Adreno GPU firstly via OpenCL. Thanks to the portabilty of OpenCL, the OpenCL backend can also run on certain Intel GPUs such as those that do not have SYCL support although the performance is not optimal.

OS Copy Chapter

Hardware Copy Chapter

Adreno GPU Copy Section

Verified devices

A6x GPUs with a recent driver and compiler are supported; they are usually found in IoT platforms.
However, A6x GPUs in phones are likely not supported due to the outdated driver and compiler.

DataType Supports Copy Chapter

Model Preparation Copy Chapter

You can refer to the general llama-quantize tool for steps to convert a model in Hugging Face safetensor format to GGUF with quantization.

Currently we support Q4_0 quantization and have optimized for it. To achieve best performance on Adreno GPU, add --pure to llama-quantize (i.e., make all weights in Q4_0). For example,

./llama-quantize --pure ggml-model-qwen2.5-3b-f16.gguf ggml-model-qwen-3b-Q4_0.gguf Q4_0

Since Q6_K is also supported, Q4_0 quantization without --pure will also work. However, the performance will be worse compared to pure Q4_0 quantization.

`MXFP4` MoE Models Copy Section

OpenAI gpt-oss models are MoE models in MXFP4. The quantized model will be in MXFP4_MOE, a mixture of MXFP4 and Q8_0.
For this quantization, there is no need to specify --pure.
For gpt-oss-20b model, you can directly download the quantized GGUF file in MXFP4_MOE from Hugging Face.

Although it is possible to quantize gpt-oss-20b model in pure Q4_0 (all weights in Q4_0), it is not recommended since MXFP4 has been optimized for MoE while Q4_0 is not. In addition, accuracy should degrade with such pure Q4_0 quantization.
Hence, using the default MXFP4_MOE quantization (see the link above) is recommended for this model.

Note that the Q4_0 model found here is a mixture of Q4_0, Q8_0 and MXFP4 and gives better performance than MXFP4_MOE quantization.

CMake Options Copy Chapter

The OpenCL backend has the following CMake options that control the behavior of the backend.

Android Copy Chapter

Ubuntu 22.04 is used for targeting Android. Make sure the following tools are accessible from command line,

• Git
• CMake 3.29
• Ninja
• Python3

I. Setup Environment Copy Section

1. Install NDK

cd ~
wget https://dl.google.com/android/repository/commandlinetools-linux-8512546_latest.zip && \
unzip commandlinetools-linux-8512546_latest.zip && \
mkdir -p ~/android-sdk/cmdline-tools && \
mv cmdline-tools latest && \
mv latest ~/android-sdk/cmdline-tools/ && \
rm -rf commandlinetools-linux-8512546_latest.zip

yes | ~/android-sdk/cmdline-tools/latest/bin/sdkmanager "ndk;26.3.11579264"

2. Install OpenCL Headers and Library

mkdir -p ~/dev/llm
cd ~/dev/llm

git clone https://github.com/KhronosGroup/OpenCL-Headers && \
cd OpenCL-Headers && \
cp -r CL ~/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include

cd ~/dev/llm

git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && \
cd OpenCL-ICD-Loader && \
mkdir build_ndk26 && cd build_ndk26 && \
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_TOOLCHAIN_FILE=$HOME/android-sdk/ndk/26.3.11579264/build/cmake/android.toolchain.cmake \
  -DOPENCL_ICD_LOADER_HEADERS_DIR=$HOME/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=24 \
  -DANDROID_STL=c++_shared && \
ninja && \
cp libOpenCL.so ~/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android

II. Build llama.cpp Copy Section

cd ~/dev/llm

git clone https://github.com/ggml-org/llama.cpp && \
cd llama.cpp && \
mkdir build-android && cd build-android

cmake .. -G Ninja \
  -DCMAKE_TOOLCHAIN_FILE=$HOME/android-sdk/ndk/26.3.11579264/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=android-28 \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_OPENCL=ON

ninja

Windows 11 Arm64 Copy Chapter

A Snapdragon X Elite device with Windows 11 Arm64 is used. Make sure the following tools are accessible from command line,

• Git
• CMake 3.29
• Clang 19
• Ninja
• Visual Studio 2022
• Powershell 7
• Python

Visual Studio provides necessary headers and libraries although it is not directly used for building.
Alternatively, Visual Studio Build Tools can be installed instead of the full Visual Studio.

Note that building using Visual Studio's cl compiler is not supported. Clang must be used. Clang depends on libraries provided by Visual Studio to work. Therefore, Visual Studio must be installed. Alternatively, Visual Studio Build Tools can be installed instead of the full Visual Studio.

Powershell 7 is used for the following commands.
If an older version of Powershell is used, these commands may not work as they are.

I. Setup Environment Copy Section

1. Install OpenCL Headers and Library

mkdir -p ~/dev/llm

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers
mkdir build && cd build
cmake .. -G Ninja `
  -DBUILD_TESTING=OFF `
  -DOPENCL_HEADERS_BUILD_TESTING=OFF `
  -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF `
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader
mkdir build && cd build
cmake .. -G Ninja `
  -DCMAKE_BUILD_TYPE=Release `
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" `
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

II. Build llama.cpp Copy Section

mkdir -p ~/dev/llm
cd ~/dev/llm

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
mkdir build && cd build

cmake .. -G Ninja `
  -DCMAKE_TOOLCHAIN_FILE="$HOME/dev/llm/llama.cpp/cmake/arm64-windows-llvm.cmake" `
  -DCMAKE_BUILD_TYPE=Release `
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" `
  -DBUILD_SHARED_LIBS=OFF `
  -DGGML_OPENCL=ON
ninja

Linux Copy Chapter

The two steps just above also apply to Linux. When building for linux, the commands are mostly the same as those for PowerShell on Windows, but in the second step they do not have the -DCMAKE_TOOLCHAIN_FILE parameter, and then in both steps the backticks are replaced with back slashes.

If not installed already, install Git, CMake, Clang, Ninja and Python, then run in the terminal the following:

I. Setup Environment Copy Section

1. Install OpenCL Headers and Library

mkdir -p ~/dev/llm

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers
mkdir build && cd build
cmake .. -G Ninja \
  -DBUILD_TESTING=OFF \
  -DOPENCL_HEADERS_BUILD_TESTING=OFF \
  -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF \
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader
mkdir build && cd build
cmake .. -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

II. Build llama.cpp Copy Section

mkdir -p ~/dev/llm
cd ~/dev/llm

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
mkdir build && cd build

cmake .. -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_OPENCL=ON
ninja

Known Issues Copy Chapter

• Flash attention does not always improve performance.
• Currently OpenCL backend works on A6xx GPUs with recent drivers and compilers (usually found in IoT platforms).
  However, it does not work on A6xx GPUs found in phones with old drivers and compilers.

TODO Copy Chapter

• Optimization for Q6_K
• Support and optimization for Q4_K
• Improve flash attention

Llama.Cpp Docs Backend SYCL (`llama.cpp/docs/backend/SYCL.md`) Copy Section

llama.cpp for SYCL

• Background
• Recommended Release
• News
• OS
• Hardware
• Docker
• Linux
• Windows
• Environment Variable
• Known Issue
• Q&A
• TODO

Background Copy Chapter

SYCL is a high-level parallel programming model designed to improve developers productivity writing code across various hardware accelerators such as CPUs, GPUs, and FPGAs. It is a single-source language designed for heterogeneous computing and based on standard C++17.

oneAPI is an open ecosystem and a standard-based specification, supporting multiple architectures including but not limited to Intel CPUs, GPUs and FPGAs. The key components of the oneAPI ecosystem include:

• DPCPP (Data Parallel C++): The primary oneAPI SYCL implementation, which includes the icpx/icx Compilers.
• oneAPI Libraries: A set of highly optimized libraries targeting multiple domains (e.g. Intel oneMKL, oneMath and oneDNN).
• oneAPI LevelZero: A high performance low level interface for fine-grained control over Intel iGPUs and dGPUs.
• Nvidia & AMD Plugins: These are plugins extending oneAPI's DPCPP support to SYCL on Nvidia and AMD GPU targets.

Llama.cpp + SYCL Copy Section

The llama.cpp SYCL backend is primarily designed for Intel GPUs.
SYCL cross-platform capabilities enable support for Nvidia GPUs as well, with limited support for AMD.

Recommended Release Copy Chapter

The following releases are verified and recommended: