ktransformers Documentation

File: README.md

🎯 Overview

KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT.

🔥 Updates

June 21, 2026: MiniMax-M3 Day0 Support! (Tutorial)
June 17, 2026: GLM-5.2 Day0 Support! (Tutorial)
May 6, 2026: KTransformers at GOSIM Paris 2026 — "Agentic AI on Edge" track. We'll present KT's inference performance on consumer hardware.
May 02, 2026: DeepSeek-V4-Flash Support! (Tutorial)
Apr 30, 2026: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate Inference and SFT Quick Start entry points.
Mar 26, 2026: Support AVX2-only CPU backend for KT-Kernel inference. (Tutorial)
Feb 13, 2026: MiniMax-M2.5 Day0 Support! (Tutorial)
Feb 12, 2026: GLM-5 Day0 Support! (Tutorial)
Jan 27, 2026: Kimi-K2.5 Day0 Support! (Tutorial) (SFT Tutorial)
Jan 22, 2026: Support CPU-GPU Expert Scheduling, Native BF16 and FP8 per channel Precision and AutoDL unified fine-tuning and inference
Dec 24, 2025: Support Native MiniMax-M2.1 inference. (Tutorial)
Dec 22, 2025: Support RL-DPO fine-tuning with LLaMA-Factory. (Tutorial)
Dec 5, 2025: Support Native Kimi-K2-Thinking inference (Tutorial)
Nov 6, 2025: Support Kimi-K2-Thinking inference (Tutorial) and fine-tune (Tutorial)
Nov 4, 2025: KTransformers Fine-Tuning × LLaMA-Factory Integration. (Tutorial)
Oct 27, 2025: Support Ascend NPU. (Tutorial)
Oct 10, 2025: Integrating into SGLang. (Roadmap, Blog)
Sept 11, 2025: Support Qwen3-Next. (Tutorial)
Sept 05, 2025: Support Kimi-K2-0905. (Tutorial)
July 26, 2025: Support SmallThinker and GLM4-MoE. (Tutorial)
July 11, 2025: Support Kimi-K2. (Tutorial)
June 30, 2025: Support 3-layer (GPU-CPU-Disk) prefix cache reuse.
May 14, 2025: Support Intel Arc GPU (Tutorial).
Apr 29, 2025: Support AMX-Int8、 AMX-BF16 and Qwen3MoE (Tutorial)
Apr 9, 2025: Experimental support for LLaMA 4 models (Tutorial).
Apr 2, 2025: Support Multi-concurrency. (Tutorial).
Mar 15, 2025: Support ROCm on AMD GPU (Tutorial).
Mar 5, 2025: Support unsloth 1.58/2.51 bits weights and IQ1_S/FP8 hybrid weights. Support 139K Longer Context for DeepSeek-V3 and R1 in 24GB VRAM.
Feb 25, 2025: Support FP8 GPU kernel for DeepSeek-V3 and R1; Longer Context.
Feb 15, 2025: Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed （+15%, up to 16 Tokens/s), update docs and online books.
Feb 10, 2025: Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see here.
Aug 28, 2024: Decrease DeepseekV2's required VRAM from 21G to 11G.
Aug 15, 2024: Update detailed tutorial for injection and multi-GPU.
Aug 14, 2024: Support llamfile as linear backend.
Aug 12, 2024: Support multiple GPU; Support new model: mixtral 8*7B and 8*22B; Support q2k, q3k, q5k dequant on gpu.
Aug 9, 2024: Support windows native.

📦 Capabilities

🚀 [Inference](./kt-kernel/README.md) - High-Performance kt-kernel Serving

CPU-optimized kernel operations for heterogeneous LLM inference.

Key Features:

AMX/AVX Acceleration: Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference
MoE Optimization: Efficient Mixture-of-Experts inference with NUMA-aware memory management
Quantization Support: CPU-side INT4/INT8 quantized weights, GPU-side GPTQ support
Easy Integration: Clean Python API for SGLang and other frameworks

Quick Start:

bash

cd kt-kernel
pip install .

Use Cases:

CPU-GPU hybrid inference for large MoE models
Integration with SGLang for production serving
Heterogeneous expert placement (hot experts on GPU, cold experts on CPU)

Performance Examples:

Model	Hardware Configuration	Total Throughput	Output Throughput
DeepSeek-R1-0528 (FP8)	8×L20 GPU + Xeon Gold 6454S	227.85 tokens/s	87.58 tokens/s (8-way concurrency)

👉 Full Documentation →

🎓 [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) - Fine-Tuning with LLaMA-Factory

KTransformers × LLaMA-Factory integration for ultra-large MoE model fine-tuning.

Key Features:

Multi-Backend Support: CPU/GPU hybrid fine-tuning with INT8/INT4 quantization
Ultra-Large MoE Support: Fine-tune models like DeepSeek-V3/R1 on limited GPU memory
Faster than ZeRO-Offload: 6-12x training speedup in benchmarked MoE SFT workloads
Lower CPU Memory: About half the CPU memory of the previous KT SFT path in the benchmarked setup
LLaMA-Factory Integration: Seamless integration with popular fine-tuning framework

Model	GPU Memory	Training Speed	Hardware
DeepSeek-V3	~80GB total	3.7 it/s	4x RTX 4090
DeepSeek-R1	~80GB total	3.7 it/s	4x RTX 4090
Qwen3-30B-A3B	~24GB total	8+ it/s	1x RTX 4090

Quick Start:

bash

cd /path/to/LLaMA-Factory
pip install -e .
pip install -r requirements/ktransformers.txt
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
  --config_file examples/ktransformers/accelerate/fsdp2_kt_int8.yaml \
  src/train.py \
  examples/ktransformers/train_lora/qwen3_5moe_lora_sft_kt.yaml

👉 Quick Start →
👉 Full Documentation →

🔥 Citation

If you use KTransformers in your research, please cite our paper:

bibtex

@inproceedings{10.1145/3731569.3764843,
  title = {KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models},
  author = {Chen, Hongtao and Xie, Weiyu and Zhang, Boxin and Tang, Jingqi and Wang, Jiahao and Dong, Jianwei and Chen, Shaoyuan and Yuan, Ziwei and Lin, Chen and Qiu, Chengyu and Zhu, Yuening and Ou, Qingliang and Liao, Jiaqi and Chen, Xianglin and Ai, Zhiyuan and Wu, Yongwei and Zhang, Mingxing},
  booktitle = {Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles},
  year = {2025}
}

👥 Contributors & Team

Developed and maintained by:

MADSys Lab @ Tsinghua University
Approaching.AI
9#AISoft
Community contributors

We welcome contributions! Please feel free to submit issues and pull requests.

💬 Community & Support

GitHub Issues: Report bugs or request features
WeChat Group: See archive/WeChatGroup.png

📦 KT original Code

The original integrated KTransformers framework has been archived to the archive/ directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance.

For the original documentation with full quick-start guides and examples, see:

archive/README.md (English)
archive/README_ZH.md (中文)

File: doc/en/api/server/api.md

OpenAI ChatCompletion

bash

POST /v1/chat/completions

Generate responses based on the selected model.

Parameters

messages: An array of message representing all historical messages. A message can be from a user or model (assistant) and includes:
- role: Either user or assistant, indicating the creator of this message.
- content: The message from the user or model.
model: The name of the selected model
stream: Either true or false. Indicates whether to use streaming response. If true, model inference results are returned via HTTP event stream.

Response

Streaming response: An event stream, each event contains a chat.completion.chunk. chunk.choices[0].delta.content is the incremental output returned by the model each time.
Non-streaming response: Not supported yet.

Example

bash

curl -X 'POST' \
  'http://localhost:9112/v1/chat/completions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "messages": [
    {
      "content": "tell a joke",
      "role": "user"
    }
  ],
  "model": "Meta-Llama-3-8B-Instruct",
  "stream": true
}'

bash

data:{"id":"c30445e8-1061-4149-a101-39b8222e79e1","object":"chat.completion.chunk","created":1720511671,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"delta":{"content":"Why ","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}

data:{"id":"c30445e8-1061-4149-a101-39b8222e79e1","object":"chat.completion.chunk","created":1720511671,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"delta":{"content":"","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}

data:{"id":"c30445e8-1061-4149-a101-39b8222e79e1","object":"chat.completion.chunk","created":1720511671,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"delta":{"content":"couldn't ","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}

...

data:{"id":"c30445e8-1061-4149-a101-39b8222e79e1","object":"chat.completion.chunk","created":1720511671,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"delta":{"content":"two-tired!","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}

event: done
data: [DONE]

Ollama ChatCompletion

bash

POST /api/generate

Generate responses using the selected model.

Parameters

prompt: A string representing the input prompt.
model: The name of the selected model
stream: Either true or false. Indicates whether to use streaming responses. If true, returns the model inference results in the form of an HTTP event stream.

Response

Streaming response: A stream of JSON responses, each line is a JSON.
- response: The incremental result of the model completion.
- done: Whether the inference has finished.
Non-streaming response: Not yet supported.

例子

bash

curl -X 'POST' \
  'http://localhost:9112/api/generate' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "Meta-Llama-3-8B-Instruct",
  "prompt": "tell me a joke",
  "stream": true
}'

bash

{"model":"Meta-Llama-3-8B-Instruct","created_at":"2024-07-09 08:13:11.686513","response":"I'll ","done":false}
{"model":"Meta-Llama-3-8B-Instruct","created_at":"2024-07-09 08:13:11.729214","response":"give ","done":false}

...

{"model":"Meta-Llama-3-8B-Instruct","created_at":"2024-07-09 08:13:33.955475","response":"for","done":false}
{"model":"Meta-Llama-3-8B-Instruct","created_at":"2024-07-09 08:13:33.956795","response":"","done":true}

File: doc/en/api/server/server.md

Backend Services (Server)

The Server offers fast heterogeneous inference capabilities of ktransformers through an API for external usage.

API

The Server provides model inference services externally through a RESTful API, with two methods of interaction: ChatCompletion and Assistant.

The ChatCompletion interface requires users to provide all historical dialogues at once, after which the model responds. AI service providers (such as OpenAI) and local inference frameworks (such as Ollama) both offer the ChatCompletion interface. To ensure compatibility with OpenAI and Ollama, the Server offers APIs that are consistent with theirs. Therefore, applications currently using OpenAI and Ollama can seamlessly switch to our Server. For example: How to use Tabby and ktransformers locally with a 236B model for code completion?.
The Assistant is suitable for applications that need to reuse a series of resources and call the model. For instance, in educational applications, developers can create an Assistant named "Second Grade Math Teacher" and set an initial prompt ("You are an experienced second-grade math teacher..."), and upload relevant materials (second grade math textbooks). After creating the Assistant, the application needs to create a Thread to store the dialogues between the user and the model (Message). When calling the model, the application creates a Run to obtain the Assistant's response. Compared to ChatCompletion, the Assistant-enabled Server handles the reuse of conversational contexts and multi-turn dialogues, making model calls in complex scenarios more convenient. The OpenAI Assistant API introduces such an Assistant interface, and the Server provides a consistent API.

These API definitions are located in server/api, and their specific usage can be seen here.

Integrating Model Inference Frameworks

The Server uses ktransformers for model calling and inference. It also supports other inference frameworks, such as the already supported transformers, and plans to support exllamav2. These functionalities are implemented in server/backend.

The model inference functionalities of the frameworks are abstracted into a base class BackendInterfaceBase. This class includes a function: inference. It takes historical dialogue information messages as input and returns the text result from the model. The inference function adopts an async generator design, allowing the Server to return model responses in a streaming manner.

python

class BackendInterfaceBase:
  async def inference(self, messages, **kwargs)->AsyncIterator[str]:
    ...

This inference function naturally implements the functionality of ChatCompletion because its inputs and outputs are historical dialogues and model responses, respectively. Thus, the ChatCompletion API can directly call the inference function to complete model inference.

Assistant is more complex than ChatCompletion, requiring the Server to store the related state of the Assistant and call the inference function appropriately. The Server maintains a set of Assistant logic in the database, storing the Assistants, Threads, and Messages created by applications. In memory, the Server maintains a ThreadContext for each Thread, gathering information related to each Thread's Assistant, etc. When a user sends a new Message, the Server calls the get_local_messages function of ThreadContext to obtain messages and then calls the inference function to get the inference results.

python

class MyThreadContext(ThreadContext):
    def get_local_messages(self):
      ...

Since different model inference frameworks have different historical dialogue input formats, ThreadContext and BackendInterface need to be used in pairs. Besides its own ktransformers, the Server also supports transformers. For integrating other model inference frameworks, refer to the implementations of TransformersInterface and TransformersThreadContext in transformers.py.

File: doc/en/api/server/tabby.md

How to Use Tabby and ktransformers Locally with 236B Large Models for Code Completion?

Tabby is an open-source code assistant that allows users to manually configure the backend framework and model, and use it across multiple IDEs/editors, such as VSCode and IntelliJ. Since Tabby can interface with Ollama on the framework side, and the ktransformers server provides a consistent API with Ollama, we can connect Tabby to the ktransformers server. This setup allows us to experience fast, heterogeneous inference in code completion scenarios.

Start ktransformers.

bash

./ktransformers --port 9112

Install Tabby: Follow the official tutorial to install Tabby on a Linux server or Windows PC with an NVIDIA GPU here.
Configure Tabby: Create ~/.tabby/config.toml and add the following configuration.

toml

[model.completion.http]
kind = "ollama/completion"
api_endpoint = "http://127.0.0.1:9112/"
model_name = "DeepSeek-Coder-V2-Instruct"
prompt_template = "<｜fim▁begin｜>{prefix}<｜fim▁hole｜>{suffix}<｜fim▁end｜>" # Prompt Template

In this configuration, kind specifies that ktransformers uses the standard Ollama API to serve Tabby; api_endpoint matches the interface bound when launching ktransformers; model_name is set to the model used by ktransformers, here DeepSeek-Coder-V2-Instruct is the backend inference model; prompt_template is the model's prompt template, which requires a corresponding template for different models to use the Fill In the Middle feature properly.
Here we demonstrate the relevant configuration for Tabby using the Ollama API to provide the Completion feature. For configuration information about other functions available in Tabby, refer to here.

Start the Tabby service: ./tabby serve.

After launching, you should see access to the /api/tags interface in the ktransformers command line (in version v0.13.0 of Tabby, this changes to access to the /api/show/ interface).
Register a Tabby account, obtain a Token: After starting the Tabby service, open the corresponding link in a browser (as shown above at 0.0.0.0:8080), and follow the tutorial to create a user and get a Token.
Start VSCode, install the Tabby extension plugin, and use the Token obtained in the previous step to connect to the Tabby Server, following here.
Open any code file and experience the fast heterogeneous inference of ktransformers.

File: doc/en/api/server/website.md

Start with website

This document provides the necessary steps to set up and run the web service for this project.

1. Starting the Web Service

1.1. Compiling the Web Code

Before you can compile the web code, make sure you have installed Node.js version 18.3 or higher

Note: The version of Node.js in the Ubuntu or Debian GNU/Linux software repository is too low, causing compilation errors. Users can also install Node.js through the Nodesource repository, provided they uninstall the outdated version first.

bash

# sudo apt-get remove nodejs npm -y && sudo apt-get autoremove -y
  sudo apt-get update -y && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
  curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/nodesource.gpg
  sudo chmod 644 /usr/share/keyrings/nodesource.gpg
  echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_23.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list
  sudo apt-get update -y
  sudo apt-get install nodejs -y

Once npm is installed, navigate to the ktransformers/website directory:

bash

cd ktransformers/website

Next, install the Vue CLI with the following command:

bash

npm install @vue/cli

Now you can build the project:

bash

npm run build

Finally you can build ktransformers with website:

text

cd ../../
pip install .

File: doc/en/operators/llamafile.md

Llamafile Operators Documentation

Llamafile Sgemm

The Llamafile Sgemm module is an efficient implementation of general matrix multiplication (GEMM) extracted from the great Llamafile project.
This module optimizes performance by utilizing various processor-specific instruction sets. For instance, it checks for different x86 instruction sets such as AVX, FMA, and AVX512, leveraging these advanced instructions to accelerate computation.
Additionally, the Llamafile Sgemm module supports multiple quantization types, including q8_0, q6_k, and q5_k, among others. This adaptability to different hardware capabilities ensures the most advanced instructions are used in any given computing environment, achieving high computational efficiency. For more information, you can view the Llamafile Sgemm module on GitHub.

CPUInfer

To power Llamafile and many future CPU kernels without the original GGML framework, we developed a simple CPUInfer multi-threaded execution framework. It currently leverages the Llamafile Sgemm module to implement key operators such as linear layers, MLP, and MoE, and will be extended to support many other operators. These operators are fundamental components for building large models. CPUInfer features a backend work-stealing thread pool and asynchronous task queue execution logic to efficiently offload parts of model parameters to the CPU, thereby maintaining high inference performance. It supports adjustments based on hardware capabilities or user configurations, providing enhanced inference performance and making it an ideal tool for running deep learning models on CPUs.

Expert-Parallel MoE

The MoE module's performance can be enhanced by using custom kernels that utilize expert parallelism. Since the routed experts are independently computable, we can utilize this inherent parallelism to speed up MoE computations. Specifically, we can allocate each expert MLP to a separate thread group, allowing for the simultaneous computation of all routed experts. This approach of expert parallelism significantly boosts MoE performance by minimizing the frequency of global synchronizations and reducing kernel launch overhead compared to sequential expert computation.

Microbenchmark

Our evaluations were conducted on an Intel(R) Xeon(R) Gold 6454S processor, utilizing real parameters from the DeepSeek-Coder-V2-Instruct model.

Linear Projection

The performance of the linear layer was assessed using an Attention Output Projection with dimensions of [5120, 16384]. Here, the input was a vector of 16384 dimensions, and the output was a vector of 5120 dimensions.

As we can see, in half-precision floating-point formats (fp16 and bf16), CPUInfer's performance exceeded that of Torch by 1.7 and 1.5 times, respectively. For 8-bit quantization, CPUInfer (supporting q8_0) and Torch (supporting qint8) demonstrated nearly equivalent performance. However, CPUInfer employs a more refined scaling approach, using different factors for each group (in q8_0 quantization, every 32 numbers form one group), whereas Torch uses a basic per-tensor quantization, potentially leading to significant precision loss. Furthermore, CPUInfer’s capability to use lower-bit quantization enhances inference speed in specific scenarios.

MoE

In the MoE module, each token selected 6 experts out of 160 for computation, with input and output dimensions of 5120, and an intermediate dimension of 1536.

For half-precision floating points and 8-bit quantization formats, CPUInfer's generation performance was 2.5 and 3.2 times better than Torch, respectively. Moreover, using the 8-bit quantization format, CPUInfer achieved faster prefill speeds compared to Torch, with shorter prompts highlighting a more pronounced performance difference.

File: doc/en/AMX.md

Qwen 3 + KTransformers 0.3 (+AMX) = AI Workstation/PC

Following DeepSeek-V3/R1, LLaMa-4, and Kimi-VL, Qwen has also released an impressive MoE model—undoubtedly, this year belongs to MoE. As a low-barrier inference system for running MoE models in local heterogeneous environments, KTransformers naturally joins the party. Thanks to the support of the Qwen team, we completed Day 0 support for the entire Qwen 3 series of MoE models. At the same time, we took this opportunity to open-source the long-awaited preliminary version of our AMX high-performance operators (BF16, Int8; an int4 variant is coming soon), officially advancing to version 0.3.

What excites me most about Qwen3MoE is that, unlike the 671 B “giant” model, its two configurations: 235B-A22 and 30B-A3B, hit the performance sweet spots for both local workstations and consumer-grade PCs. Accordingly, we ran benchmarks in two typical setups:

Server CPU (Xeon 4) + RTX 4090

Consumer-grade CPU (Core i9-14900KF + dual-channel DDR5-4000 MT/s) + RTX 4090

Note: Because the PC's memory has a low frequency, large capacity, and multiple sticks, it downclocks severely and only operates at 4000MT. Using higher - frequency memory can boost performance.

The results are as follows:

https://github.com/user-attachments/assets/fafe8aec-4e22-49a8-8553-59fb5c6b00a2

You can see that, thanks to the AMX instruction optimizations, we achieve up to 347 tokens/s prefill performance in the workstation scenario. On consumer-grade CPUs, we’re able to run the large model (235B-A22) and deliver smooth performance on the smaller 30B-A3B. Even in terms of resource overhead, it appears that a high-end gaming laptop can handle 30B-A3B smoothly. After talking about the concept of AIPC for so long, we can finally see its feasibility.

Here is the Qwen3MoE startup command:

python

# llamafile backend
python ktransformers/server/main.py --architectures Qwen3MoeForCausalLM --model_path <model_dir> --gguf_path <gguf_dir> --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --backend_type balance_serve
# AMX backend
python ktransformers/server/main.py --architectures Qwen3MoeForCausalLM --model_path <model_dir> --gguf_path <gguf_dir> --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve-amx.yaml --backend_type balance_serve

Note: At present, Qwen3MoE running with AMX can only read BF16 GGUF; support for loading from safetensor will be added later.

To make it easier for everyone to understand the AMX optimizations we’ve open-sourced, we’ve prepared a brief document. We also extend our gratitude to Intel for their assistance.

Introduction to AMX Instruction Set

Intel Advanced Matrix Extensions (AMX) are a set of specialized instruction extensions introduced for the x86 architecture starting with Sapphire Rapids (4th generation Xeon Scalable processors) and onward. AMX accelerates large-scale matrix computations at the hardware level, particularly for the compute-intensive parts of deep learning inference and machine learning workloads. By introducing the concept of Tile registers, it loads 2D sub-matrices into dedicated Tile registers and performs matrix multiply-accumulate operations at the register level, significantly improving throughput and energy efficiency.

Each CPU core contains 8 dedicated registers (tmm0–tmm7), with each register capable of holding up to 16 rows × 64 bytes of data to store 2D sub-matrices. Additionally, there is a 64-byte configuration register (TILECFG) used to describe each tmm register's number of rows, columns, and row stride.

The main AMX instructions are summarized as follows:

Instruction Category	Instruction Names	Description
Configuration Instructions	LDTILECFG, STTILECFG, TILERELEASE, TILEZERO	Configure/reset Tile registers and metadata
Load/Store Instructions	TILELOADD, TILELOADDT1, TILESTORED	Transfer data between memory and Tile registers
INT8 Computation Instructions	TDPBSSD, TDPBUSD, TDPBUUD, TDPBSUD	Perform multiply and accumulate operations on int8 sub-matrices within Tiles
BF16 Computation Instructions	TDPBF16PS	Perform multiply and accumulate operations on bfloat16 sub-matrices within Tiles

To simplify development, Intel provides corresponding intrinsics, allowing C/C++ developers to leverage AMX's performance benefits without writing lengthy assembly code. For example:

c++

#include 

_tile_loadconfig(cfg_ptr);
_tile_loadd(tmm0, A_ptr, lda);
_tile_loadd(tmm1, B_ptr, ldb);
_tile_zero(tmm2)
_tile_dpbf16ps(tmm2, tmm0, tmm1);
_tile_stored(tmm2, C_ptr, ldc);
_tile_release();

The above code copies sub-matrices from memory (A_ptr, B_ptr) to Tile registers, calls the AMX BF16 compute instruction to multiply two sub-matrices, and then copies the result to memory (C_ptr).

Taking INT8 as an example, AMX can perform the multiplication of two 16×64 sub-matrices (32,768 multiply/add operations) with a single instruction in 16 CPU cycles, enabling each core to complete 2048 multiply/add operations per cycle — 8 times the performance of AVX-512. On an Intel Xeon 4 CPU, a single core can theoretically provide 4 TOPS of compute power, making it highly suitable for compute-intensive tasks on the CPU.

AMX Kernel in KTransformers

Before version v0.3, KTransformers performed CPU matrix multiplications based on operators provided by llamafile. Unfortunately, llamafile's implementation had not yet been optimized for the AMX instruction set. This resulted in performance bottlenecks, even in strong hardware environments (such as Xeon 4th Gen + 4090), where inference speeds for large models like DeepSeek-V3 reached only 91 tokens/s during the prefill phase. The CPU thus remained a significant bottleneck. In long prompt scenarios, such performance is clearly unsatisfactory. To fully unleash CPU potential, we introduced a brand-new AMX optimization path along with multiple technical improvements in v0.3.

1. AMX Tiling-aware Memory Layout

AMX provides a high-throughput Tile register computation model, reducing instruction count and boosting theoretical throughput through coarse-grained matrix operations. However, to truly exploit AMX's potential, memory access efficiency is critical: because AMX transfers entire Tiles at once, misaligned Tiles and chaotic access patterns can cause severe cache misses, nullifying throughput gains.

Thus, in v0.3, we stopped directly memory-mapping GGUF-format files and introduced AMX Tiling-aware memory preprocessing during model loading. Specifically, expert weight matrices in MoE models are pre-rearranged into Tile-friendly sub-matrices whose shapes precisely match AMX Tile register dimensions, eliminating dynamic transposition overhead during inference. During rearrangement, we strictly align each sub-matrix's start address to 64 bytes to avoid cache line splits, and arrange sub-matrices sequentially according to computation access patterns, maximizing L1/L2 cache hit rates using compiler and hardware sequential prefetch capabilities.

For Int8 quantized formats, we adopted Symmetric Group-wise Quantization, with each column forming a group sharing a scale factor stored separately to maintain memory alignment for Tile data.

This AMX Tiling-aware memory layout design reduces memory latency while providing optimal input conditions for downstream computation kernels.

2. Cache-friendly AMX Kernel

During inference, we designed around the CPU’s multi-level cache hierarchy to perform computations in-place in high-speed caches, minimizing DRAM access frequency and overhead.

As shown in the figure,

① Expert weight matrices are first column-wise partitioned into multiple tasks dynamically scheduled across threads. Input activations are shared among tasks and typically reside in the shared L3 cache due to locality.
② Within each task, expert weights are row-wise partitioned into blocks, with block sizes finely tuned to ensure input activations, weights, and intermediate results stay within L2 cache, avoiding DRAM access.
③ ④ ⑤ Each block is treated as a set of sub-matrices matching AMX Tile registers, and during Tile-level computation, input Tiles (tmm0–tmm1) and expert Tiles (tmm2–tmm3) are loaded, and four AMX multiplication instructions directly generate and accumulate products into Tile registers (tmm4–tmm7), with output activations accumulated in Tile registers or L1 cache, avoiding additional data movement.

In short, we leveraged the cache hierarchy: every data element of expert weights and output activations accesses DRAM only once, with the other accesses hitting L2 or higher caches; input activations are accessed from DRAM only once and later hit in L3 or higher caches. This significantly reduces main memory traffic and improves overall execution efficiency.

3. AVX-512 Kernel Adaptation for Low Arithmetic Intensity Scenarios

Although AMX is highly efficient for large-scale matrix multiplication, it performs poorly under low arithmetic intensity, such as vector-matrix operations in the decode phase. This is because dispatching AMX Tiles involves fixed instruction overhead, which becomes wasteful when the data volume is insufficient to fill a Tile, causing reduced throughput.

To address this, we introduced a lightweight AVX-512 kernel as a complement. This kernel follows the same memory layout as the AMX kernel but replaces heavy AMX matrix-matrix multiplications with fine-grained AVX-512 vector-matrix multiplications, lowering latency for small matrices.

KTransformers dynamically selects between AMX and AVX-512 kernels at runtime based on arithmetic intensity: AMX kernels are automatically selected during long prompt prefill phases (where each expert handles more than 4 tokens on average), while short prompt prefill and decode phases dynamically switch to AVX-512 kernels. This ensures optimal efficiency under different arithmetic intensity conditions.

4. MoE Operator Fusion and Dynamic Scheduling

MoE models have many experts per layer, each requiring three matrix multiplications (Gate, Up, Down projections), leading to many small matrix multiplication tasks. Independently scheduling each small task would cause massive synchronization overhead between threads, dragging down overall inference speed.

Thus, we fused the same type of matrix computations for all experts in a layer into large unified tasks. Furthermore, as there are no data dependencies between Gate and Up projections, their computations can also be fused, ultimately consolidating a layer’s matrix multiplications into two major tasks, greatly reducing scheduling overhead.

To address load imbalance — especially during the prefill phase where expert activations can be highly skewed — we introduced a dynamic task scheduling strategy. Each matrix multiplication task is further split into multiple fine-grained sub-tasks, evenly distributed among CPU threads initially. Once a thread completes its assigned tasks, it atomically "steals" tasks from others, greatly mitigating load imbalance and achieving near-optimal CPU resource utilization.

Thanks to these optimizations, our kernel can achieve 21 TFLOPS of BF16 throughput and 35 TOPS of Int8 throughput on Xeon4 CPUs — about 4× faster than PyTorch’s general AMX kernel. For DeepSeek-V3, pairing a Xeon4 CPU with a single RTX 4090 GPU achieves 418 tokens/s end-to-end throughput, close to the performance of multi-machine, multi-GPU setups. KTransformers’ AMX kernel is the first AMX kernel specifically designed for MoE inference scenarios, significantly lowering the hardware barrier for large model deployment and enabling more developers to enjoy GPU cluster level inference experiences at lower cost.

Usage

Checking AMX Support

Before enabling the AMX-optimized kernels, it is important to verify whether your CPU supports the AMX instruction set. You can check AMX availability with the following command:

bash

lscpu | grep -i amx

If your system supports AMX, you should see output similar to:

bash

Flags: ... amx-bf16 amx-int8 amx-tile ...

If no amx-related flags are found, your CPU may not support AMX, or AMX may be disabled in BIOS settings. In that case, please ensure that:

You are using a Sapphire Rapids (Xeon 4th Gen) or newer CPU.
AMX support is enabled in your system BIOS under CPU feature settings.

Enabling AMX in KTransformers

KTransformers allows users to easily switch between different backends through simple YAML configuration modifications. To enable AMX, modify the injection configuration of your experts by specifying backend as AMXInt8 or AMXBF16:

yaml

- match:
    name: "^model\\.layers\\..*\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts    # custom MoE Kernel with expert parallelism
    kwargs:
      prefill_device: "cuda"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op: "KExpertsCPU"
      out_device: "cuda"
      backend: "AMXInt8"  # or "AMXBF16" or "llamafile" (default)

Note: Currently, using AMXInt8 requires reading weights from a BF16 GGUF file and performing online quantization during model loading. This may cause slightly slower load times. Future versions will provide pre-quantized weights to eliminate this overhead.

File: doc/en/balance-serve.md

Balance Serve backend (multi-concurrency) for ktransformers

KTransformers v0.2.4 Release Notes

We are excited to announce the official release of the long-awaited KTransformers v0.2.4!
In this version, we’ve added highly desired multi-concurrency support to the community through a major refactor of the whole architecture, updating more than 10,000 lines of code.
By drawing inspiration from the excellent architecture of sglang, we have implemented high-performance asynchronous concurrent scheduling in C++, including features like continuous batching, chunked prefill, and more. Thanks to GPU sharing in concurrent scenarios, overall throughput is also improved to a certain extent. The following is a demonstration:

https://github.com/user-attachments/assets/faa3bda2-928b-45a7-b44f-21e12ec84b8a

🚀 Key Updates

Multi-Concurrency Support
- Added capability to handle multiple concurrent inference requests. Supports receiving and executing multiple tasks simultaneously.
- We implemented custom_flashinfer based on the high-performance and highly flexible operator library flashinfer, and achieved a variable batch size CUDA Graph, which further enhances flexibility while reducing memory and padding overhead.
- In our benchmarks, overall throughput improved by approximately 130% under 4-way concurrency.
- With support from Intel, we tested KTransformers v0.2.4 on the latest Xeon6 + MRDIMM-8800 platform. By increasing concurrency, the total output throughput increased from 17 tokens/s to 40 tokens/s. We observed that the bottleneck has now shifted to the GPU. Using a higher-end GPU than the 4090D could further improve performance.
Engine Architecture Optimization

Inspired by the scheduling framework of sglang, we refactored KTransformers with a clearer three-layer architecture through an update of 11,000 lines of code, now supporting full multi-concurrency:
- Server：Handles user requests and serves the OpenAI-compatible API.
- Inference Engine：Executes model inference and supports chunked prefill.
- Scheduler：Manages task scheduling and requests orchestration. Supports continuous batching by organizing queued requests into batches in a FCFS manner and sending them to the inference engine.
Project Structure Reorganization
All C/C++ code is now centralized under the /csrc directory.
Parameter Adjustments
Removed some legacy and deprecated launch parameters for a cleaner configuration experience.
We plan to provide a complete parameter list and detailed documentation in future releases to facilitate flexible configuration and debugging.

📚 Upgrade Notes

Due to parameter changes, users who have installed previous versions are advised to delete the ~/.ktransformers directory and reinitialize.
To enable multi-concurrency, please refer to the latest documentation for configuration examples.

What's Changed

Implemented custom_flashinfer @Atream @ovowei @qiyuxinlin
Implemented balance_serve engine based on FlashInfer @qiyuxinlin @ovowei
Implemented a continuous batching scheduler in C++ @ErvinXie
release: bump version v0.2.4 by @Atream @Azure-Tang @ErvinXie @qiyuxinlin @ovowei @KMSorSMS @SkqLiao

Download the Docker image for testing v0.2.4

Visit the link to pull the image, using v0.2.4-AVX512 as an example.

bash

docker pull approachingai/ktransformers:v0.2.4-AVX512
docker run -it --gpus all --privileged --shm-size 64g --name ktrans --network=host -v /mnt:/mnt approachingai/ktransformers:v0.2.4-AVX512 /bin/bash
# Open a new terminal
docker exec -it ktrans bash

Installation Guide

⚠️ Please note that installing this project will replace flashinfer in your environment. It is strongly recommended to create a new conda environment!!!

2. Set Up Conda Environment

We recommend using Miniconda3/Anaconda3 for environment management:

bash

# Download Miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

# Create environment
conda create --name ktransformers python=3.11
conda activate ktransformers

# Install required libraries
conda install -c conda-forge libstdcxx-ng

# Verify GLIBCXX version (should include 3.4.32)
strings ~/anaconda3/envs/ktransformers/lib/libstdc++.so.6 | grep GLIBCXX

Note: Adjust the Anaconda path if your installation directory differs from ~/anaconda3

2. Install dependencies

bash

sudo apt install libtbb-dev libssl-dev libcurl4-openssl-dev libaio1 libaio-dev libfmt-dev libgflags-dev zlib1g-dev patchelf
pip3 install packaging ninja cpufeature numpy openai
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

3. Build ktransformers

bash

# Clone repository
git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive


# Install single NUMA dependencies
USE_BALANCE_SERVE=1  bash ./install.sh
# For those who have two cpu and 1T RAM（Dual NUMA）:
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh

Running DeepSeek-R1-Q4KM Models

1. Run for 24GB VRAM GPUs

Use our optimized configuration for constrained VRAM:

bash

python ktransformers/server/main.py \
  --port 10002 \
  --model_path  \
  --gguf_path  \
  --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
  --max_new_tokens 1024 \
  --cache_lens 32768 \
  --chunk_size 256 \
  --max_batch_size 4 \
  --backend_type balance_serve \
  --force_think # useful for R1

It features the following arguments:

--max_new_tokens: Maximum number of tokens generated per request.
--cache_lens: Total length of kvcache allocated by the scheduler. All requests share a kvcache space.
--max_batch_size: Maximum number of requests (prefill + decode) processed in a single run by the engine. (Supported only by balance_serve)
--chunk_size: Maximum number of tokens processed in a single run by the engine.
corresponding to 32768 tokens, and the space occupied will be released after the requests are completed.
--backend_type: balance_serve is a multi-concurrency backend engine introduced in version v0.2.4. The original single-concurrency engine is ktransformers.
--model_path: Path to safetensor config path (only config required, not model safetensors).
Please note that, since ver 0.2.4, the last segment of ${model_path} directory name MUST be a local directory that contains the model's configuration files. Hugging Face links (e.g., deepseek-ai/DeepSeek-R1) are not supported at the moment.
--force_think: Force responding the reasoning tag of DeepSeek R1.

The relationship between max_batch_size, cache_lens, and max_new_tokens should satisfy:
cache_lens > max_batch_size * max_new_tokens, otherwise the concurrency will decrease.

2. access server

text

curl -X POST http://localhost:10002/v1/chat/completions \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "hello"}
    ],
    "model": "DeepSeek-R1",
    "temperature": 0.3,
    "top_p": 1.0,
    "stream": true
  }'

Docker & Self-Hosted Deployment Verified OCI

File: README.md Copy Chapter

🎯 Overview Copy Chapter

🔥 Updates Copy Chapter

📦 Capabilities Copy Chapter

🚀 [Inference](./kt-kernel/README.md) - High-Performance kt-kernel Serving Copy Section

🎓 [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) - Fine-Tuning with LLaMA-Factory Copy Section

🔥 Citation Copy Chapter

👥 Contributors & Team Copy Chapter

💬 Community & Support Copy Chapter

📦 KT original Code Copy Chapter

File: doc/en/api/server/api.md Copy Chapter

API

OpenAI ChatCompletion Copy Chapter

Parameters Copy Section

Response Copy Section

Example Copy Section

Ollama ChatCompletion Copy Chapter

Parameters Copy Section

Response Copy Section

例子 Copy Section

File: doc/en/api/server/server.md Copy Chapter

Backend Services (Server)

API Copy Chapter

Integrating Model Inference Frameworks Copy Chapter

File: doc/en/api/server/tabby.md Copy Chapter

How to Use Tabby and ktransformers Locally with 236B Large Models for Code Completion?

File: doc/en/api/server/website.md Copy Chapter

Start with website

1. Starting the Web Service Copy Chapter

1.1. Compiling the Web Code Copy Section

File: doc/en/operators/llamafile.md Copy Chapter

Llamafile Operators Documentation

Llamafile Sgemm Copy Chapter

CPUInfer Copy Chapter

Expert-Parallel MoE Copy Chapter

Microbenchmark Copy Chapter

Linear Projection Copy Section

MoE Copy Section

File: doc/en/AMX.md Copy Chapter

Qwen 3 + KTransformers 0.3 (+AMX) = AI Workstation/PC

Introduction to AMX Instruction Set

AMX Kernel in KTransformers

1. AMX Tiling-aware Memory Layout Copy Chapter

2. Cache-friendly AMX Kernel Copy Chapter

3. AVX-512 Kernel Adaptation for Low Arithmetic Intensity Scenarios Copy Chapter

4. MoE Operator Fusion and Dynamic Scheduling Copy Chapter

Usage

Checking AMX Support Copy Chapter

Enabling AMX in KTransformers Copy Chapter

File: doc/en/balance-serve.md Copy Chapter

Balance Serve backend (multi-concurrency) for ktransformers

KTransformers v0.2.4 Release Notes Copy Chapter

🚀 Key Updates Copy Section

📚 Upgrade Notes Copy Section

What's Changed Copy Section

Download the Docker image for testing v0.2.4 Copy Chapter

Installation Guide Copy Chapter

2. Set Up Conda Environment Copy Section

2. Install dependencies Copy Section

3. Build ktransformers Copy Section

Running DeepSeek-R1-Q4KM Models Copy Chapter

1. Run for 24GB VRAM GPUs Copy Section

2. access server Copy Section

Explore Related Technologies & Topics

File: README.md

🎯 Overview

🔥 Updates

📦 Capabilities

🚀 [Inference](./kt-kernel/README.md) - High-Performance kt-kernel Serving

🎓 [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) - Fine-Tuning with LLaMA-Factory

🔥 Citation

👥 Contributors & Team

💬 Community & Support

📦 KT original Code

File: doc/en/api/server/api.md

OpenAI ChatCompletion

Parameters

Response

Example

Ollama ChatCompletion

Parameters

Response

例子

File: doc/en/api/server/server.md

API

Integrating Model Inference Frameworks

File: doc/en/api/server/tabby.md

File: doc/en/api/server/website.md

1. Starting the Web Service

1.1. Compiling the Web Code

File: doc/en/operators/llamafile.md

Llamafile Sgemm

CPUInfer

Expert-Parallel MoE

Microbenchmark

Linear Projection

MoE

File: doc/en/AMX.md

1. AMX Tiling-aware Memory Layout

2. Cache-friendly AMX Kernel

3. AVX-512 Kernel Adaptation for Low Arithmetic Intensity Scenarios

4. MoE Operator Fusion and Dynamic Scheduling

Checking AMX Support

Enabling AMX in KTransformers

File: doc/en/balance-serve.md

KTransformers v0.2.4 Release Notes

🚀 Key Updates

📚 Upgrade Notes

What's Changed

Download the Docker image for testing v0.2.4

Installation Guide

2. Set Up Conda Environment

2. Install dependencies

3. Build ktransformers

Running DeepSeek-R1-Q4KM Models

1. Run for 24GB VRAM GPUs

2. access server