BitNet Documentation

1. Project Overview & Quickstart (microsoft/BitNet) Copy Chapter

File: README.md Copy Chapter

📰 News

07/23/2026: 📣 We released VibeASR.cpp — a real-time multilingual ASR inference engine on CPU using BitNet I2_S quantization, achieving RTF < 1 with very few threads on x86 (AVX2) and ARM (NEON) platforms. [Code] [Models] [Report]

07/20/2026: 📣 We released BitNet-embedding-0.6B and BitNet-embedding-270M on Hugging Face — the first 1-bit embedding models that deliver competitive embedding quality with significantly faster inference on CPUs.

• 1.42x to 2.28x speedup over F16 on BitNet-embedding-0.6B prefill (8 threads)
• 1.32x to 1.74x speedup over F16 on BitNet-embedding-270M prefill (8 threads)
• Supports I2_S conversion with optimized kernels on x86 CPUs
• Lossless inference with 2 bits per weight

07/16/2026: 📣 Released BitNet Embeddings 0.6B/270M: I2_S Conversion and Inference Optimization — detailed guide for converting and running BitNet embedding models with optimized I2_S kernels.

01/15/2026: 📣 Released BitNet CPU Inference Optimization — parallel kernel implementations with configurable tiling and embedding quantization support, achieving 1.15x to 2.1x additional speedup over the original implementation.

05/20/2025: 📣 Released BitNet Official GPU inference kernel — extending 1-bit inference beyond CPUs.

04/14/2025: 📣 Released BitNet Official 2B Parameter Model on Hugging Face — the first official BitNet b1.58 model trained with 4T tokens.

02/18/2025: 📑 Bitnet.cpp: Efficient Edge Inference for Ternary LLMs — system-level paper on bitnet.cpp's architecture and design.

11/08/2024: 📑 BitNet a4.8: 4-bit Activations for 1-bit LLMs — enabling 4-bit activations for further efficiency gains.

10/21/2024: 📑 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs — the technical report behind bitnet.cpp.

10/17/2024: 📣 bitnet.cpp 1.0 released.

03/21/2024: 📑 The-Era-of-1-bit-LLMs: Training Tips, Code, FAQ

02/27/2024: 📑 The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — the foundational paper introducing BitNet b1.58.

10/17/2023: 📑 BitNet: Scaling 1-bit Transformers for Large Language Models — the original BitNet paper.

Overview Copy Chapter

bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels that support fast and lossless inference of 1.58-bit models on CPU and GPU (NPU support coming next).

Try it out via this online demo, or build and run it on your own CPU or GPU.

bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. Please refer to the technical report for more details.

Model Releases Copy Chapter

1. [BitNet-b1.58-2B-4T](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T) - 1-bit Large Language Model Copy Section

BitNet-b1.58-2B-4T is the first official BitNet b1.58 model with 2.4B parameters, trained on 4 trillion tokens. It is a ternary (1.58-bit) language model that delivers competitive performance with full-precision models of similar size while enabling significantly faster and more energy-efficient inference.

• Fast CPU Inference: Achieves up to 6.17x speedup on x86 CPUs and 5.07x on ARM CPUs compared to full-precision models.
• Energy Efficient: Reduces energy consumption by up to 82.2% on x86 and 70.0% on ARM.
• GPU Support: Official GPU inference kernel available for accelerated deployment.
• Chat-Ready: Supports conversational mode for interactive use.

🤗 Hugging Face | 🔗 Online Demo | 📄 Technical Report

2. [BitNet-embedding-0.6B](https://huggingface.co/microsoft/BitNet-embedding-0.6B) - 1-bit Embedding Model Copy Section

BitNet-embedding-0.6B is a 0.6B-parameter 1-bit embedding model that achieves competitive embedding quality with significantly faster CPU inference. It is the first model to demonstrate that ternary weights can deliver strong performance on embedding tasks.

• 1.42x to 2.28x speedup over F16 on prefill (8 threads, x86)
• Lossless Quality: Competitive embedding quality with 2 bits per weight
• I2_S Kernel: Supports optimized I2_S conversion on x86 CPUs

🤗 Hugging Face | 📄 I2_S Guide

3. [BitNet-embedding-270M](https://huggingface.co/microsoft/BitNet-embedding-270M) - Lightweight 1-bit Embedding Model Copy Section

BitNet-embedding-270M is a compact 270M-parameter 1-bit embedding model designed for resource-constrained environments, offering fast inference with minimal memory footprint.

• 1.32x to 1.74x speedup over F16 on prefill (8 threads, x86)
• Lossless Quality: Competitive embedding quality with 2 bits per weight
• Lightweight: Only 270M parameters for edge deployment scenarios

🤗 Hugging Face | 📄 I2_S Guide

Supported Models Copy Chapter

❗️We use existing 1-bit LLMs available on Hugging Face to demonstrate the inference capabilities of bitnet.cpp. We hope the release of bitnet.cpp will inspire the development of 1-bit LLMs in large-scale settings in terms of model size and training tokens.

Installation Copy Chapter

Requirements Copy Section

• python>=3.10
• cmake>=3.22
• clang>=18

  • For Windows users, install Visual Studio 2022. In the installer, toggle on at least the following options(this also automatically installs the required additional tools like CMake):

    • Desktop-development with C++
    • C++-CMake Tools for Windows
    • Git for Windows
    • C++-Clang Compiler for Windows
    • MS-Build Support for LLVM-Toolset (clang)

  • For Debian/Ubuntu users, you can download with Automatic installation script

    bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"

• conda (highly recommend)

Build from source Copy Section

[!IMPORTANT]
If you are using Windows, please remember to always use a Developer Command Prompt / PowerShell for VS2022 for the following commands. Please refer to the FAQs below if you see any issues.

1. Clone the repo

git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

2. Install the dependencies

# (Recommended) Create a new conda environment
conda create -n bitnet-cpp python=3.10
conda activate bitnet-cpp

pip install -r requirements.txt

3. Build the project

# Manually download the model and run with local path
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

usage: setup_env.py [-h] [--hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}] [--model-dir MODEL_DIR] [--log-dir LOG_DIR] [--quant-type {i2_s,tl1}] [--quant-embd]
[--use-pretuned]

Setup the environment for running inference

optional arguments:
-h, --help show this help message and exit
--hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}, -hr {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}
Model used for inference
--model-dir MODEL_DIR, -md MODEL_DIR
Directory to save/load the model
--log-dir LOG_DIR, -ld LOG_DIR
Directory to save the logging info
--quant-type {i2_s,tl1}, -q {i2_s,tl1}
Quantization type
--quant-embd Quantize the embeddings to f16
--use-pretuned, -p Use the pretuned kernel parameters

Usage Copy Chapter

Basic usage Copy Section

# Run inference with the quantized model
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv

usage: run_inference.py [-h] [-m MODEL] [-n N_PREDICT] -p PROMPT [-t THREADS] [-c CTX_SIZE] [-temp TEMPERATURE] [-cnv]

Run inference

optional arguments:
-h, --help show this help message and exit
-m MODEL, --model MODEL
Path to model file
-n N_PREDICT, --n-predict N_PREDICT
Number of tokens to predict when generating text
-p PROMPT, --prompt PROMPT
Prompt to generate text from
-t THREADS, --threads THREADS
Number of threads to use
-c CTX_SIZE, --ctx-size CTX_SIZE
Size of the prompt context
-temp TEMPERATURE, --temperature TEMPERATURE
Temperature, a hyperparameter that controls the randomness of the generated text
-cnv, --conversation Whether to enable chat mode or not (for instruct models.)
(When this option is turned on, the prompt specified by -p will be used as the system prompt.)

Demo Copy Section

A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:

https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1

Benchmark Copy Section

We provide scripts to run the inference benchmark providing a model.

usage: e2e_benchmark.py -m MODEL [-n N_TOKEN] [-p N_PROMPT] [-t THREADS]  
   
Setup the environment for running the inference  
   
required arguments:  
  -m MODEL, --model MODEL  
                        Path to the model file. 
   
optional arguments:  
  -h, --help  
                        Show this help message and exit. 
  -n N_TOKEN, --n-token N_TOKEN  
                        Number of generated tokens. 
  -p N_PROMPT, --n-prompt N_PROMPT  
                        Prompt to generate text from. 
  -t THREADS, --threads THREADS  
                        Number of threads to use.

Here's a brief explanation of each argument:

• -m, --model: The path to the model file. This is a required argument that must be provided when running the script.
• -n, --n-token: The number of tokens to generate during the inference. It is an optional argument with a default value of 128.
• -p, --n-prompt: The number of prompt tokens to use for generating text. This is an optional argument with a default value of 512.
• -t, --threads: The number of threads to use for running the inference. It is an optional argument with a default value of 2.
• -h, --help: Show the help message and exit. Use this argument to display usage information.

For example:

python utils/e2e_benchmark.py -m /path/to/model -n 200 -p 256 -t 4

This command would run the inference benchmark using the model located at /path/to/model, generating 200 tokens from a 256 token prompt, utilizing 4 threads.

For the model layout that do not supported by any public model, we provide scripts to generate a dummy model with the given model layout, and run the benchmark on your machine:

python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large --outfile models/dummy-bitnet-125m.tl1.gguf --outtype tl1 --model-size 125M

# Run benchmark with the generated model, use -m to specify the model path, -p to specify the prompt processed, -n to specify the number of token to generate
python utils/e2e_benchmark.py -m models/dummy-bitnet-125m.tl1.gguf -p 512 -n 128

Convert from `.safetensors` Checkpoints Copy Section

# Prepare the .safetensors model file
huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 --local-dir ./models/bitnet-b1.58-2B-4T-bf16

# Convert to gguf model
python ./utils/convert-helper-bitnet.py ./models/bitnet-b1.58-2B-4T-bf16

Acknowledgements Copy Chapter

This project is based on the llama.cpp framework. We would like to thank all the authors for their contributions to the open-source community. Also, bitnet.cpp's kernels are built on top of the Lookup Table methodologies pioneered in T-MAC. For inference of general low-bit LLMs beyond ternary models, we recommend using T-MAC.

FAQ (Frequently Asked Questions)📌 Copy Section

Q1: The build dies with errors building llama.cpp due to issues with std::chrono in log.cpp?

A:
This is an issue introduced in recent version of llama.cpp. Please refer to this commit in the discussion to fix this issue.

Q2: How to build with clang in conda environment on windows?

A:
Before building the project, verify your clang installation and access to Visual Studio tools by running:

clang -v

This command checks that you are using the correct version of clang and that the Visual Studio tools are available. If you see an error message such as:

'clang' is not recognized as an internal or external command, operable program or batch file.

It indicates that your command line window is not properly initialized for Visual Studio tools.

• If you are using Command Prompt, run:

"C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\VsDevCmd.bat" -startdir=none -arch=x64 -host_arch=x64

• If you are using Windows PowerShell, run the following commands:

Import-Module "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\Microsoft.VisualStudio.DevShell.dll" Enter-VsDevShell 3f0e31ad -SkipAutomaticLocation -DevCmdArguments "-arch=x64 -host_arch=x64"

These steps will initialize your environment and allow you to use the correct Visual Studio tools.

File: docs/codegen.md Copy Chapter

Codegen for TL1 and TL2 Copy Chapter

codegen_tl1.py and codegen_tl2.py are using params to generate kernel codes in different devices to achieve fastest performance for TL1 and TL2.

We cutting weight into multiple compute blocks to best utilize hardware capabilities.

Example Copy Section

bitnet_b1_58-large:

• Make sure Matmul kernels shapes
  For example, bitnet_b1_58-large Matmul kernel shapes are:
  [1536, 4096]
  [1536, 1536]
  [4096, 1536]

• Make sure each BM, BK, bm for each kernel to meet the requirements below

• Generate codes
  For example, for bitnet_b1_58-large, we can gencode like:

# For TL1
python utils/codegen_tl1.py --model bitnet_b1_58-large --BM 256,128,256 --BK 128,64,128 --bm 32,64,32

# For TL2
python utils/codegen_tl2.py --model bitnet_b1_58-large --BM 256,128,256 --BK 96,192,96 --bm 32,32,32

TL1: Copy Section

For TL1, we cut weight into M / BM weights, each weight shape is (BM, K). Then we cut weight into K / BK weights, each weight shape is (BM, BK). As for (BM, BK) weight, we cut it the same way into (bm, compute_num / bm) compute blocks, and finish computing in it.

Thus, we need to make sure

• M % BM == 0
• K % BK == 0
• BM % bm == 0
• bm choose in [32, 64]

TL2: Copy Section

For TL2, things got a little more complicated. Due to TL2 needs BK % 6 == 0, we need to split K into threeK and twoK, in which compute in TL2 for (M, threeK), compute in TL1 for (M, two_K).

Thus, we needs to make sure

• M % BM == 0
• K % BK % 32 == 0
• BM % bm == 0
• bm choose in [32]

File: gpu/README.md Copy Chapter

BitNet Inference Kernel

This repository provides a highly efficient GEMV kernel implementation for the BitNet model, optimized for W2A8 inference — 2-bit weights and 8-bit activations. It is tailored for use with the BitNet-b1.58-2B-4T model.

Features Copy Chapter

• Support for W2A8 (2-bit weight × 8-bit activation) GEMV computation
• Custom CUDA kernels with low-latency execution
• Optimizations for memory access, decoding, and compute throughput

Usage Copy Chapter

Installation and kernel performance tests:

# (Recommended) Create a new conda environment
conda create --name bitnet-gpu "python<3.13"
conda activate bitnet-gpu

# Install dependencies
pip install -r requirements.txt

# Build the kernel
cd bitnet_kernels
bash compile.sh
cd ..

# Run performance tests
python test.py

End-to-end inference:

# Download and convert the BitNet-b1.58-2B model
mkdir checkpoints
huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 --local-dir ./checkpoints/bitnet-b1.58-2B-4T-bf16
python ./convert_safetensors.py --safetensors_file ./checkpoints/bitnet-b1.58-2B-4T-bf16/model.safetensors --output checkpoints/model_state.pt --model_name 2B
python ./convert_checkpoint.py --input ./checkpoints/model_state.pt
rm ./checkpoints/model_state.pt

# Inference
python3 ./generate.py ./checkpoints/ --interactive --chat_format

Optimizations Copy Chapter

Weight Permutation Copy Section

The weight matrix is divided into 16×32 blocks to optimize memory access patterns.

Within each block, values are stored contiguously in memory and permuted to facilitate efficient access and processing.

See convert_checkpoint.py for details.

Fast Decoding Copy Section

Every 16 two-bit values are packed into a single 32-bit integer using the following interleaving pattern:

[0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15]

This layout is designed to accelerate decoding by enabling efficient extraction of 4 values at a time into int8.

`dp4a` Instruction Copy Section

We use the dp4a instruction to accelerate low-precision dot product operations.

This instruction performs a dot product between two 4-element vectors (each stored in a 32-bit word as 8-bit integers) and accumulates the result into a 32-bit integer.

It significantly improves GEMV throughput when processing quantized weights and activations.

Performance Copy Chapter

Kernel Benchmarks Copy Section

Tested on NVIDIA A100 40GB GPU, our custom W2A8 kernel shows significant speedups over standard BF16 implementations:

End-to-End Generation Latency Copy Section

Compared to a similarly-sized BF16 model (Gemma-2-2B using vLLM), BitNet-b1.58-2B with our kernel achieves consistent speedups across workloads:

Note: Comparison uses equivalent-sized models (2B parameters) on NVIDIA A100 40GB GPU.

File: src/README.md Copy Chapter

BitNet CPU Inference Optimization

This update provides significant performance improvements for BitNet inference on CPU through paralleled kernel implementations, native I2_S GEMM/GEMV support, configurable tiling block size and embedding quantization.

Update Copy Chapter

• Parallel Weight & Activation Computation
  Implemented parallel processing of weights and activations in the W2A8 vet_dot kernel, achieving improved throughput on both x86 and ARM architectures.

• Native I2_S GEMM & GEMV Support
  Integrated I2_S GEMM and GEMV operations into ggml library, making them fully compatible with the llama.cpp architecture. This enables seamless integration with existing inference pipelines.

• Configurable Tiling & Parallelism
  Introduced configurable GEMM & GEMV block sizes and parallelism levels, allowing performance fine-tuning for different CPU architectures.

• Embedding Quantization
  Added support for embedding layer quantization with Q6_K format, reducing memory footprint and improving inference speed while maintaining high accuracy.

Usage Copy Chapter

Configuration Options Copy Section

The include/gemm-config.h file controls kernel behavior:

#define ROW_BLOCK_SIZE 4
#define COL_BLOCK_SIZE 128
#define PARALLEL_SIZE 4

Modify these values based on your CPU cache size and architecture for optimal performance. Users can fine-tune performance on their machine through include/gemm-config.h.

Enabling Embedding Quantization Copy Section

To use embedding quantization for additional speedup:

Using setup_env.py:

python setup_env.py --quant-embd

This automatically converts embeddings to Q6_K format.

Manual conversion:

build/bin/llama-quantize --token-embedding-type Q6_K models/BitNet-b1.58-2B-4T/ggml-model-f32.gguf models/BitNet-b1.58-2B-4T/ggml-model-i2_s-embed-q6_k.gguf I2_S 1 1

Optimizations Copy Chapter

1. Weight & Activation Parallelism Copy Section

The kernel implements two parallelization strategies:

• Weight Parallel: Processes multiple weight rows/columns in a single kernel call, reducing kernel launch overhead.

• Activation Parallel: Built on top of weight parallel, amortizes the I2_S weight unpacking cost across multiple activation elements.

Recommendation: For I2_S quantization format, activation parallel is recommended due to the unpack operation benefits. The current kernel defaults to activation parallel.

Kernel Performance Comparison:

2. GEMM/GEMV Integration with llama.cpp Copy Section

Integrated I2_S quantization format into llama.cpp's compute graph:

• GEMV Operations: Optimized matrix-vector multiplication for token generation.
• GEMM Operations: Efficient matrix-matrix multiplication for prompt processing.
• Tiling Strategy: Configurable block sizes for optimal cache utilization.

3. Configuration Fine-tuning Copy Section

Fine-tuning kernel parameters for optimal performance on specific hardware:

Example Configuration (x86, AMD EPYC 7V13):

• Method: Activation Parallel
• Threads: 8
• Workload: 128 prompt tokens (pp128)

Fine-tuning Parameters:

• Parallelism Degree: [2, 4, 8]
• Row Block Size: [2, 4, 8, 16, 32]
• Column Block Size: [32, 64, 128, 256, 512, 1024]

Fine-tuning Results:

Optimal Configuration: Under this setup (x86, 8 threads, pp128), the best performance is achieved with parallelism degree = 4, row block size = 4, and column block size = 128.

4. Embedding Quantization Copy Section

Evaluated multiple embedding quantization formats to balance memory usage, model quality, and inference speed:

Perplexity Comparison:

Inference Speed Comparison:

Recommendation: Based on comprehensive evaluation of memory footprint, perplexity preservation, and inference speed, Q6_K is selected as the optimal embedding quantization format.

Performance Copy Chapter

Comparison of optimized parallel kernels vs. original implementation:

Test Configuration:

• Model: BitNet-b1.58-2B-4T
• Hardware: AMD EPYC 7V13
• Threads: 1 / 2 / 4 / 8 / 12 / 16
• Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
• Method: Activation Parallel

Test Configuration:

• Model: BitNet-b1.58-2B-4T
• Hardware: Intel i7-13800H
• Threads: 1 / 2 / 4 / 6
• Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
• Method: Activation Parallel

Test Configuration:

• Model: BitNet-b1.58-2B-4T
• Hardware: Cobalt 100
• Threads: 1 / 2 / 4 / 8
• Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
• Method: Activation Parallel

Technical Details Copy Chapter

Key Files Modified Copy Section

• src/ggml-bitnet-mad.cpp: Parallel kernel implementations
• 3rdparty/llama.cpp/ggml/src/ggml.c: GEMM/GEMV integration
• include/gemm-config.h: Configuration file

Supported Architectures Copy Section

• ✅ x86-64 with AVX2
• ✅ ARM with NEON
• ✅ ARM with DOTPROD extension

2. Official Technical Reference & Guides (microsoft/microsoft.github.io) Copy Chapter

[microsoft.github.io](https://microsoft.github.io)

Many open source projects released by Microsoft products, services and teams also have
deployments powered by GitHub Pages on this domain.

To discover Microsoft-released open source projects, learn about Microsoft's open source
approach and other information, check us out online at opensource.microsoft.com.

If we can help answer questions about the open source program at Microsoft, you can
contact opensource at microsoft com - although we are not able to provide support
for projects through this mechanism.

Microsoft GitHub resources Copy Chapter

• Default repo templates including SECURITY.MD and code of conduct files: microsoft/repo-templates
• Code of Conduct (FAQ)

Microsoft open source resources Copy Chapter

• Open Source Blog
• The Open Source Show on Channel 9
• https://opensource.microsoft.com/

Resources for Microsoft employees Copy Chapter

• docs.opensource.microsoft.com
• Link your GitHub Account and join GitHub organizations automatically

How to identify an official Microsoft organization on GitHub Copy Chapter

• Official Microsoft GitHub orgs have a professional avatar and include a web address at Microsoft.com or an associated official domain
• Official Microsoft orgs are Verified on GitHub - look for the domain verification indicator on GitHub