ggml Documentation

Tensor library for machine learning

15,486 stars C++ C++ 12 files · ~17,538 tokens #machine-learning
RAW Doc

Technical Documentation: ggml-org/ggml

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

ggml

Quick start

Build from source:

bash
git clone https://github.com/ggml-org/ggml
cd ggml

mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8

For a minimal, fully commented example (matrix multiplication), see examples/simple.

Description

The main goal of ggml is to be a simple, portable, and efficient tensor library for machine learning with minimal setup.

  • Plain C/C++ implementation without any dependencies
  • Cross-platform - x86, ARM, RISC-V, LoongArch, PowerPC, s390x, and WebAssembly
  • SIMD-optimized kernels for x86, ARM, and RISC-V
  • Broad backend support - CPU, GPU, NPU, and browser
  • 2- to 8-bit integer quantization, plus MXFP4 and NVFP4 microscaling formats
  • Zero memory allocations during runtime

Documentation

Contributing

  • For changes to the core ggml library (including to the CMake build system), please open a PR in llama.cpp - doing so will make your PR more visible, better tested, and more likely to be reviewed

Overview (`examples/gpt-2/README.md`)

gpt-2

This is a C++ example running GPT-2 inference using the ggml library.

The program runs on the CPU - no video card is required.

The Cerebras-GPT models are also supported.

The example supports the following GPT-2 models:

Model Description Disk Size
117M Small model 240 MB
345M Medium model 680 MB
774M Large model 1.5 GB
1558M XL model 3.0 GB

Sample performance on MacBook M1 Pro:

Model Size Time / Token
GPT-2 117M 5 ms
GPT-2 345M 12 ms
GPT-2 774M 23 ms
GPT-2 1558M 42 ms

TODO: add tables for Cerebras-GPT models

Sample output:

text
/* Detailed source-code truncated for AI context efficiency. */

Downloading and converting the original models (GPT-2)

You can download the original model files using the download-model.sh Bash script. The models are
in Tensorflow format, so in order to use them with ggml, you need to convert them to appropriate format. This is done
via the convert-ckpt-to-ggml.py python script.

Here is the entire process for the GPT-2 117M model (download from official site + conversion):

bash
cd ggml/build
../examples/gpt-2/download-model.sh 117M

Downloading model 117M ...
models/gpt-2-117M/checkpoint                      100%[=============================>]      77  --.-KB/s    in 0s
models/gpt-2-117M/encoder.json                    100%[=============================>]   1018K  1.20MB/s    in 0.8s
models/gpt-2-117M/hparams.json                    100%[=============================>]      90  --.-KB/s    in 0s
models/gpt-2-117M/model.ckpt.data-00000-of-00001  100%[=============================>] 474.70M  1.21MB/s    in 8m 39s
models/gpt-2-117M/model.ckpt.index                100%[=============================>]   5.09K  --.-KB/s    in 0s
models/gpt-2-117M/model.ckpt.meta                 100%[=============================>] 460.11K   806KB/s    in 0.6s
models/gpt-2-117M/vocab.bpe                       100%[=============================>] 445.62K   799KB/s    in 0.6s
Done! Model '117M' saved in 'models/gpt-2-117M/'

Run the convert-ckpt-to-ggml.py script to convert the model to ggml format.

  python /Users/john/ggml/examples/gpt-2/convert-ckpt-to-ggml.py models/gpt-2-117M/ 1

This conversion requires that you have python and Tensorflow installed on your computer. Still, if you want to avoid
this, you can download the already converted ggml models as described below.

Downloading and converting the original models (Cerebras-GPT)

Clone the respective repository from here: https://huggingface.co/cerebras

Use the convert-cerebras-to-ggml.py script to convert the model to ggml format:

bash
cd ggml/build
git clone https://huggingface.co/cerebras/Cerebras-GPT-111M models/
python ../examples/gpt-2/convert-cerebras-to-ggml.py models/Cerebras-GPT-111M/

Downloading the ggml model directly (GPT-2)

For convenience, I will be hosting the converted ggml model files in order to make it easier to run the examples. This
way, you can directly download a single binary file and start using it. No python or Tensorflow is required.

Here is how to get the 117M ggml model:

bash
cd ggml/build
../examples/gpt-2/download-ggml-model.sh 117M

Downloading ggml model 117M ...
models/gpt-2-117M/ggml-model.bin         100%[===============================>] 239.58M  8.52MB/s    in 28s
Done! Model '117M' saved in 'models/gpt-2-117M/ggml-model.bin'
You can now use it like this:

  $ ./bin/gpt-2 -m models/gpt-2-117M/ggml-model.bin -p "This is an example"

At some point, I might decide to stop hosting these models. So in that case, simply revert to the manual process above.

Quantizing the models

You can also try to quantize the ggml models via 4-bit integer quantization.
Keep in mind that for smaller models, this will render them completely useless.
You generally want to quantize larger models.

bash
# quantize GPT-2 F16 to Q4_0 (faster but less precise)
./bin/gpt-2-quantize models/gpt-2-1558M/ggml-model-f16.bin models/gpt-2-1558M/ggml-model-q4_0.bin 2
./bin/gpt-2 -m models/gpt-2-1558M/ggml-model-q4_0.bin -p "This is an example"

# quantize Cerebras F16 to Q4_1 (slower but more precise)
./bin/gpt-2-quantize models/Cerebras-GPT-6.7B/ggml-model-f16.bin models/Cerebras-GPT-6.7B/ggml-model-q4_1.bin 3
./bin/gpt-2 -m models/Cerebras-GPT-6.7B/ggml-model-q4_1.bin -p "This is an example"

Batched generation example

You can try the batched generation from a given prompt using the gpt-2-batched binary.

Sample output:

text
/* Detailed source-code truncated for AI context efficiency. */

Overview (`examples/gpt-j/README.md`)

gpt-j

Local GPT-J inference on your computer using C/C++

No video card required. You just need to have 16 GB of RAM.

Motivation

The GPT-J 6B model is the open-source alternative to OpenAI's GPT-3. It's basically a neural network that allows you to
generate coherent, human-like text given a certain context (prompt).

The GPT-J model is quite big - the compact version of the model uses 16-bit floating point representation of the weights
and is still 12 GB big. This means that in order to run inference on your computer, you would need to have a video card
with at least 12 GB of video RAM. Alternatively, you can try to run the python implementations on the CPU, but that
would probably not be very efficient as they are primarily optimized for running on a GPU (or at least this is my guess -
I don't have much experience with python).

I wanted to try and run the model on my MacBook, so I decided to implement the model inference from scratch using my own
custom build tensor library. The tensor library (called ggml, written in C) is in
early development stage, but it already allows me to run the GPT-J model.

On my 32GB MacBook M1 Pro, I achieve an inference speed of about 125 ms/token or about ~6 words per second (1 word
typically consists of 1 or 2 tokens).

Here is a sample run with prompt int main(int argc, char ** argv) {:

bash
$ time ./bin/gpt-j -p "int main(int argc, char ** argv) {"

gptj_model_load: loading model from 'models/gpt-j-6B/ggml-model.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx   = 2048
gptj_model_load: n_embd  = 4096
gptj_model_load: n_head  = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot   = 64
gptj_model_load: f16     = 1
gptj_model_load: ggml ctx size = 13334.86 MB
gptj_model_load: memory_size =  1792.00 MB, n_mem = 57344
gptj_model_load: ................................... done
gptj_model_load: model size = 11542.79 MB / num tensors = 285
main: number of tokens in prompt = 13

int main(int argc, char ** argv) {
    (void)argc;
    (void)argv;

    {
        struct sockaddr_in addr;
        int addrlen;
        char * ip = "192.168.1.4";
        int i;

        if ( (addrlen = sizeof(addr)) == -1 )
            return -1;

        for (i = 0; i < 10; ++i) {
            addr.sin_family = AF_INET;
            addr.sin_addr.s_addr = inet_addr(ip);

main: mem per token = 16430420 bytes
main:     load time =  6211.48 ms
main:   sample time =    13.74 ms
main:  predict time = 26420.34 ms / 124.62 ms per token
main:    total time = 33035.37 ms

real	0m33.171s
user	3m32.269s
sys      0m3.686s

$

It took ~6.2 seconds to load the model to memory. After that, it took ~26.4 seconds to generate 200 tokens of what
looks like to be the beginning of a networking program in C. Pretty cool!

Here is another run, just for fun:

text
/* Detailed source-code truncated for AI context efficiency. */

Implementation details

The high level implementation of the model is contained in the main.cpp file. The core computations are
performed by the ggml library.

Matrix multiplication

The most performance critical part of the implementation is of course the matrix multiplication routine. 99% of the time
is spent here, so it was important to optimize this as much as possible.

On Arm64, I utilize the 128-bit NEON intrinsics for 16-bit floating point operations:

https://github.com/ggerganov/ggml/blob/fb558f78d905f85c54813602649ddd628ffe0f3a/src/ggml.c#L187-L243

These instructions allow each core to operate simultaneously on 64 16-bit floats. I'm no expert in SIMD, but after quite
some trials this was the most efficient code for dot product of a row and column that I could come up with. Combined
with the parallel computation on 8 CPU threads, I believe I'm close to the maximum performance that one could possibly
get on the M1 CPU. Still, I'm curious to know if there is a more efficient way to implement this.

Attempt to use the M1 GPU

One interesting property of the GPT-J transformer architecture is that it allows you to perform part of the inference in
parallel - i.e. the Feed-forward network can be computed in parallel to the Self-attention layer:

https://github.com/ggerganov/ggml/blob/fb558f78d905f85c54813602649ddd628ffe0f3a/examples/gpt-j/main.cpp#L507-L531

So I thought why not try and bring in the M1 GPU to compute half of the neural network in parallel to the CPU and
potentially gain some extra performance. Thanks to the M1's shared memory model, it was relatively easy to offload part
of the computation to the GPU using Apple's Metal Performance
Shaders. The GPU shares the host memory, so there is
no need to copy the data back and forth as you would normally do with Cuda or OpenCL. The weight matrices are directly
available to be used by the GPU.

However, to my surprise, using MPS together with the CPU did not lead to any performance improvement at all. My
conclusion was that the 8-thread NEON CPU computation is already saturating the memory bandwidth of the M1 and since
the CPU and the GPU on the MacBook are sharing that bandwidth, it does not help to offload the computation to the GPU.
Another observation was that the MPS GPU matrix multiplication using 16-bit floats had the same performance as the
8-thread NEON CPU implementation. Again, I explain this with a saturated memory channel. But of course, my explanation
could be totally wrong and somehow the implementation wasn't utilizing the resources correctly.

In the end, I decided to not use MPS or the GPU all together.

Zero memory allocations

Another property of my implementation is that it does not perform any memory allocations once the model is loaded into
memory. All required memory is allocated at the start of the program with a single malloc (technically 2 calls, but
that is not important).

Usage

If you want to give this a try and you are on Linux or Mac OS, simply follow these instructions:

bash
# Download the ggml-compatible GPT-J 6B model (requires 12GB disk space)
../examples/gpt-j/download-ggml-model.sh 6B

# Run the inference (requires 16GB of CPU RAM)
./bin/gpt-j -m models/gpt-j-6B/ggml-model.bin -p "This is an example"

# Input prompt through pipe and run the inference.
echo "This is an example" > prompt.txt
cat prompt.txt | ./bin/gpt-j -m models/gpt-j-6B/ggml-model.bin

To run the gpt-j tool, you need the 12GB ggml-model.bin file which contains the GPT-J model in
ggml compatible format. In the instructions above, the binary file
is downloaded from my repository on Hugging Face using the download-ggml-model.sh script.
You can also, download the file manually from this link:

https://huggingface.co/ggerganov/ggml/tree/main

Alternatively, if you don't want to download the 12GB ggml model file, you can perform the conversion yourself using
python.

First, you need to download the full GPT-J model from here: https://huggingface.co/EleutherAI/gpt-j-6B

Note that the full model is quite big - about 72 GB. After you download it, you need to convert it to ggml format using
the convert-h5-to-ggml.py script. This will generate the ggml-model.bin file, which you can
then use with the gpt-j program.

GPT-2

I also implemented a tool for CPU inference using the smaller GPT-2 models. They have worse quality compared to GPT-J,
but are much faster to execute.

For example, the Small GPT-2 model is only 240 MB big and the inference speed on my MacBook is about 200 tokens/sec.

For more details, checkout the GPT-2 example here: gpt-2

Overview (`examples/magika/README.md`)

Google Magika inference

Simple example that shows how to use GGML for inference with the Google Magika file type detection model.

Usage

bash
$ python examples/magika/convert.py /path/to/model.h5
  • Invoke the program with the model file and a list of files to identify:
bash
$ build/bin/magika model.h5.gguf examples/sam/example.jpg examples/magika/convert.py README.md src/ggml.c /bin/gcc write.exe jfk.wav
  examples/sam/example.jpg      : jpeg (100.00%) pptx (0.00%) smali (0.00%) shell (0.00%) sevenzip (0.00%)
  examples/magika/convert.py    : python (99.99%) javascript (0.00%) txt (0.00%) asm (0.00%) scala (0.00%)
  README.md                     : markdown (100.00%) txt (0.00%) yaml (0.00%) ppt (0.00%) shell (0.00%)
  src/ggml.c                    : c (99.95%) txt (0.04%) asm (0.01%) yaml (0.00%) html (0.00%)
  /bin/gcc                      : elf (99.98%) odex (0.02%) pptx (0.00%) smali (0.00%) shell (0.00%)
  write.exe                     : pebin (100.00%) ppt (0.00%) smali (0.00%) shell (0.00%) sevenzip (0.00%)
  jfk.wav                       : wav (100.00%) ppt (0.00%) shell (0.00%) sevenzip (0.00%) scala (0.00%)

Overview (`examples/mnist/README.md`)

MNIST Examples for GGML

This directory contains simple examples of how to use GGML for training and inference using the MNIST dataset.
All commands listed in this README assume the working directory to be examples/mnist.
Please note that training in GGML is a work-in-progress and not production ready.

Obtaining the data

A description of the dataset can be found on Yann LeCun's website.
While it is also in principle possible to download the dataset from this website these downloads are frequently throttled and
it is recommended to use HuggingFace instead.
The dataset will be downloaded automatically when running mnist-train-fc.py.

Fully connected network

For our first example we will train a fully connected network.
To train a fully connected model in PyTorch and save it as a GGUF file, run:

bash
$ python3 mnist-train-fc.py mnist-fc-f32.gguf

...

Test loss: 0.066377+-0.010468, Test accuracy: 97.94+-0.14%

Model tensors saved to mnist-fc-f32.gguf:
fc1.weight       (500, 784)
fc1.bias         (500,)
fc2.weight       (10, 500)
fc2.bias         (10,)

The training script includes an evaluation of the model on the test set.
To evaluate the model on the CPU using GGML, run:

text
/* Detailed source-code truncated for AI context efficiency. */

In addition to the evaluation on the test set the GGML evaluation also prints a random image from the test set as well as the model prediction for said image.
To train a fully connected model on the CPU using GGML run:

bash
$ ../../build/bin/mnist-train mnist-fc mnist-fc-f32.gguf data/MNIST/raw/train-images-idx3-ubyte data/MNIST/raw/train-labels-idx1-ubyte

It can then be evaluated with the same binary as above.

Convolutional network

To train a convolutional network using TensorFlow run:

bash
$ python3 mnist-train-cnn.py mnist-cnn-f32.gguf

...

Test loss: 0.047947
Test accuracy: 98.46%
GGUF model saved to 'mnist-cnn-f32.gguf'

The saved model can be evaluated on the CPU using the mnist-eval binary:

text
/* Detailed source-code truncated for AI context efficiency. */

Like with the fully connected network the convolutional network can also be trained using GGML:

bash
$ ../../build/bin/mnist-train mnist-cnn mnist-cnn-f32.gguf data/MNIST/raw/train-images-idx3-ubyte data/MNIST/raw/train-labels-idx1-ubyte

As always, the evaluation is done using mnist-eval and like with the fully connected network the GGML graph is exported to mnist-cnn-f32.ggml.

Hardware Acceleration

Both the training and evaluation code is agnostic in terms of hardware as long as the corresponding GGML backend has implemented the necessary operations.
A specific backend can be selected by appending the above commands with a backend name.
The compute graphs then schedule the operations to preferentially use the specified backend.
Note that if a backend does not implement some of the necessary operations a CPU fallback is used instead which may result in bad performance.

Web demo

The evaluation code can be compiled to WebAssembly using Emscripten (may need to re-login to update $PATH after installation).
First, copy the GGUF file of either of the trained models to examples/mnist and name it mnist-f32.gguf.
Copy the test set to examples/mnist and name it t10k-images-idx3-ubyte.
Symlinking these files will not work!
Compile the code like so:

bash
$ cd ../../
$ mkdir -p build-em
$ emcmake cmake .. -DGGML_BUILD_EXAMPLES=ON \
    -DCMAKE_C_FLAGS="-pthread -matomics -mbulk-memory" \
    -DCMAKE_CXX_FLAGS="-pthread -matomics -mbulk-memory"
$ make mnist

The compilation output is copied into examples/mnist/web.
To run it, you need an HTTP server.
For example:

bash
$ python3 examples/mnist/server.py

Serving directory '/home/danbev/work/ai/ggml/examples/mnist/web' at http://localhost:8000
Application context root: http://localhost:8000/

The web demo can then be accessed via the link printed on the console.
Simply draw a digit on the canvas and the model will try to predict what it's supposed to be.
Alternatively, click the "Random" button to retrieve a random digit from the test set.
Be aware that like all neural networks the one we trained is susceptible to distributional shift:
if the numbers you draw look different than the ones in the training set
(e.g. because they're not centered) the model will perform comparatively worse.
An online demo can be accessed here.

Overview (`examples/python/README.md`)

Simple autogenerated Python bindings for ggml

This folder contains:

  • Scripts to generate full Python bindings from ggml headers (+ stubs for autocompletion in IDEs)
  • Some barebones utils (see ggml/utils.py):
    • ggml.utils.init builds a context that's freed automatically when the pointer gets GC'd
    • ggml.utils.copy copies between same-shaped tensors (numpy or ggml), w/ automatic (de/re)quantization
    • ggml.utils.numpy returns a numpy view over a ggml tensor; if it's quantized, it returns a copy (requires allow_copy=True)
  • Very basic examples (anyone wants to port llama2.c?)

Provided you set GGML_LIBRARY=.../path/to/libggml_shared.so (see instructions below), it's trivial to do some operations on quantized tensors:

python
# Make sure libllama.so is in your [DY]LD_LIBRARY_PATH, or set GGML_LIBRARY=.../libggml_shared.so

from ggml import lib, ffi
from ggml.utils import init, copy, numpy
import numpy as np

ctx = init(mem_size=12*1024*1024)
n = 256
n_threads = 4

a = lib.ggml_new_tensor_1d(ctx, lib.GGML_TYPE_Q5_K, n)
b = lib.ggml_new_tensor_1d(ctx, lib.GGML_TYPE_F32, n) # Can't both be quantized
sum = lib.ggml_add(ctx, a, b) # all zeroes for now. Will be quantized too!

gf = ffi.new('struct ggml_cgraph*')
lib.ggml_build_forward_expand(gf, sum)

copy(np.array([i for i in range(n)], np.float32), a)
copy(np.array([i*100 for i in range(n)], np.float32), b)

lib.ggml_graph_compute_with_ctx(ctx, gf, n_threads)

print(numpy(a, allow_copy=True))
#  0.    1.0439453   2.0878906   3.131836    4.1757812   5.2197266. ...
print(numpy(b))
#  0.  100.        200.        300.        400.        500.         ...
print(numpy(sum, allow_copy=True))
#  0.  105.4375    210.875     316.3125    421.75      527.1875     ...

Prerequisites

You'll need a shared library of ggml to use the bindings.

Build libggml_shared.so or libllama.so

As of this writing the best is to use ggerganov/llama.cpp's generated libggml_shared.so or libllama.so, which you can build as follows:

bash
git clone https://github.com/ggerganov/llama.cpp
# On a CUDA-enabled system add -DLLAMA_CUDA=1
# On a Mac add -DLLAMA_METAL=1
cmake llama.cpp \
  -B llama_build \
  -DCMAKE_C_FLAGS=-Ofast \
  -DLLAMA_NATIVE=1 \
  -DLLAMA_LTO=1 \
  -DBUILD_SHARED_LIBS=1 \
  -DLLAMA_MPI=1 \
  -DLLAMA_BUILD_TESTS=0 \
  -DLLAMA_BUILD_EXAMPLES=0
( cd llama_build && make -j )

# On Mac, this will be libggml_shared.dylib instead
export GGML_LIBRARY=$PWD/llama_build/libggml_shared.so
# Alternatively, you can just copy it to your system's lib dir, e.g /usr/local/lib

(Optional) Regenerate the bindings and stubs

If you added or changed any signatures of the C API, you'll want to regenerate the bindings (ggml/cffi.py) and stubs (ggml/init.pyi).

Luckily it's a one-liner using regenerate.py:

bash
pip install -q cffi

python regenerate.py

By default it assumes llama.cpp was cloned in ../../../llama.cpp (alongside the ggml folder). You can override this with:

bash
C_INCLUDE_DIR=$LLAMA_CPP_DIR python regenerate.py

You can also edit api.h to control which files should be included in the generated bindings (defaults to llama.cpp/ggml*.h)

In fact, if you wanted to only generate bindings for the current version of the ggml repo itself (instead of llama.cpp; you'd loose support for k-quants), you could run:

bash
API=../../include/ggml.h python regenerate.py

Develop

Run tests:

bash
pytest

Alternatives

This example's goal is to showcase cffi-generated bindings that are trivial to use and update, but there are already alternatives in the wild:

  • https://github.com/abetlen/ggml-python: these bindings seem to be hand-written and use ctypes. It has high-quality API reference docs that can be used with these bindings too, but it doesn't expose Metal, CUDA, MPI or OpenCL calls, doesn't support transparent (de/re)quantization like this example does (see ggml.utils module), and won't pick up your local changes.

  • https://github.com/abetlen/llama-cpp-python: these expose the C++ llama.cpp interface, which this example cannot easily be extended to support (cffi only generates bindings of C libraries)

  • pybind11 and nanobind are two alternatives to cffi that support binding C++ libraries, but it doesn't seem either of them have an automatic generator (writing bindings is rather time-consuming).

Overview (`examples/sam/README.md`)

SAM.cpp

Inference of Meta's Segment Anything Model in pure C/C++

Description

The example currently supports only the ViT-B SAM model checkpoint.

Next steps

  • Reduce memory usage by utilizing the new ggml-alloc
  • Remove redundant graph nodes
  • Make inference faster
  • Fix the difference in output masks compared to the PyTorch implementation
  • Filter masks based on stability score
  • Add support for user input
  • Support F16 for heavy F32 ops
  • Test quantization
  • Support bigger model checkpoints
  • GPU support

Quick start

Setup Python and build examples according to main README.

bash
# Download PTH model
wget -P examples/sam/ https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth

# Convert PTH model to ggml
python examples/sam/convert-pth-to-ggml.py examples/sam/sam_vit_b_01ec64.pth examples/sam/ 1

# run inference
./bin/sam -t 16 -i ../examples/sam/example.jpg -m ../examples/sam/ggml-model-f16.bin

Downloading and converting the model checkpoints

You can download a model checkpoint and convert it to ggml format using the script convert-pth-to-ggml.py:

Example output on M2 Ultra

text
$ ▶ make -j sam && time ./bin/sam -t 8 -i img.jpg
[ 28%] Built target common
[ 71%] Built target ggml
[100%] Built target sam
main: seed = 1693224265
main: loaded image 'img.jpg' (680 x 453)
sam_image_preprocess: scale = 0.664062
main: preprocessed image (1024 x 1024)
sam_model_load: loading model from 'models/sam-vit-b/ggml-model-f16.bin' - please wait ...
sam_model_load: n_enc_state      = 768
sam_model_load: n_enc_layer      = 12
sam_model_load: n_enc_head       = 12
sam_model_load: n_enc_out_chans  = 256
sam_model_load: n_pt_embd        = 4
sam_model_load: ftype            = 1
sam_model_load: qntvr            = 0
operator(): ggml ctx size = 202.32 MB
sam_model_load: ...................................... done
sam_model_load: model size =   185.05 MB / num tensors = 304
embd_img
dims: 64 64 256 1 f32
First & Last 10 elements:
-0.05117 -0.06408 -0.07154 -0.06991 -0.07212 -0.07690 -0.07508 -0.07281 -0.07383 -0.06779
0.01589 0.01775 0.02250 0.01675 0.01766 0.01661 0.01811 0.02051 0.02103 0.03382
sum:  12736.272313

Skipping mask 0 with iou 0.705935 below threshold 0.880000
Skipping mask 1 with iou 0.762136 below threshold 0.880000
Mask 2: iou = 0.947081, stability_score = 0.955437, bbox (371, 436), (144, 168)


main:     load time =    51.28 ms
main:    total time =  2047.49 ms

real	0m2.068s
user	0m16.343s
sys	0m0.214s

Input point is (414.375, 162.796875) (currently hardcoded)

Input image:

Output mask (mask_out_2.png in build folder):

References

Overview (`examples/simple/README.md`)

Simple

This example simply performs a matrix multiplication, solely for the purpose of demonstrating a basic usage of ggml and backend handling. The code is commented to help understand what each part does.

Traditional matrix multiplication goes like this (multiply row-by-column):

$$
A \times B = C
$$

$$
\begin{bmatrix}
2 & 8 \
5 & 1 \
4 & 2 \
8 & 6 \
\end{bmatrix}
\times
\begin{bmatrix}
10 & 9 & 5 \
5 & 9 & 4 \
\end{bmatrix}
=
\begin{bmatrix}
60 & 90 & 42 \
55 & 54 & 29 \
50 & 54 & 28 \
110 & 126 & 64 \
\end{bmatrix}
$$

In ggml, we pass the matrix $B$ in transposed form and multiply row-by-row. The result $C$ is also transposed:

$$
ggml\_mul\_mat(A, B^T) = C^T
$$

$$
ggml\_mul\_mat(
\begin{bmatrix}
2 & 8 \
5 & 1 \
4 & 2 \
8 & 6 \
\end{bmatrix}
,
\begin{bmatrix}
10 & 5 \
9 & 9 \
5 & 4 \
\end{bmatrix}
)
=
\begin{bmatrix}
60 & 55 & 50 & 110 \
90 & 54 & 54 & 126 \
42 & 29 & 28 & 64 \
\end{bmatrix}
$$

The simple-ctx doesn't support gpu acceleration. simple-backend demonstrates how to use other backends like CUDA and Metal.

Overview (`examples/test-cmake/README.md`)

cmake-test

This directory can be built as a separate project to test/verify a ggml
installation.

Usage

The following will configure, build, and install ggml using two different
configurations. One will use dynamically linked backends (GGML_BACKEND_DL=ON)
and one without.

Configuring, build, and install ggml:

console
./build-install.sh

Build this project twice using the installations created above:

console
./build.sh

wip

Overview (`examples/yolo/README.md`)

This example shows how to implement YOLO object detection with ggml using pretrained model.

YOLOv3-tiny

Download the model weights:

bash
$ wget https://pjreddie.com/media/files/yolov3-tiny.weights
$ sha1sum yolov3-tiny.weights 
40f3c11883bef62fd850213bc14266632ed4414f  yolov3-tiny.weights

Convert the weights to GGUF format:

bash
$ ./convert-yolov3-tiny.py yolov3-tiny.weights
yolov3-tiny.weights converted to yolov3-tiny.gguf

Alternatively, you can download the converted model from HuggingFace

Object detection:

bash
$ wget https://raw.githubusercontent.com/pjreddie/darknet/master/data/dog.jpg
$ ./yolov3-tiny -m yolov3-tiny.gguf -i dog.jpg
load_model: using CUDA backend
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA T1200 Laptop GPU, compute capability 7.5, VMM: yes
Layer  0 output shape:  416 x 416 x   16 x   1
Layer  1 output shape:  208 x 208 x   16 x   1
Layer  2 output shape:  208 x 208 x   32 x   1
Layer  3 output shape:  104 x 104 x   32 x   1
Layer  4 output shape:  104 x 104 x   64 x   1
Layer  5 output shape:   52 x  52 x   64 x   1
Layer  6 output shape:   52 x  52 x  128 x   1
Layer  7 output shape:   26 x  26 x  128 x   1
Layer  8 output shape:   26 x  26 x  256 x   1
Layer  9 output shape:   13 x  13 x  256 x   1
Layer 10 output shape:   13 x  13 x  512 x   1
Layer 11 output shape:   13 x  13 x  512 x   1
Layer 12 output shape:   13 x  13 x 1024 x   1
Layer 13 output shape:   13 x  13 x  256 x   1
Layer 14 output shape:   13 x  13 x  512 x   1
Layer 15 output shape:   13 x  13 x  255 x   1
Layer 18 output shape:   13 x  13 x  128 x   1
Layer 19 output shape:   26 x  26 x  128 x   1
Layer 20 output shape:   26 x  26 x  384 x   1
Layer 21 output shape:   26 x  26 x  256 x   1
Layer 22 output shape:   26 x  26 x  255 x   1
dog: 57%
car: 52%
truck: 56%
car: 62%
bicycle: 59%
Detected objects saved in 'predictions.jpg' (time: 0.057000 sec.)

Overview (`src/ggml-openvino/README.md`)

Compiled model cache

GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR exports compiled CPU/GPU graphs with their weights. It bypasses the plugin-level GGML_OPENVINO_CACHE_DIR and uses OPTIMIZE_SPEED, so weightless caching is disabled.

One directory can hold blobs for different models and compilation settings. Run each intended workload once to export its dynamic graph:

sh
GGML_OPENVINO_DEVICE=GPU \
GGML_OPENVINO_NATIVE_SOFTPLUS=1 \
GGML_OPENVINO_DISABLE_KV_SLICE=1 \
GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all \
GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=/path/to/qwen-cache \
./build/ReleaseOV/bin/llama-bench -m /path/to/model.gguf -r 1

GGML_OPENVINO_SPILL_DIR remains optional for this first run. Wait for the model cache WROTE message and completion of the workload before stopping it. Compatible prefill and decode graphs share one blob and manifest. A graph with different ports or incompatible shapes gets an exact entry instead; interrupted exports are not cache hits.

On later runs, supply the same compilation settings and enable GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY=1:

sh
GGML_OPENVINO_DEVICE=GPU \
GGML_OPENVINO_NATIVE_SOFTPLUS=1 \
GGML_OPENVINO_DISABLE_KV_SLICE=1 \
GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all \
GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=/path/to/qwen-cache \
GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY=1 \
./build/ReleaseOV/bin/llama-bench -m /path/to/model.gguf -r 1

Cache-only mode allocates backend address space without filling weight pages. On Windows, this also uses system commit capacity. The model-buffer size in the loader log is this virtual size. Weight uploads only record source identity; they do not read or requantize the weights. Graph conversion and compilation are skipped. Runtime buffers are still allocated and populated normally.

Cache-only mode uses the settings provided by the current process. Keep these values exactly the same, including set versus unset: GGML_OPENVINO_REQUANT_KQUANT, GGML_OPENVINO_NATIVE_SOFTPLUS, GGML_OPENVINO_DISABLE_KV_SLICE, GGML_OPENVINO_MANUAL_GQA_ATTN, GGML_OPENVINO_STATEFUL_EXECUTION, GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT, GGML_OPENVINO_DISABLE_REMOTE_OUTPUTS, GGML_OPENVINO_REDUCE_COMPILE_MEM, GGML_OPENVINO_MEMORY_OPTIMIZE, GGML_OPENVINO_PROFILING, and GGML_OPENVINO_DEBUG_NODE. On GPU, also repeat GGML_OPENVINO_MOE_OP=0 if used. GGML_OPENVINO_SPILL_DIR is optional on the first run and ignored in cache-only mode; host-weight release is disabled in cache-only mode.

A missing or incompatible graph fails with an error instead of compiling with absent weights. The fingerprint uses the dynamic graph's topology, ports, model parameters, weights, settings, and OpenVINO version; changing only dynamic token or KV sizes does not require a new entry. A different workload can still require another graph; populate it first without cache-only mode.

Restrictions

  • Cache-only mode requires Linux or Windows and mmap loading (--load-mode mmap, or the default when all selected devices support mmap). Do not use tensor validation or mlock when trying to avoid weight reads.
  • On Windows, the backend commits virtual memory for its buffers without touching weight pages. Large models can still reach the system commit limit.
  • The model must execute entirely on OpenVINO, with dynamic CPU/GPU graphs and in-process caching enabled. Static/NPU execution and CPU fallback are unsupported in cache-only mode.
  • GGUF metadata, tokenizer data, tensor descriptors, and graph construction are still needed. The llama.cpp loader is unchanged: depending on its prefetch settings, it may request pages with MAP_POPULATE or read-ahead on Linux, or PrefetchVirtualMemory on Windows. Non-mmap loading also reads the payload before the backend sees it.
  • File identity, size, modification/change timestamps, tensor offsets, graph structure, settings, and OpenVINO version identify cache entries. Linux uses device/inode and Windows uses volume serial/file index. Replacing, copying, or modifying a GGUF invalidates its entries. This avoids reading weight bytes and ties the cache to the local source files. Keep those files unchanged throughout loading and inference.
  • Use the same target device and compatible OpenVINO/plugin installation. Import support depends on the plugin; the tested CPU plugin cannot import MoE graphs containing GatherMatmulCompressed. GPU MoE and CPU dense graph imports were tested.
  • Blobs contain weights and can approach model size for each compiled graph. Import still reads those blobs and initializes the device.

Gguf (`docs/gguf.md`)

GGUF

GGUF is a file format for storing models for inference with GGML and executors based on GGML. GGUF is a binary format that is designed for fast loading and saving of models, and for ease of reading. Models are traditionally developed using PyTorch or another framework, and then converted to GGUF for use in GGML.

It is a successor file format to GGML, GGMF and GGJT, and is designed to be unambiguous by containing all the information needed to load a model. It is also designed to be extensible, so that new information can be added to models without breaking compatibility.

For more information about the motivation behind GGUF, see Historical State of Affairs.

Specification

GGUF is a format based on the existing GGJT, but makes a few changes to the format to make it more extensible and easier to use. The following features are desired:

  • Single-file deployment: they can be easily distributed and loaded, and do not require any external files for additional information.
  • Extensible: new features can be added to GGML-based executors/new information can be added to GGUF models without breaking compatibility with existing models.
  • mmap compatibility: models can be loaded using mmap for fast loading and saving.
  • Easy to use: models can be easily loaded and saved using a small amount of code, with no need for external libraries, regardless of the language used.
  • Full information: all information needed to load a model is contained in the model file, and no additional information needs to be provided by the user.

The key difference between GGJT and GGUF is the use of a key-value structure for the hyperparameters (now referred to as metadata), rather than a list of untyped values. This allows for new metadata to be added without breaking compatibility with existing models, and to annotate the model with additional information that may be useful for inference or for identifying the model.

GGUF Naming Convention

GGUF follow a naming convention of [<Sidecar>]<SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf where each component is delimitated by a - if present. Ultimately this is intended to make it easier for humans to at a glance get the most important details of a model. It is not intended to be perfectly parsable in the field due to the diversity of existing gguf filenames.

The components are:

  1. Sidecar: (Optional) Prefix marking the file as an auxiliary module loaded alongside a base model, rather than a standalone model. When present, sits at the very front of the filename followed by -. Lowercase by convention.
    • mmproj : Multimodal projector (vision/audio encoder and projection layer for use with a base LLM)
    • mtp : Multi-Token Prediction heads (speculative-decoding draft module, intended to be loaded alongside a base model of matching architecture and version). Note that oftentimes the MTP weights can be distributed inside the base model, in which case there is no separate mtp- sidecar file.
  2. BaseName: A descriptive name for the model base type or architecture.
    • This can be derived from gguf metadata general.basename substituting spaces for dashes.
  3. SizeLabel: Parameter weight class (useful for leader boards) represented as <expertCount>x<count><scale-prefix>
    • This can be derived from gguf metadata general.size_label if available or calculated if missing.
    • Rounded decimal point is supported in count with a single letter scale prefix to assist in floating point exponent shown below
      • Q: Quadrillion parameters.
      • T: Trillion parameters.
      • B: Billion parameters.
      • M: Million parameters.
      • K: Thousand parameters.
    • Additional -<attributes><count><scale-prefix> can be appended as needed to indicate other attributes of interest
  4. FineTune: A descriptive name for the model fine tuning goal (e.g. Chat, Instruct, etc...)
    • This can be derived from gguf metadata general.finetune substituting spaces for dashes.
  5. Version: (Optional) Denotes the model version number, formatted as v<Major>.<Minor>
    • If model is missing a version number then assume v1.0 (First Public Release)
    • This can be derived from gguf metadata general.version
  6. Encoding: Indicates the weights encoding scheme that was applied to the model. Content, type mixture and arrangement however are determined by user code and can vary depending on project needs.
  7. Type: Indicates the kind of gguf file and the intended purpose for it
    • If missing, then file is by default a typical gguf tensor model file
    • LoRA : GGUF file is a LoRA adapter
    • vocab : GGUF file with only vocab data and metadata
  8. Shard: (Optional) Indicates and denotes that the model has been split into multiple shards, formatted as <ShardNum>-of-<ShardTotal>.
    • ShardNum : Shard position in this model. Must be 5 digits padded by zeros.
      • Shard number always starts from 00001 onwards (e.g. First shard always starts at 00001-of-XXXXX rather than 00000-of-XXXXX).
    • ShardTotal : Total number of shards in this model. Must be 5 digits padded by zeros.

Validating Above Naming Convention

At a minimum all model files should have at least BaseName, SizeLabel, Version, in order to be easily validated as a file that is keeping with the GGUF Naming Convention. An example of this issue is that it is easy for Encoding to be mistaken as a FineTune if Version is omitted.

To validate you can use this regular expression ^(?:(?<Sidecar>mmproj|mtp)-)?(?[A-Za-z0-9\s]*(?:(?:-(?:(?:[A-Za-z\s][A-Za-z0-9\s]*)|(?:[0-9\s]*)))*))-(?:(?<SizeLabel>(?:\d+x)?(?:\d+\.)?\d+[A-Za-z](?:-[A-Za-z]+(\d+\.)?\d+[A-Za-z]+)?)(?:-(?<FineTune>[A-Za-z0-9\s-]+))?)?-(?:(?<Version>v\d+(?:\.\d+)*))(?:-(?<Encoding>(?!LoRA|vocab)[\w_]+))?(?:-(?<Type>LoRA|vocab))?(?:-(?<Shard>\d{5}-of-\d{5}))?\.gguf$ which will check that you got the minimum BaseName, SizeLabel and Version present in the correct order.

For example:

  • Mixtral-8x7B-v0.1-KQ2.gguf:

    • Model Name: Mixtral
    • Expert Count: 8
    • Parameter Count: 7B
    • Version Number: v0.1
    • Weight Encoding Scheme: KQ2

  • Hermes-2-Pro-Llama-3-8B-F16.gguf:

    • Model Name: Hermes 2 Pro Llama 3
    • Expert Count: 0
    • Parameter Count: 8B
    • Version Number: v1.0
    • Weight Encoding Scheme: F16
    • Shard: N/A

  • Grok-100B-v1.0-Q4_0-00003-of-00009.gguf

    • Model Name: Grok
    • Expert Count: 0
    • Parameter Count: 100B
    • Version Number: v1.0
    • Weight Encoding Scheme: Q4_0
    • Shard: 3 out of 9 total shards

  • mtp-Qwen3-27B-v1.0-Q4_K_M.gguf

    • Sidecar: mtp (Multi-Token Prediction draft module)
    • Model Name: Qwen3
    • Expert Count: 0
    • Parameter Count: 27B (of the main model — sidecar tensors are smaller)
    • Version Number: v1.0
    • Weight Encoding Scheme: Q4_K_M

  • mmproj-Qwen2-VL-7B-v1.0-F16.gguf

    • Sidecar: mmproj (multimodal projector)
    • Model Name: Qwen2-VL
    • Expert Count: 0
    • Parameter Count: 7B (of the main model — sidecar tensors are smaller)
    • Version Number: v1.0
    • Weight Encoding Scheme: F16

Example Node.js Regex Function

text
/* Detailed source-code truncated for AI context efficiency. */

File Structure

diagram by @mishig25 (GGUF v3)

GGUF files are structured as follows. They use a global alignment specified in the general.alignment metadata field, referred to as ALIGNMENT below. Where required, the file is padded with 0x00 bytes to the next multiple of general.alignment.

Fields, including arrays, are written sequentially without alignment unless otherwise specified.

Models are little-endian by default. They can also come in big-endian for use with big-endian computers; in this case, all values (including metadata values and tensors) will also be big-endian. At the time of writing, there is no way to determine if a model is big-endian; this may be rectified in future versions. If no additional information is provided, assume the model is little-endian.

text
/* Detailed source-code truncated for AI context efficiency. */

Standardized key-value pairs

The following key-value pairs are standardized. This list may grow in the future as more use cases are discovered. Where possible, names are shared with the original model definitions to make it easier to map between the two.

Not all of these are required, but they are all recommended. Keys that are required are bolded. For omitted pairs, the reader should assume that the value is unknown and either default or error as appropriate.

The community can develop their own key-value pairs to carry additional data. However, these should be namespaced with the relevant community name to avoid collisions. For example, the rustformers community might use rustformers. as a prefix for all of their keys.

If a particular community key is widely used, it may be promoted to a standardized key.

By convention, most counts/lengths/etc are uint64 unless otherwise specified. This is to allow for larger models to be supported in the future. Some models may use uint32 for their values; it is recommended that readers support both.

General

Required

  • general.architecture: string: describes what architecture this model implements. All lowercase ASCII, with only [a-z0-9]+ characters allowed. Known values include:
    • llama
    • mpt
    • gptneox
    • gptj
    • gpt2
    • bloom
    • falcon
    • mamba
    • rwkv
  • general.quantization_version: uint32: The version of the quantization format. Not required if the model is not quantized (i.e. no tensors are quantized). If any tensors are quantized, this must be present. This is separate to the quantization scheme of the tensors itself; the quantization version may change without changing the scheme's name (e.g. the quantization scheme is Q5_K, and the quantization version is 4).
  • general.alignment: uint32: the global alignment to use, as described above. This can vary to allow for different alignment schemes, but it must be a multiple of 8. Some writers may not write the alignment. If the alignment is not specified, assume it is 32.

General metadata

  • general.name: string: The name of the model. This should be a human-readable name that can be used to identify the model. It should be unique within the community that the model is defined in.
  • general.author: string: The author of the model.
  • general.version: string: The version of the model.
  • general.organization: string: The organization of the model.
  • general.basename: string: The base model name / architecture of the model
  • general.finetune: string: What has the base model been optimized toward.
  • general.description: string: free-form description of the model including anything that isn't covered by the other fields
  • general.quantized_by: string: The name of the individual who quantized the model
  • general.size_label: string: Size class of the model, such as number of weights and experts. (Useful for leader boards)
  • general.license: string: License of the model, expressed as a SPDX license expression (e.g. "MIT OR Apache-2.0). Do not include any other information, such as the license text or the URL to the license.
  • general.license.name: string: Human friendly license name
  • general.license.link: string: URL to the license.
  • general.url: string: URL to the model's homepage. This can be a GitHub repo, a paper, etc.
  • general.doi: string: Digital Object Identifier (DOI) https://www.doi.org/
  • general.uuid: string: Universally unique identifier
  • general.repo_url: string: URL to the model's repository such as a GitHub repo or HuggingFace repo
  • general.tags: string[]: List of tags that can be used as search terms for a search engine or social media
  • general.languages: string[]: What languages can the model speak. Encoded as ISO 639 two letter codes
  • general.datasets: string[]: Links or references to datasets that the model was trained upon
  • general.file_type: uint32: An enumerated value describing the type of the majority of the tensors in the file. Optional; can be inferred from the tensor types.
    • ALL_F32 = 0
    • MOSTLY_F16 = 1
    • MOSTLY_Q4_0 = 2
    • MOSTLY_Q4_1 = 3
    • MOSTLY_Q4_1_SOME_F16 = 4
    • MOSTLY_Q4_2 = 5 (support removed)
    • MOSTLY_Q4_3 = 6 (support removed)
    • MOSTLY_Q8_0 = 7
    • MOSTLY_Q5_0 = 8
    • MOSTLY_Q5_1 = 9
    • MOSTLY_Q2_K = 10
    • MOSTLY_Q3_K_S = 11
    • MOSTLY_Q3_K_M = 12
    • MOSTLY_Q3_K_L = 13
    • MOSTLY_Q4_K_S = 14
    • MOSTLY_Q4_K_M = 15
    • MOSTLY_Q5_K_S = 16
    • MOSTLY_Q5_K_M = 17
    • MOSTLY_Q6_K = 18

Source metadata

Information about where this model came from. This is useful for tracking the provenance of the model, and for finding the original source if the model is modified. For a model that was converted from GGML, for example, these keys would point to the model that was converted from.

  • general.source.url: string: URL to the source of the model's homepage. This can be a GitHub repo, a paper, etc.

  • general.source.doi: string: Source Digital Object Identifier (DOI) https://www.doi.org/

  • general.source.uuid: string: Source Universally unique identifier

  • general.source.repo_url: string: URL to the source of the model's repository such as a GitHub repo or HuggingFace repo

  • general.base_model.count: uint32: Number of parent models

  • general.base_model.{id}.name: string: The name of the parent model.

  • general.base_model.{id}.author: string: The author of the parent model.

  • general.base_model.{id}.version: string: The version of the parent model.

  • general.base_model.{id}.organization: string: The organization of the parent model.

  • general.base_model.{id}.url: string: URL to the source of the parent model's homepage. This can be a GitHub repo, a paper, etc.

  • general.base_model.{id}.doi: string: Parent Digital Object Identifier (DOI) https://www.doi.org/

  • general.base_model.{id}.uuid: string: Parent Universally unique identifier

  • general.base_model.{id}.repo_url: string: URL to the source of the parent model's repository such as a GitHub repo or HuggingFace repo

LLM

In the following, [llm] is used to fill in for the name of a specific LLM architecture. For example, llama for LLaMA, mpt for MPT, etc. If mentioned in an architecture's section, it is required for that architecture, but not all keys are required for all architectures. Consult the relevant section for more information.

  • [llm].context_length: uint64: Also known as n_ctx. length of the context (in tokens) that the model was trained on. For most architectures, this is the hard limit on the length of the input. Architectures, like RWKV, that are not reliant on transformer-style attention may be able to handle larger inputs, but this is not guaranteed.
  • [llm].embedding_length: uint64: Also known as n_embd. Embedding layer size.
  • [llm].block_count: uint64: The number of blocks of attention+feed-forward layers (i.e. the bulk of the LLM). Does not include the input or embedding layers.
  • [llm].feed_forward_length: uint64: Also known as n_ff. The length of the feed-forward layer.
  • [llm].use_parallel_residual: bool: Whether or not the parallel residual logic should be used.
  • [llm].tensor_data_layout: string: When a model is converted to GGUF, tensors may be rearranged to improve performance. This key describes the layout of the tensor data. This is not required; if not present, it is assumed to be reference.
    • reference: tensors are laid out in the same order as the original model
    • further options can be found for each architecture in their respective sections
  • [llm].expert_count: uint32: Number of experts in MoE models (optional for non-MoE arches).
  • [llm].expert_used_count: uint32: Number of experts used during each token token evaluation (optional for non-MoE arches).

Attention

  • [llm].attention.head_count: uint64: Also known as n_head. Number of attention heads.
  • [llm].attention.head_count_kv: uint64: The number of heads per group used in Grouped-Query-Attention. If not present or if present and equal to [llm].attention.head_count, the model does not use GQA.
  • [llm].attention.max_alibi_bias: float32: The maximum bias to use for ALiBI.
  • [llm].attention.clamp_kqv: float32: Value (C) to clamp the values of the Q, K, and V tensors between ([-C, C]).
  • [llm].attention.layer_norm_epsilon: float32: Layer normalization epsilon.
  • [llm].attention.layer_norm_rms_epsilon: float32: Layer RMS normalization epsilon.
  • [llm].attention.key_length: uint32: The optional size of a key head, $d_k$. If not specified, it will be n_embd / n_head.
  • [llm].attention.value_length: uint32: The optional size of a value head, $d_v$. If not specified, it will be n_embd / n_head.

RoPE

  • [llm].rope.dimension_count: uint64: The number of rotary dimensions for RoPE.
  • [llm].rope.freq_base: float32: The base frequency for RoPE.

Scaling

The following keys describe RoPE scaling parameters:

  • [llm].rope.scaling.type: string: Can be none, linear, or yarn.
  • [llm].rope.scaling.factor: float32: A scale factor for RoPE to adjust the context length.
  • [llm].rope.scaling.original_context_length: uint32_t: The original context length of the base model.
  • [llm].rope.scaling.finetuned: bool: True if model has been finetuned with RoPE scaling.

Note that older models may not have these keys, and may instead use the following key:

  • [llm].rope.scale_linear: float32: A linear scale factor for RoPE to adjust the context length.

It is recommended that models use the newer keys if possible, as they are more flexible and allow for more complex scaling schemes. Executors will need to support both indefinitely.

SSM

  • [llm].ssm.conv_kernel: uint32: The size of the rolling/shift state.
  • [llm].ssm.inner_size: uint32: The embedding size of the states.
  • [llm].ssm.state_size: uint32: The size of the recurrent state.
  • [llm].ssm.time_step_rank: uint32: The rank of time steps.

Models

The following sections describe the metadata for each model architecture. Each key specified must be present.

LLaMA

  • llama.context_length
  • llama.embedding_length
  • llama.block_count
  • llama.feed_forward_length
  • llama.rope.dimension_count
  • llama.attention.head_count
  • llama.attention.layer_norm_rms_epsilon

Optional

  • llama.rope.scale
  • llama.attention.head_count_kv
  • llama.tensor_data_layout:
    • Meta AI original pth:
      python
      def permute(weights: NDArray, n_head: int) -> NDArray:
    return (weights.reshape(n_head, 2, weights.shape[0] // n_head // 2, *weights.shape[1:])
                .swapaxes(1, 2)
                .reshape(weights.shape))
  • llama.expert_count
  • llama.expert_used_count

MPT

  • mpt.context_length
  • mpt.embedding_length
  • mpt.block_count
  • mpt.attention.head_count
  • mpt.attention.alibi_bias_max
  • mpt.attention.clip_kqv
  • mpt.attention.layer_norm_epsilon

GPT-NeoX

  • gptneox.context_length
  • gptneox.embedding_length
  • gptneox.block_count
  • gptneox.use_parallel_residual
  • gptneox.rope.dimension_count
  • gptneox.attention.head_count
  • gptneox.attention.layer_norm_epsilon

Optional

  • gptneox.rope.scale

GPT-J

  • gptj.context_length
  • gptj.embedding_length
  • gptj.block_count
  • gptj.rope.dimension_count
  • gptj.attention.head_count
  • gptj.attention.layer_norm_epsilon

Optional

  • gptj.rope.scale

GPT-2

  • gpt2.context_length
  • gpt2.embedding_length
  • gpt2.block_count
  • gpt2.attention.head_count
  • gpt2.attention.layer_norm_epsilon

BLOOM

  • bloom.context_length
  • bloom.embedding_length
  • bloom.block_count
  • bloom.feed_forward_length
  • bloom.attention.head_count
  • bloom.attention.layer_norm_epsilon

Falcon

  • falcon.context_length
  • falcon.embedding_length
  • falcon.block_count
  • falcon.attention.head_count
  • falcon.attention.head_count_kv
  • falcon.attention.use_norm
  • falcon.attention.layer_norm_epsilon

Optional

  • falcon.tensor_data_layout:

    • jploski (author of the original GGML implementation of Falcon):

      python
      # The original query_key_value tensor contains n_head_kv "kv groups",
# each consisting of n_head/n_head_kv query weights followed by one key
# and one value weight (shared by all query heads in the kv group).
# This layout makes it a big pain to work with in GGML.
# So we rearrange them here,, so that we have n_head query weights
# followed by n_head_kv key weights followed by n_head_kv value weights,
# in contiguous fashion.

if "query_key_value" in src:
    qkv = model[src].view(
        n_head_kv, n_head // n_head_kv + 2, head_dim, head_dim * n_head)

    q = qkv[:, :-2 ].reshape(n_head * head_dim, head_dim * n_head)
    k = qkv[:, [-2]].reshape(n_head_kv * head_dim, head_dim * n_head)
    v = qkv[:, [-1]].reshape(n_head_kv * head_dim, head_dim * n_head)

    model[src] = torch.cat((q,k,v)).reshape_as(model[src])

Mamba

  • mamba.context_length
  • mamba.embedding_length
  • mamba.block_count
  • mamba.ssm.conv_kernel
  • mamba.ssm.inner_size
  • mamba.ssm.state_size
  • mamba.ssm.time_step_rank
  • mamba.attention.layer_norm_rms_epsilon

RWKV

The vocabulary size is the same as the number of rows in the head matrix.

  • rwkv.architecture_version: uint32: The only allowed value currently is 4. Version 5 is expected to appear some time in the future.
  • rwkv.context_length: uint64: Length of the context used during training or fine-tuning. RWKV is able to handle larger context than this limit, but the output quality may suffer.
  • rwkv.block_count: uint64
  • rwkv.embedding_length: uint64
  • rwkv.feed_forward_length: uint64

Whisper

Keys that do not have types defined should be assumed to share definitions with llm. keys.
(For example, whisper.context_length is equivalent to llm.context_length.)
This is because they are both transformer models.

  • whisper.encoder.context_length

  • whisper.encoder.embedding_length

  • whisper.encoder.block_count

  • whisper.encoder.mels_count: uint64

  • whisper.encoder.attention.head_count

  • whisper.decoder.context_length

  • whisper.decoder.embedding_length

  • whisper.decoder.block_count

  • whisper.decoder.attention.head_count

Prompting

TODO: Include prompt format, and/or metadata about how it should be used (instruction, conversation, autocomplete, etc).

LoRA

TODO: Figure out what metadata is needed for LoRA. Probably desired features:

  • match an existing model exactly, so that it can't be misapplied
  • be marked as a LoRA so executors won't try to run it by itself

Should this be an architecture, or should it share the details of the original model with additional fields to mark it as a LoRA?

Tokenizer

The following keys are used to describe the tokenizer of the model. It is recommended that model authors support as many of these as possible, as it will allow for better tokenization quality with supported executors.

GGML

GGML supports an embedded vocabulary that enables inference of the model, but implementations of tokenization using this vocabulary (i.e. llama.cpp's tokenizer) may have lower accuracy than the original tokenizer used for the model. When a more accurate tokenizer is available and supported, it should be used instead.

It is not guaranteed to be standardized across models, and may change in the future. It is recommended that model authors use a more standardized tokenizer if possible.

  • tokenizer.ggml.model: string: The name of the tokenizer model.
    • llama: Llama style SentencePiece (tokens and scores extracted from HF tokenizer.model)
    • replit: Replit style SentencePiece (tokens and scores extracted from HF spiece.model)
    • gpt2: GPT-2 / GPT-NeoX style BPE (tokens extracted from HF tokenizer.json)
    • rwkv: RWKV tokenizer
  • tokenizer.ggml.tokens: array[string]: A list of tokens indexed by the token ID used by the model.
  • tokenizer.ggml.scores: array[float32]: If present, the score/probability of each token. If not present, all tokens are assumed to have equal probability. If present, it must have the same length and index as tokens.
  • tokenizer.ggml.token_type: array[int32]: The token type (1=normal, 2=unknown, 3=control, 4=user defined, 5=unused, 6=byte). If present, it must have the same length and index as tokens.
  • tokenizer.ggml.merges: array[string]: If present, the merges of the tokenizer. If not present, the tokens are assumed to be atomic.
  • tokenizer.ggml.added_tokens: array[string]: If present, tokens that were added after training.

Special tokens

  • tokenizer.ggml.bos_token_id: uint32: Beginning of sequence marker
  • tokenizer.ggml.eos_token_id: uint32: End of sequence marker
  • tokenizer.ggml.unknown_token_id: uint32: Unknown token
  • tokenizer.ggml.separator_token_id: uint32: Separator token
  • tokenizer.ggml.padding_token_id: uint32: Padding token

Hugging Face

Hugging Face maintains their own tokenizers library that supports a wide variety of tokenizers. If your executor uses this library, it may be able to use the model's tokenizer directly.

Other

Other tokenizers may be used, but are not necessarily standardized. They may be executor-specific. They will be documented here as they are discovered/further developed.

Computation graph

This is a future extension and still needs to be discussed, and may necessitate a new GGUF version. At the time of writing, the primary blocker is the stabilization of the computation graph format.

A sample computation graph of GGML nodes could be included in the model itself, allowing an executor to run the model without providing its own implementation of the architecture. This would allow for a more consistent experience across executors, and would allow for more complex architectures to be supported without requiring the executor to implement them.

Standardized tensor names

To minimize complexity and maximize compatibility, it is recommended that models using the transformer architecture use the following naming convention for their tensors:

Base layers

AA.weight AA.bias

where AA can be:

  • token_embd: Token embedding layer
  • pos_embd: Position embedding layer
  • output_norm: Output normalization layer
  • output: Output layer

Attention and feed-forward layer blocks

blk.N.BB.weight blk.N.BB.bias

where N signifies the block number a layer belongs to, and where BB could be:

  • attn_norm: Attention normalization layer

  • attn_norm_2: Attention normalization layer

  • attn_qkv: Attention query-key-value layer

  • attn_q: Attention query layer

  • attn_k: Attention key layer

  • attn_v: Attention value layer

  • attn_output: Attention output layer

  • ffn_norm: Feed-forward network normalization layer

  • ffn_up: Feed-forward network "up" layer

  • ffn_gate: Feed-forward network "gate" layer

  • ffn_down: Feed-forward network "down" layer

  • ffn_gate_inp: Expert-routing layer for the Feed-forward network in MoE models

  • ffn_gate_exp: Feed-forward network "gate" layer per expert in MoE models

  • ffn_down_exp: Feed-forward network "down" layer per expert in MoE models

  • ffn_up_exp: Feed-forward network "up" layer per expert in MoE models

  • ssm_in: State space model input projections layer

  • ssm_conv1d: State space model rolling/shift layer

  • ssm_x: State space model selective parametrization layer

  • ssm_a: State space model state compression layer

  • ssm_d: State space model skip connection layer

  • ssm_dt: State space model time step layer

  • ssm_out: State space model output projection layer

Version History

This document is actively updated to describe the current state of the metadata, and these changes are not tracked outside of the commits.

However, the format itself has changed. The following sections describe the changes to the format itself.

v3

Adds big-endian support.

v2

Most countable values (lengths, etc) were changed from uint32 to uint64 to allow for larger models to be supported in the future.

v1

Initial version.

Historical State of Affairs

The following information is provided for context, but is not necessary to understand the rest of this document.

Overview

At present, there are three GGML file formats floating around for LLMs:

  • GGML (unversioned): baseline format, with no versioning or alignment.
  • GGMF (versioned): the same as GGML, but with versioning. Only one version exists.
  • GGJT: Aligns the tensors to allow for use with mmap, which requires alignment. v1, v2 and v3 are identical, but the latter versions use a different quantization scheme that is incompatible with previous versions.

GGML is primarily used by the examples in ggml, while GGJT is used by llama.cpp models. Other executors may use any of the three formats, but this is not 'officially' supported.

These formats share the same fundamental structure:

  • a magic number with an optional version number
  • model-specific hyperparameters, including
    • metadata about the model, such as the number of layers, the number of heads, etc.
    • a ftype that describes the type of the majority of the tensors,
      • for GGML files, the quantization version is encoded in the ftype divided by 1000
  • an embedded vocabulary, which is a list of strings with length prepended. The GGMF/GGJT formats embed a float32 score next to the strings.
  • finally, a list of tensors with their length-prepended name, type, and (aligned, in the case of GGJT) tensor data

Notably, this structure does not identify what model architecture the model belongs to, nor does it offer any flexibility for changing the structure of the hyperparameters. This means that the only way to add new hyperparameters is to add them to the end of the list, which is a breaking change for existing models.

Drawbacks

Unfortunately, over the last few months, there are a few issues that have become apparent with the existing models:

  • There's no way to identify which model architecture a given model is for, because that information isn't present
    • Similarly, existing programs cannot intelligently fail upon encountering new architectures
  • Adding or removing any new hyperparameters is a breaking change, which is impossible for a reader to detect without using heuristics
  • Each model architecture requires its own conversion script to their architecture's variant of GGML
  • Maintaining backwards compatibility without breaking the structure of the format requires clever tricks, like packing the quantization version into the ftype, which are not guaranteed to be picked up by readers/writers, and are not consistent between the two formats

Why not other formats?

There are a few other formats that could be used, but issues include:

  • requiring additional dependencies to load or save the model, which is complicated in a C environment
  • limited or no support for 4-bit quantization
  • existing cultural expectations (e.g. whether or not the model is a directory or a file)
  • lack of support for embedded vocabularies
  • lack of control over direction of future development

Ultimately, it is likely that GGUF will remain necessary for the foreseeable future, and it is better to have a single format that is well-documented and supported by all executors than to contort an existing format to fit the needs of GGML.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#machine-learning Browse machine-learning catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →