llama-cpp-python Documentation

File: README.md

Python Bindings for [`llama.cpp`](https://github.com/ggerganov/llama.cpp)

Simple Python bindings for @ggerganov's llama.cpp library.
This package provides:

Low-level access to C API via ctypes interface.
High-level Python API for text completion
- OpenAI-like API
- LangChain compatibility
- LlamaIndex compatibility
OpenAI compatible web server

Documentation is available at https://llama-cpp-python.readthedocs.io/en/latest.

Installation

Requirements:

Python 3.8+
C compiler
- Linux: gcc or clang
- Windows: Visual Studio or MinGW
- MacOS: Xcode

To install the package, run:

bash

pip install llama-cpp-python

This will also build llama.cpp from source and install it alongside this python package.

If this fails, add --verbose to the pip install see the full cmake build log.

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with basic CPU support.

bash

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

Installation Configuration

llama.cpp supports a number of hardware acceleration backends to speed up inference as well as backend specific options. See the llama.cpp README for a full list.

All llama.cpp cmake build options can be set via the CMAKE_ARGS environment variable or via the --config-settings / -C cli flag during installation.

Environment Variables

bash

# Linux and Mac
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" \
  pip install llama-cpp-python

powershell

# Windows
$env:CMAKE_ARGS = "-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install llama-cpp-python

CLI / requirements.txt

They can also be set via pip install -C / --config-settings command and saved to a requirements.txt file:

bash

pip install --upgrade pip # ensure pip is up to date
pip install llama-cpp-python \
  -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

txt

# requirements.txt

llama-cpp-python -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

Supported Backends

Below are some common backends, their build commands and any additional environment variables required.

OpenBLAS (CPU)

To install with OpenBLAS, set the GGML_BLAS and GGML_BLAS_VENDOR environment variables before installing:

bash

CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python

CUDA

To install with CUDA support, set the GGML_CUDA=on environment variable before installing:

bash

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with CUDA support. As long as your system meets some requirements:

CUDA Version is 11.8, 12.1, 12.2, 12.3, 12.4, 12.5, 13.0 or 13.2
NVIDIA GPU compute capability is 6.0 through 8.9 for CUDA 11.8 wheels, 6.0 or newer for CUDA 12 wheels, or 7.5 or newer for CUDA 13 wheels
Python Version is 3.10, 3.11 or 3.12

bash

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/<cuda-version>

Where <cuda-version> is one of the following:

cu118: CUDA 11.8
cu121: CUDA 12.1
cu122: CUDA 12.2
cu123: CUDA 12.3
cu124: CUDA 12.4
cu125: CUDA 12.5
cu130: CUDA 13.0
cu132: CUDA 13.2

For example, to install the CUDA 12.1 wheel:

bash

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

Metal

To install with Metal (MPS), set the GGML_METAL=on environment variable before installing:

bash

CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with Metal support. As long as your system meets some requirements:

MacOS Version is 11.0 or later
Python Version is 3.10, 3.11 or 3.12

bash

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

HIP (ROCm)

To install with HIP / ROCm support for AMD cards, set the GGML_HIP=on environment variable before installing:

bash

CMAKE_ARGS="-DGGML_HIP=on" pip install llama-cpp-python

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with ROCm support for Linux:

bash

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/rocm72

Or a pre-built wheel with HIP Radeon support for Windows:

powershell

pip install llama-cpp-python `
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/hip-radeon

Vulkan

To install with Vulkan support, set the GGML_VULKAN=on environment variable before installing:

bash

CMAKE_ARGS="-DGGML_VULKAN=on" pip install llama-cpp-python

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with Vulkan support for Linux or Windows:

bash

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/vulkan

SYCL

To install with SYCL support, set the GGML_SYCL=on environment variable before installing:

bash

source /opt/intel/oneapi/setvars.sh   
CMAKE_ARGS="-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx" pip install llama-cpp-python

RPC

To install with RPC support, set the GGML_RPC=on environment variable before installing:

bash

source /opt/intel/oneapi/setvars.sh   
CMAKE_ARGS="-DGGML_RPC=on" pip install llama-cpp-python

Windows Notes

Error: Can't find 'nmake' or 'CMAKE_C_COMPILER'

If you run into issues where it complains it can't find 'nmake' '?' or CMAKE_C_COMPILER, you can extract w64devkit as mentioned in llama.cpp repo and add those manually to CMAKE_ARGS before running pip install:

$env:CMAKE_GENERATOR = "MinGW Makefiles"
$env:CMAKE_ARGS = "-DGGML_OPENBLAS=on -DCMAKE_C_COMPILER=C:/w64devkit/bin/gcc.exe -DCMAKE_CXX_COMPILER=C:/w64devkit/bin/g++.exe"

See the above instructions and set CMAKE_ARGS to the BLAS backend you want to use.

MacOS Notes

Detailed MacOS Metal GPU install documentation is available at docs/install/macos.md

M1 Mac Performance Issue

Note: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:

bash

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh

Otherwise, while installing it will build the llama.cpp x86 version which will be 10x slower on Apple Silicon (M1) Mac.

M Series Mac Error: (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Try installing with

bash

CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DGGML_METAL=on" pip install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python

Upgrading and Reinstalling

To upgrade and rebuild llama-cpp-python add --upgrade --force-reinstall --no-cache-dir flags to the pip install command to ensure the package is rebuilt from source.

High-level API

API Reference

The high-level API provides a simple managed interface through the Llama class.

Below is a short example demonstrating how to use the high-level API for basic text completion:

python

from llama_cpp import Llama

llm = Llama(
      model_path="./models/7B/llama-model.gguf",
      # n_gpu_layers=-1, # Uncomment to use GPU acceleration
      # seed=1337, # Uncomment to set a specific seed
      # n_ctx=2048, # Uncomment to increase the context window
)
output = llm(
      "Q: Name the planets in the solar system? A: ", # Prompt
      max_tokens=32, # Generate up to 32 tokens, set to None to generate up to the end of the context window
      stop=["Q:", "\n"], # Stop generating just before the model would generate a new question
      echo=True # Echo the prompt back in the output
) # Generate a completion, can also call create_completion
print(output)

By default llama-cpp-python generates completions in an OpenAI compatible format:

python

{
  "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1679561337,
  "model": "./models/7B/llama-model.gguf",
  "choices": [
    {
      "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
      "index": 0,
      "logprobs": None,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 28,
    "total_tokens": 42
  }
}

Text completion is available through the __call__ and create_completion methods of the Llama class.

Pulling models from Hugging Face Hub

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method.
You'll need to install the huggingface-hub package to use this feature (pip install huggingface-hub).

python

llm = Llama.from_pretrained(
    repo_id="lmstudio-community/Qwen3.5-0.8B-GGUF",
    filename="*Q8_0.gguf",
    verbose=False
)

By default from_pretrained will download the model to the huggingface cache directory, you can then manage installed model files with the hf tool.

Chat Completion

The high-level API also provides a simple interface for chat completion.

Chat completion requires that the model knows how to format the messages into a single prompt.
The Llama class does this using pre-registered chat formats (ie. chatml, llama-2, gemma, etc) or by providing a custom chat handler object.

The model will format the messages into a single prompt using the following order of precedence:

Use the chat_handler if provided
Use the chat_format if provided
Use the tokenizer.chat_template from the gguf model's metadata (should work for most new models, older models may not have this)
else, fallback to the llama-2 chat format

Set verbose=True to see the selected chat format.

python

from llama_cpp import Llama
llm = Llama(
      model_path="path/to/llama-2/llama-model.gguf",
      chat_format="llama-2"
)
llm.create_chat_completion(
      messages = [
          {"role": "system", "content": "You are an assistant who perfectly describes images."},
          {
              "role": "user",
              "content": "Describe this image in detail please."
          }
      ]
)

Chat completion is available through the create_chat_completion method of the Llama class.

For OpenAI API v1 compatibility, you use the create_chat_completion_openai_v1 method which will return pydantic models instead of dicts.

JSON and JSON Schema Mode

To constrain chat responses to only valid JSON or a specific JSON Schema use the response_format argument in create_chat_completion.

JSON Mode

The following example will constrain the response to valid JSON strings only.

python

from llama_cpp import Llama
llm = Llama(model_path="path/to/model.gguf", chat_format="chatml")
llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that outputs in JSON.",
        },
        {"role": "user", "content": "Who won the world series in 2020"},
    ],
    response_format={
        "type": "json_object",
    },
    temperature=0.7,
)

JSON Schema Mode

To constrain the response further to a specific JSON Schema add the schema to the schema property of the response_format argument.

python

from llama_cpp import Llama
llm = Llama(model_path="path/to/model.gguf", chat_format="chatml")
llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that outputs in JSON.",
        },
        {"role": "user", "content": "Who won the world series in 2020"},
    ],
    response_format={
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {"team_name": {"type": "string"}},
            "required": ["team_name"],
        },
    },
    temperature=0.7,
)

Function Calling

The high-level API supports OpenAI compatible function and tool calling. This is possible through the functionary pre-trained models chat format or through the generic chatml-function-calling chat format.

python

from llama_cpp import Llama
llm = Llama(model_path="path/to/chatml/llama-model.gguf", chat_format="chatml-function-calling")
llm.create_chat_completion(
      messages = [
        {
          "role": "system",
          "content": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. The assistant calls functions with appropriate input when necessary"

        },
        {
          "role": "user",
          "content": "Extract Jason is 25 years old"
        }
      ],
      tools=[{
        "type": "function",
        "function": {
          "name": "UserDetail",
          "parameters": {
            "type": "object",
            "title": "UserDetail",
            "properties": {
              "name": {
                "title": "Name",
                "type": "string"
              },
              "age": {
                "title": "Age",
                "type": "integer"
              }
            },
            "required": [ "name", "age" ]
          }
        }
      }],
      tool_choice={
        "type": "function",
        "function": {
          "name": "UserDetail"
        }
      }
)

Functionary v2

The various gguf-converted files for this set of models can be found here. Functionary is able to intelligently call functions and also analyze any provided function outputs to generate coherent responses. All v2 models of functionary supports parallel function calling. You can provide either functionary-v1 or functionary-v2 for the chat_format when initializing the Llama class.

Due to discrepancies between llama.cpp and HuggingFace's tokenizers, it is required to provide HF Tokenizer for functionary. The LlamaHFTokenizer class can be initialized and passed into the Llama class. This will override the default llama.cpp tokenizer used in Llama class. The tokenizer files are already included in the respective HF repositories hosting the gguf files.

python

from llama_cpp import Llama
from llama_cpp.llama_tokenizer import LlamaHFTokenizer
llm = Llama.from_pretrained(
  repo_id="meetkai/functionary-small-v2.2-GGUF",
  filename="functionary-small-v2.2.q4_0.gguf",
  chat_format="functionary-v2",
  tokenizer=LlamaHFTokenizer.from_pretrained("meetkai/functionary-small-v2.2-GGUF")
)

NOTE: There is no need to provide the default system messages used in Functionary as they are added automatically in the Functionary chat handler. Thus, the messages should contain just the chat messages and/or system messages that provide additional context for the model (e.g.: datetime, etc.).

llama-cpp-python supports such as llava1.5 which allow the language model to read information from both text and images.

Below are the supported multi-modal models and their respective chat handlers (Python API) and chat formats (Server API).

Model	`LlamaChatHandler`	`chat_format`
llava-v1.5-7b	`Llava15ChatHandler`	`llava-1-5`
llava-v1.5-13b	`Llava15ChatHandler`	`llava-1-5`
llava-v1.6-34b	`Llava16ChatHandler`	`llava-1-6`
moondream2	`MoondreamChatHandler`	`moondream2`
nanollava	`NanoLlavaChatHandler`	`nanollava`
llama-3-vision-alpha	`Llama3VisionAlphaChatHandler`	`llama-3-vision-alpha`
minicpm-v-2.6	`MiniCPMv26ChatHandler`	`minicpm-v-2.6`
qwen2.5-vl	`Qwen25VLChatHandler`	`qwen2.5-vl`
gemma-4	`Gemma4ChatHandler`	`gemma4`
GGUF models with an mtmd projector and embedded chat template	`MTMDChatHandler`	`mtmd`

Try Gemma 4 12B in Google Colab ->

Try Gemma 4 12B QAT in Google Colab ->

Then you'll need to use a custom chat handler to load the clip model and process the chat messages and images.

python

from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llava15ChatHandler
chat_handler = Llava15ChatHandler(clip_model_path="path/to/llava/mmproj.bin")
llm = Llama(
  model_path="./path/to/llava/llama-model.gguf",
  chat_handler=chat_handler,
  n_ctx=2048, # n_ctx should be increased to accommodate the image embedding
)
llm.create_chat_completion(
    messages = [
        {"role": "system", "content": "You are an assistant who perfectly describes images."},
        {
            "role": "user",
            "content": [
                {"type" : "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" } }
            ]
        }
    ]
)

You can also pull the model from the Hugging Face Hub using the from_pretrained method.

python

from llama_cpp import Llama
from llama_cpp.llama_chat_format import MoondreamChatHandler

chat_handler = MoondreamChatHandler.from_pretrained(
  repo_id="vikhyatk/moondream2",
  filename="*mmproj*",
)

llm = Llama.from_pretrained(
  repo_id="vikhyatk/moondream2",
  filename="*text-model*",
  chat_handler=chat_handler,
  n_ctx=2048, # n_ctx should be increased to accommodate the image embedding
)

response = llm.create_chat_completion(
    messages = [
        {
            "role": "user",
            "content": [
                {"type" : "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" } }

            ]
        }
    ]
)
print(response["choices"][0]["text"])

Note: Multi-modal models also support tool calling and JSON mode.

Loading a Local Image

Images can be passed as base64 encoded data URIs. The following example demonstrates how to do this.

python

import base64

def image_to_base64_data_uri(file_path):
    with open(file_path, "rb") as img_file:
        base64_data = base64.b64encode(img_file.read()).decode('utf-8')
        return f"data:image/png;base64,{base64_data}"

# Replace 'file_path.png' with the actual path to your PNG file
file_path = 'file_path.png'
data_uri = image_to_base64_data_uri(file_path)

messages = [
    {"role": "system", "content": "You are an assistant who perfectly describes images."},
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": data_uri }},
            {"type" : "text", "text": "Describe this image in detail please."}
        ]
    }
]

Speculative Decoding

llama-cpp-python supports speculative decoding which allows the model to generate completions based on a draft model.

The fastest way to use speculative decoding is through the LlamaPromptLookupDecoding class.

Just pass this as a draft model to the Llama class during initialization.

python

from llama_cpp import Llama
from llama_cpp.llama_speculative import LlamaPromptLookupDecoding

llama = Llama(
    model_path="path/to/model.gguf",
    draft_model=LlamaPromptLookupDecoding(num_pred_tokens=10) # num_pred_tokens is the number of tokens to predict 10 is the default and generally good for gpu, 2 performs better for cpu-only machines.
)

Embeddings

To generate text embeddings use create_embedding or embed. Note that you must pass embedding=True to the constructor upon model creation for these to work properly.

python

import llama_cpp

llm = llama_cpp.Llama(model_path="path/to/model.gguf", embedding=True)

embeddings = llm.create_embedding("Hello, world!")

# or create multiple embeddings at once

embeddings = llm.create_embedding(["Hello, world!", "Goodbye, world!"])

There are two primary notions of embeddings in a Transformer-style model: token level and sequence level. Sequence level embeddings are produced by "pooling" token level embeddings together, usually by averaging them or using the first token.

Models that are explicitly geared towards embeddings will usually return sequence level embeddings by default, one for each input string. Non-embedding models such as those designed for text generation will typically return only token level embeddings, one for each token in each sequence. Thus the dimensionality of the return type will be one higher for token level embeddings.

It is possible to control pooling behavior in some cases using the pooling_type flag on model creation. You can ensure token level embeddings from any model using LLAMA_POOLING_TYPE_NONE. The reverse, getting a generation oriented model to yield sequence level embeddings is currently not possible, but you can always do the pooling manually.

Adjusting the Context Window

The context window of the Llama models determines the maximum number of tokens that can be processed at once. By default, this is set to 512 tokens, but can be adjusted based on your requirements.

For instance, if you want to work with larger contexts, you can expand the context window by setting the n_ctx parameter when initializing the Llama object:

python

llm = Llama(model_path="./models/7B/llama-model.gguf", n_ctx=2048)

OpenAI Compatible Web Server

llama-cpp-python offers a web server which aims to act as a drop-in replacement for the OpenAI API.
This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).

To install the server package and get started:

bash

pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf

Similar to Hardware Acceleration section above, you can also install with GPU (cuBLAS) support like this:

bash

CMAKE_ARGS="-DGGML_CUDA=on" FORCE_CMAKE=1 pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35

Navigate to http://localhost:8000/docs to see the OpenAPI documentation.

To bind to 0.0.0.0 to enable remote connections, use python3 -m llama_cpp.server --host 0.0.0.0.
Similarly, to change the port (default is 8000), use --port.

You probably also want to set the prompt format. For chatml, use

bash

python3 -m llama_cpp.server --model models/7B/llama-model.gguf --chat_format chatml

That will format the prompt according to how model expects it. You can find the prompt format in the model card.
For possible options, see llama_cpp/llama_chat_format.py and look for lines starting with "@register_chat_format".

If you have huggingface-hub installed, you can also use the --hf_model_repo_id flag to load a model from the Hugging Face Hub.

bash

python3 -m llama_cpp.server --hf_model_repo_id lmstudio-community/Qwen3.5-0.8B-GGUF --model '*Q8_0.gguf'

Web Server Features

Docker image

A Docker image is available on GHCR. To run the server:

bash

docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/llama-model.gguf ghcr.io/abetlen/llama-cpp-python:latest

Docker on termux (requires root) is currently the only known way to run this on phones, see termux support issue

Low-level API

API Reference

The low-level API is a direct ctypes binding to the C API provided by llama.cpp.
The entire low-level API can be found in llama_cpp/llama_cpp.py and directly mirrors the C API in llama.h.

Below is a short example demonstrating how to use the low-level API to tokenize a prompt:

python

import llama_cpp
import ctypes
llama_cpp.llama_backend_init()  # Must be called once at the start of each program
model_params = llama_cpp.llama_model_default_params()
ctx_params = llama_cpp.llama_context_default_params()
prompt = b"Q: Name the planets in the solar system? A: "
# use bytes for char * params
model = llama_cpp.llama_model_load_from_file(b"./models/7b/llama-model.gguf", model_params)
ctx = llama_cpp.llama_init_from_model(model, ctx_params)
vocab = llama_cpp.llama_model_get_vocab(model)
max_tokens = ctx_params.n_ctx
# use ctypes arrays for array params
tokens = (llama_cpp.llama_token * int(max_tokens))()
n_tokens = llama_cpp.llama_tokenize(vocab, prompt, len(prompt), tokens, max_tokens, True, False)
llama_cpp.llama_free(ctx)
llama_cpp.llama_model_free(model)

Check out the examples folder for more examples of using the low-level API.

Documentation

Documentation is available via https://llama-cpp-python.readthedocs.io/.
If you find any issues with the documentation, please open an issue or submit a PR.

Development

This package is under active development and I welcome any contributions.
See CONTRIBUTING.md for contribution workflow, PR title, changelog, testing, and style guidelines.

To get started, clone the repository and install the package in editable / development mode:

bash

git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python

# Upgrade pip (required for editable mode)
pip install --upgrade pip

# Install with pip
pip install -e .

# install development tooling (tests, docs, ruff)
pip install -e '.[dev]'

# if you want to use the fastapi / openapi server
pip install -e '.[server]'

# to install all optional dependencies
pip install -e '.[all]'

# to clear the local build cache
make clean

Now try running the tests

bash

pytest

And check formatting / linting before opening a PR:

bash

python -m ruff check llama_cpp tests
python -m ruff format --check llama_cpp tests

# or use the Makefile targets
make lint
make format

There's a Makefile available with useful targets.
A typical workflow would look like this:

bash

make build
make test

You can also test out specific commits of llama.cpp by checking out the desired commit in the vendor/llama.cpp submodule and then running make clean and pip install -e . again. Any changes in the llama.h API will require
changes to the llama_cpp/llama_cpp.py file to match the new API (additional changes may be required elsewhere).

FAQ

Are there pre-built binaries / binary wheels available?

The recommended installation method is to install from source as described above.
The reason for this is that llama.cpp is built with compiler optimizations that are specific to your system.
Using pre-built binaries would require disabling these optimizations or supporting a large number of pre-built binaries for each platform.

That being said there are some pre-built binaries available through the Releases as well as some community provided wheels.

In the future, I would like to provide pre-built binaries and wheels for common platforms and I'm happy to accept any useful contributions in this area.
This is currently being tracked in #741

How does this compare to other Python bindings of `llama.cpp`?

I originally wrote this package for my own use with two goals in mind:

Provide a simple process to install llama.cpp and access the full C API in llama.h from Python
Provide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to use llama.cpp

Any contributions and changes to this package will be made with these goals in mind.

License

This project is licensed under the terms of the MIT license.

File: docker/README.md

Install Docker Server

[!IMPORTANT]
This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this README.md with a PR!

Install Docker Engine

Simple Dockerfiles for building the llama-cpp-python server with external model bin files

openblas_simple

A simple Dockerfile for non-GPU OpenBLAS, where the model is located outside the Docker image:

text

cd ./openblas_simple
docker build -t openblas_simple .
docker run --cap-add SYS_RESOURCE -e USE_MLOCK=0 -e MODEL=/var/model/<model-path> -v <model-root-path>:/var/model -t openblas_simple

where <model-root-path>/<model-path> is the full path to the model file on the Docker host system.

cuda_simple

[!WARNING]
Nvidia GPU CuBLAS support requires an Nvidia GPU with sufficient VRAM (approximately as much as the size in the table below) and Docker Nvidia support (see container-toolkit/install-guide)

A simple Dockerfile for CUDA-accelerated CuBLAS, where the model is located outside the Docker image:

text

cd ./cuda_simple
docker build -t cuda_simple .
docker run --gpus=all --cap-add SYS_RESOURCE -e USE_MLOCK=0 -e MODEL=/var/model/<model-path> -v <model-root-path>:/var/model -t cuda_simple

where <model-root-path>/<model-path> is the full path to the model file on the Docker host system.

"Open-Llama-in-a-box"

Download an Apache V2.0 licensed 3B params Open LLaMA model and install into a Docker image that runs an OpenBLAS-enabled llama-cpp-python server:

text

$ cd ./open_llama
./build.sh
./start.sh

Manually choose your own Llama model from Hugging Face

python3 ./hug_model.py -a TheBloke -t llama
You should now have a model in the current directory and model.bin symlinked to it for the subsequent Docker build and copy step. e.g.

text

docker $ ls -lh *.bin
-rw-rw-r-- 1 user user 4.8G May 23 18:30 <downloaded-model-file>q5_1.bin
lrwxrwxrwx 1 user user   24 May 23 18:30 model.bin -> <downloaded-model-file>q5_1.bin

[!NOTE]
Make sure you have enough disk space to download the model. As the model is then copied into the image you will need at least
TWICE as much disk space as the size of the model:

Model	Quantized size
3B	3 GB
7B	5 GB
13B	10 GB
33B	25 GB
65B	50 GB

[!NOTE]
If you want to pass or tune additional parameters, customise ./start_server.sh before running docker build ...

File: docs/install/macos.md

title: MacOS Install with Metal GPU

(1) Make sure you have xcode installed... at least the command line parts

text

# check the path of your xcode install 
xcode-select -p

# xcode installed returns
# /Applications/Xcode-beta.app/Contents/Developer

# if xcode is missing then install it... it takes ages;
xcode-select --install

(2) Install the conda version for MacOS that supports Metal GPU

text

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh

(3) Make a conda environment

text

conda create -n llama python=3.9.16
conda activate llama

(4) Install the LATEST llama-cpp-python...which happily supports MacOS Metal GPU as of version 0.1.62
(you needed xcode installed in order pip to build/compile the C++ code)

text

pip uninstall llama-cpp-python -y
CMAKE_ARGS="-DGGML_METAL=on" pip install -U llama-cpp-python --no-cache-dir
pip install 'llama-cpp-python[server]'

# you should now have llama-cpp-python v0.1.62 or higher installed
llama-cpp-python         0.1.68

(5) Download a v3 gguf v2 model

ggufv2
file name ends with Q4_0.gguf - indicating it is 4bit quantized, with quantisation method 0

https://huggingface.co/TheBloke/CodeLlama-7B-GGUF

(6) run the llama-cpp-python API server with MacOS Metal GPU support

text

# config your ggml model path
# make sure it is gguf v2
# make sure it is q4_0
export MODEL=[path to your llama.cpp ggml models]]/[ggml-model-name]]Q4_0.gguf
python3 -m llama_cpp.server --model $MODEL  --n_gpu_layers 1

Note: If you omit the --n_gpu_layers 1 then CPU will be used

File: docs/api-reference.md

title: API Reference

High Level API

High-level Python bindings for llama.cpp.

::: llama_cpp.Llama
options:
members:
- init
- tokenize
- detokenize
- reset
- eval
- sample
- generate
- create_embedding
- embed
- create_completion
- call
- create_chat_completion
- create_chat_completion_openai_v1
- set_cache
- save_state
- load_state
- token_bos
- token_eos
- from_pretrained
show_root_heading: true

::: llama_cpp.LlamaGrammar
options:
members:
- from_string
- from_json_schema

::: llama_cpp.LlamaCache
options:
show_root_heading: true

::: llama_cpp.LlamaState
options:
show_root_heading: true

::: llama_cpp.LogitsProcessor
options:
show_root_heading: true

::: llama_cpp.LogitsProcessorList
options:
show_root_heading: true

::: llama_cpp.StoppingCriteria
options:
show_root_heading: true

::: llama_cpp.StoppingCriteriaList
options:
show_root_heading: true

Low Level API

Low-level Python bindings for llama.cpp using Python's ctypes library.

::: llama_cpp.llama_cpp
options:
show_if_no_docstring: true
# filter only members starting with llama_
filters:
- "^llama_"

::: llama_cpp.llama_cpp
options:
show_if_no_docstring: true
show_root_heading: false
show_root_toc_entry: false
heading_level: 4
# filter only members starting with LLAMA_
filters:
- "^LLAMA_"

Misc

::: llama_cpp.llama_types
options:
show_if_no_docstring: true

File: docs/server.md

OpenAI Compatible Server

llama-cpp-python offers an OpenAI API compatible web server.

This web server can be used to serve local models and easily connect them to existing clients.

Setup

Installation

The server can be installed by running the following command:

bash

pip install llama-cpp-python[server]

Running the server

The server can then be started by running the following command:

bash

python3 -m llama_cpp.server --model <model_path>

You can also pass chat-template kwargs at model load time from the CLI:

bash

python3 -m llama_cpp.server \
  --model <model_path> \
  --chat_format chatml \
  --chat_template_kwargs '{"enable_thinking": true}'

Server options

For a full list of options, run:

bash

python3 -m llama_cpp.server --help

NOTE: All server options are also available as environment variables. For example, --model can be set by setting the MODEL environment variable.

Check out the server config reference below settings for more information on the available options.
CLI arguments and environment variables are available for all of the fields defined in ServerSettings and ModelSettings

Additionally the server supports configuration check out the configuration section for more information and examples.

Guides

Code Completion

llama-cpp-python supports code completion via GitHub Copilot.

NOTE: Without GPU acceleration this is unlikely to be fast enough to be usable.

You'll first need to download one of the available code completion models in GGUF format:

replit-code-v1_5-GGUF

Then you'll need to run the OpenAI compatible web server with a increased context size substantially for GitHub Copilot requests:

bash

python3 -m llama_cpp.server --model <model_path> --n_ctx 16192

Then just update your settings in .vscode/settings.json to point to your code completion server:

json

{
    // ...
    "github.copilot.advanced": {
        "debug.testOverrideProxyUrl": "http://<host>:",
        "debug.overrideProxyUrl": "http://<host>:"
    }
    // ...
}

Function Calling

llama-cpp-python supports structured function calling based on a JSON schema.
Function calling is completely compatible with the OpenAI function calling API and can be used by connecting with the official OpenAI Python client.

You'll first need to download one of the available function calling models in GGUF format:

functionary

Then when you run the server you'll need to also specify either functionary-v1 or functionary-v2 chat_format.

Note that since functionary requires a HF Tokenizer due to discrepancies between llama.cpp and HuggingFace's tokenizers as mentioned here, you will need to pass in the path to the tokenizer too. The tokenizer files are already included in the respective HF repositories hosting the gguf files.

bash

python3 -m llama_cpp.server --model <model_path_to_functionary_v2_model> --chat_format functionary-v2 --hf_pretrained_model_name_or_path <model_path_to_functionary_v2_tokenizer>

Check out this example notebook for a walkthrough of some interesting use cases for function calling.

Multimodal Models

llama-cpp-python supports the llava1.5 family of multi-modal models which allow the language model to
read information from both text and images.

You'll first need to download one of the available multi-modal models in GGUF format:

Then when you run the server you'll need to also specify the path to the clip model used for image embedding and the llava-1-5 chat_format

bash

python3 -m llama_cpp.server --model <model_path> --clip_model_path <clip_model_path> --chat_format llava-1-5

Then you can just use the OpenAI API as normal

python3

from openai import OpenAI

client = OpenAI(base_url="http://<host>:/v1", api_key="sk-xxx")
response = client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": ""
                    },
                },
                {"type": "text", "text": "What does the image say"},
            ],
        }
    ],
)
print(response)

Configuration and Multi-Model Support

The server supports configuration via a JSON config file that can be passed using the --config_file parameter or the CONFIG_FILE environment variable.

bash

python3 -m llama_cpp.server --config_file <config_file>

Config files support all of the server and model options supported by the cli and environment variables however instead of only a single model the config file can specify multiple models.

The server supports routing requests to multiple models based on the model parameter in the request which matches against the model_alias in the config file.

At the moment only a single model is loaded into memory at, the server will automatically load and unload models as needed.

For a single-model config, chat_template_kwargs can be set directly on the model entry:

json

{
    "models": [
        {
            "model": "models/Qwen3.5-0.8B/qwen3.5-0.8b-q8_0.gguf",
            "chat_format": "chatml",
            "chat_template_kwargs": {
                "enable_thinking": true
            }
        }
    ]
}

json

{
    "host": "0.0.0.0",
    "port": 8080,
    "models": [
        {
            "model": "models/OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
            "model_alias": "gpt-3.5-turbo",
            "chat_format": "chatml",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
            "model_alias": "gpt-4",
            "chat_format": "chatml",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/ggml_llava-v1.5-7b/ggml-model-q4_k.gguf",
            "model_alias": "gpt-4-vision-preview",
            "chat_format": "llava-1-5",
            "clip_model_path": "models/ggml_llava-v1.5-7b/mmproj-model-f16.gguf",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/mistral-7b-v0.1-GGUF/ggml-model-Q4_K.gguf",
            "model_alias": "text-davinci-003",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/replit-code-v1_5-3b-GGUF/replit-code-v1_5-3b.Q4_0.gguf",
            "model_alias": "copilot-codex",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 1024,
            "n_ctx": 9216
        }
    ]
}

The config file format is defined by the ConfigFileSettings class.

Server Options Reference

::: llama_cpp.server.settings.ConfigFileSettings
options:
show_if_no_docstring: true

::: llama_cpp.server.settings.ServerSettings
options:
show_if_no_docstring: true

::: llama_cpp.server.settings.ModelSettings
options:
show_if_no_docstring: true

File: examples/ray/README.md

This is an example of doing LLM inference with Ray and Ray Serve.

First, install the requirements:

bash

$ pip install -r requirements.txt

Deploy a GGUF model to Ray Serve with the following command:

bash

$ serve run llm:llm_builder model_path='../models/mistral-7b-instruct-v0.2.Q4_K_M.gguf'

This will start an API endpoint at http://localhost:8000/. You can query the model like this:

bash

$ curl -k -d '{"prompt": "tell me a joke", "max_tokens": 128}' -X POST http://localhost:8000

Docker & Self-Hosted Deployment Verified OCI

File: README.md Copy Chapter

Python Bindings for [`llama.cpp`](https://github.com/ggerganov/llama.cpp)

Installation Copy Chapter

Installation Configuration Copy Section

Supported Backends Copy Section

Windows Notes Copy Section

MacOS Notes Copy Section

Upgrading and Reinstalling Copy Section

High-level API Copy Chapter

Pulling models from Hugging Face Hub Copy Section

Chat Completion Copy Section

JSON and JSON Schema Mode Copy Section

JSON Mode

JSON Schema Mode

Function Calling Copy Section

Multi-modal Models Copy Section

Speculative Decoding Copy Section

Embeddings Copy Section

Adjusting the Context Window Copy Section

OpenAI Compatible Web Server Copy Chapter

Web Server Features Copy Section

Docker image Copy Chapter

Low-level API Copy Chapter

Documentation Copy Chapter

Development Copy Chapter

FAQ Copy Chapter

Are there pre-built binaries / binary wheels available? Copy Section

How does this compare to other Python bindings of `llama.cpp`? Copy Section

License Copy Chapter

File: docker/README.md Copy Chapter

Install Docker Server Copy Section

Simple Dockerfiles for building the llama-cpp-python server with external model bin files Copy Chapter

openblas_simple Copy Section

cuda_simple Copy Section

"Open-Llama-in-a-box" Copy Section

Manually choose your own Llama model from Hugging Face Copy Section

File: docs/install/macos.md Copy Chapter

title: MacOS Install with Metal GPU Copy Chapter

File: docs/api-reference.md Copy Chapter

title: API Reference Copy Chapter

High Level API Copy Chapter

Low Level API Copy Chapter

Misc Copy Chapter

File: docs/server.md Copy Chapter

OpenAI Compatible Server

Setup Copy Chapter

Installation Copy Section

Running the server Copy Section

Server options Copy Section

Guides Copy Chapter

Code Completion Copy Section

Function Calling Copy Section

Multimodal Models Copy Section

Configuration and Multi-Model Support Copy Chapter

Server Options Reference Copy Chapter

File: examples/ray/README.md Copy Chapter

Explore Related Technologies & Topics

File: README.md

Installation

Installation Configuration

Supported Backends

Windows Notes

MacOS Notes

Upgrading and Reinstalling

High-level API

Pulling models from Hugging Face Hub

Chat Completion

JSON and JSON Schema Mode

Function Calling

Multi-modal Models

Speculative Decoding

Embeddings

Adjusting the Context Window

OpenAI Compatible Web Server

Web Server Features

Docker image

Low-level API

Documentation

Development

FAQ

Are there pre-built binaries / binary wheels available?

How does this compare to other Python bindings of `llama.cpp`?

License

File: docker/README.md

Install Docker Server

Simple Dockerfiles for building the llama-cpp-python server with external model bin files

openblas_simple

cuda_simple

"Open-Llama-in-a-box"

Manually choose your own Llama model from Hugging Face

File: docs/install/macos.md

title: MacOS Install with Metal GPU

File: docs/api-reference.md

title: API Reference

High Level API

Low Level API

Misc

File: docs/server.md

Setup

Installation

Running the server

Server options

Guides

Code Completion

Function Calling

Multimodal Models

Configuration and Multi-Model Support

Server Options Reference

File: examples/ray/README.md