llamafile Documentation

File: README.md Copy Chapter

llamafile

llamafile lets you distribute and run LLMs with a single file.

llamafile is a Mozilla Builders project (see its announcement blog post), now revamped by Mozilla.ai.

Our goal is to make open LLMs much more
accessible to both developers and end users. We're doing that by
combining llama.cpp with Cosmopolitan Libc into one
framework that collapses all the complexity of LLMs down to
a single-file executable (called a "llamafile") that runs
locally on most operating systems and CPU architectures, with no installation.

llamafile also includes whisperfile, a single-file speech-to-text tool built on whisper.cpp and the same Cosmopolitan packaging. It supports transcription and translation of audio files across all the same platforms, with no installation required.

v0.10.* Copy Chapter

llamafile versions starting from 0.10.0 use a new build system, aimed at keeping our code more easily
aligned with the latest versions of llama.cpp. This means they support more recent models and functionalities,
but at the same time they might be missing some of
the features you were accustomed to (check out this doc for a high-level description of what has been done). If you liked
the "classic experience" more, you will always be able to access the previous versions from our
releases page. Our pre-built llamafiles always
show which version of the server they have been bundled with (0.9.* example, 0.10.* example), so you will always know
which version of the software you are downloading.

We want to hear from you!
Whether you are a new user or a long-time fan, please share what you find most valuable about llamafile and what would make it more useful for you.
Read more via the blog and add your voice to the discussion here.

Quick Start Copy Chapter

Download and run your first llamafile in minutes:

# Download an example model (Qwen3.5 0.8B)
curl -LO https://huggingface.co/mozilla-ai/llamafile_0.10/resolve/main/Qwen3.5-0.8B-Q8_0.llamafile

# Make it executable (macOS/Linux/BSD)
chmod +x Qwen3.5-0.8B-Q8_0.llamafile

# Run it
./Qwen3.5-0.8B-Q8_0.llamafile

We chose this model because that's the smallest one we have
built a llamafile for, so most likely to work out-of-the-box for you.
If you have powerful hardware and/or GPUs, feel free to choose
larger and more expressive models which should provide more accurate
responses.

Windows users: Rename the file to add .exe extension before running.

Note - Only executables under 4GB can run on Windows, so any llamafile above 4GB won't work. Download the llamafile binary and run it with any external weights/models(GGUF).

Documentation Copy Chapter

Check the full documentation at docs.mozilla.ai/llamafile, or directly jump into one of the following subsections:

• Quickstart
• Pre-built llamafiles
• Running a llamafile
• Creating llamafiles
• Source installation
• Technical details
• Supported Systems
• Troubleshooting
• Whisperfile

Licensing Copy Chapter

While the llamafile project is Apache 2.0-licensed, our changes
to llama.cpp and whisper.cpp are licensed under MIT (just like the projects
themselves) so as to remain compatible and upstreamable in the future,
should that be desired.

The llamafile logo on this page was generated with the assistance of DALL·E 3.

File: docs/creating_llamafiles.md Copy Chapter

Creating a llamafile

A llamafile bundles the llamafile executable, model weights, and a set of
default arguments into a single self-contained file using the
APE (Actually Portable Executable) format,
which supports ZIP as a container for extra data. If you have already
downloaded a llamafile, you can inspect its contents with
unzip -vl <filename.llamafile> (or on Windows, rename it to .zip and
open it in your ZIP GUI).

Prerequisites Copy Chapter

llamafile uses zipalign to bundle files
into the executable. It is included as a git submodule and built alongside
llamafile, so if you have already compiled llamafile you have the zipalign
executable in the o//third_party/zipalign folder. To build it on its own:

make o//third_party/zipalign

[!NOTE]
The zipalign tool referenced here is not the
Android zipalign. See the
GitHub repo above for an in-depth description and up-to-date code.

What you need Copy Chapter

• The llamafile executable — download a prebuilt binary from the
  releases page, or build
  from source following
  these instructions.

• Model weights in GGUF format — download from Hugging Face
  (search here), or use weights
  already on disk from
  another application.

• A .args file — specifies default arguments (at minimum, the model
  path so it loads automatically).

Examples Copy Chapter

TUI, text-only Copy Section

Let's see how this works in practice with a simple, text-only language
model, e.g. Qwen3-0.6B:

• Search for the model weights in GGUF format
  (for the sake of this example we'll download these with Q8 quantization)
• Create a file named .args with the following content:

-m
/zip/Qwen3-0.6B-Q8_0.gguf
-fa
on
--temp
0.6
--top-k
20
--top-p
0.95
--min-p
0
--presence-penalty
1.5
-c
40960
-n
32768
--no-context-shift
--no-mmap
...

[!NOTE]
There is one argument per line. Most arguments are optional — the model
name is the only required one (the above replicates the parameters suggested
here). The /zip/ path
prefix is required whenever referencing a file packaged inside the llamafile.
The ... token is replaced with any additional CLI arguments the user passes
at runtime.

• Copy the llamafile executable and run zipalign to embed the weights and args:

cp o//llamafile/llamafile Qwen3-0.6B-Q8.llamafile

o//third_party/zipalign/zipalign -j0 \
  Qwen3-0.6B-Q8.llamafile \
  Qwen3-0.6B-Q8_0.gguf \
  .args

./Qwen3-0.6B-Q8.llamafile

Congratulations, you've just made your own LLM executable that's easy to
share with your friends!

Your new llamafile will start loading the Qwen model in the TUI. You can also
run it as a web server with:

./Qwen3-0.6B-Q8.llamafile --server

Server, multimodal Copy Section

Now, let us build another llamafile running a multimodal model served
via HTTP. If you want to be able to just say:

./llava.llamafile

...and have it run the web server without having to specify arguments,
embed both the weights and the following .args file
(weights used in this example are downloaded from here):

-m
/zip/llava-v1.6-mistral-7b.Q8_0.gguf
--mmproj
/zip/mmproj-model-f16.gguf
--server
--host
0.0.0.0
-ngl
9999
--no-mmap
...

Next, add both the weights and the argument file to the executable:

cp o//llamafile/llamafile llava.llamafile

o//third_party/zipalign/zipalign -j0 \
  llava.llamafile \
  llava-v1.6-mistral-7b.Q8_0.gguf \
  mmproj-model-f16.gguf \
  .args

./llava.llamafile

Distribution Copy Chapter

One good way to share a llamafile with your friends is by posting it on
Hugging Face. If you do that, then it's recommended that you mention in
your Hugging Face commit message what git revision or released version
of llamafile you used when building your llamafile. That way everyone
online will be able verify the provenance of its executable content. If
you've made changes to the llama.cpp or cosmopolitan source code, then
the Apache 2.0 license requires you to explain what changed. One way you
can do that is by embedding a notice in your llamafile using zipalign
that describes the changes, and mention it in your Hugging Face commit.

File: docs/index.md Copy Chapter

llamafile

llamafile lets you distribute and run LLMs with a single file.

llamafile is a Mozilla Builders project (see its announcement blog post), now revamped by Mozilla.ai.

Our goal is to make open LLMs much more
accessible to both developers and end users. We're doing that by
combining llama.cpp with Cosmopolitan Libc into one
framework that collapses all the complexity of LLMs down to
a single-file executable (called a "llamafile") that runs
locally on most operating systems and CPU archiectures, with no installation.

llamafile also includes whisperfile, a single-file speech-to-text tool built on whisper.cpp and the same Cosmopolitan packaging. It supports transcription and translation of audio files across all the same platforms, with no installation required.

v0.10.* Copy Chapter

llamafile versions starting from 0.10.0 use a new build system, aimed at keeping our code more easily
aligned with the latest versions of llama.cpp. This means they support more recent models and functionalities,
but at the same time they might be missing some of
the features you were accustomed to (check out this doc for a high-level description of what has been done). If you liked
the "classic experience" more, you will always be able to access the previous versions from our
releases page. Our pre-built llamafiles always
show which version of the server they have been bundled with (0.9.* example, 0.10.* example), so you will always know
which version of the software you are downloading.

We want to hear from you!
Whether you are a new user or a long-time fan, please share what you find most valuable about llamafile and what would make it more useful for you.
Read more via the blog and add your voice to the discussion here.

How llamafile works Copy Chapter

A llamafile is an executable LLM that you can run on your own
computer. It contains the weights for a given open LLM, as well
as everything needed to actually run that model on your computer.
There's nothing to install or configure (with a few caveats, discussed
in subsequent sections of this document).

This is all accomplished by combining llama.cpp with Cosmopolitan Libc,
which provides some useful capabilities:

1. llamafiles can run on multiple CPU microarchitectures. We
   added runtime dispatching to llama.cpp that lets new Intel systems use
   modern CPU features without trading away support for older computers.

2. llamafiles can run on multiple CPU architectures. We do
   that by concatenating AMD64 and ARM64 builds with a shell script that
   launches the appropriate one. Our file format is compatible with WIN32
   and most UNIX shells. It's also able to be easily converted (by either
   you or your users) to the platform-native format, whenever required.

3. llamafiles can run on six OSes (macOS, Windows, Linux,
   FreeBSD, OpenBSD, and NetBSD). If you make your own llama files, you'll
   only need to build your code once, using a Linux-style toolchain. The
   GCC-based compiler we provide is itself an Actually Portable Executable,
   so you can build your software for all six OSes from the comfort of
   whichever one you prefer most for development.

4. The weights for an LLM can be embedded within the llamafile.
   We added support for PKZIP to the GGML library. This lets uncompressed
   weights be mapped directly into memory, similar to a self-extracting
   archive. It enables quantized weights distributed online to be prefixed
   with a compatible version of the llama.cpp software, thereby ensuring
   its originally observed behaviors can be reproduced indefinitely.

5. Finally, with the tools included in this project you can create your
   own llamafiles, using any compatible model weights you want. You can
   then distribute these llamafiles to other people, who can easily make
   use of them regardless of what kind of computer they have.

Licensing Copy Chapter

While the llamafile project is Apache 2.0-licensed, our changes
to llama.cpp are licensed under MIT (just like the llama.cpp project
itself) so as to remain compatible and upstreamable in the future,
should that be desired.

The llamafile logo on this page was generated with the assistance of DALL·E 3.

File: docs/quickstart.md Copy Chapter

Getting Started with llamafile

The easiest way to try it for yourself is to download our example llamafile
for the Qwen3.5 model (license:
Apache 2.0).
Qwen3.5 is a recent LLM that can do more than just chat; you can also upload
images and ask it questions about them. With llamafile, this all happens
locally: no data ever leaves your computer.

NOTE: we chose this model because that's the smallest one we have
built a llamafile for, so most likely to work out-of-the-box for you.
Please let us know if you are still having issues with that! If, on the
other hand, you have powerful hardware and/or GPUs, feel free to choose
larger and more expressive models which should provide more accurate
responses.

1. Download Qwen3.5-0.8B-Q8_0.llamafile (1.77 GB).

2. Open your computer's terminal.

   • If you're using macOS, Linux, or BSD, you'll need to grant permission
     for your computer to execute this new file. (You only need to do this
     once.)

     sh Copy

     chmod +x Qwen3.5-0.8B-Q8_0.llamafile

   • If you're on Windows, rename the file by adding ".exe" on the end.

3. Run the llamafile. e.g.:

   sh Copy

   ./Qwen3.5-0.8B-Q8_0.llamafile

4. A chat interface will open in the terminal window. That's it: you can immediately
   start writing. You can also upload an image by using the /upload command and specifying the path to the image, or write
   /help to see the available commands).

5. Note that when llamafile is running, you can also chat with it using
   llama.cpp's Web UI: just open a
   browser window and connect to http://localhost:8080/.

6. When you're done chatting, Control-C to shut down llamafile.

Having trouble? See the Troubleshooting page.

JSON API Quickstart Copy Chapter

As llamafile relies on llama.cpp for serving models, it comes with all its
features. When it is started, in addition to hosting a web UI chat server at
http://127.0.0.1:8080/, it also exposes an endpoint compatible with
OpenAI API
and Anthropic's Messages API.
For further details on what fields and endpoints are available, refer to the
APIs documentation and llama.cpp server's
README.

Curl API Client Example

The simplest way to get started using the API is to copy and paste the
following curl command into your terminal.

curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
  "model": "LLaMA_CPP",
  "messages": [
      {
          "role": "system",
          "content": "You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
      },
      {
          "role": "user",
          "content": "Write a limerick about python exceptions"
      }
    ]
}' | python3 -c '
import json
import sys
json.dump(json.load(sys.stdin), sys.stdout, indent=2)
print()
'

The response that's printed should look like the following:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "In the world of Python, where magic breaks and errors occur,\nA script fails when it should not have failed.\nWith a `KeyError`, I can't access the key,\nSo I tell you to use the `except` clause!"
      }
    }
  ],
  "created": 1773659260,
  "model": "Qwen3.5-0.8B-Q8_0.gguf",
  "system_fingerprint": "b1773565177-7f5ee5496",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 52,
    "prompt_tokens": 49,
    "total_tokens": 101
  },
  "id": "chatcmpl-KOqwN6C0oRzINGZuFqZ95bU1iPfc6RFO",
  "timings": {
    "cache_n": 0,
    "prompt_n": 49,
    "prompt_ms": 54.944,
    "prompt_per_token_ms": 1.1213061224489795,
    "prompt_per_second": 891.8171228887594,
    "predicted_n": 52,
    "predicted_ms": 405.856,
    "predicted_per_token_ms": 7.804923076923076,
    "predicted_per_second": 128.1242608215722
  }
}

Python API Client example

If you've already developed your software using the openai Python
package (that's published by OpenAI)
then you should be able to port your app to talk to llamafile instead,
by making a few changes to base_url and api_key. This example
assumes you've run pip3 install openai to install OpenAI's client
software, which is required by this example. Their package is just a
simple Python wrapper around the OpenAI API interface, which can be
implemented by any server.

#!/usr/bin/env python3
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
    api_key = "sk-no-key-required"
)
completion = client.chat.completions.create(
    model="LLaMA_CPP",
    messages=[
        {"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
        {"role": "user", "content": "Write a limerick about python exceptions"}
    ]
)
print(completion.choices[0].message)

The above code will return a Python object like this:

ChatCompletionMessage(content="A script that crashes like a ghost,\nWhen it tries to solve the problem deep and fast.\nThe error message pops up in a bright light,\nAnd tells us what's wrong when we try to fix it.", refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None)

Using llamafile with external weights Copy Chapter

Even though our pre-built llamafiles have the weights built-in, you don't
have to use llamafile that way. Instead, you can download just the
llamafile software (without any weights included) from our releases page.
You can then use it alongside any external weights you may have on hand.
External weights are particularly useful for Windows users because they
enable you to work around Windows' 4GB executable file size limit.

For Windows users, here's an example for the gpt-oss LLM (whose size is >12GB):

curl -L -o llamafile.exe https://huggingface.co/mozilla-ai/llamafile_0.10/resolve/main/llamafile_0.10.1
curl -L -o gpt-oss.gguf https://huggingface.co/unsloth/gpt-oss-20b-GGUF/resolve/main/gpt-oss-20b-Q5_K_S.gguf
./llamafile.exe -m gpt-oss.gguf

Windows users may need to change ./llamafile.exe to .\llamafile.exe when running the above command.

Which release file do I download? Copy Section

On the releases page each
release ships several independent single-file programs. They are not
bundled together inside the llamafile executable: if you want to download all of them
at once, choose the zip download; otherwise, pick the one that matches the task you
want to perform:

`llamafile` vs `llamafile-thin`

Both files contain the same executable code. The difference is whether pre-built GPU backend
libraries are bundled together with it:

• llamafile-thin is the base binary, so it is the smaller download. It
  seamlessly runs on different operating systems and CPU architectures. To run on GPU,
  it requires a matching backend library to be available on your system. On Apple Silicon this
  is fully transparent: if the system does not have the required library, llamafile compiles
  one on the fly. On the other systems, you can build GPU libraries yourself with llamafile/cuda.sh
  (NVIDIA), llamafile/rocm.sh (AMD), or llamafile/vulkan.sh (NVIDIA + AMD + others).
  Each library requires the respective GPU toolchain to be installed before you run a build.
  As an alternative, you can find libraries for the latest release on our
  HuggingFace repo.
• llamafile is that same binary with ggml-cuda and ggml-vulkan
  bundled, as x86_64 Linux (.so) and Windows (.dll) artifacts (a
  single set of libraries, with no separate ARM build). Those backends work
  with no build step of your own — you only need a driver that provides the
  corresponding runtime API. The bundled libraries are what makes this file
  larger.

Note that ROCm support is still experimental and thus not automatically bundled in either file.
AMD users who do not want to use the Vulkan backend have to build ggml-rocm with
llamafile/rocm.sh, or download the pre-built ggml-rocm dylibs from llamafile's HuggingFace
repo, regardless of which llamafile executable they are running.

The bundled libraries are x86_64 Linux and Windows artifacts only. On any
other platform (macOS, or a non-x86_64 architecture such as an aarch64/ARM64
Linux host, even one with an NVIDIA GPU) neither file carries an applicable
pre-built backend, so the two downloads behave the same there and GPU support
comes from the usual runtime path instead.

Use llamafile unless you want the smallest download and only need CPU inference,
you have an Apple Silicon Mac, or have already set up your own GPU backend libraries.

Running llamafile with models downloaded by third-party applications Copy Chapter

This section answers the question "I already have a model downloaded locally by application X, can I use it with llamafile?". The general answer is "yes, as long as those models are locally stored in GGUF format" but its implementation can be more or less hacky depending on the application. A few examples (tested on a Mac) follow.

LM Studio Copy Section

LM Studio stores downloaded models in ~/.cache/lm-studio/models/lmstudio-community, in subdirectories with the same name of the models, minus their quantization level. So if you have downloaded e.g. the gpt-oss-20b-MXFP4.gguf file, it will be stored in ~/.cache/lm-studio/models/lmstudio-community/gpt-oss-20b-GGUF/ and you can run llamafile as follows:

llamafile -m ~/.cache/lm-studio/models/lmstudio-community/gpt-oss-20b-GGUF/gpt-oss-20b-MXFP4.gguf

Ollama Copy Section

When you download a new model with ollama, all its metadata will be stored in a manifest file under ~/.ollama/models/manifests/registry.ollama.ai/library/. The directory and manifest file name are the model name as returned by ollama list. For instance, for llama3:latest the manifest file will be named .ollama/models/manifests/registry.ollama.ai/library/llama3/latest.

The manifest maps each file related to the model (e.g. GGUF weights, license, prompt template, etc) to a sha256 digest. The digest corresponding to the element whose mediaType is application/vnd.ollama.image.model is the one referring to the model's GGUF file.

Each sha256 digest is also used as a filename in the ~/.ollama/models/blobs directory (if you look into that directory you'll see only those sha256-* filenames). This means you can directly run llamafile by passing the sha256 digest as the model filename. So if e.g. the llama3:latest GGUF file digest is sha256-00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29, you can run llamafile as follows:

cd ~/.ollama/models/blobs
llamafile -m sha256-00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29

Note that Ollama's GGUF weights do not always work with llama.cpp (see e.g. here),
and as llamafile relies on llama.cpp this trick might not always work for you.

File: docs/security.md Copy Chapter

Security

llamafile can sandbox itself with two Cosmopolitan
Libc primitives:

• pledge() restricts which system
  calls the process may make. On Linux it installs a SECCOMP BPF filter;
  on OpenBSD it calls the native pledge(2). On by default.
• unveil() restricts which parts of
  the filesystem the process can see. On Linux it uses the
  Landlock LSM
  (kernel 5.13+); on OpenBSD it calls the native unveil(2). Opt-in,
  via --confine-reads.

Neither needs any kernel configuration or privileges. SECCOMP filtering
requires Linux 3.5+ on x86-64 and 5.13+ (for Landlock) on either
architecture; on older kernels, and on macOS, Windows, and other BSDs, the
calls are no-ops and llamafile logs that sandboxing is unavailable and keeps
running. The pledge() sandbox can be turned off entirely with --unsecure.

The default: pledge() Copy Chapter

The llamafile --server process runs under stdio anet rpath on Linux
(stdio inet rpath on OpenBSD). After startup this means:

• No outbound network. anet allows accept() but not connect(), so
  the only networking the server can do is answer connections it received.
  If the server is ever compromised (say, through a bug in the GGUF parser
  or an HTTP handler), this is the single most valuable restriction: it cuts
  the attacker's ability to exfiltrate anything over the network.
• No writing, creating, deleting, executing, or forking. A compromised
  server can't modify your files, drop a payload, or launch a program.
  (--slot-save-path and a prompt cache add write access to their
  directories only.)
• Reads are allowed (rpath) — anywhere the process could already read.
  This is deliberate: a server routinely opens files whose paths it only
  learns at request time (multimodal media, static assets), which cannot be
  known when a path allow-list would have to be locked. If you want to
  restrict which files are readable, see --confine-reads below.

--cli runs under stdio rpath tty and --chat under
stdio rpath wpath cpath tty — neither has any network access at all.

Opt-in read confinement: `--confine-reads` Copy Chapter

Passing --confine-reads to the server additionally applies unveil(),
confining reads to the executable and the directories holding the weights
(the model and its shards, --mmproj, LoRA adapters, a draft model, control
vectors, --media-path, a static --path web root) plus the two
name-resolution files a non-numeric --host needs. The rest of the
filesystem — /etc/passwd, your SSH keys, other users' files — becomes
invisible to the server even though it is world-readable.

Two things to know before relying on it:

• It confines directories, not individual files. The model's parent
  directory is unveiled (so multi-part GGUF shards beside it load), which
  means anything else in that directory is readable too. Put weights in a
  dedicated directory if you don't want their neighbours exposed.
• It's incompatible with reading files by arbitrary path at request time.
  Because the readable set is locked at startup, a request that references a
  file outside the unveiled directories will be denied. The built-in flows
  (media under --media-path, static files under --path) are covered;
  bespoke setups that read elsewhere are not.
• It requires a filesystem Landlock can govern. On the handful it cannot
  (some network mounts, virtiofs, 9p) llamafile keeps the pledge() sandbox
  but skips confinement rather than refusing to load the model, and logs
  that it did so.

When the sandbox is relaxed or skipped Copy Chapter

llamafile logs a notice in each case:

• Outbound features (--rpc for distributed inference, server-side
  tools, the MCP proxy) legitimately need to connect() out, so the
  networking promise is relaxed from anet to inet for them. Writes and
  exec stay blocked.
• GPU mode gets no sandbox. GPU backends are loaded dynamically and their
  drivers need device access (ioctl, /dev/*) that no promise set covers,
  so the sandbox is skipped whenever a GPU backend is loaded. This is the
  common production case — a GPU server is not sandboxed. Pass
  --gpu disable to force CPU inference with the sandbox.
• Combined mode (the default llamafile -m model.gguf, TUI + server in
  one process) hosts an in-process HTTP client that must connect() to the
  server, so it's skipped. Run llamafile --server for the sandboxed server.
• Model downloads (-hf, --model-url) happen before the sandbox is
  installed; once the download completes the process is sandboxed normally.

A note on the penalty mode Copy Chapter

On Linux, sandbox violations are configured to return EPERM (permission
denied) rather than killing the process, so a blocked syscall surfaces as an
ordinary I/O error. If a server refuses to start with a pledge failed
error (an exotic kernel or an outer seccomp policy), --unsecure disables
the sandbox. OpenBSD's native pledge(2) always terminates a violating
process with SIGABRT instead; that behavior is not configurable there.

Verifying the sandbox Copy Chapter

Unit tests exercise every promise set llamafile uses, assert that allowed
operations work while blocked ones fail, and check that unveil() confines
reads to the unveiled directory (run on Linux):

.cosmocc/4.0.2/bin/make o//tests/sandbox_test && o//tests/sandbox_test

Integration tests verify the running server end-to-end — every thread of the
server process carries the SECCOMP filter, completions still work inside the
sandbox, a bundled /zip/ llamafile loads under it, --confine-reads
confines while the default does not, and --unsecure really disables it:

cd tests/integration
./run_tests.sh --executable ../../o/llamafile/llamafile \
    --model ../../models/TinyLLama-v0.1-5M-F16.gguf -m sandbox

You can also check a running server by hand:

llamafile --server -m model.gguf &
grep Seccomp /proc/$!/status   # "Seccomp: 2" means the filter is active

Caveats Copy Chapter

Your llamafile is able to protect itself against the outside world, but that
doesn't mean you're protected from llamafile. Sandboxing is self-imposed. If
you obtained your llamafile from an untrusted source then its author could
have simply modified it to not do that. In that case, you can run the
untrusted llamafile inside another sandbox, such as a virtual machine, to
make sure it behaves how you expect.

File: docs/source_installation.md Copy Chapter

Developing on llamafile requires a modern version of the GNU make
command (called gmake on some systems), sha256sum (otherwise cc
will be used to build it), wget (or curl), and unzip available at
https://cosmo.zip/pub/cosmos/bin/.
Windows users need cosmos bash shell too.

Dependency Setup Copy Section

Some dependencies are managed as git submodules with llamafile-specific patches.
Before building, you need to initialize and configure these dependencies:

make setup

The patches modify code in the git submodules. These modifications remain as local
changes in the submodule working directories.

make setup also downloads the Cosmopolitan
C compiler for you, saving it under the .cosmocc directory.

Building Copy Section

.cosmocc/4.0.2/bin/make -j8
sudo .cosmocc/4.0.2/bin/make install PREFIX=/usr/local

Build outputs will appear in the ./o directory, e.g.:

• ./o/llama.cpp/server/llama-server: the original llama.cpp inference server, compiled with cosmocc
• o/llamafile/llamafile: the llamafile executable, running both as a TUI and a server (with the --server flag)
• o/third_party/zipalign/zipalign: the zipalign tool used to bundle llamafile executable, model weights, and default args into llamafiles

NOTE: Calling make should automatically run cosmocc's make when required.
If that does not happen for any reason, you can still directly run the one provided
by cosmocc: .cosmocc/4.0.2/bin/make.

Testing Copy Section

Optionally, you can verify the build with:

make check

This runs our unit tests to ensure everything is built correctly.

Some integration tests in tests/integration are available to test llamafile
with real models. Check the README to learn how to run them.

Running llamafile Copy Section

After the build, you can run llamafile as:

./o/llamafile/llamafile --model <gguf_model>

or just the llama.cpp server as:

./o/llamafile/llamafile --model <gguf_model> --server

or the llamafile CLI command as:

./o/llamafile/llamafile --model <gguf_model> --cli -p "Hello world"

Documentation Copy Chapter

There's a manual page for each of the llamafile programs installed when you
run sudo make install. Most commands will also display that information when
passing the --help flag.

File: docs/support.md Copy Chapter

Supported OSes Copy Chapter

llamafile supports the following operating systems, which require a minimum
stock install:

• Linux 2.6.18+ (i.e. every distro since RHEL5 c. 2007)
• Darwin (macOS) 23.1.0+ [1] (GPU is only supported on ARM64)
• Windows 10+ (AMD64 only)
• FreeBSD 13+
• NetBSD 9.2+ (AMD64 only)
• OpenBSD 7.0 to 7.4 (AMD64 only)

On Windows, llamafile runs as a native portable executable. On UNIX
systems, llamafile extracts a small loader program named ape to
$TMPDIR/.ape-1.10 which is used to map your model into memory.

[1] Darwin kernel versions 15.6+ should be supported, but we currently
have no way of testing that.

Supported CPUs Copy Chapter

llamafile supports the following CPUs:

• AMD64 microprocessors must have AVX. Otherwise llamafile will
  print an error and refuse to run. This means that if you have an Intel
  CPU, it needs to be Intel Core or newer (circa 2006+), and if you have
  an AMD CPU, then it needs to be K8 or newer (circa 2003+). Support for
  AVX512, AVX2, FMA, F16C, and VNNI are conditionally enabled at runtime
  if you have a newer CPU. For example, Zen4 has very good AVX512 that
  can speed up BF16 llamafiles.

• ARM64 microprocessors must have ARMv8a+. This means everything
  from Apple Silicon to 64-bit Raspberry Pis will work, provided your
  weights fit into memory.

GPU support Copy Chapter

llamafile ships GPU acceleration for Apple Metal, NVIDIA, AMD, and Vulkan.
There is no Intel oneAPI/SYCL backend, but Vulkan covers a lot of the
hardware that CUDA and ROCm do not. On hardware outside the table below,
llamafile runs on the CPU.

CUDA, ROCm, and Vulkan dynamic libraries are loaded as follows. The executable looks
for prebuilt ggml-(cuda|rocm|vulkan).(so|dll) files in this order:

• in the same directory (deliberately first, so a hand-built DSO overrides everything else)
• in the llamafile bundle
• in ~/.llamafile/v/LLAMAFILE_VERSION
• in $HOME

Check out Building the GPU libraries for more info on how these libraries are built.
On macOS, Vulkan runs through MoltenVK; Apple Silicon users normally want the built-in Metal
backend instead. If any library loads but reports no usable device, llamafile skips it and continues
with the next backend rather than failing.

The 0.10.* series has not been tested on every GPU and platform yet, so
treat the AMD and Windows paths in particular as best-effort.

IQ-quantized models on NVIDIA GPUs Copy Section

The CUDA library bundled in our releases is size-optimized and leaves out
the IQ-quant kernels (the IQ1_*, IQ2_*, IQ3_*, IQ4_* quantizations).
When you offload an IQ-quantized model to an NVIDIA GPU, llamafile keeps just
those layers on the CPU automatically — the output is correct, those specific
layers simply don't get GPU acceleration. Every other quantization (Q*_*,
MXFP4, NVFP4, F16, BF16, …) runs fully on the GPU. The Apple Metal and
AMD (ROCm) builds are not size-optimized and include full IQ-quant GPU support.
For full IQ acceleration on NVIDIA, build or supply a full (non-minimized) CUDA
library — see Building the GPU libraries.

GPU on MacOS ARM64 is supported by compiling a small module using the
Xcode Command Line Tools, which need to be installed. This is a one time
cost that happens the first time you run your llamafile. The DSO built
by llamafile is stored in $TMPDIR/.llamafile or $HOME/.llamafile.
Offloading to GPU is enabled by default when a Metal GPU is present.
This can be disabled by passing -ngl 0 or --gpu disable to force
llamafile to perform CPU inference.

Owners of NVIDIA and AMD graphics cards need to pass the -ngl 999 flag
to enable maximum offloading. If multiple GPUs are present then the work
will be divided evenly among them by default, so you can load larger
models. Multiple GPU support may be broken on AMD Radeon systems. If
that happens to you, then use export HIP_VISIBLE_DEVICES=0 which
forces llamafile to only use the first GPU.

Windows users are encouraged to use our release binaries, because they
contain prebuilt DLLs for both NVIDIA and AMD graphics cards, which only
depend on the graphics driver being installed. If llamafile detects that
NVIDIA's CUDA SDK or AMD's ROCm HIP SDK are installed, then llamafile
will try to build a faster DLL that uses cuBLAS or rocBLAS. In order for
llamafile to successfully build a cuBLAS module, it needs to be run on
the x64 MSVC command prompt. You can use CUDA via WSL by enabling
Nvidia CUDA on
WSL
and running your llamafiles inside of WSL. Using WSL has the added
benefit of letting you run llamafiles greater than 4GB on Windows.

On Linux, NVIDIA users will need to install the CUDA SDK (ideally using
the shell script installer) and ROCm users need to install the HIP SDK.
They're detected by looking to see if nvcc or hipcc are on the PATH.
For AMD systems, make sure the executable directory containing hipcc is
on your PATH and that it can be executed by your user; a hipcc: Permission denied message means ROCm was found but can't be run, so GPU
offload will not be available until the SDK permissions or installation
are fixed. Running with --gpu amd or --gpu nvidia is a useful way to
turn an otherwise quiet CPU fallback into an explicit startup error while
you diagnose the toolchain.

If you have both an AMD GPU and an NVIDIA GPU in your machine, then
you may need to qualify which one you want used, by passing either
--gpu amd or --gpu nvidia.

In the event that GPU support couldn't be compiled and dynamically
linked on the fly for any reason, llamafile will fall back to CPU
inference.

Verifying GPU acceleration Copy Section

Because llamafile silently falls back to the CPU when a GPU backend can't
be set up, it isn't always obvious whether offloading actually happened.
To check:

• Pass -ngl 999 on NVIDIA and AMD to request maximum offloading (Metal
  offloads by default).
• Force a specific backend with --gpu nvidia, --gpu amd, or
  --gpu vulkan. This turns an otherwise quiet CPU fallback into an
  explicit startup error, which makes a missing or misconfigured
  CUDA/ROCm/Vulkan installation easy to spot.
• Watch the startup logs for the messages about building and loading the
  GPU module. If you don't see them, llamafile is running on the CPU.

NOTE that the 0.10.* build of llamafile has not been tested on all
GPUs/platforms yet, so we welcome your feedback both whether there are
any issues or if everything runs smoothly on your specific setup!

File: docs/technical_details.md Copy Chapter

Here is a succinct overview of the tricks we used to create the fattest
executable format ever. The long story short is llamafile is a shell
script that launches itself and runs inference on embedded weights in
milliseconds without needing to be copied or installed. What makes that
possible is mmap(). Both the llama.cpp executable and the weights are
concatenated onto the shell script. A tiny loader program is then
extracted by the shell script, which maps the executable into memory.
The llama.cpp executable then opens the shell script again as a file,
and calls mmap() again to pull the weights into memory and make them
directly accessible to both the CPU and GPU.

ZIP weights embedding Copy Section

The trick to embedding weights inside llama.cpp executables is to ensure
the local file is aligned on a page size boundary. That way, assuming
the zip file is uncompressed, once it's mmap()'d into memory we can pass
pointers directly to GPUs like Apple Metal, which require that data be
page size aligned. Since no existing ZIP archiving tool has an alignment
flag, we had to write about 500 lines of code to
insert the ZIP files ourselves. However, once there, every existing ZIP
program should be able to read them, provided they support ZIP64. This
makes the weights much more easily accessible than they otherwise would
have been, had we invented our own file format for concatenated files.

Microarchitectural portability Copy Section

On Intel and AMD microprocessors, llama.cpp spends most of its time in
the matmul quants, which are usually written thrice for SSSE3, AVX, and
AVX2. llamafile pulls each of these functions out into a separate file
that can be #includeed multiple times, with varying
__attribute__((__target__("arch"))) function attributes. Then, a
wrapper function is added which uses Cosmopolitan's X86_HAVE(FOO)
feature to runtime dispatch to the appropriate implementation.

Architecture portability Copy Section

llamafile solves architecture portability by building llama.cpp twice:
once for AMD64 and again for ARM64. It then wraps them with a shell
script which has an MZ prefix. On Windows, it'll run as a native binary.
On Linux, it'll extract a small 8kb executable called APE
Loader
to ${TMPDIR:-${HOME:-.}}/.ape that'll map the binary portions of the
shell script into memory. It's possible to avoid this process by running
the
assimilate
program that comes included with the cosmocc compiler. What the
assimilate program does is turn the shell script executable into
the host platform's native executable format. This guarantees a fallback
path exists for traditional release processes when it's needed.

GPU support Copy Section

Cosmopolitan Libc uses static linking, since that's the only way to get
the same executable to run on six OSes. This presents a challenge for
llama.cpp, because it's not possible to statically link GPU support. The
way we solve that is by checking if a compiler is installed on the host
system. For Apple, that would be Xcode, and for other platforms, that
would be nvcc. llama.cpp has a single file implementation of each GPU
module, named ggml-metal.m (Objective C) and ggml-cuda.cu (Nvidia
C). llamafile embeds those source files within the zip archive and asks
the platform compiler to build them at runtime, targeting the native GPU
microarchitecture. If it works, then it's linked with platform C library
dlopen() implementation. See llamafile/cuda.c and
llamafile/metal.c.

In order to use the platform-specific dlopen() function, we need to ask
the platform-specific compiler to build a small executable that exposes
these interfaces. On ELF platforms, Cosmopolitan Libc maps this helper
executable into memory along with the platform's ELF interpreter. The
platform C library then takes care of linking all the GPU libraries, and
then runs the helper program which longjmp()'s back into Cosmopolitan.
The executable program is now in a weird hybrid state where two separate
C libraries exist which have different ABIs. For example, thread local
storage works differently on each operating system, and programs will
crash if the TLS register doesn't point to the appropriate memory. The
way Cosmopolitan Libc solves that on AMD is by using SSE to recompile
the executable at runtime to change %fs register accesses into %gs
which takes a millisecond. On ARM, Cosmo uses the x28 register for TLS
which can be made safe by passing the -ffixed-x28 flag when compiling
GPU modules. Lastly, llamafile uses the __ms_abi__ attribute so that
function pointers passed between the application and GPU modules conform
to the Windows calling convention. Amazingly enough, every compiler we
tested, including nvcc on Linux and even Objective-C on MacOS, all
support compiling WIN32 style functions, thus ensuring your llamafile
will be able to talk to Windows drivers, when it's run on Windows,
without needing to be recompiled as a separate file for Windows. See
cosmopolitan/dlopen.c
for further details.

File: docs/troubleshooting.md Copy Chapter

Gotchas and troubleshooting Copy Chapter

On any platform, if your llamafile process is immediately killed, check
if you have CrowdStrike and then ask to be whitelisted.

Mac Copy Section

On macOS with Apple Silicon you need to have Xcode Command Line Tools
installed for llamafile to be able to bootstrap itself.

If you use zsh and have trouble running llamafile, try saying sh -c ./llamafile. This is due to a bug that was fixed in zsh 5.9+. The same
is the case for Python subprocess, old versions of Fish, etc.

Mac error "... cannot be opened because the developer cannot be verified"

1. Immediately launch System Settings, then go to Privacy & Security. llamafile should be listed at the bottom, with a button to Allow.
2. If not, then change your command in the Terminal to be sudo spctl --master-disable; [llama launch command]; sudo spctl --master-enable. This is because --master-disable disables all checking, so you need to turn it back on after quitting llama.

Linux Copy Section

On some Linux systems, you might get errors relating to run-detectors
or WINE. This is due to binfmt_misc registrations. You can fix that by
adding an additional registration for the APE file format llamafile
uses:

sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo chmod +x /usr/bin/ape
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"

Windows Copy Section

As mentioned above, on Windows you may need to rename your llamafile by
adding .exe to the filename.

Also as mentioned above, Windows also has a maximum file size limit of 4GB
for executables. The LLaVA server executable above is just 30MB shy of
that limit, so it'll work on Windows, but with larger models like
WizardCoder 13B, you need to store the weights in a separate file. An
example is provided above; see "Using llamafile with external weights."

On WSL, there are many possible gotchas. One thing that helps solve them
completely is this:

[Unit]
Description=cosmopolitan APE binfmt service
After=wsl-binfmt.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"

[Install]
WantedBy=multi-user.target

Put that in /etc/systemd/system/cosmo-binfmt.service.

Ensure that the APE loader is installed to /usr/bin/ape:

sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo chmod +x /usr/bin/ape

Then run sudo systemctl enable --now cosmo-binfmt.

Another thing that's helped WSL users who experience issues, is to
disable the WIN32 interop feature:

sudo sh -c "echo -1 > /proc/sys/fs/binfmt_misc/WSLInterop"

In Windows 11 with WSL 2 the location of the interop flag has changed, as such
the following command be required instead/additionally:

sudo sh -c "echo -1 > /proc/sys/fs/binfmt_misc/WSLInterop-late"

In the instance of getting a Permission Denied on disabling interop
through CLI, it can be permanently disabled by adding the following in
/etc/wsl.conf

[interop]
enabled=false