Technical Documentation: Blaizzy/mlx-vlm

ℹ️ Provenance: Hybrid Fusion: Blaizzy/mlx-vlm (README + 4 In-Tree Chapters) · CodeWiki Reference · Recency: Active (< 180 days)

1. Project Overview & Quickstart (Blaizzy/mlx-vlm)

[](https://github.com/Blaizzy/mlx-vlm/actions/workflows/python-publish.yml)

MLX-VLM

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) and Omni Models (VLMs with audio and video support) on your Mac using MLX.

- Installation
- Usage
- Command Line Interface (CLI)
- Thinking Budget
- Speculative Decoding
- DFlash (Qwen3.5)
- Gemma 4 MTP
- Gemma 4 EAGLE-3
- MiniMax M3 EAGLE-3
- Chat UI with Gradio
- Python Script
- Server (FastAPI)
- Continuous Batching
- Automatic Prefix Caching (APC)
- KV Cache Quantization
- 1-bit Affine Inference
- Activation Quantization (CUDA)
- Multi-Image Chat Support
- Supported Models
- Usage Examples
- Model-Specific Documentation
- Vision Feature Caching
- TurboQuant KV Cache
- Distributed Inference
- Fine-tuning

Model-Specific Documentation

Some models have detailed documentation with prompt formats, examples, and best practices:

| Model | Documentation |
|-------|---------------|
| DeepSeek-OCR | Docs |
| DeepSeek-OCR-2 | Docs |
| Unlimited-OCR | Docs |
| DOTS-OCR | Docs |
| DOTS-MOCR | Docs |
| ERNIE 4.5 VL | Docs |
| GLM-OCR | Docs |
| Phi-4 Reasoning Vision | Docs |
| MiniCPM-o | Docs |
| PaddleOCR-VL | Docs |
| Phi-4 Multimodal | Docs |
| MolmoPoint | Docs |
| LocateAnything | Docs |
| Moondream2 | Docs |
| Moondream3 | Docs |
| Gemma 4 | Docs |
| MiniMax M3 | Docs |
| Falcon-OCR | Docs |
| Granite Vision 3.2 | Docs |
| Granite 4.0 Vision | Docs |
| MiniCPM-V 4.6 | Docs |

Installation

The easiest way to get started is to install the mlx-vlm package using pip:

pip install -U mlx-vlm

The Gradio chat UI needs an extra dependency that is not
part of the base install:

pip install -U 'mlx-vlm[ui]'

Quote the package name so that shells which expand square brackets, such as
zsh, do not treat [ui] as a glob pattern.

Agent Skills

This repo ships an agent-skills bundle under skills/ for common MLX-VLM workflows — usage, conversion, development, and support. Skills load into a coding agent (Claude Code, Codex, Gemini) so it follows the right project conventions instead of guessing.

| Skill | Description |
|-------|-------------|
| cli-inference | Run and debug command-line inference (mlx_vlm.generate) — text/image/audio inputs and image-generation flags. |
| server-inference | Run and debug the local server across the models, chat, responses, messages, audio, image, cache, and metrics endpoints. |
| convert-quantize | Convert and quantize Hugging Face models to MLX (mlx_vlm.convert) — bits/group size, quant modes, RTN/AWQ, mixed recipes. |
| add-new-model | Port a new architecture into mlx_vlm/models — config, weight-name mapping, reuse a similar model, add a test class. |
| benchmarking | Produce credible, reproducible perf numbers and fork-vs-main A/B tables for PRs. |
| contributing | Shape a change to pass review — code/config/test placement, pre-commit hooks, and PR expectations. |
| hf-cache-models | List MLX-VLM-supported (and, with --check-arch, loadable) models in the local Hugging Face cache. |
| reproducible-github-issues | Turn CLI or server failures into concise, reproducible GitHub issues. |

Validate the bundle at any time:

python3 skills/scripts/validate_skills.py

Install from a local checkout:

Claude Code

/plugin marketplace add /path/to/mlx-vlm
/plugin install mlx-vlm-skills@mlx-vlm
Codex CLI

codex plugin marketplace add /path/to/mlx-vlm
codex plugin add mlx-vlm-skills@mlx-vlm
Gemini CLI

gemini extensions install /path/to/mlx-vlm/skills

Usage

Command Line Interface (CLI)

Generate output from a model using the CLI:

Text generation

mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Hello, how are you?"
Image generation

mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg
Audio generation (New)

mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you hear" --audio /path/to/audio.wav
Multi-modal generation (Image + Audio)

mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you see and hear" --image /path/to/image.jpg --audio /path/to/audio.wav

#### Thinking Budget

For thinking models (e.g., Qwen3.5), you can limit the number of tokens spent in the thinking block:

mlx_vlm.generate --model mlx-community/Qwen3.5-2B-4bit \
  --thinking-budget 50 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>" \
  --enable-thinking \
  --prompt "Solve 2+2"

| Flag | Description |
|------|-------------|
| --enable-thinking | Activate thinking mode in the chat template |
| --thinking-budget | Max tokens allowed inside the thinking block |
| --thinking-start-token | Token that opens a thinking block (default: <think>) |
| --thinking-end-token | Token that closes a thinking block (default: </think>) |

When the budget is exceeded, the model is forced to emit \n</think> and transition to the answer. If --enable-thinking is passed but the model's chat template does not support it, the budget is applied only if the model generates the start token on its own.

On the server, thinking mode is disabled by default. Start the server with --enable-thinking to make thinking mode the default for requests that do not specify it:

mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking

You can also set server defaults for the thinking budget and delimiter tokens:

mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"

Requests can override the server defaults with enable_thinking, thinking_budget, thinking_start_token, or thinking_end_token.

Speculative Decoding

Speed up generation by drafting several candidate tokens with a small "drafter" model and verifying them in a single target forward pass. Three drafter families are supported.

| Flag | Description |
|------|-------------|
| --draft-model | HuggingFace repo or local path for the drafter |
| --draft-kind | Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP) |
| --draft-block-size | Override the drafter's configured block size |

See docs/usage.md for Python API examples including batch generation.

#### DFlash (Qwen3.5 and Muse Glimmer)

A lightweight block-diffusion drafter that predicts multiple tokens per round, typically 2–3× faster.

Text generation with speculative decoding

mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0 --enable-thinking
Also works with images

mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --image examples/images/cats.jpg \
  --prompt "Describe this image." \
  --max-tokens 256 --temperature 0 --enable-thinking
Server with speculative decoding

mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash

Muse Glimmer's published assistant checkpoint is auto-detected as DFlash:

mlx_vlm.generate --model meta-models/Muse-Glimmer-30B \ --draft-model meta-models/Muse-Glimmer-30B-assistant \ --prompt "Write a quicksort in Python." \ --max-tokens 512 --temperature 0

mlx_vlm.server --model meta-models/Muse-Glimmer-30B \ --draft-model meta-models/Muse-Glimmer-30B-assistant

DFlash draft-cache windowing is available from the Python API. During
speculative decoding the target model still verifies every proposed token with
its full KV cache; this knob only changes the DFlash drafter cache. When
draft_window_size is set, the drafter keeps at most that many recent committed
tokens in its own KV cache instead of attending over the full generated prefix.
That reduces draft-side cache length and memory, but it can lower acceptance
because the drafter has less context than the target verifier. On MLX, the full
draft cache is usually faster for Qwen3.5 DFlash, so windowing defaults to
None; set it only when you want to experiment with this compact recent-token
cache tradeoff:

python

from mlx_vlm import load
from mlx_vlm.generate import generate
from mlx_vlm.speculative.drafters import load_drafter
model, processor = load("Qwen/Qwen3.5-4B")
draft_model, draft_kind = load_drafter("z-lab/Qwen3.5-4B-DFlash")
draft_model.config.draft_window_size = 256  # None disables windowingresult = generate(
    model,
    processor,
    "Write a quicksort in Python.",
    max_tokens=512,
    temperature=0,
    draft_model=draft_model,
    draft_kind=draft_kind,
)

#### Gemma 4 MTP

Multi-Token Prediction: Google's 4-layer "assistant" drafter that shares K/V with the target and drafts multiple tokens autoregressively from a constant position. Pass --draft-kind mtp to dispatch the MTP round-loop.

mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4 \
  --prompt "Explain speculative decoding in 3 sentences." \
  --max-tokens 256 --temperature 0
Server

mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4

Supported pairings (target ↔ drafter):

| Target | Drafter |
|---------------------------------|------------------------------------------|
| mlx-community/gemma-4-E2B-it-bf16 | mlx-community/gemma-4-E2B-it-assistant-bf16 |
| mlx-community/gemma-4-E4B-it-bf16 | mlx-community/gemma-4-E4B-it-assistant-bf16 |
| mlx-community/gemma-4-26B-A4B-it-bf16 | mlx-community/gemma-4-26B-A4B-it-assistant-bf16 |
| mlx-community/gemma-4-31B-it-bf16 | mlx-community/gemma-4-31B-it-assistant-bf16 |

Measured speedups (greedy, byte-identical output): up to 3.94× on 26B-A4B and 2.29× on 31B at B=4. See mlx_vlm/speculative/drafters/gemma4_assistant/README.md for full sweeps and architecture notes.

#### Gemma 4 EAGLE-3

EAGLE-3 drafts from three target hidden-state captures with a lightweight one-layer speculator. The Red Hat Speculators checkpoint auto-detects as --draft-kind eagle3.

mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3 \
  --prompt "Explain speculative decoding in 3 sentences." \
  --max-tokens 256 --temperature 0
Server

mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3

#### MiniMax M3 EAGLE-3

MiniMax M3 supports the released Inferact/MiniMax-M3-EAGLE3 drafter. Convert
the target with mlx_vlm.convert because mlx_lm.convert does not know the
minimax_m3_vl model type.

mlx_vlm.convert \ --hf-path MiniMaxAI/MiniMax-M3 \ --mlx-path ~/MiniMax-M3-4bit \ --quantize --q-bits 4 \ --trust-remote-code mlx_vlm.convert \ --hf-path Inferact/MiniMax-M3-EAGLE3 \ --mlx-path ~/MiniMax-M3-EAGLE3

mlx_vlm.generate \ --model ~/MiniMax-M3-4bit \ --draft-model ~/MiniMax-M3-EAGLE3 \ --draft-kind eagle3 \ --draft-block-size 3 \ --prompt "Explain MiniMax Sparse Attention in one paragraph." \ --max-tokens 256 --temperature 0

The public MiniMax M3 BF16 checkpoint advertises MTP metadata but does not
publish mtp or nextn tensors, so use the released EAGLE-3 drafter for that
checkpoint.

MiniMax M3 also supports image/video prompts, MiniMax thinking tags, MiniMax
tool-call parsing, MSA index caches, and MXFP8 config loading. See
mlx_vlm/models/minimax_m3_vl/README.md
for model-specific conversion and runtime notes.

Chat UI with Gradio

The Gradio chat UI requires the optional ui extra, which the base mlx-vlm
install does not include:

pip install -U 'mlx-vlm[ui]'

Then launch the chat interface:

mlx_vlm.chat_ui --model mlx-community/Qwen2-VL-2B-Instruct-4bit

Python Script

Here's an example of how to use MLX-VLM in a Python script:

python

import mlx.core as mx
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load the model

model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = load_config(model_path)
Prepare input

image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
image = [Image.open("...")] can also be used with PIL.Image.Image objects

prompt = "Describe this image."
Apply chat template

formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=len(image)
)
Generate output

output = generate(model, processor, formatted_prompt, image, verbose=False)
print(output)

#### Audio Example

python

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load model with audio support

model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config
Prepare audio input

audio = ["/path/to/audio1.wav", "/path/to/audio2.mp3"]
prompt = "Describe what you hear in these audio files."
Apply chat template with audio

formatted_prompt = apply_chat_template(
    processor, config, prompt, num_audios=len(audio)
)
Generate output with audio

output = generate(model, processor, formatted_prompt, audio=audio, verbose=False)
print(output)

#### Multi-Modal Example (Image + Audio)

python

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load multi-modal model

model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config
Prepare inputs

image = ["/path/to/image.jpg"]
audio = ["/path/to/audio.wav"]
prompt = ""
Apply chat template

formatted_prompt = apply_chat_template(
    processor, config, prompt,
    num_images=len(image),
    num_audios=len(audio)
)
Generate output

output = generate(model, processor, formatted_prompt, image, audio=audio, verbose=False)
print(output)

Server (FastAPI)

Start the server:

mlx_vlm.server --port 8080
Preload a model at startup (Hugging Face repo or local path)

mlx_vlm.server --model <hf_repo_or_local_path>
Preload separate model kinds at startup

mlx_vlm.server --model <language_model> \
  --image-model <image_generation_model> \
  --tts-model <text_to_speech_model> \
  --stt-model <speech_to_text_model>
Preload a model with adapter

mlx_vlm.server --model <hf_repo_or_local_path> --adapter-path <adapter_path>
With trust remote code enabled (required for some models)

mlx_vlm.server --trust-remote-code
Enable thinking mode by default for requests that do not override it

mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking
Configure thinking defaults at startup

mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"
Require bearer authentication for API endpoints

mlx_vlm.server --api-key <secret-token>

#### Server Options

- --model: Preload a language model at server startup, accepts a Hugging Face repo ID or local path (optional, loads lazily on first request if omitted)
- --image-model: Preload an image generation model at server startup
- --tts-model: Preload a text-to-speech model at server startup
- --stt-model: Preload a speech-to-text model at server startup
- --embedding-model: Preload an embedding model at server startup
- --reranker-model: Preload a supported reranker model at server startup
- --adapter-path: Path for adapter weights to use with the preloaded model
- --draft-model: Speculative drafter path or HF id (e.g. z-lab/Qwen3.5-4B-DFlash, RedHatAI/gemma-4-31B-it-speculator.eagle3, google/gemma-4-31B-it-assistant, Inferact/MiniMax-M3-EAGLE3) — enables speculative decoding for ~2× or higher throughput
- --draft-kind: Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP)
- --draft-block-size: Override the drafter's configured block size
- --host: Host address (default: 0.0.0.0)
- --port: Port number (default: 8080)
- --trust-remote-code: Trust remote code when loading models from Hugging Face Hub
- --enable-thinking: Enable thinking mode by default for requests that do not set enable_thinking
- --thinking-budget: Default maximum number of tokens allowed inside a thinking block
- --thinking-start-token: Default token that opens a thinking block
- --thinking-end-token: Default token that closes a thinking block (--thinking-eos-token is also accepted)
- --kv-bits: Number of bits for KV cache quantization (e.g. 8 for uniform, 3.5 for TurboQuant)
- --kv-quant-scheme: KV cache quantization backend (uniform or turboquant)
- --kv-key-bits / --kv-value-bits: Override the bit-width for keys or values individually (see Per-tensor KV quantization)
- --kv-key-scheme / --kv-value-scheme: Override the quantization backend for keys or values individually
- --kv-group-size: Group size for uniform KV cache quantization (default: 64)
- --max-kv-size: Maximum KV cache size in tokens
- --vision-cache-size: Max number of cached vision features (default: 20)
- --log-progress-interval: Decoded tokens between progress log messages; 0 disables periodic decode progress (default: 10)
- --api-key: Bearer token required for inference, model discovery, and management endpoints
- --log-level: Logging level — DEBUG, INFO, WARNING, ERROR, CRITICAL (default: INFO)

At INFO, the server logs request start/completion, chunked-prefill progress,
time to first token, periodic decode throughput, and the final token counts. Set
--log-level DEBUG to emit decode progress for every token and add its token
number, token ID, and decoded text to the same log entry. Decode progress uses
rate for the instantaneous inter-token rate; decode completion uses the same
field name for aggregate decode throughput measured across completed token
intervals.

OpenAI-compatible streaming responses expose throughput under
timings.predicted_per_second. Token-bearing SSE chunks report the instantaneous
inter-token rate, while terminal and usage chunks report the aggregate rate as
(tokens - 1) / (last_token_time - first_token_time). The first token reports
null because it has no preceding token interval.

You can also set trust remote code via environment variable:

MLX_TRUST_REMOTE_CODE=true mlx_vlm.server

The server provides multiple endpoints for different use cases and supports dynamic model loading/unloading with caching (one model at a time).

Continuous Batching

The server supports continuous batching for higher throughput when handling multiple concurrent requests. New requests join the active batch immediately without waiting for existing requests to finish, and mixed batches of image and text-only requests are supported.

Continuous batching is enabled automatically when the server loads a model. You can pre-load a model at startup so it's ready to serve immediately:

mlx_vlm.server --port 8080 --model mlx-community/Qwen2.5-VL-3B-Instruct-4bit

Verify via the health endpoint:

curl http://localhost:8080/health
{"status":"healthy","loaded_model":"...","apc_enabled":false}

If --model is omitted, the model is loaded on the first request.

Automatic Prefix Caching (APC)

Automatic Prefix Caching reuses block-level K/V cache state across requests that share the same prefix. It is useful for repeated long documents, long chat histories, or retrieval contexts where each request appends a short new suffix.

APC has two tiers:

- Warm memory: keeps reusable APCBlock tensors in process memory. This is the fastest path, but it keeps both the reusable block pool and the runtime KVCache.
- Warm disk: persists cached prefixes as safetensors shards so they survive process restarts. Warm-disk reads build the layer-major prompt cache directly without promoting restored blocks into the APCBlock pool; writes can still populate both memory and disk tiers.

#### Python Script

Use APCManager directly when calling stream_generate:

python

from pathlib import Path
from mlx_vlm import load, stream_generate
from mlx_vlm.apc import APCManager, DiskBlockStore
from mlx_vlm.prompt_utils import apply_chat_template
model_id = "Qwen/Qwen3-VL-4B-Instruct"
model, processor = load(model_id)
disk = DiskBlockStore(
    Path("~/.cache/mlx-vlm/caching").expanduser(),
    namespace=model_id,
    max_bytes=3 * (1 << 30),  # 3 GB disk cap; use None for uncapped
)
apc = APCManager(num_blocks=4096, block_size=16, disk=disk)
document = Path("long_document.txt").read_text()
try:
    # First request computes the full prefix and stores reusable K/V blocks.
    prompt1 = apply_chat_template(
        processor,
        model.config,
        prompt=f"{document}\n\nSummarize the key decisions.",
        num_images=0,
    )
    for _ in stream_generate(
        model, processor, prompt1, max_tokens=128, temperature=0.0, apc_manager=apc
    ):
        pass
    # Second request shares the same document prefix and only prefills the suffix.
    prompt2 = apply_chat_template(
        processor,
        model.config,
        prompt=f"{document}\n\nList the open engineering risks.",
        num_images=0,
    )
    for chunk in stream_generate(
        model, processor, prompt2, max_tokens=128, temperature=0.0, apc_manager=apc
    ):
        print(chunk.text, end="", flush=True)    print(apc.stats_snapshot())
finally:
    apc.close()

To compare cold, warm-memory, warm-disk, and disk-eviction behavior with a
model, use the same direct API path:

text

/ Detailed source-code truncated for AI context efficiency. /

#### Server

Enable in-memory APC for the server with environment variables:

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080

APC works with KV-cache quantization (--kv-bits):

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --kv-bits 8 --port 8080

Enable the persistent disk tier:

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
APC_DISK_PATH=~/.cache/mlx-vlm/caching \
APC_DISK_MAX_GB=3 \
APC_DISK_SHARD_MAX_BLOCKS=256 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080

Repeated requests with the same long prefix will hit APC automatically:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "X-APC-Tenant: demo" \
  -d '{
    "model": "Qwen/Qwen3-VL-4B-Instruct",
    "messages": [{
      "role": "user",
      "content": "Paste a long shared document here.\n\nNow answer question A."
    }],
    "max_tokens": 128
  }'

Use the same X-APC-Tenant value for requests that may share cached prefixes. Use different tenant values to isolate cache entries between users or workspaces.

Inspect and reset APC state:

curl http://localhost:8080/v1/cache/stats
curl -X POST http://localhost:8080/v1/cache/reset

Common APC environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| APC_ENABLED | 0 | Set to 1 to enable APC |
| APC_NUM_BLOCKS | 2048 | Number of in-memory APC blocks |
| APC_BLOCK_SIZE | 16 | Tokens per APC block |
| APC_DISK_PATH | unset | Directory for persistent disk shards |
| APC_DISK_MAX_GB | 0 | Disk cap in GB; 0 means uncapped |
| APC_DISK_SHARD_MAX_BLOCKS | 256 | Max blocks per disk segment shard |
| APC_MAX_POOL_TENSORS | 450000 | Stops adding memory blocks before the Metal resource limit; disk writes continue |
| APC_LAYER_MAJOR_MEMORY_MIN_TOKENS | 50000 | Store long warm-memory prefixes as compact layer-major snapshots instead of per-block tensors |
| APC_HASH | fast | Set to sha256 for a stable cryptographic hash |
| APC_TRACE | unset | Set to 1 for greppable store/reject/self-check log lines |

APC is disabled automatically for models that use a custom cache layout. APC works with --kv-bits (including TurboQuant): the live KV cache stays quantized; the reusable APC pool stores dequantized float K/V, so pool size does not shrink with quant.
When APC is enabled on the server, a non-fatal layout self-check runs at model load.

#### KV Cache Quantization

Reduce KV cache memory during continuous batching with --kv-bits. Both uniform quantization and TurboQuant are supported. Compatible with Automatic Prefix Caching (APC_ENABLED=1).

Uniform 8-bit KV cache quantization

mlx_vlm.server --model google/gemma-4-26b-a4b-it --kv-bits 8
TurboQuant 3.5-bit (3-bit keys + 4-bit values)

mlx_vlm.server --model google/gemma-4-26b-a4b-it --kv-bits 3.5 --kv-quant-scheme turboquant

Full-attention layers use quantized batch caches while sliding-window layers keep their fixed-size rotating caches. The last full-attention layer stays unquantized (sensitive in deep models).

##### Per-tensor KV quantization

Keys and values do not have to share a bit-width or a backend. A fractional --kv-bits already splits the budget — 3.5 gives 3-bit keys and 4-bit values — and --kv-key-bits / --kv-value-bits override either side:

8-bit keys, 3-bit values, both TurboQuant

mlx_vlm.generate --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --kv-bits 3.5 --kv-quant-scheme turboquant \
  --kv-key-bits 8 --kv-value-bits 3

--kv-key-scheme / --kv-value-scheme go further and select a different backend per tensor, which builds a hybrid cache:

uniform 8-bit keys beside TurboQuant 3-bit values

mlx_vlm.generate --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --kv-bits 8 --kv-quant-scheme uniform \
  --kv-value-bits 3 --kv-value-scheme turboquant

Two limitations apply to mixed schemes specifically:

- The hybrid cache dequantizes on every step instead of using a fused kernel, so it is slower than either homogeneous path.
- Mixed schemes are not supported during continuous batching or for batch prefix caches, and raise NotImplementedError there. Mixed bit-widths under a single scheme work everywhere.

Note that values are given the extra bit by default for a reason: value error passes straight through the attention output, whereas key error is partly reabsorbed by the softmax. Measured on Qwen3.5, spending an equal budget key-heavy was consistently worse than value-heavy, so prefer measuring before overriding.

Tested with gemma-4-26b-a4b-it at 20K context:

| Config | Gen tok/s | KV Cache | KV Reduction |
|--------|-----------|----------|--------------|
| No quant | 50.3 | 0.624 GB | 1x |
| Uniform 8-bit | 52.6 | 0.469 GB | 1.33x |
| TurboQuant 3.5-bit | 25.6 | 0.365 GB | 1.71x |

Models with all full-attention layers (e.g. Qwen, LLaMA) see larger reductions — up to 3.6x at 8-bit and 6.4x at 4-bit.

#### Log Probabilities

The /chat/completions endpoint supports OpenAI-compatible per-token log probabilities. Pass logprobs: true (and optionally top_logprobs: N, up to 20) in the request:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
    "messages": [{"role":"user","content":"Say hi in 3 words."}],
    "max_tokens": 8,
    "logprobs": true,
    "top_logprobs": 3
  }'

Each choice gets a logprobs.content[] list with one entry per generated token: {token, logprob, bytes, top_logprobs: [{token, logprob, bytes}, ...]}. Works for both streaming and non-streaming.

top_logprobs requires the server to be started with a non-zero cap on how many alternatives it will compute per token (default 0 = disabled, max 20). Set it via the --top-logprobs-k flag or the TOP_LOGPROBS_K env var:

mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit --top-logprobs-k 5
or

TOP_LOGPROBS_K=5 mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit

Per-request top_logprobs is clamped to TOP_LOGPROBS_K. When TOP_LOGPROBS_K=0, requests with logprobs: true still return chosen-token logprobs; only the top_logprobs list stays empty. Leaving the cap at 0 keeps the vocab-wide sort out of the decode graph, so deployments that don't need logprobs pay zero overhead.

#### Structured Outputs

The /v1/chat/completions and /v1/responses endpoints support OpenAI-compatible json_schema structured outputs. The server constrains generation to the supplied JSON schema and supports both streaming and non-streaming responses.

You can define the schema with Pydantic:

python

from typing import Literal
from pydantic import BaseModel, ConfigDict, Field

class AnimalResult(BaseModel):
    model_config = ConfigDict(extra="forbid")
    animal: Literal["dog", "cat", "bird", "unknown"]
    species: str = Field(max_length=60)
    description: str = Field(max_length=200)
schema = AnimalResult.model_json_schema()

Call the local server with the OpenAI Python client:

python

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="mlx-community/Qwen3.5-4B-MLX-4bit",
    messages=[
        {"role": "user", "content": "Return a dog object."},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "AnimalResult",
            "strict": True,
            "schema": schema,
        },
    },
)result = AnimalResult.model_validate_json(response.choices[0].message.content)
print(result)

Example output:

text

animal='dog' species='Canis lupus familiaris' description='A domesticated canine known for companionship and loyalty.'

Chat completions use top-level response_format. The same format works for text-only and multimodal requests:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3.5-4B-MLX-4bit",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Identify the main animal in this image."},
        {"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}}
      ]
    }],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "AnimalResult",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "animal": {"type": "string", "enum": ["dog", "cat", "bird", "unknown"]},
            "species": {"type": "string", "maxLength": 60},
            "description": {"type": "string", "maxLength": 200}
          },
          "required": ["animal", "species", "description"],
          "additionalProperties": false
        }
      }
    },
    "max_tokens": 256
  }'

Structured outputs are also supported with:

- Streaming chat completions by setting "stream": true
- The responses API via text.format on /v1/responses
- Text-only requests using the same response_format shape

Structured outputs are not currently supported with speculative decoding.

#### How It Works

- A dedicated generation thread runs a BatchGenerator that processes multiple requests in parallel
- Image requests are prefilled individually with their own vision embeddings, then join the shared decoding batch
- Text-only requests are batched together for efficient prefill
- After prefill, all requests decode together in a single batch, sharing GPU compute

#### Available Endpoints

- /models and /v1/models - List models available locally
- /chat/completions and /v1/chat/completions - OpenAI-compatible chat-style interaction endpoint with support for images, audio, and text
- /responses and /v1/responses - OpenAI-compatible responses endpoint
- /embeddings and /v1/embeddings - OpenAI-compatible embeddings endpoint backed by native MLX embedding models
- /v1/rerank - Rank text or multimodal documents by relevance to a query
- /audio/speech and /v1/audio/speech - OpenAI-compatible text-to-speech endpoint backed by mlx-audio TTS models
- /audio/transcriptions and /v1/audio/transcriptions - OpenAI-compatible speech-to-text endpoint backed by mlx-audio STT models
- /audio/translations and /v1/audio/translations - OpenAI-compatible audio translation endpoint for STT models that expose a translation task
- /v1/realtime - WebSocket-based realtime full-duplex speech. See the Nemotron VoiceChat guide for supported models and usage.
- /health - Check server status
- /metrics and /v1/metrics - Inspect rolling request metrics, throughput, and runtime counters
- /unload - Unload all loaded model caches from memory

#### Usage Examples

##### List available models

curl "http://localhost:8080/models"

##### Embeddings

curl -X POST "http://localhost:8080/v1/embeddings" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sentence-transformers/all-MiniLM-L6-v2",
    "input": ["The quick brown fox.", "A fast auburn fox."]
  }'

Preload a default with --embedding-model <repo-or-path>. Supported architectures: BERT, XLM-RoBERTa, ModernBERT, Qwen3-Embedding, EmbeddingGemma (gemma3), LFM2, SigLIP (text), Qwen3-VL-Embedding, and Llama-Nemotron-VL, plus LLM2Vec bidirectional Llama. ColBERT-style multi-vector models (ColIdefics3, ColQwen2.5) are also available for late-interaction use.

##### Reranking

curl -X POST "http://localhost:8080/v1/rerank" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-Reranker-0.6B-4bit",
    "query": "What is the capital of France?",
    "documents": ["Berlin is in Germany.", "Paris is the capital of France."],
    "top_n": 1,
    "return_documents": true
  }'

Preload a default with --reranker-model <repo-or-path>. Supported text rerankers include one-label BERT, XLM-RoBERTa, and ModernBERT sequence-classification checkpoints, plus Qwen3 generative rerankers. Qwen3-VL rerankers also accept objects containing text, image, image_url, video, or video_url. Sequence-classification rerankers accept text pairs and do not support custom instructions.

##### Text Input

curl -X POST "http://localhost:8080/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you"
      }
    ],
    "stream": true,
    "max_tokens": 100
  }'

##### Image Input

curl -X POST "http://localhost:8080/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen2.5-VL-32B-Instruct-8bit",
    "messages":
    [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "This is today's chart for energy demand in California. Can you provide an analysis of the chart and comment on the implications for renewable energy in California?"
          },
          {
            "type": "input_image",
            "image_url": "/path/to/repo/examples/images/renewables_california.png"
          }
        ]
      }
    ],
    "stream": true,
    "max_tokens": 1000
  }'

##### Audio Input

curl -X POST "http://localhost:8080/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/gemma-3n-E2B-it-4bit",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Describe what you hear in these audio files" },
          { "type": "input_audio", "input_audio": "/path/to/audio1.wav" },
          { "type": "input_audio", "input_audio": "https://example.com/audio2.mp3" }
        ]
      }
    ],
    "stream": true,
    "max_tokens": 500
  }'

##### Text-to-Speech

curl -X POST "http://localhost:8080/v1/audio/speech" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/pocket-tts",
    "input": "Hello from MLX VLM.",
    "voice": "fantine",
    "response_format": "mp3"
  }' \
  --output speech.mp3

##### Speech-to-Text

curl -X POST "http://localhost:8080/v1/audio/transcriptions" \
  -F model=mlx-community/parakeet-tdt-0.6b-v3 \
  -F file=@/path/to/audio.mp3 \
  -F response_format=json

##### Multi-Modal (Image + Audio)

curl -X POST "http://localhost:8080/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/gemma-3n-E2B-it-4bit",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "input_image", "image_url": "/path/to/image.jpg"},
          {"type": "input_audio", "input_audio": "/path/to/audio.wav"}
        ]
      }
    ],
    "max_tokens": 100
  }'

##### Responses Endpoint

curl -X POST "http://localhost:8080/responses" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "input_text", "text": "What is in this image?"},
          {"type": "input_image", "image_url": "/path/to/image.jpg"}
        ]
      }
    ],
    "max_tokens": 100
  }'

#### Request Parameters

- model: Model identifier (required)
- messages: Chat messages for chat/OpenAI endpoints
- max_tokens: Maximum tokens to generate
- temperature: Sampling temperature
- top_p: Top-p sampling parameter
- top_k: Top-k sampling cutoff
- min_p: Min-p sampling threshold
- repetition_penalty: Penalty applied to repeated tokens
- enable_thinking: Override the server thinking-mode default for a request (true or false)
- thinking_budget: Maximum tokens allowed inside the thinking block
- thinking_start_token: Token that opens a thinking block
- thinking_end_token: Token that closes a thinking block
- stream: Enable streaming responses

1-bit Affine Inference

MLX-VLM can load existing affine 1-bit MLX checkpoints without a custom MLX
build. When a checkpoint declares "bits": 1, compatible Linear and
Embedding layers are replaced automatically with an inference-only module
that JIT-compiles its Metal kernel from Python.

The checkpoint must use MLX's packed uint32 weight layout, include scales
and biases, and declare a group size of 32, 64, or 128:

json

{
  "quantization": {
    "group_size": 64,
    "bits": 1,
    "mode": "affine"
  }
}

Load and generate normally; no extra inference flag is needed:

python

from mlx_vlm import generate, loadmodel, processor = load("path/to/1bit-model")
result = generate(model, processor, "Describe this image", image=["image.jpg"])

This path is for inference from an already quantized checkpoint. Converting a
floating-point model to 1-bit still requires a quantizer that can produce the
packed weights and affine parameters.

Activation Quantization (CUDA)

When running on NVIDIA GPUs with MLX CUDA, models quantized with mxfp8 or nvfp4 modes require activation quantization to work properly. This converts QuantizedLinear layers to QQLinear layers which quantize both weights and activations.

Command Line

Use the -qa or --quantize-activations flag:

mlx_vlm.generate --model /path/to/mxfp8-model --prompt "Describe this image" --image /path/to/image.jpg -qa

Python API

Pass quantize_activations=True to the load function:

python

from mlx_vlm import load, generate
Load with activation quantization enabled

model, processor = load(
    "path/to/mxfp8-quantized-model",
    quantize_activations=True
)
Generate as usual

output = generate(model, processor, "Describe this image", image=["image.jpg"])

Supported Quantization Modes

- mxfp8 - 8-bit MX floating point
- nvfp4 - 4-bit NVIDIA floating point

Note: This feature is required for mxfp/nvfp quantized models on CUDA. On Apple Silicon (Metal), these models work without the flag.

Multi-Image Chat Support

MLX-VLM supports analyzing multiple images simultaneously with select models. This feature enables more complex visual reasoning tasks and comprehensive analysis across multiple images in a single conversation.

Usage Examples

#### Python Script

python

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = model.config
images = ["path/to/image1.jpg", "path/to/image2.jpg"]
prompt = "Compare these two images."
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=len(images)
)output = generate(model, processor, formatted_prompt, images, verbose=False)
print(output)

#### Command Line

mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Compare these images" --image path/to/image1.jpg path/to/image2.jpg

Video Understanding

MLX-VLM also supports video analysis such as captioning, summarization, and more, with select models.

Supported Models

The following models support video chat:

1. Qwen2-VL
2. Qwen2.5-VL
3. Idefics3
4. LLaVA
5. MiniMax M3

With more coming soon.

Usage Examples

#### Command Line

mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Describe this video" --video path/to/video.mp4 --fps 1.0

These examples demonstrate how to use multiple images with MLX-VLM for more complex visual reasoning tasks.

Vision Feature Caching

In multi-turn conversations about an image, the vision encoder runs on every turn even though the image hasn't changed. VisionFeatureCache stores projected vision features in an LRU cache keyed by image path, so the expensive vision encoder is only called once per unique image.

How It Works

1. First turn (cache miss) -- encode_image() runs the full vision pipeline (vision tower + projector), stores the result in the cache, and passes it to the language model.
2. Subsequent turns (cache hit) -- the cached features are passed directly via cached_image_features, skipping the vision encoder entirely.
3. Image switch -- when the image changes, it's a new cache key so features are computed and cached. Switching back to a previous image is a cache hit.

The cache holds up to 8 entries (configurable) and uses LRU eviction.

CLI

All chat interfaces use VisionFeatureCache automatically:

Gradio chat UI

python -m mlx_vlm.chat_ui --model google/gemma-4-26b-a4b-it
Interactive chat with Rich UI (load images with /image command)

python -m mlx_vlm.chat --model google/gemma-4-26b-a4b-it
Inline chat mode

python -m mlx_vlm.generate \
  --model google/gemma-4-26b-a4b-it \
  --image path/to/image.jpg \
  --chat \
  --max-tokens 200

Python

python

from mlx_vlm import load, stream_generate, VisionFeatureCache
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load("google/gemma-4-26b-a4b-it")
cache = VisionFeatureCache()
image = "path/to/image.jpg"
Turn 1 -- cache miss, encodes image

prompt1 = apply_chat_template(processor, model.config, "Describe this image.", num_images=1)
for chunk in stream_generate(model, processor, prompt1, image=[image],
                              max_tokens=200, vision_cache=cache):
    print(chunk.text, end="")
Turn 2 -- cache hit, skips vision encoder

prompt2 = apply_chat_template(processor, model.config, "What colors do you see?", num_images=1)
for chunk in stream_generate(model, processor, prompt2, image=[image],
                              max_tokens=200, vision_cache=cache):
    print(chunk.text, end="")

Server

The server caches vision features automatically across requests for the same image. No configuration needed -- the cache is created when a model loads and cleared on unload.

mlx_vlm.server --model google/gemma-4-26b-a4b-it

Multi-turn conversations via /v1/chat/completions (streaming and non-streaming) and /responses all benefit. The same image sent across multiple requests will only be encoded once.

Performance

Tested on google/gemma-4-26b-a4b-it over 10 multi-turn conversation turns:

| Metric | Without Cache | With Cache |
|--------|--------------|------------|
| Prompt TPS | ~48 | ~550-825 |
| Speedup | -- | 11x+ |
| Peak Memory | 52.66 GB | 52.66 GB (flat) |

Generation speed (~31 tok/s) and memory are unaffected -- only prompt processing gets faster.

TurboQuant KV Cache

TurboQuant compresses the KV cache during generation, enabling longer context lengths with less memory while maintaining quality.

Quick Start

3.5-bit KV cache quantization (3-bit keys + 4-bit values)

mlx_vlm generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --prompt "Your long prompt here..."

python

from mlx_vlm import generateresult = generate(
    model, processor, prompt,
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
    max_tokens=256,
)

Server with TurboQuant

mlx_vlm server \
  --model google/gemma-4-26b-a4b-it \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant

How It Works

TurboQuant uses random rotation + codebook quantization (arXiv:2504.19874) to compress KV cache entries from 16-bit to 2-4 bits per dimension:

- Keys & Values: MSE codebook quantization with Hadamard rotation
- Fractional bits (e.g. 3.5): uses lower bits for keys, higher for values (3-bit K + 4-bit V)

Custom Metal kernels fuse score computation and value aggregation directly on packed quantized data, avoiding full dequantization during decode.

Performance

Tested on Qwen3.5-4B-4bit at 128k context:

| Metric | Baseline | TurboQuant 3.5-bit |
|--------|----------|-------------------|
| KV Memory | 4.1 GB | 0.97 GB (76% reduction) |
| Peak Memory | 18.3 GB | 17.3 GB (-1.0 GB) |

At 512k+ contexts, TurboQuant's per-layer attention is faster than FP16 SDPA due to reduced memory bandwidth requirements.

Tested on gemma-4-31b-it at 128k context:

| Metric | Baseline | TurboQuant 3.5-bit |
|--------|----------|-------------------|
| KV Memory | 13.3 GB | 4.9 GB (63% reduction) |
| Peak Memory | 75.2 GB | 65.8 GB (-9.4 GB) |

Supported Bit Widths

| Bits | Compression | Best For |
|------|------------|----------|
| 2 | ~8x | Maximum compression, some quality loss |
| 3 | ~5x | Good balance of quality and compression |
| 3.5 | ~4.5x | Recommended default (3-bit keys + 4-bit values) |
| 4 | ~4x | Best quality, moderate compression |

Compatibility

TurboQuant automatically quantizes KVCache layers (global attention). Models with RotatingKVCache (sliding window) or ArraysCache (MLA/absorbed keys) keep their native cache format for those layers since they are already memory-efficient.

TurboQuant is supported in both single-request generation and continuous batching on the server. In continuous batching mode, KV states are stored in TurboQuant's compressed format and dequantized at attention time (custom Metal kernels are not yet batch-aware).

Distributed Inference

mlx-vlm supports distributed inference across multiple computers. It works by sharding the language model (not the vision tower), because the LLM is much larger and vision embeddings only need to be computed once.

The parallel implementation is compatible with mlx-lm sharding primitives.

See docs/usage.md for command-line examples.

Fine-tuning

MLX-VLM supports fine-tuning models with LoRA and QLoRA. Fine-tuning (and the
eval scripts) need the training extra, which is not installed by default:

bash

pip install "mlx-vlm[train]"

LoRA & QLoRA

To learn more about LoRA, please refer to the LoRA.md file.

2. In-Tree Documentation Chapters (Blaizzy/mlx-vlm)

Index

MLX-VLM

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on Apple silicon using MLX.

---

Contributing

To work on MLX-VLM in editable mode run:

bash

pip install -e .

Check that the model weights are available in the safetensors format, convert if necessary and add the model file to mlx_vlm/models.

Tests can be run from the mlx_vlm/ directory:

bash

python -m unittest discover tests/

Please format code using pre-commit before submitting a pull request.

---

Mkdocs.Yml

site_name: MLX-VLM
site_description: MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.
site_author: Prince Canuma
repo_name: Blaizzy/mlx-vlm
site_url: https://Blaizzy.github.io/mlx-vlm
repo_url: https://github.com/Blaizzy/mlx-vlm

theme:
palette:
- scheme: default
primary: black
toggle:
icon: material/toggle-switch-off-outline
name: Switch to dark mode
- scheme: slate
primary: black
accent: indigo
toggle:
icon: material/toggle-switch
name: Switch to light mode
name: material
icon:
repo: fontawesome/brands/github
features:
- navigation.instant
- navigation.tracking
- navigation.top
- navigation.footer
- search.highlight
- search.share
- content.code.copy
custom_dir: "docs/overrides"
font:
text: Google Sans
code: Regular

plugins:
- search
- mkdocstrings
- mkdocs-jupyter:
include_source: True
ignore_h1_titles: True
execute: True
allow_errors: false
ignore: ["conf.py"]
execute_ignore: ["*ignore.ipynb"]

markdown_extensions:
- admonition
- abbr
- attr_list
- def_list
- footnotes
- meta
- md_in_html
- pymdownx.superfences
- pymdownx.highlight:
linenums: true
- toc:
permalink: true

extra:
social:
- icon: fontawesome/brands/github
link: https://github.com/Blaizzy
- icon: fontawesome/brands/twitter
link: https://twitter.com/Prince_Canuma
version:
provider: mike
consent:
title: Cookie consent
description: >-
We use cookies to recognize your repeated visits and preferences, as well
as to measure the effectiveness of our documentation and whether users
find what they're searching for. With your consent, you're helping us to
make our documentation better.

extra_css:
- stylesheets/extra.css

nav:
- Home: index.md
- Installation: installation.md
- CLI Reference: cli_reference.md
- Examples: examples.md
- Contributing: contributing.md
- Community Projects: community_projects.md
- Report Issues: report_issues.md
- Changelog: changelog.md

docs_dir: docs

---

CONTRIBUTING

Contributing to MLX VLM

Below are some tips to port Vision LLMs available on Hugging Face to MLX.

Next, from this directory, do an editable install:

shell

pip install -e .

Then check if the model has weights in the
safetensors format. If not
follow instructions to
convert it.

After that, add the model file to the
mlx_vlm/models
directory. You can see other examples there. We recommend starting from a model
that is similar to the model you are porting.

Make sure the name of the new model file is the same as the model_type in the
config.json, for example
llava.

To determine the model layer names, we suggest either:

- Refer to the Transformers implementation if you are familiar with the
codebase.
- Load the model weights and check the weight names which will tell you about
the model structure.
- Look at the names of the weights by inspecting model.safetensors.index.json
in the Hugging Face repo.

Additionally, add a test for the new model type to the model
tests.

From the src/ directory, you can run the tests with:

shell

python -m unittest discover tests/

Pull Requests

1. Fork and submit pull requests to the repo.
2. If you've added code that should be tested, add tests.
3. Every PR should have passing tests and at least one review.
4. For code formatting install pre-commit using something like pip install pre-commit and run pre-commit install.
This should install hooks for running black and clang-format to ensure
consistent style for C++ and python code.

You can also run the formatters manually as follows on individual files:

bash

clang-format -i file.cpp

bash

black file.py

or,

bash

# single file pre-commit run --files file1.py

# specific files pre-commit run --files file1.py file2.py

or run pre-commit run --all-files to check all files in the repo.

Issues

We use GitHub issues to track public bugs. Please ensure your description is
clear and has sufficient instructions to be able to reproduce the issue.

License

By contributing to mlx-examples, you agree that your contributions will be licensed
under the LICENSE file in the root directory of this source tree.

---

--- METRICS ---
- Files Extracted: 5
- Estimated Token Budget: ~14287 tokens
- Recency Window: Active (< 180 days)
- Canonical Reference: https://codewiki.google/github.com/Blaizzy/mlx-vlm

Technical Documentation: Blaizzy/mlx-vlm

1. Project Overview & Quickstart (Blaizzy/mlx-vlm) Copy Chapter

MLX-VLM

Table of Contents Copy Chapter

Model-Specific Documentation Copy Chapter

Installation Copy Chapter

Agent Skills Copy Chapter

Claude Code

Codex CLI

Gemini CLI

Usage Copy Chapter

Command Line Interface (CLI) Copy Section

Text generation

Image generation

Audio generation (New)

Multi-modal generation (Image + Audio)

Speculative Decoding Copy Section

Text generation with speculative decoding

Also works with images

Server with speculative decoding

Server

Server

Chat UI with Gradio Copy Section

Python Script Copy Section

Load the model

Prepare input

image = [Image.open("...")] can also be used with PIL.Image.Image objects

Apply chat template

Generate output

Load model with audio support

Prepare audio input

Apply chat template with audio

Generate output with audio

Load multi-modal model

Prepare inputs

Apply chat template

Generate output

Server (FastAPI) Copy Section

Preload a model at startup (Hugging Face repo or local path)

Preload separate model kinds at startup

Preload a model with adapter

With trust remote code enabled (required for some models)

Enable thinking mode by default for requests that do not override it

Configure thinking defaults at startup

Require bearer authentication for API endpoints

Continuous Batching Copy Section

{"status":"healthy","loaded_model":"...","apc_enabled":false}

Automatic Prefix Caching (APC) Copy Section

Uniform 8-bit KV cache quantization

TurboQuant 3.5-bit (3-bit keys + 4-bit values)

8-bit keys, 3-bit values, both TurboQuant

uniform 8-bit keys beside TurboQuant 3-bit values

or

1-bit Affine Inference Copy Chapter

Activation Quantization (CUDA) Copy Chapter

Command Line Copy Section

Python API Copy Section

Load with activation quantization enabled

Generate as usual

Supported Quantization Modes Copy Section

Multi-Image Chat Support Copy Chapter

Usage Examples Copy Section

Video Understanding Copy Chapter

Supported Models Copy Section

Usage Examples Copy Section

Vision Feature Caching Copy Chapter

How It Works Copy Section

CLI Copy Section

Gradio chat UI

Interactive chat with Rich UI (load images with /image command)

Inline chat mode

Python Copy Section

Turn 1 -- cache miss, encodes image

Turn 2 -- cache hit, skips vision encoder

Server Copy Section

Performance Copy Section

TurboQuant KV Cache Copy Chapter

Quick Start Copy Section

3.5-bit KV cache quantization (3-bit keys + 4-bit values)

Server with TurboQuant

1. Project Overview & Quickstart (Blaizzy/mlx-vlm)

Table of Contents

Model-Specific Documentation

Installation

Agent Skills

Usage

Command Line Interface (CLI)

Speculative Decoding

Chat UI with Gradio

Python Script

Server (FastAPI)

Continuous Batching

Automatic Prefix Caching (APC)

1-bit Affine Inference

Activation Quantization (CUDA)

Command Line

Python API

Supported Quantization Modes

Multi-Image Chat Support

Usage Examples

Video Understanding

Supported Models

Usage Examples

Vision Feature Caching

How It Works

CLI

Python

Server

Performance

TurboQuant KV Cache

Quick Start

How It Works

Performance

Supported Bit Widths

Compatibility

Distributed Inference

LoRA & QLoRA

2. In-Tree Documentation Chapters (Blaizzy/mlx-vlm)

Index

Contributing

Mkdocs.Yml

CONTRIBUTING

Pull Requests

Issues

License