omlx Documentation

<h1 align="center">oMLX</h1>
LLM inference, optimized for your Mac Continuous batching and tiered KV caching, managed directly from your menu bar.

<a href="mailto:[email protected]">[email protected]</a> · <a href="https://omlx.ai/me">https://omlx.ai/me</a>

<a href="#install">Install</a> ·
<a href="#quickstart">Quickstart</a> ·
<a href="#features">Features</a> ·
<a href="#models">Models</a> ·
<a href="#cli-configuration">CLI Configuration</a> ·
<a href="https://omlx.ai/benchmarks">Benchmarks</a> ·
<a href="https://omlx.ai">oMLX.ai</a>

English ·
<a href="README.zh.md">中文</a> ·
<a href="README.ko.md">한국어</a> ·
<a href="README.ja.md">日本語</a>

---

Every LLM server I tried made me choose between convenience and control. I wanted to pin everyday models in memory, auto-swap heavier ones on demand, set context limits - and manage it all from a menu bar.

> oMLX persists KV cache across a hot in-memory tier and cold SSD tier - even when context changes mid-conversation, all past context stays cached and reusable across requests, making local LLMs practical for real coding work with tools like Claude Code. That's why I built it.

Install

macOS App

Download the .dmg from Releases, drag to Applications, done. The app includes in-app auto-update, so future upgrades are just one click. The macOS app also installs a lightweight ~/.omlx/bin/omlx CLI shim so terminal commands and Apple Shortcuts can control the app-managed server.

Homebrew

bash

brew tap jundot/omlx https://github.com/jundot/omlx
brew install omlx
Upgrade to the latest version

brew update && brew upgrade omlx
Run as a background service (auto-restarts on crash)

omlx start
Optional: MCP (Model Context Protocol) support

/opt/homebrew/opt/omlx/libexec/bin/pip install mcp

Optional GLM-5.2 / MiniMax M3 native custom kernels currently require a HEAD build:

bash

brew install omlx --HEAD --with-custom-kernel

From Source

bash

git clone https://github.com/jundot/omlx.git
cd omlx
pip install -e .          # Core only
pip install -e ".[mcp]"   # With MCP (Model Context Protocol) support
GLM-5.2 / MiniMax M3 / Qwen3.5 native custom kernels (strongly recommended

if you serve those families -- see note below)

OMLX_WITH_CUSTOM_KERNEL=1 pip install -e .

Requires macOS 15.0+ (Sequoia), Python 3.11–3.13, and Apple Silicon (M1/M2/M3/M4).

Note on native custom kernels: a plain pip install -e . does NOT build

them, and the affected model families then silently fall back to much slower

generic paths -- for GLM-5.2 the fused DSA prefill is roughly 30x faster with

the kernels (measured 845 vs ~29 tok/s on an M3 Ultra), and the fallback also

uses more memory (#2137). Building them requires the Metal toolchain, which

Command Line Tools alone do not provide (xcrun: error: unable to find utility

"metal"): install full Xcode, or use the official DMG which ships the kernels

precompiled. Homebrew can build them with brew install omlx --HEAD

--with-custom-kernel, but that build also needs full Xcode. To verify your

install:

> ``bash


python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_status())"

`

`Quickstart`

`macOS App`

Launch oMLX from your Applications folder. The Welcome screen guides you through three steps - model directory, server start, and first model download. That's it. To connect OpenClaw, OpenCode, Codex, Hermes Agent, or Copilot, see Integrations.

 <img src="docs/images/Screenshot 2026-02-10 at 00.36.32.png" alt="oMLX Welcome Screen" width="360"> <img src="docs/images/Screenshot 2026-02-10 at 00.34.30.png" alt="oMLX Menubar" width="240"> 

`CLI`

bash

Managed background server (macOS app or Homebrew install)

omlx start
omlx stop
omlx restart
Foreground server attached to this terminal

omlx serve --model-dir ~/models

The server discovers LLMs, VLMs, embedding models, and rerankers from subdirectories automatically. Any OpenAI-compatible client can connect to http://localhost:8000/v1. A built-in chat UI is also available at http://localhost:8000/admin/chat.

`Homebrew Service`

If you installed via Homebrew, you can run oMLX as a managed background service:

bash

omlx start                    # Start via brew services
omlx stop                     # Stop
omlx restart                  # Restartbrew services start omlx    # Start (auto-restarts on crash)
brew services stop omlx     # Stop
brew services restart omlx  # Restart
brew services info omlx     # Check status

The service runs omlx serve with zero-config defaults (~/.omlx/models, port 8000). omlx start, omlx stop, and omlx restart are the portable lifecycle commands; Homebrew installs delegate them to brew services. To customize, either set environment variables (OMLX_MODEL_DIR, OMLX_PORT, etc.) or run omlx serve --model-dir /your/path once to persist settings to ~/.omlx/settings.json.

Logs are written to two locations: - Service log:$(brew --prefix)/var/log/omlx.log(stdout/stderr) - Server log:~/.omlx/logs/server.log (structured application log)

`Features`

Supports text LLMs, vision-language models (VLM), OCR models, embeddings, and rerankers on Apple Silicon.

`Admin Dashboard`

Web UI at /admin for real-time monitoring, model management, chat, benchmark, and per-model settings. Supports English, Korean, Japanese, Chinese, French, Russian, Spanish, and Brazilian Portuguese. All CDN dependencies are vendored for fully offline operation.

 <img src="docs/images/Screenshot 2026-02-10 at 00.45.34.png" alt="oMLX Admin Dashboard" width="720"> 

`Experimental Multi-Mac Inference`

Source builds can split one downloaded language model across unequal-memory Macs using MLX pipeline ranks over Ring or Thunderbolt RDMA/JACCL. The Cluster dashboard handles read-only peer discovery, strict SSH/runtime verification, byte-aware unequal shard planning, measured compute/link rebalancing, headroom-aware execution tuning, activation, and a live shard/performance map on both Macs. Interactive, balanced, and throughput profiles expose coalesced batching, prompt-cache affinity, rotating-KV limits, Ring connection tuning, and a capability-gated experimental token-only output path. See Distributed inference across Macs for setup, security boundaries, current limitations, and the physical-hardware validation checklist.

`Vision-Language Models`

Run VLMs with the same continuous batching and tiered KV cache stack as text LLMs. Supports multi-image chat, base64/URL/file image inputs, and tool calling with vision context. OCR models (DeepSeek-OCR, DOTS-OCR, GLM-OCR) are auto-detected with optimized prompts.

`Tiered KV Cache (Hot + Cold)`

Block-based KV cache management inspired by vLLM, with prefix sharing and Copy-on-Write. The cache operates across two tiers:

- Hot tier (RAM): Frequently accessed blocks stay in memory for fast access. - Cold tier (SSD): When the hot cache fills up, blocks are offloaded to SSD in safetensors format. On the next request with a matching prefix, they're restored from disk instead of recomputed from scratch - even after a server restart.

 <img src="docs/images/omlx_hot_cold_cache.png" alt="oMLX Hot & Cold Cache" width="720"> 

`Continuous Batching`

Handles concurrent requests through mlx-lm's BatchGenerator. Max concurrent requests is configurable via CLI or admin panel.

`Claude Code Optimization`

Context scaling support for running smaller context models with Claude Code. Scales reported token counts so that auto-compact triggers at the right timing, and SSE keep-alive prevents read timeouts during long prefill.

`Multi-Model Serving`

Load LLMs, VLMs, embedding models, and rerankers within the same server. Models are managed through a combination of automatic and manual controls:

- LRU eviction: Least-recently-used models are evicted automatically when memory runs low. - Manual load/unload: Interactive status badges in the admin panel let you load or unload models on demand. - Model pinning: Pin frequently used models to keep them always loaded. - Per-model TTL: Set an idle timeout per model to auto-unload after a period of inactivity. - Process memory enforcement: Total memory limit (default: system RAM - 8GB) prevents system-wide OOM.

`Per-Model Settings`

Configure sampling parameters, chat template kwargs, TTL, model alias, model type override, and more per model directly from the admin panel. Changes apply immediately without server restart.

- Model alias: set a custom API-visible name. /v1/modelsreturns the alias, and requests accept both the alias and directory name. - Model type override: manually set a model as LLM or VLM regardless of auto-detection. - Profiles: save named bundles of per-model settings and switch between them from the admin panel. A profile can optionally be exposed as its own model:/v1/models then also lists <model>:<profile> (e.g. qwen3-8b:thinking), which serves on the same engine as the base model with the profile's settings overlaid per request — no extra memory, no reload. When the base model has an alias, the exposed ID is advertised as <alias>:<profile>; the directory-name form keeps working, just like for the base model.

 <img src="docs/images/omlx_ChatTemplateKwargs.png" alt="oMLX Chat Template Kwargs" width="480"> 

`Built-in Chat`

Chat directly with any loaded model from the admin panel. Supports conversation history, model switching, dark mode, reasoning model output, and image upload for VLM/OCR models.

 <img src="docs/images/ScreenShot_2026-03-14_104350_610.png" alt="oMLX Chat" width="720"> 

`Model Downloader`

Search and download MLX models from HuggingFace directly in the admin dashboard. Browse model cards, check file sizes, and download with one click.

 <img src="docs/images/downloader_omlx.png" alt="oMLX Model Downloader" width="720"> 

`Integrations`

Set up OpenClaw, OpenCode, Codex, Hermes Agent, Copilot, and Pi directly from the admin dashboard with a single click. No manual config editing required.

 <img src="docs/images/omlx_integrations.png" alt="oMLX Integrations" width="720"> 

`Performance Benchmark`

One-click benchmarking from the admin panel. Measures prefill (PP) and text generation (TG) tokens per second, with partial prefix cache hit testing for realistic performance numbers.

 <img src="docs/images/benchmark_omlx.png" alt="oMLX Benchmark Tool" width="720"> 

`macOS Menubar App`

Native Swift / SwiftUI menubar app (not Electron). Start, stop, and monitor the server without opening a terminal. Includes persistent serving stats (survives restarts), auto-restart on crash, and Sparkle-driven auto-update.

 <img src="docs/images/Screenshot 2026-02-10 at 00.51.54.png" alt="oMLX Menubar Stats" width="400"> 

`API Compatibility`

Drop-in replacement for OpenAI and Anthropic APIs. Supports streaming usage stats (stream_options.include_usage), Anthropic adaptive thinking, and vision inputs (base64, URL).

| Endpoint | Description | |----------|-------------| |POST /v1/chat/completions| Chat completions (streaming) | |POST /v1/completions| Text completions (streaming) | |POST /v1/messages| Anthropic Messages API | |POST /v1/embeddings| Text embeddings | |POST /v1/rerank| Document reranking | |GET /v1/models | List available models |

`Tool Calling & Structured Output`

Supports all function calling formats available in mlx-lm, JSON schema validation, and MCP tool integration. Tool calling requires the model's chat template to support the tools parameter. The following model families are auto-detected via mlx-lm's built-in tool parsers:

| Model Family | Format | |---|---| | Llama, Qwen, DeepSeek, etc. | JSON<tool_call>| | Qwen3.5 Series | XML<function=...>| | Gemma |<start_function_call>| | GLM (4.7, 5) |<arg_key>/<arg_value>XML | | MiniMax | Namespaced<minimax:tool_call>| | Mistral |[TOOL_CALLS]| | Kimi K2 |<\|tool_calls_section_begin\|>| | Longcat |<longcat_tool_call> |

Models not listed above may still work if their chat template accepts tools and their output uses a recognized <tool_call> XML format. For tool-enabled streaming, assistant text is emitted incrementally while known tool-call control markup is suppressed from visible content; structured tool calls are emitted after parsing the completed turn.

`Models`

Point --model-dir at a directory containing MLX-format model subdirectories. Two-level organization folders (e.g., mlx-community/model-name/) are also supported.

text

~/models/
├── Step-3.5-Flash-8bit/
├── Qwen3-Coder-Next-8bit/
├── gpt-oss-120b-MXFP4-Q8/
├── Qwen3.5-122B-A10B-4bit/
└── bge-m3/

Models are auto-detected by type. You can also download models directly from the admin dashboard.

| Type | Models | |------|--------| | LLM | Any model supported by mlx-lm | | VLM | Qwen3.5 Series, GLM-4V, Pixtral, and other mlx-vlm models | | OCR | DeepSeek-OCR, DOTS-OCR, GLM-OCR | | Embedding | BERT, BGE-M3, ModernBERT | | Reranker | ModernBERT, XLM-RoBERTa |

`CLI Configuration`

bash

Managed background server (macOS app or Homebrew install)

omlx start
omlx stop
omlx restart
Start with default settings (memory guard tier = balanced, manage via admin UI)

omlx serve --model-dir ~/models
Choose a memory guard tier at startup

omlx serve --model-dir ~/models --memory-guard safe
Set a custom memory guard ceiling in GB

omlx serve --model-dir ~/models --memory-guard-gb 48
Enable SSD cache for KV blocks

omlx serve --model-dir ~/models --paged-ssd-cache-dir ~/.omlx/cache
Set in-memory hot cache size

omlx serve --model-dir ~/models --hot-cache-max-size 20%
Adjust max concurrent requests (default: 8)

omlx serve --model-dir ~/models --max-concurrent-requests 16
With MCP tools

omlx serve --model-dir ~/models --mcp-config mcp.json
HuggingFace mirror endpoint (for restricted regions)

omlx serve --model-dir ~/models --hf-endpoint https://hf-mirror.com
API key authentication

omlx serve --model-dir ~/models --api-key your-secret-key
Localhost-only: skip verification via admin panel global settings

All settings can also be configured from the web admin panel at /admin. Settings are persisted to ~/.omlx/settings.json, and CLI flags take precedence.

<details> <summary>Architecture</summary>

text

FastAPI Server (OpenAI / Anthropic API)
    │
    ├── EnginePool (multi-model, LRU eviction, TTL, manual load/unload)
    │   ├── BatchedEngine (LLMs, continuous batching)
    │   ├── VLMEngine (vision-language models)
    │   ├── EmbeddingEngine
    │   └── RerankerEngine
    │
    ├── ProcessMemoryEnforcer (total memory limit, TTL checks)
    │
    ├── Scheduler (FCFS, configurable concurrency)
    │   └── mlx-lm BatchGenerator
    │
    └── Cache Stack
        ├── PagedCacheManager (GPU, block-based, CoW, prefix sharing)
        ├── Hot Cache (in-memory tier, write-back)
        └── PagedSSDCacheManager (SSD cold tier, safetensors format)

</details>

`Development`

`CLI Server`

bash

git clone https://github.com/jundot/omlx.git
cd omlx
pip install -e ".[dev]"
pytest -m "not slow"

`macOS App`

The native SwiftUI app lives at apps/omlx-mac/. Requires Xcode 26.5+ and Python 3.11+. venvstacks is declared as a dev dependency so pip install -e ".[dev]" (or uv sync --dev) brings the pinned version in. The build script also falls back to uvx venvstacks or pipx run venvstacks if you prefer a host-global tool runner.

bash

Stage a runnable oMLX.app (xcodebuild + venvstacks Python layers + ad-hoc sign)

apps/omlx-mac/Scripts/build.sh release
Result lands at apps/omlx-mac/build/Stage/oMLX.app

open apps/omlx-mac/build/Stage/oMLX.app
Force a fresh venvstacks rebuild (otherwise it's cached by fingerprint)

apps/omlx-mac/Scripts/build.sh release --rebuild-donor
Stage with optional GLM-5.2 / MiniMax M3 native custom kernels

apps/omlx-mac/Scripts/build.sh release --with-custom-kernel

First cold build takes 10–20 minutes (venvstacks Python layer assembly). Subsequent builds reuse the cached packaging/_export/` and finish in about 4 minutes. See packaging/README.md for the layer configuration and apps/omlx-mac/ for the Swift sources.

Contributing

Contributions are welcome! See Contributing Guide for details.

- Bug fixes and improvements
- Performance optimizations
- Documentation improvements

License

Apache 2.0

Acknowledgments

- MLX and mlx-lm by Apple
- mlx-vlm - Vision-language model inference on Apple Silicon
- vllm-mlx - oMLX started from vllm-mlx v0.1.0 and evolved significantly with multi-model serving, tiered KV caching, VLM with full paged cache support, an admin panel, and a macOS menu bar app
- venvstacks - Portable Python environment layering for the macOS app bundle
- mlx-embeddings - Embedding model support for Apple Silicon
- dflash-mlx - Block diffusion speculative decoding on Apple Silicon
- MTPLX - Lightning MTP's verify-shape Metal kernels are powered by MTPLX by Youssof Altoukhi, which also inspired the depth-k pipeline
- SiliconScope - The menu bar statistics take their design and rendering approach from SiliconScope by Kennt Kim, which also inspired the energy-efficient re-render gating

Install Copy Chapter

macOS App Copy Section

Homebrew Copy Section

Upgrade to the latest version

Run as a background service (auto-restarts on crash)

Optional: MCP (Model Context Protocol) support

From Source Copy Section

GLM-5.2 / MiniMax M3 / Qwen3.5 native custom kernels (strongly recommended

if you serve those families -- see note below)

Quickstart Copy Chapter

macOS App Copy Section

CLI Copy Section

Managed background server (macOS app or Homebrew install)

Foreground server attached to this terminal

Homebrew Service Copy Section

Features Copy Chapter

Admin Dashboard Copy Section

Experimental Multi-Mac Inference Copy Section

Vision-Language Models Copy Section

Tiered KV Cache (Hot + Cold) Copy Section

Continuous Batching Copy Section

Claude Code Optimization Copy Section

Multi-Model Serving Copy Section

Per-Model Settings Copy Section

Built-in Chat Copy Section

Model Downloader Copy Section

Integrations Copy Section

Performance Benchmark Copy Section

macOS Menubar App Copy Section

API Compatibility Copy Section

Tool Calling & Structured Output Copy Section

Models Copy Chapter

CLI Configuration Copy Chapter

Managed background server (macOS app or Homebrew install)

Start with default settings (memory guard tier = balanced, manage via admin UI)

Choose a memory guard tier at startup

Set a custom memory guard ceiling in GB

Enable SSD cache for KV blocks

Set in-memory hot cache size

Adjust max concurrent requests (default: 8)

With MCP tools

HuggingFace mirror endpoint (for restricted regions)

API key authentication

Localhost-only: skip verification via admin panel global settings

Development Copy Chapter

CLI Server Copy Section

macOS App Copy Section

Stage a runnable oMLX.app (xcodebuild + venvstacks Python layers + ad-hoc sign)

Result lands at apps/omlx-mac/build/Stage/oMLX.app

Force a fresh venvstacks rebuild (otherwise it's cached by fingerprint)

Stage with optional GLM-5.2 / MiniMax M3 native custom kernels

Contributing Copy Chapter

License Copy Chapter

Acknowledgments Copy Chapter

Install

macOS App

Homebrew

From Source

`Quickstart`

`macOS App`

`CLI`

`Homebrew Service`

`Features`

`Admin Dashboard`

`Experimental Multi-Mac Inference`

`Vision-Language Models`

`Tiered KV Cache (Hot + Cold)`

`Continuous Batching`

`Claude Code Optimization`

`Multi-Model Serving`

`Per-Model Settings`

`Built-in Chat`

`Model Downloader`

`Integrations`

`Performance Benchmark`

`macOS Menubar App`

`API Compatibility`

`Tool Calling & Structured Output`

`Models`

`CLI Configuration`

`Development`

`CLI Server`

`macOS App`

Contributing

License

Acknowledgments