shimmy Documentation

Technical Documentation: Michael-A-Kuykendall/shimmy

ℹ️ Provenance: Hybrid Fusion: Michael-A-Kuykendall/shimmy (README + 10 In-Tree Chapters) · CodeWiki Reference · Recency: Active (< 180 days)

1. Project Overview & Quickstart (Michael-A-Kuykendall/shimmy) Copy Chapter

Shimmy will be free forever. No asterisks. No "free for now." No pivot to paid.

💝 Support Shimmy's Growth Copy Section

🚀 If Shimmy helps you, consider sponsoring — 100% of support goes to keeping it free forever.

• $5/month: Coffee tier ☕ - Eternal gratitude + sponsor badge
• $25/month: Bug prioritizer 🐛 - Priority support + name in SPONSORS.md
• $100/month: Corporate backer 🏢 - Logo placement + monthly office hours
• $500/month: Infrastructure partner 🚀 - Direct support + roadmap input

🎯 Become a Sponsor | See our amazing sponsors 🙏

What Is Shimmy? Copy Chapter

Shimmy is a single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.

Under the hood it runs on Airframe, a pure-Rust WebGPU (WGSL) transformer engine built from scratch. No C++ toolchain, no backend flags, no compilation required. Version history lives in the CHANGELOG; see the Airframe CHANGELOG for engine release notes.

Why this matters:

• No C++ toolchain required — Rust only, top to bottom
• F32 precision throughout for deterministic, high-quality output
• WGSL compute shaders work on any GPU via WebGPU (NVIDIA, AMD, Intel, integrated)
• Model spec auto-derived from GGUF metadata — no hardcoded per-model constants
• YaRN RoPE scaling for extended context via SHIMMY_MAX_CTX (see Extended Context)

🎯 Supported Models Copy Chapter

11 model families · 25 certified model/quant combinations — every model below passes Shimmy's 5-gate GPU math verification pipeline (dequant, structural peel, numerical, decode≡prefill, logits) against the certification ledger. GGUF files load as-is; no recompilation, no hardcoded per-model constants.

Features at a Glance Copy Chapter

• ⚡ TurboShimmy INT4 KV Cache — ~7× less KV VRAM with one flag (--kv-quant int4). Run Llama-3.2-3B on 4 GB GPUs.
• 🚀 OpenAI SDK Compatibility — drop-in replacement; VSCode Copilot, Cursor, Continue.dev, any OpenAI SDK.
• 🔧 Extended Context — YaRN RoPE scaling via SHIMMY_MAX_CTX.
• 📦 Migrating from v1.x — the llama.cpp backend was removed in v2.0; see the migration guide.
• 🧠 MOE support — Mixture-of-Experts CPU offloading is on the Airframe roadmap.
• 🏆 Certification — Every model passes a 5-gate mathematical verification pipeline. See docs/CERTIFICATION.md for how it works.

Quick Start (30 seconds) Copy Chapter

# 1) Download pre-built binary (Windows example)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe

# 2) Point it at a GGUF model
set SHIMMY_BASE_GGUF=C:\path\to\model.gguf && ./shimmy.exe serve &

# 3) See registered models
./shimmy list

# 4) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}' \
  | jq -r '.choices[0].message.content'

Full install, model acquisition, GPU, and VRAM sizing: docs/quickstart.md

Documentation Hub Copy Chapter

Full documentation lives in docs/. Use this table to find what you need:

Getting Started Copy Section

Models & Performance Copy Section

API & Integration Copy Section

Engine Deep Dives Copy Section

FAQ & Troubleshooting Copy Section

Certification & Methodology Copy Section

Development Testing Copy Chapter

Shimmy maintains high code quality through comprehensive testing:

# Full test suite (default features = GPU engine)
cargo test --features airframe,huggingface

# Quick CPU-only tests (no GPU required)
cargo test --lib --no-default-features --features huggingface -- --test-threads=1

See docs/ppt-invariant-testing.md for technical details.

Community & Support Copy Chapter

• 🐛 Bug Reports: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 💝 Sponsorship: GitHub Sponsors

Star History Copy Section

🚀 Momentum Snapshot Copy Section

🌟 ** stars and climbing fast**
⏱ <1s startup
🦀 100% Rust, no Python

📰 As Featured On Copy Section

🔥 Hacker News • Front Page Again • IPE Newsletter

Companies: Need invoicing? Email [email protected]

Performance Comparison Copy Chapter

License & Philosophy Copy Chapter

MIT License - forever and always.

Philosophy: Infrastructure should be invisible. Shimmy is infrastructure.

Testing Philosophy: Reliability through comprehensive validation and property-based testing.

Forever maintainer: Michael A. Kuykendall
Promise: This will never become a paid product
Mission: Making local model inference simple and reliable

2. In-Tree Documentation Chapters (Michael-A-Kuykendall/shimmy) Copy Chapter

File: README.md Copy Chapter

Shimmy will be free forever. No asterisks. No "free for now." No pivot to paid.

💝 Support Shimmy's Growth Copy Section

🚀 If Shimmy helps you, consider sponsoring — 100% of support goes to keeping it free forever.

• $5/month: Coffee tier ☕ - Eternal gratitude + sponsor badge
• $25/month: Bug prioritizer 🐛 - Priority support + name in SPONSORS.md
• $100/month: Corporate backer 🏢 - Logo placement + monthly office hours
• $500/month: Infrastructure partner 🚀 - Direct support + roadmap input

🎯 Become a Sponsor | See our amazing sponsors 🙏

What Is Shimmy? Copy Chapter

Shimmy is a single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.

Under the hood it runs on Airframe, a pure-Rust WebGPU (WGSL) transformer engine built from scratch. No C++ toolchain, no backend flags, no compilation required. Version history lives in the CHANGELOG; see the Airframe CHANGELOG for engine release notes.

Why this matters:

• No C++ toolchain required — Rust only, top to bottom
• F32 precision throughout for deterministic, high-quality output
• WGSL compute shaders work on any GPU via WebGPU (NVIDIA, AMD, Intel, integrated)
• Model spec auto-derived from GGUF metadata — no hardcoded per-model constants
• YaRN RoPE scaling for extended context via SHIMMY_MAX_CTX (see Extended Context)

🎯 Supported Models Copy Chapter

11 model families · 25 certified model/quant combinations — every model below passes Shimmy's 5-gate GPU math verification pipeline (dequant, structural peel, numerical, decode≡prefill, logits) against the certification ledger. GGUF files load as-is; no recompilation, no hardcoded per-model constants.

Features at a Glance Copy Chapter

• ⚡ TurboShimmy INT4 KV Cache — ~7× less KV VRAM with one flag (--kv-quant int4). Run Llama-3.2-3B on 4 GB GPUs.
• 🚀 OpenAI SDK Compatibility — drop-in replacement; VSCode Copilot, Cursor, Continue.dev, any OpenAI SDK.
• 🔧 Extended Context — YaRN RoPE scaling via SHIMMY_MAX_CTX.
• 📦 Migrating from v1.x — the llama.cpp backend was removed in v2.0; see the migration guide.
• 🧠 MOE support — Mixture-of-Experts CPU offloading is on the Airframe roadmap.
• 🏆 Certification — Every model passes a 5-gate mathematical verification pipeline. See docs/CERTIFICATION.md for how it works.

Quick Start (30 seconds) Copy Chapter

# 1) Download pre-built binary (Windows example)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe

# 2) Point it at a GGUF model
set SHIMMY_BASE_GGUF=C:\path\to\model.gguf && ./shimmy.exe serve &

# 3) See registered models
./shimmy list

# 4) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}' \
  | jq -r '.choices[0].message.content'

Full install, model acquisition, GPU, and VRAM sizing: docs/quickstart.md

Documentation Hub Copy Chapter

Full documentation lives in docs/. Use this table to find what you need:

Getting Started Copy Section

Models & Performance Copy Section

API & Integration Copy Section

Engine Deep Dives Copy Section

FAQ & Troubleshooting Copy Section

Certification & Methodology Copy Section

Development Testing Copy Chapter

Shimmy maintains high code quality through comprehensive testing:

# Full test suite (default features = GPU engine)
cargo test --features airframe,huggingface

# Quick CPU-only tests (no GPU required)
cargo test --lib --no-default-features --features huggingface -- --test-threads=1

See docs/ppt-invariant-testing.md for technical details.

Community & Support Copy Chapter

• 🐛 Bug Reports: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 💝 Sponsorship: GitHub Sponsors

Star History Copy Section

🚀 Momentum Snapshot Copy Section

🌟 ** stars and climbing fast**
⏱ <1s startup
🦀 100% Rust, no Python

📰 As Featured On Copy Section

🔥 Hacker News • Front Page Again • IPE Newsletter

Companies: Need invoicing? Email [email protected]

Performance Comparison Copy Chapter

License & Philosophy Copy Chapter

MIT License - forever and always.

Philosophy: Infrastructure should be invisible. Shimmy is infrastructure.

Testing Philosophy: Reliability through comprehensive validation and property-based testing.

Forever maintainer: Michael A. Kuykendall
Promise: This will never become a paid product
Mission: Making local model inference simple and reliable

File: deploy/README.md Copy Chapter

Shimmy Cloud Deployment

One-click deployment configurations for popular cloud platforms.

Quick Deploy Buttons Copy Chapter

Railway Copy Section

Render Copy Section

Fly.io Copy Section

# Install flyctl and deploy
curl -L https://fly.io/install.sh | sh
fly deploy

Docker (Any Platform) Copy Section

# Local development
docker-compose up

# Production with Nginx
docker-compose --profile production up

Platform-Specific Instructions Copy Chapter

Railway.app Copy Section

1. Click the "Deploy on Railway" button above
2. Connect your GitHub account
3. Fork this repository
4. Railway will automatically build and deploy
5. Your Shimmy instance will be available at https://your-app.railway.app

Render.com Copy Section

1. Click the "Deploy to Render" button above
2. Connect your GitHub repository
3. Render will use the render.yaml configuration
4. Your service will be available with automatic HTTPS

Fly.io Copy Section

1. Install the Fly CLI: curl -L https://fly.io/install.sh | sh
2. Clone this repository: git clone https://github.com/Michael-A-Kuykendall/shimmy.git
3. Navigate to the project: cd shimmy
4. Create and deploy: fly deploy
5. Access your app: fly open

Google Cloud Run Copy Section

# Build and deploy to Cloud Run
gcloud builds submit --tag gcr.io/PROJECT-ID/shimmy
gcloud run deploy --image gcr.io/PROJECT-ID/shimmy --platform managed

AWS App Runner Copy Section

1. Create apprunner.yaml in your repository root:

version: 1.0
runtime: docker
build:
  commands:
    build:
      - echo "Building Shimmy with Docker"
run:
  runtime-version: latest
  command: shimmy serve --bind 0.0.0.0:8080
  network:
    port: 8080

DigitalOcean App Platform Copy Section

1. Create app via DigitalOcean control panel
2. Connect your GitHub repository
3. DigitalOcean will detect the Dockerfile automatically
4. Set environment variables as needed

Environment Variables Copy Chapter

Resource Requirements Copy Chapter

Minimal Copy Section

• CPU: 0.5 vCPU
• Memory: 512MB RAM
• Storage: 100MB (binary only)

Recommended Copy Section

• CPU: 1 vCPU
• Memory: 1GB RAM
• Storage: 1GB+ (for model caching)

High Performance Copy Section

• CPU: 2+ vCPU
• Memory: 4GB+ RAM
• Storage: 10GB+ SSD

Security Considerations Copy Chapter

1. Authentication: Shimmy doesn't include built-in authentication. Use a reverse proxy (Nginx, Cloudflare) for auth.

2. Rate Limiting: The included Nginx configuration has basic rate limiting. Adjust as needed.

3. HTTPS: Most cloud platforms provide automatic HTTPS. For self-hosted deployments, configure SSL certificates.

4. Firewall: Only expose port 11434 (or your configured port) to the public internet.

Monitoring Copy Chapter

Health Checks Copy Section

All configurations include health checks at /health endpoint.

Logs Copy Section

Set RUST_LOG=debug for detailed logging. Most platforms provide log aggregation.

Metrics Copy Section

For production deployments, consider adding:

• Prometheus metrics
• Jaeger tracing
• Custom monitoring dashboards

Scaling Copy Chapter

Horizontal Scaling Copy Section

Shimmy is stateless and can be horizontally scaled. Use a load balancer to distribute requests.

Vertical Scaling Copy Section

For better performance with large models:

• Increase memory for model caching
• Add more CPU cores for parallel processing
• Use SSD storage for faster model loading

Troubleshooting Copy Chapter

Common Issues Copy Section

1. Out of Memory: Increase memory allocation or use memory-mapped loading
2. Slow Startup: Enable model caching and use persistent storage
3. Connection Timeout: Increase proxy timeout settings for large model inference

Debug Mode Copy Section

# Enable debug logging
RUST_LOG=debug shimmy serve

Container Debugging Copy Section

# Access running container
docker exec -it shimmy-container /bin/bash

# Check logs
docker logs shimmy-container

File: docs/benchmark-evidence/README.md Copy Chapter

MoE CPU Offloading Benchmark Evidence

<<<<<<< HEAD <<<<<<< HEAD **Date**: October 8, 2025

Date: October 8, 2025

main
=======
Date: October 8, 2025
main
Purpose: Raw benchmark data and logs for audit verification

Contents Copy Chapter

Streaming vs Non-Streaming Benchmarks Copy Section

• phi35-streaming-bench.log - Phi-3.5-MoE 41.9B performance comparison
• gpt-oss-streaming-bench.log - GPT-OSS 20B performance comparison
• deepseek-streaming-bench.log - DeepSeek MoE 16B performance comparison

Each log contains:

• 4 test prompts (short, medium, long, very long)
• Non-streaming TPS measurements
• Streaming TPS measurements with actual token counts
• TTFT (Time To First Token) estimates
• Performance delta calculations

Model Loading and Offloading Logs Copy Section

• shimmy-phi35.log - Phi-3.5-MoE server startup with CPU offloading
• shimmy-gpt-oss.log - GPT-OSS server startup with CPU offloading
• shimmy-deepseek.log - DeepSeek server startup with CPU offloading

Each log contains:

• Model architecture detection (expert count, active experts)
• Expert tensor CPU offloading confirmation
• Memory distribution (GPU vs CPU allocation)
• Context configuration

Verification Copy Chapter

These logs provide evidence for claims in the MoE CPU Offloading White Paper:

1. Expert Detection: Search for expert_count and expert_used_count in loading logs
2. CPU Offloading: Search for CUDA_Host buffer overrides in loading logs
3. Memory Savings: Search for CPU_Mapped and CUDA0 model buffer size in loading logs
4. Performance Data: Raw TPS and TTFT measurements in streaming-bench logs

Reproduction Copy Chapter

To reproduce these results:

# Start shimmy server with CPU offloading
cd /home/ubuntu/shimmy
SHIMMY_BASE_GGUF=/path/to/model.gguf \
  ./target/release/shimmy serve --bind 127.0.0.1:11435 --cpu-moe > server.log 2>&1 &

# Run streaming benchmark
./scripts/benchmark-moe-streaming.sh <model-name> > benchmark.log

# Compare results with evidence files in this directory

File Integrity Copy Chapter

Evidence preserved for audit verification and reproducibility

File: docs/releases/RELEASE_NOTES_v1.6.0.md Copy Chapter

Shimmy v1.6.0 - Windows CUDA Support + Critical Stability Fixes

🎯 Headline Features Copy Chapter

✨ Native Windows CUDA Support Copy Section

Shimmy is now the first lightweight Rust LLM tool with full Windows MSVC CUDA support.

• Fixed critical llama-cpp-rs Windows MSVC build issue via custom fork
• CUDA binaries build successfully on Windows (24MB binary, 11m 25s compile time)
• All 4 GPU backends now verified working on Windows: CUDA, Vulkan, OpenCL, HuggingFace
• Fork available at: Michael-A-Kuykendall/llama-cpp-rs (branch: fix-windows-msvc-cuda-stdbool)

Technical Achievement: Solved bindgen's inability to locate MSVC standard C headers by extracting paths from cc crate's environment detection and passing as -isystem arguments. This fix benefits the entire Rust + llama.cpp ecosystem.

🐛 Issue #72 Fixed: GPU Backend Flag Now Works Copy Section

Previous versions accepted the --gpu-backend flag but ignored it completely, assigning all layers to CPU.

What was broken:

• CLI parsed --gpu-backend auto|vulkan|opencl|cuda ✅
• Engine had gpu_backend field ✅
• BUT: Field was never used in model loading ❌
• AND: CLI value was never passed to engine constructor ❌

What's fixed:

• Added LlamaEngine::new_with_backend(Option<&str>) constructor
• Implemented auto-detection with priority: CUDA > Vulkan > OpenCL > CPU
• GPU backend now properly wired through all CLI commands (serve, generate, gpu-info)
• Verified with 13 comprehensive regression tests

User Impact: Your GPU will actually be used now when you specify --gpu-backend vulkan 🎉

🔧 Critical Stability Fixes Copy Chapter

Resolved RwLock Deadlock in Concurrent Operations Copy Section

Fixed an infinite hang when 20+ concurrent model loading operations occurred.

Root Cause: ModelManager::load_model() held a write lock while calling async functions that tried to acquire read locks on the same data, creating a circular dependency.

Solution: Extract data with locks, then drop locks before calling other functions. Simple pattern, massive impact.

Before: test_concurrent_load_unload hung indefinitely
After: Passes in 0.00s

Fixed All Test Failures Across Feature Combinations Copy Section

• 295/295 tests passing with full CUDA backend
• 284/284 tests passing with minimal features (huggingface only)
• 295/295 tests passing with any GPU backend (vulkan/opencl/cuda)

What we fixed:

1. PPT contract tests now properly guarded by feature flags
2. Removed flaky property tests that used broken property_test() wrapper
3. Fixed test_local_file_detection to work without llama backend
4. All tests now deterministic, no random failures

📊 Build Verification Copy Chapter

All 4 backends verified working on Windows with full test coverage.

🧹 Code Quality Improvements Copy Chapter

• Removed 700+ lines of dead code (unused specs, ModelCache, RouteManager)
• Fixed all clippy warnings
• Comprehensive audit and cleanup before release
• Updated all "sub-20MB" references back to "sub-5MB" (actual: 4.8MB)
• Removed AI marketing fluff from README

🔬 Technical Details Copy Chapter

Windows MSVC CUDA Fix Architecture Copy Section

The fix leverages cc::Build to extract MSVC's INCLUDE environment variable and passes those paths as -isystem arguments to bindgen's libclang:

// Extract MSVC include paths from cc crate
let include_paths = cc::Build::new()
    .target("x86_64-pc-windows-msvc")
    .get_compiler()
    .env()
    .iter()
    .filter(|(k, _)| k == "INCLUDE")
    .flat_map(|(_, v)| v.to_string_lossy().split(';').map(String::from).collect::<Vec<_>>())
    .collect::<Vec<_>>();

// Pass to bindgen
for path in include_paths {
    builder = builder.clang_arg(format!("-isystem{}", path));
}

Similar to the Android fix in llama-cpp-rs (lines 390-414), but adapted for MSVC's environment.

🚀 What's Next Copy Chapter

• Consider upstream PR to llama-cpp-rs with Windows MSVC fix
• Continue monitoring regression gates for stability
• Explore optimizations for faster CUDA build times

📦 Installation Copy Chapter

# Minimal build (HuggingFace only)
cargo install shimmy --no-default-features --features huggingface

# Vulkan (fastest compile, 45s)
cargo install shimmy --no-default-features --features huggingface,llama-vulkan

# OpenCL (AMD/Intel GPUs)
cargo install shimmy --no-default-features --features huggingface,llama-opencl

# CUDA (NVIDIA GPUs, longer compile)
cargo install shimmy --no-default-features --features huggingface,llama-cuda

🙏 Credits Copy Chapter

• Issue #72 reported by @D0wn10ad
• Windows CUDA testing and validation by the community
• llama-cpp-rs team for the excellent bindings foundation

Shimmy: 4.8MB. No Python. No bloat. Now with Windows CUDA support.

142x smaller than Ollama. 2x faster model loading. 100% Rust.

File: docs/releases/RELEASE_NOTES_v1.7.0.md Copy Chapter

Shimmy v1.7.0 - Mixture of Experts CPU Offloading Release

<<<<<<< HEAD **Released:** January 9, 2025

Released: January 9, 2025

main
Branch: feat/moe-cpu-offload

🎯 Headline Features Copy Chapter

Mixture of Experts (MoE) CPU Offloading Support Copy Section

Major new capability enabling large MoE models on consumer GPUs - requested by @razvanab in Issue #81.

This release adds full support for offloading Mixture of Experts (MoE) model weights to CPU memory, dramatically reducing VRAM requirements while maintaining usable inference performance. Now you can run massive models like GPT-OSS 20B, Phi-3.5-MoE 42B, and DeepSeek-16B on GPUs with limited VRAM.

New CLI Flags:

• --cpu-moe - Offload all MoE expert tensors to CPU memory
• --n-cpu-moe N - Offload N expert layers to CPU (partial offloading)

Performance Achievements:

• 78%-94% VRAM reduction across tested models
• 2.5x-6.9x speed penalty (acceptable for development/prototyping)
• Successfully validated on Lambda Cloud GH200 (96GB VRAM)

Example Usage:

# Full CPU offload (maximum VRAM savings)
shimmy serve --cpu-moe --model gpt-oss-20b.gguf

# Partial offload (balance VRAM vs speed)
shimmy serve --n-cpu-moe 64 --model phi-3.5-moe.gguf

# Generate with offloading
shimmy generate --cpu-moe --model deepseek-16b.gguf --prompt "Hello"

Technical Implementation:

• Rust bindings to llama.cpp's MoE offloading functionality via llama-cpp-2 fork
• Integration through engine adapter with global CLI flags
• Verified with 144 expert tensors successfully offloaded on GPT-OSS 20B
• Comprehensive testing: 36/36 test runs passing (3 models × 2 configs × 3 runs × 2 quantizations)

📦 Quantized Models Released Copy Chapter

Six professionally quantized MoE models uploaded to HuggingFace with comprehensive model cards following bartowski/Microsoft standards:

Phi-3.5-MoE Quantizations (from 79GB F16) Copy Section

1. Q2_K - Ultra-Compressed (15GB, 81% reduction)

• Repository: MikeKuykendall/phi-3.5-moe-q2-k-cpu-offload-gguf
• Direct Download: phi-3.5-moe-q2-k-cpu-offload.gguf (15.0 GB)
• Use Case: Maximum compression, development/testing, low VRAM systems
• Quality: Acceptable for most tasks, noticeable quality loss vs F16

2. Q4_K_M - Recommended (24GB, 70% reduction)

• Repository: MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf
• Direct Download: phi-3.5-moe-q4-k-m-cpu-offload.gguf (23.8 GB)
• Use Case: Best quality/size balance, general production use
• Quality: Minimal quality loss vs F16, recommended for most users

3. Q8_0 - High Quality (42GB, 47% reduction)

• Repository: MikeKuykendall/phi-3.5-moe-q8-0-cpu-offload-gguf
• Direct Download: phi-3.5-moe-q8-0-cpu-offload.gguf (41.7 GB)
• Use Case: Maximum quality, near F16 performance
• Quality: Virtually identical to F16

DeepSeek-MoE-16B Quantizations (from 31GB F16) Copy Section

4. Q2_K - Ultra-Compressed (6.3GB, 80% reduction)

• Repository: MikeKuykendall/deepseek-moe-16b-q2-k-cpu-offload-gguf
• Direct Download: deepseek-moe-16b-q2-k-cpu-offload.gguf (6.32 GB)
• Use Case: Maximum compression, development/testing
• Quality: Acceptable for most tasks, noticeable quality loss vs F16

5. Q4_K_M - Recommended (11GB, 65% reduction)

• Repository: MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf
• Direct Download: deepseek-moe-16b-q4-k-m-cpu-offload.gguf (10.9 GB)
• Use Case: Best quality/size balance, general production use
• Quality: Minimal quality loss vs F16, recommended for most users

6. Q8_0 - High Quality (17GB, 45% reduction)

• Repository: MikeKuykendall/deepseek-moe-16b-q8-0-cpu-offload-gguf
• Direct Download: deepseek-moe-16b-q8-0-cpu-offload.gguf (16.7 GB)
• Use Case: Maximum quality, near F16 performance
• Quality: Virtually identical to F16

Model Card Features:

• Proper YAML metadata (language, license, tags, base_model, pipeline_tag)
• Real performance benchmarks from controlled A/B testing
• VRAM usage with/without CPU offloading
• Token generation speeds (TPS) with detailed methodology
• Usage examples for shimmy CLI integration
• Quantization methodology and technical specifications

Complete Model Comparison Table Copy Section

⭐ = Recommended quantization level for production use

Testing Validation:

• 36 baseline tests completed (100% success rate)
• N=3 statistical runs per configuration for reliability
• Controlled A/B comparisons (with/without --cpu-moe)
• Lambda Cloud GH200 infrastructure (96GB VRAM, 72 CPU cores)
• shimmy v1.6.0 used for all test runs

🔧 Technical Details Copy Chapter

Upstream Contributions Copy Section

llama-cpp-rs Fork Integration:

• Using custom fork: utilityai/llama-cpp-rs (branch: feat/moe-cpu-offload)
• Added Rust bindings: with_cpu_moe_all(), with_n_cpu_moe(n) methods
• Submitted upstream PR: utilityai/llama-cpp-rs#839 (CUDA stdbool fix)
• Clean integration via Cargo dependency override in Cargo.toml

Implementation Architecture:

CLI Flags (--cpu-moe, --n-cpu-moe)
    ↓
Global Config (MoeConfig struct)
    ↓
Engine Adapter (apply_moe_config)
    ↓
llama-cpp-2 Bindings (LlamaParams)
    ↓
llama.cpp MoE Offloading (native C++)

Performance Benchmarks Copy Section

Phi-3.5-MoE Q4_K_M (24GB model):

• Baseline (no offload): 11.55 TPS, ~23GB VRAM
• With --cpu-moe: 4.69 TPS, ~2MB VRAM (2.5x speed penalty, 99.9% VRAM reduction)

GPT-OSS 20B Q8_0 (17GB model):

• Baseline (no offload): 12.3 TPS, ~15GB VRAM
• With --cpu-moe: 1.78 TPS, ~2MB VRAM (6.9x speed penalty, 99.9% VRAM reduction)

DeepSeek-16B Q8_0 (17GB model):

• Baseline (no offload): 14.2 TPS, ~16GB VRAM
• With --cpu-moe: 3.1 TPS, ~2MB VRAM (4.6x speed penalty, 99.9% VRAM reduction)

TTFT (Time to First Token):

• Minimal impact: <500ms increase with CPU offloading
• Dominated by model loading, not offloading configuration

Code Quality Improvements Copy Section

Systematic Audit Cleanup (Phases 1-3):

• Phase 1 (I2 Pattern): Renamed 22 Java-style getters to Rust conventions
  • get_model() → model(), get_metrics() → metrics(), etc.
  • All call sites updated, 295/295 tests passing
    <<<<<<< HEAD

=======

main

• Phase 2 (N5 Pattern): Fixed 14 production unwraps with proper error handling

  • src/metrics.rs (5 unwraps), src/openai_compat.rs (3 unwraps)
  • Replaced with match, unwrap_or_else, unwrap_or patterns
  • 226+ test unwraps remain (acceptable - tests should panic)

• Phase 3 (A3_stringly Pattern): Converted 16+ string errors to typed ShimmyError

  • New variants: WorkflowStepNotFound, MlxNotAvailable, ToolExecutionFailed, etc.
  • Typed errors in workflow.rs, safetensors_adapter.rs, tools.rs, preloading.rs
  • Engine layer kept with anyhow::Result (clean boundary for third-party errors)

Build Verification:

• All 295 unit tests passing
• Zero compiler warnings (achieved clean build)
• Clippy clean (removed unnecessary conversions, unused imports)
• Formatting verified with cargo fmt

Startup Diagnostics Enhancement Copy Section

New Serve Command Output:

🚀 Shimmy v1.7.0
🖥️  Backend: CUDA (GPU acceleration enabled)
🧠 MoE: CPU offload enabled (all experts)
📚 Models: 0 available
🌐 Starting server on 127.0.0.1:11435
📚 Models: 3 available
✅ Ready to serve requests
   • POST /api/generate (streaming + non-streaming)
   • GET /health (health check + metrics)
   • GET /v1/models (OpenAI-compatible)

Benefits:

• Immediate configuration feedback before first request
• GPU backend visibility (CPU/CUDA/Vulkan/OpenCL/auto-detected)
• MoE config shown at startup (when feature enabled)
• Model discovery progress (shows count twice: before/after scan)
• Error prevention (wrong config visible instantly)

Implementation:

• Zero performance overhead (<1ms)
• Works with RUST_LOG=off (uses stdout)
• Emoji markers for visual scanning
• 7 new unit tests, 204/204 bin tests passing

🐛 Critical Fixes Copy Chapter

Issue #85: Template Compilation Errors in crates.io Installation Copy Section

Problem: cargo install shimmy failed with template generation errors

• Nested tokio runtime panics during template file generation
• Async functions causing runtime conflicts

Solution:

• Remove async from template generation functions (they were synchronous)
• Eliminate nested tokio runtime causing panics
• Template files properly included in package, runtime issue was the blocker

Verification:

• Fresh install from crates.io: cargo install shimmy --features llama
• Template generation working correctly
• All integration tests passing

Issue #84: Startup Diagnostics Implementation Copy Section

Problem: No visibility into shimmy configuration until first request fails

• Wrong GPU backend only discovered after server starts
• Missing MoE config not shown until generation attempted
• No model count feedback during discovery

Solution: Added comprehensive startup diagnostics (see Technical Details above)

Testing:

• Manual testing on Windows with CUDA
• 7 new unit tests for diagnostic output formatting
• Regression tests: 204/204 bin tests, 295/295 lib tests passing

MoE Config Application Fix Copy Section

Problem: --cpu-moe flags ignored when auto-registering discovered models in serve command

Root Cause: Serve command created new LlamaEngine without MoE configuration

Solution:

• Apply MoE config to both initial engine AND enhanced_engine
• Ensure expert tensor offloading works in serve mode
• Verified: 144 expert tensors offloaded to CPU with GPT-OSS 20B model

Testing:

• Manual verification with GPT-OSS 20B (144 experts offloaded)
• Phi-3.5-MoE and DeepSeek-16B validation
• All serve mode configurations tested

📚 Documentation Updates Copy Chapter

HuggingFace Model Cards Copy Section

Professional Standards:

• All 6 model cards follow bartowski/Microsoft style
• Real performance benchmarks (not estimates)
• Comprehensive YAML metadata (language, license, tags, base_model, pipeline_tag)
• Usage examples with shimmy CLI integration
• Quantization methodology and technical specifications

Metadata Audit & Corrections:

• Fixed "empty or missing yaml metadata" warnings
• Corrected DeepSeek base_model references (was pointing to wrong model)
• All repos rendering correctly on HuggingFace
• Proper tag relationships (GGUF, quantized, transformers)

Internal Documentation Organization Copy Section

Moved to docs/internal/:

• EXECUTION-PLAN-QUANTIZATION-TO-HF.md
• MODEL-CARD-PLAN.md
• MOE-TESTING-STATUS.md
• QUANTIZATION-PERFORMANCE-SUMMARY.md
• QUANTIZATION-STATUS-REPORT.md
• QUANTIZATION-TESTING-PLAN.md
• QUANTIZATION-UPLOAD-COMPLETE.md
• UPLOAD-COMMANDS.md
• HUGGINGFACE-AUDIT-2025-10-09.md

Benefits:

• Cleaner repository root
• Internal planning docs separated from user-facing documentation
• Historical context preserved for future development

🔮 What's Next Copy Chapter

Planned Enhancements Copy Section

• Additional quantization levels: Q3_K_M, Q5_K_M for quality/size balance
• More MoE models: Qwen-3-235B, Mixtral variants with CPU offloading
• Benchmark suite: Automated A/B testing framework for MoE configs
• Dynamic offloading: Runtime adjustment of expert tensor placement
• VRAM monitoring: Real-time VRAM usage tracking during inference

Community Contributions Copy Section

• Upstream PR pending: utilityai/llama-cpp-rs#839
• Testing feedback welcome on Issue #81
• Additional model requests via GitHub issues

📥 Installation Copy Chapter

From Source (Recommended for MoE Support) Copy Section

git clone https://github.com/Michael-A-Kuykendall/shimmy.git
cd shimmy
git checkout feat/moe-cpu-offload
cargo build --release --features llama
./target/release/shimmy --version

From crates.io (Standard Features) Copy Section

cargo install shimmy --features llama
shimmy --version

Quick Start with MoE Models Copy Section

# Example 1: Phi-3.5-MoE Q4_K_M (Recommended - Best Balance)
# Download the model (24GB)
wget https://huggingface.co/MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf/resolve/main/phi-3.5-moe-q4-k-m-cpu-offload.gguf \
  -O phi-3.5-moe-q4-k-m.gguf

# Run with CPU offloading
shimmy serve --cpu-moe --model phi-3.5-moe-q4-k-m.gguf

# Test generation
curl -X POST http://localhost:11435/api/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain quantum computing in simple terms", "max_tokens": 100}'

# Example 2: DeepSeek-16B Q2_K (Smallest - Maximum VRAM Savings)
# Download the model (6.3GB)
wget https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q2-k-cpu-offload-gguf/resolve/main/deepseek-moe-16b-q2-k-cpu-offload.gguf \
  -O deepseek-moe-16b-q2-k.gguf

# Run with CPU offloading
shimmy serve --cpu-moe --model deepseek-moe-16b-q2-k.gguf

# Example 3: Phi-3.5-MoE Q8_0 (Highest Quality - Near F16)
# Download the model (42GB)
wget https://huggingface.co/MikeKuykendall/phi-3.5-moe-q8-0-cpu-offload-gguf/resolve/main/phi-3.5-moe-q8-0-cpu-offload.gguf \
  -O phi-3.5-moe-q8-0.gguf

# Run with partial CPU offloading (64 layers)
shimmy serve --n-cpu-moe 64 --model phi-3.5-moe-q8-0.gguf

# Example 4: Using huggingface-cli (Alternative Download Method)
# Install: pip install huggingface-hub
huggingface-cli download MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf \
  phi-3.5-moe-q4-k-m-cpu-offload.gguf --local-dir ./models

shimmy serve --cpu-moe --model ./models/phi-3.5-moe-q4-k-m-cpu-offload.gguf

Quantization Selection Guide Copy Section

Choose Q2_K if:

• You have very limited disk space (<10GB available)
• You're doing rapid prototyping/testing
• Quality is less critical than VRAM savings
• You want the absolute smallest model size

Choose Q4_K_M if (RECOMMENDED):

• You want the best balance of quality and size
• You're deploying to production
• You need reliable performance across diverse tasks
• You have 12-30GB disk space available

Choose Q8_0 if:

• You need maximum quality (virtually identical to F16)
• You have sufficient disk space (17-42GB)
• You're doing critical work requiring best possible output
• You can afford slightly larger VRAM usage

🙏 Credits Copy Chapter

Special Thanks:

• @razvanab for suggesting MoE CPU offloading in Issue #81 - this entire release exists because of your feature request! 🎉
• Lambda Labs for providing GH200 GPU infrastructure for comprehensive testing
• llama.cpp team for the upstream MoE offloading implementation
• bartowski for setting the standard with professional HuggingFace model cards

Contributors:

• Michael A. Kuykendall (@Michael-A-Kuykendall) - Lead development, quantization, testing
• Claude Code (Anthropic) - Code refactoring assistance, documentation

🔗 Related Links Copy Chapter

• Issue #81: Feature Request - MoE CPU Offloading
• Issue #84: Startup Diagnostics
• Issue #85: Template Compilation Fix
• PR #839: llama-cpp-rs CUDA stdbool Fix
• HuggingFace Models: MikeKuykendall Profile
• Previous Release: v1.6.0 Release Notes

📊 Detailed Changelog Copy Chapter

New Features Copy Section

• --cpu-moe flag for full MoE CPU offloading
• --n-cpu-moe N flag for partial MoE CPU offloading
• Startup diagnostics with GPU backend and MoE config visibility
• 6 quantized MoE models on HuggingFace with professional documentation

Bug Fixes Copy Section

• Fixed --cpu-moe flags being ignored in serve command
• Resolved template compilation errors in crates.io installation
• Fixed ANSI color output (respects NO_COLOR and TERM env vars)
• Corrected HuggingFace model card metadata (YAML, base_model references)

Code Quality Copy Section

• Renamed 22 Java-style getters to Rust conventions (I2 pattern)
• Fixed 14 production unwraps with proper error handling (N5 pattern)
• Converted 16+ string errors to typed ShimmyError (A3_stringly pattern)
• Achieved zero compiler warnings and clean clippy output

Documentation Copy Section

• 6 professional HuggingFace model cards with real benchmarks
• Organized 9 internal planning docs into docs/internal/
• Created comprehensive v1.7.0 release notes
• Updated copilot instructions with audit progress

Testing Copy Section

• 36/36 quantization baseline tests passing (N=3 statistical runs)
• 295/295 unit tests passing
• 204/204 bin tests passing
• Validated on Lambda Cloud GH200 (96GB VRAM, 72 cores)

Infrastructure Copy Section

• Lambda Cloud GH200 testing environment
• HuggingFace integration for model distribution
• Custom llama-cpp-rs fork with MoE bindings
• Cargo dependency override for upstream contributions

Full Changelog: https://github.com/Michael-A-Kuykendall/shimmy/compare/v1.6.0...feat/moe-cpu-offload

File: docs/releases/RELEASE_PREP_V1.7.2.md Copy Chapter

Release Preparation Report - v1.7.2

Generated: October 9, 2025
Branch: feature/mlx-native-support
Target Version: v1.7.2
Previous Version: v1.7.1

✅ Merge Status Copy Chapter

PR #97 Merged Successfully Copy Section

• PR: feat(moe): complete v1.7.0 MOE CPU offloading implementation
• Commits: 8 squashed commits
• Status: ✅ Merged into main at commit 67bb6af
• Content: Complete MOE CPU offloading with 3 validated models

✅ MOE Work Integrated into MLX Branch Copy Section

• Merge Commit: c488b59
• Branch State: feature/mlx-native-support now has both MLX + MOE work
• Commits Ahead: 9 commits ahead of origin/main

📦 What's New in v1.7.2 Copy Chapter

1. MOE CPU Offloading (from PR #97) Copy Section

• CLI Flags:
  • --cpu-moe: Offload ALL expert tensors to CPU
  • --n-cpu-moe N: Offload first N expert layers to CPU
• Performance Metrics:
  • GPT-OSS 20B: 71.5% VRAM reduction, 6.9x speed penalty
  • Phi-3.5-MoE 42B: 99.9% VRAM reduction, 2.5x speed penalty
  • DeepSeek MoE 16B: 99.9% VRAM reduction, 4.6x speed penalty
• Testing: 36 test result files, N=3 statistical validation
• Documentation: Complete whitepapers, technical reports, model cards

2. MLX Apple Silicon Support (new) Copy Section

• Platform: Native Apple Silicon (M-series) support
• Backend: MLX engine for Metal GPU acceleration
• Feature Flag: --features mlx and apple feature set
• Files:
  • src/engine/mlx.rs: MLX engine implementation
  • .github/workflows/mlx-apple-silicon.yml: CI workflow (✅ passing)
  • tests/mlx_support_regression_test.rs: Regression tests
  • MLX_IMPLEMENTATION_PLAN.md: Implementation documentation
• Status: ✅ Workflow passing on GitHub Actions (macos-14 runner)
• Fix: Resolved --bin shimmy binary ambiguity issue

3. Infrastructure Improvements Copy Section

• Pre-commit hooks disabled (preventing workflow friction)
• Release gates system integrated
• Comprehensive regression test suite
• GitHub Actions workflow improvements

🧪 Release Gates Checklist Copy Chapter

Pre-Release Validation Copy Section

Compilation & Build

• cargo build --release (no warnings)
• cargo build --release --all-features (all features compile)
• cargo build --release --no-default-features --features llama (llama only)
• cargo build --release --no-default-features --features mlx (MLX only)
• cargo build --release --no-default-features --features apple (Apple set)
• cargo build --release --no-default-features --features gpu (Windows GPU)

Testing

• cargo test --all-features (all tests pass)
• cargo test --features mlx mlx (MLX-specific tests)
• tests/mlx_support_regression_test.rs (Issue #68 regression)
• tests/packaging_regression_test.rs (crates.io package)
• tests/template_compilation_regression_test.rs (template files)
• tests/apple_silicon_detection_test.rs (GPU detection)
• tests/release_gate_integration.rs (release gates)

Lint & Format

• cargo fmt -- --check (formatting)
• cargo clippy --all-features (no warnings)
• cargo deny check (license/dependency check)

Documentation

• README.md updated with new features
• CHANGELOG.md has v1.7.2 entry
• All new features documented in docs/
• Model cards updated (if applicable)

Regression Tests (User-Reported Issues)

• Issue #68: MLX support in macOS binaries
• Issue #80: LLM model filtering in discovery
• Issue #81: MOE CPU offloading
• Issue #83: Template compilation in crates.io
• Issue #87: Apple Silicon GPU detection
• Issue #92: Bind address 'auto' panic

GitHub Actions

• CI workflow passing
• MLX Apple Silicon workflow passing
• Release workflow configured
• All checks green before merge

🔍 Known Issues & Risks Copy Chapter

Resolved Copy Section

• ✅ Pre-commit hooks causing merge friction → Disabled
• ✅ MLX workflow binary ambiguity → Fixed with --bin shimmy
• ✅ PR #97 merge conflicts → Resolved and merged

Outstanding Copy Section

• ⚠️ Need to verify no compilation warnings
• ⚠️ Need to run full regression test suite
• ⚠️ Need to update version number in Cargo.toml (currently 1.7.1)

📝 Release Steps Copy Chapter

1. Pre-Release Testing (NOW) Copy Section

# Build all feature combinations
cargo build --release --all-features
cargo build --release --no-default-features --features apple
cargo build --release --no-default-features --features gpu

# Run full test suite
cargo test --all-features
cargo test --features mlx mlx

# Check for warnings
cargo clippy --all-features -- -D warnings

# Format check
cargo fmt -- --check

2. Version Bump Copy Section

# Update Cargo.toml version to 1.7.2
# Update CHANGELOG.md with release notes
# Commit version bump
git commit -am "chore: bump version to 1.7.2"

3. Merge to Main Copy Section

git checkout main
git merge feature/mlx-native-support
git push origin main

4. Tag Release Copy Section

git tag -a v1.7.2 -m "Release v1.7.2: MLX Apple Silicon + MOE CPU Offloading"
git push origin v1.7.2

5. GitHub Release Copy Section

• Create release from tag v1.7.2
• Upload release binaries (from CI)
• Include release notes from CHANGELOG

📊 Commits Since v1.7.1 Copy Chapter

c488b59 merge: integrate MOE CPU offloading from main into MLX branch
88e1878 chore: stage MLX branch changes before merge
29c1e40 fix(mlx): specify --bin shimmy in workflow to resolve binary ambiguity
a352af6 fix MLX workflow Python environment issue
a41631b remove pre-commit hooks causing issues
6ca1d00 fix: make release gates OS-agnostic and fix fragile pre-commit hooks
0b0e8e2 feat(mlx): implement native Apple Silicon MLX support with pre-commit quality gates
67bb6af feat(moe): complete v1.7.0 MOE CPU offloading implementation (#97)

Total: 8 new commits for v1.7.2

🎯 Next Actions Copy Chapter

1. RUN RELEASE GATES - Execute full test suite
2. CHECK COMPILATION WARNINGS - Ensure clean build
3. UPDATE VERSION - Bump to 1.7.2 in Cargo.toml
4. UPDATE CHANGELOG - Document all changes
5. MERGE TO MAIN - Final integration
6. TAG & RELEASE - Create v1.7.2 release

Status: Ready for release gates testing
Blocker: None (all merges complete)
Risk Level: Low (both features tested independently)

File: docs/releases/release-notes-v1.7.0.md Copy Chapter

🚀 Shimmy v1.7.0: The MoE Revolution is Here!

💥 BREAKTHROUGH: Run 42B+ Models on Consumer Hardware Copy Chapter

Shimmy v1.7.0 unleashes the MoE (Mixture of Experts) CPU Offloading Revolution - enabling massive expert models to run on everyday GPUs with up to 99.9% VRAM reduction.

🔥 What's New & Game-Changing Copy Chapter

⚡ MoE CPU Offloading Technology Copy Section

Transform impossible into possible:
<<<<<<< HEAD

• **`--cpu-moe`**: Automatically offload MoE layers to CPU

• --cpu-moe: Automatically offload MoE layers to CPU

main

• --n-cpu-moe N: Fine-tune performance with precise layer control
• Massive Memory Savings: 15GB models → 4GB VRAM usage
• Enterprise Ready: Deploy 42B parameter models on 8GB consumer cards

📊 Real Performance Gains (Validated) Copy Section

• GPT-OSS 20B: 71.5% VRAM reduction (15GB → 4.3GB actual measurement)
• Phi-3.5-MoE 42B: Runs on consumer hardware for the first time
• DeepSeek 16B: Intelligent CPU-GPU hybrid execution
• Smart Tradeoffs: Accept 2-7x slower inference for 10-100x memory savings

🛠️ Technical Excellence Copy Section

• First-Class Rust: Enhanced llama.cpp bindings with MoE support
• Cross-Platform: Windows MSVC CUDA, macOS ARM64 Metal, Linux x86_64/ARM64
• Production Tested: 295/295 tests passing, comprehensive validation pipeline
• Still Tiny: Sub-5MB binary maintains legendary efficiency

🎯 Use Cases Unlocked Copy Chapter

🏢 Enterprise Deployment Copy Section

• Cost Revolution: Run large models without GPU farm investments
  <<<<<<< HEAD
• Scalable AI: Deploy expert models on existing infrastructure
• Flexible Performance: Balance speed vs. memory for any workload
• On-Premises Ready: Keep sensitive data in-house with minimal hardware

🔬 Research & Development Copy Section

=======

• Scalable AI: Deploy expert models on existing infrastructure
• Flexible Performance: Balance speed vs. memory for any workload
• On-Premises Ready: Keep sensitive data in-house with minimal hardware

🔬 Research & Development Copy Section

main

• Democratized Access: Test large models on developer laptops
• Rapid Iteration: Prototype MoE architectures efficiently
• Educational Power: Advanced AI models accessible to everyone
• Hybrid Intelligence: Combine CPU and GPU resources intelligently

🚀 Quick Start Your MoE Journey Copy Chapter

Installation Options Copy Section

# Install from crates.io (LIVE NOW!)
cargo install shimmy

# Or grab platform binaries below ⬇️

🤖 Ready-to-Use MoE Models Copy Section

Curated collection on HuggingFace - optimized for CPU offloading:

🥇 **Recommended Starting Points**

# Download and run Phi-3.5-MoE 42B (Q4 K-M) - Best balance of quality/performance
huggingface-cli download MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf
./shimmy serve --cpu-moe --model-path phi-3.5-moe-q4-k-m.gguf

# Or DeepSeek-MoE 16B (Q4 K-M) - Faster alternative
huggingface-cli download MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf
./shimmy serve --cpu-moe --model-path deepseek-moe-16b-q4-k-m.gguf

📊 **Complete Model Collection**

🎯 **Model Selection Guide**

• 🥇 First Time? → Phi-3.5-MoE Q4 K-M (best balance)
  <<<<<<< HEAD

• **💪 High-End GPU (8GB+)?** → Phi-3.5-MoE Q8.0 (maximum quality)

• 💪 High-End GPU (8GB+)? → Phi-3.5-MoE Q8.0 (maximum quality)

main

• 💻 Limited VRAM (4GB)? → DeepSeek-MoE Q4 K-M (budget friendly)
• ⚡ Speed Critical? → DeepSeek-MoE Q2 K (blazing fast)
• 🔬 Research/Validation? → GPT-OSS 21B (proven baseline)

⚡ Launch Commands Copy Section

# Enable MoE CPU offloading magic
./shimmy serve --cpu-moe --port 11435 --model-path your-model.gguf

# Fine-tune performance for your hardware
./shimmy serve --n-cpu-moe 8 --port 11435 --model-path your-model.gguf

# Standard OpenAI-compatible API - zero changes to your code!
curl -X POST http://localhost:11435/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "your-model", "prompt": "Explain quantum computing in simple terms"}'

📦 Cross-Platform Binaries Copy Chapter

Choose your platform and start the revolution:

All binaries include zero Python dependencies and native SafeTensors support.

🌟 Why This Changes Everything Copy Chapter

Before Shimmy v1.7.0: "I need a $10,000 GPU to run expert models"

After Shimmy v1.7.0: "I'm running 42B models on my gaming laptop"

This isn't just an update - it's sustainable AI democratization. Organizations can now:

• ✅ Deploy cutting-edge models without infrastructure overhaul
• ✅ Experiment with state-of-the-art architectures on existing hardware
• ✅ Scale AI capabilities based on actual needs, not hardware limits
• ✅ Maintain complete data sovereignty with on-premises deployment

📈 Validated & Transparent Copy Chapter

• Multi-Model Testing: 3 models validated across all platforms
  <<<<<<< HEAD

• **Real Baselines**: Controlled A/B testing with actual measurements

• Real Baselines: Controlled A/B testing with actual measurements

main

• Production Quality: Comprehensive release gate system
• Open Development: Technical validation report available

🤝 Join the Revolution Copy Chapter

• 🚀 Start Now: cargo install shimmy
• 📚 Learn More: Technical Documentation
• 🐛 Report Issues: GitHub Issues
• 🔗 Upstream: Supporting llama-cpp-rs PR #839

<<<<<<< HEAD **Ready to revolutionize your AI deployment?** The future of efficient model serving is here. Download Shimmy v1.7.0 and experience the MoE revolution! 🚀

Ready to revolutionize your AI deployment? The future of efficient model serving is here. Download Shimmy v1.7.0 and experience the MoE revolution! 🚀

main

File: docs/zh-CN/CHAT_TEMPLATES.md Copy Chapter

对话模板参考

本文档说明 Shimmy 如何将对话消息格式化为模型输入，介绍三大模板族的格式细节，并解释自动识别逻辑——帮助你在输出异常时快速定位问题。

为什么模板很重要 Copy Chapter

大型语言模型在训练时使用了特定的提示格式。当输入格式与训练时不匹配，模型会产生奇怪的行为——输出提前截断、重复原始输入内容，或者持续生成无意义的文本。

这是实际工作中最常见的静默失败场景。模型运行正常，API 返回 200，但输出明显不对。

三大模板族 Copy Chapter

1. ChatML Copy Section

使用此模板的模型：TinyLlama、Phi-2、StarCoder2（已通过验证），以及大多数未被识别为 Llama-3 的模型（默认回退）。

格式：

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

停止 token：<|im_end|>、<|im_start|>

完整示例：

<|im_start|>system
你是一个有帮助的助手。<|im_end|>
<|im_start|>user
中国的首都是哪里？<|im_end|>
<|im_start|>assistant
中国的首都是北京。<|im_end|>

2. Llama-3 Copy Section

使用此模板的模型：Llama-3.2-1B-Instruct、Llama-3.2-3B-Instruct，以及所有 Meta Llama-3 系列。

格式：

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>

{user_message}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

停止 token：<|eot_id|>、<|end_of_text|>

关键细节：<|end_header_id|> 后面有两个换行符，这是正确格式的必要组成部分。停止 token 必须是 <|eot_id|>——不是 \n，不是 ###，不是 <|im_end|>。

3. OpenChat Copy Section

使用此模板的模型：专门采用 OpenChat 训练格式的模型。

格式：

GPT4 Correct User: {user_message}<|end_of_turn|>GPT4 Correct Assistant:

停止 token：此模板无额外停止 token（使用模型默认的 EOS token）。

自动识别逻辑 Copy Chapter

Shimmy 根据模型文件名自动推断模板类型，规则如下（按优先级排序）：

实现细节：识别逻辑在 infer_template() 函数中，对模型名称进行小写比较，因此大小写不影响识别结果。

当自动识别失败时：可通过 API 请求中的 template 字段手动指定：

{
  "model": "local",
  "template": "chatml",
  "messages": [...]
}

可选值："chatml"、"llama3"、"openchat"。

各模型模板对照 Copy Chapter

* StarCoder2 是代码补全模型，不是指令模型。虽然可以接受 ChatML 格式输入，但建议直接使用 /v1/completions 接口效果更好。

补全接口 vs 对话接口 Copy Chapter

Shimmy 同时支持两个接口：

/v1/chat/completions（对话格式）：

{
  "model": "local",
  "messages": [
    {"role": "system", "content": "你是代码助手"},
    {"role": "user", "content": "写一个快速排序"}
  ]
}

/v1/completions（原始补全格式，适合代码模型）：

{
  "model": "local",
  "prompt": "def quicksort(arr):",
  "max_tokens": 200,
  "temperature": 0.2
}

对于 StarCoder2 和 GPT-2，优先使用 /v1/completions——它们在补全任务上的效果远好于模拟对话格式。

自定义停止 Token Copy Chapter

通过 API 可以添加额外的停止条件：

{
  "model": "local",
  "messages": [...],
  "stop": ["###", "---", "\n\n"]
}

注意事项：

• 停止 token 必须是词表中的单个 token，不能是多 token 序列
• 字符串 "###" 如果对应多个 token 则不会生效
• 建议使用模型原生的停止 token（如上表中的格式）

调试模板问题 Copy Chapter

常见症状与原因：

# 查看原始请求的模板渲染结果
RUST_LOG=debug shimmy serve 2>&1 | grep -i template

延伸阅读 Copy Chapter

• 故障排查指南 — 生成质量问题的系统排查
• 量化格式详解 — 模型格式与兼容性
• GPU 推理管线 — Token 采样链路

💝 如果 Shimmy 对您有帮助，欢迎赞助支持——所有款项 100% 用于保持项目永久免费。

File: docs/zh-CN/EXTENDED_CONTEXT.md Copy Chapter

扩展上下文窗口

本文档介绍 Airframe 如何通过 YaRN RoPE 缩放突破模型原生上下文长度限制——包含数学原理、各模型的显存计算方法，以及在不同硬件上的实际配置建议。

什么是上下文窗口？ Copy Chapter

上下文窗口是模型在生成响应时能够"看到"的最大 token 数量。超出这个限制，模型要么截断输入，要么产生无意义的输出——它根本无法引用窗口之外的内容。

常见模型的原生上下文长度：

RoPE 位置编码原理 Copy Chapter

Transformer 模型通过**旋转位置编码（RoPE）**感知 token 在序列中的位置。对于序列位置 t、注意力头维度中的第 i 个维度：

$$\theta(t, i) = \frac{t}{\text{base}^{2i / d_{\text{head}}}}$$

其中 base 通常为 10000，d_head 为注意力头的维度。

每个 Q 和 K 向量按此角度旋转，注意力得分的计算自然地编码了 token 之间的相对距离。

问题所在：模型训练时只见过 t ≤ native_ctx 的位置。当 t 超出训练范围，旋转角度进入模型从未见过的区域，输出质量迅速下降。

YaRN：上下文长度扩展方案 Copy Chapter

**YaRN（Yet another RoPE extensioN）**通过对位置序列进行缩放，将超出范围的位置映射回模型熟悉的区域：

$$\theta_{\text{YaRN}}(t, i) = \frac{t / s}{\text{base}^{2i / d_{\text{head}}}}$$

缩放因子 s 的计算公式：

$$s = \frac{\text{max_ctx}}{\text{native_ctx}}$$

触发条件：当 SHIMMY_MAX_CTX > native_ctx 时，Airframe 自动启用 YaRN。无需手动配置——设置环境变量即可：

SHIMMY_MAX_CTX=8192 ./shimmy serve

若 SHIMMY_MAX_CTX ≤ native_ctx，RoPE 按原始参数运行，不引入任何额外开销。

显存（VRAM）计算 Copy Chapter

KV 缓存是上下文扩展的显存瓶颈。计算公式：

$$\text{KV 缓存（字节）} = n_{\text{layers}} \times n_{\text{kv_heads}} \times d_{\text{head}} \times \text{max_ctx} \times 2 \times 4$$

各模型实际用量：

TinyLlama 1.1B Copy Section

（22 层，4 个 KV 头，head_dim = 64）

Llama-3.2-1B Copy Section

（16 层，8 个 KV 头，head_dim = 64）

Llama-3.2-3B Copy Section

（28 层，8 个 KV 头，head_dim = 128）

注意：上述数值仅为 KV 缓存。总显存需求还需加上模型权重本身。

按显存容量的推荐配置 Copy Chapter

YaRN 质量说明 Copy Chapter

YaRN 是一种近似方法，不能完全还原模型在超长上下文上的性能：

• 1× 以内（max_ctx ≤ native_ctx）：完整训练精度，无损失
• 1–2×（略超原生上下文）：质量轻微下降，日常任务基本无感
• 2–4×：远端内容的注意力质量逐渐下降，长文档理解可能受影响
• 4× 以上：输出质量显著退化，仅适合实验性用途

对于 TinyLlama（原生 2K），SHIMMY_MAX_CTX=8192 是一个合理的实用上限。

针脚测试（Needle-in-a-Haystack）基准结果 Copy Chapter

以下为 RTX 3060 12GB 上的实测结果（TinyLlama 1.1B Q4_0）：

"注入深度"指目标信息在整个上下文中的位置百分比。头部和尾部内容的检索准确率通常高于中间位置。

环境变量配置 Copy Chapter

延伸阅读 Copy Chapter

• 量化格式详解 — 压缩格式与显存占用
• 故障排查指南 — 上下文溢出与 OOM 错误处理
• GPU 推理管线 — KV 缓存在 GPU 内存中的布局

💝 如果 Shimmy 对您有帮助，欢迎赞助支持——所有款项 100% 用于保持项目永久免费。

File: docs/zh-CN/GPU_PIPELINE.md Copy Chapter

GPU 推理管线

本文档深入解析 Airframe 在 GPU 上运行 Transformer 推理的完整过程——着色器调度架构、无绑定资源模型、KV 缓存管理，以及采样链路。

适合阅读对象：准备修改着色器的贡献者、排查 GPU 层面故障的开发者，以及希望理解底层机制的用户。

架构总览 Copy Chapter

HTTP 请求抵达服务器
       │
       ▼
shimmy openai_compat 层
  - 解析 JSON 请求
  - 应用对话模板 → 提示词字符串
  - 构建 SamplingParams（温度、top_p、惩罚系数、停止 token）
       │
       ▼
airframe runtime::gpu::GpuRuntime::generate()
  - Tokenize（分词）提示词
  - 预填充阶段：处理提示词 token → 填充 KV 缓存
  - 解码阶段：逐 token 自回归生成
  - 反 Tokenize → 响应字符串
       │
       ▼
shimmy → HTTP 响应

无绑定资源模型 Copy Chapter

传统 WebGPU 需要将每个缓冲区单独绑定到绑定槽。大型 Transformer 模型有数千个张量——逐一绑定会触及 WebGPU 的绑定数量上限，并在每次调度时产生巨大开销。

Airframe 采用无绑定（Bindless）设计：每层的权重张量打包进一个大型存储缓冲区，WGSL 着色器通过推送常量（WebGPU 中以 uniform 缓冲区实现）中的字节偏移量来索引：

层缓冲区布局（每个 Transformer 层一个）：
┌────────────────────────────────────────────────────────────┐
│  attn_norm_weight  │  attn_q.weight  │  attn_k.weight  │  │
│  attn_v.weight  │  attn_o.weight  │  ffn_norm_weight  │  │
│  ffn_gate.weight  │  ffn_up.weight  │  ffn_down.weight  │  │
└────────────────────────────────────────────────────────────┘
        ↑
  每个张量区域以 GGUF 量化格式存储。
  着色器通过元数据缓冲区获取各区域的字节偏移量。

调试提示：如遇缓冲区绑定上限或绑定组创建失败的错误，问题通常不在无绑定权重缓冲区本身，而是 KV 缓存或激活缓冲区（这些仍独立绑定）。

预填充阶段 — 分块处理 Copy Chapter

预填充阶段一次性处理完整的输入提示词，填充 KV 缓存。这是单次请求中 GPU 计算最密集的部分。

Airframe 将长提示词切分为 512 token 的分块，以避免 GPU 命令编码器超时：

提示词 = 2048 token
          │
 ┌────────▼────────┐
 │  分块 1：512    │ → KV 缓存 0..511   ← 前向传播，写入 KV
 │  分块 2：512    │ → KV 缓存 512..1023
 │  分块 3：512    │ → KV 缓存 1024..1535
 │  分块 4：512    │ → KV 缓存 1536..2047
 └─────────────────┘
          │
  只使用最后一个分块的 logits 来生成第一个采样 token。
  所有分块的 KV 状态保留在 GPU 缓冲区中。

为什么是 512？ WebGPU 对 GPU 命令执行时间有上限限制，超时会被操作系统终止。512 token 的分块在所有已测试硬件上（RTX 3060 到集成显卡）都能安全完成。

调试跟踪：设置 AIRFRAME_TRACE_PREFILL_CHUNKS=1 可以记录分块边界和处理时间。

解码阶段 — 自回归生成 Copy Chapter

预填充完成后，解码器逐 token 生成输出。每个解码步骤：

1. 嵌入：通过嵌入查找表将上一个 token 转换为向量
2. 前向传播：穿过所有 Transformer 层（使用 KV 缓存）
   • 注意力机制使用 KV 缓存——当前 Q 关注所有过去的 K/V 对
   • 每步只需计算位置 t 的新 Q/K/V 向量
3. 输出映射：将 LM Head 应用到激活值，得到词表上的 logits
4. 采样：从 logits 分布中选择下一个 token
5. 检查停止条件：EOS token、额外停止 token、max_tokens 上限
6. 以新 token 返回第 1 步重复

每个解码步骤需要完整的一次模型前向传播。对于 22 层的 TinyLlama，每生成一个 token 就要执行 22 次注意力计算 + 22 次 FFN 计算。

解码是内存带宽瓶颈，不是算力瓶颈。速度限制来自从显存读取权重张量，而非浮点计算能力。显存带宽更大的 GPU 生成速度更快。

Transformer 层计算（WGSL 着色器解析） Copy Chapter

每个 Transformer 层按以下顺序执行：

输入激活值（形状：[seq_len, n_embd]）
  │
  ├── RMS Norm → normalized_x（稳定数值）
  │
  ├── Q 投影：normalized_x × attn_q.weight → Q [seq, n_heads, head_dim]
  ├── K 投影：normalized_x × attn_k.weight → K [seq, n_kv_heads, head_dim]
  ├── V 投影：normalized_x × attn_v.weight → V [seq, n_kv_heads, head_dim]
  │
  ├── RoPE：对 Q 和 K 应用旋转位置编码
  │     θ(t, i) = (t / yarn_scale) / base^(2i / head_dim)
  │     YaRN：当 ctx > native_ctx 时 yarn_scale = max_ctx / native_ctx
  │
  ├── 将 K、V 写入位置 t 的 KV 缓存
  │
  ├── 注意力得分：Q × Kᵀ / √head_dim → scores [seq, n_heads, seq]
  ├── Softmax（在 seq 维度） → weights
  ├── 加权求和：weights × V → attn_out [seq, n_heads, head_dim]
  │
  ├── 输出投影：attn_out × attn_o.weight → residual_add
  │
  ├── 残差连接：x = x + residual_add
  │
  ├── RMS Norm → normalized_for_ffn
  │
  ├── FFN gate：normalized_for_ffn × ffn_gate.weight → gate
  ├── FFN up：  normalized_for_ffn × ffn_up.weight   → up
  ├── SwiGLU 激活：gate × sigmoid(gate) × up
  ├── FFN down：swiglu_out × ffn_down.weight → ffn_residual
  │
  └── 残差连接：x = x + ffn_residual → 输出激活值

反量化在矩阵乘法内部实时进行：每个矩阵乘法着色器在读取量化权重块时立即解码，然后与激活值相乘，结果直接累加到输出，不产生额外的显存分配。

KV 缓存 Copy Chapter

KV 缓存存储过去的键值对，避免在每个解码步骤重新计算。

缓冲区布局：

key_cache:   [n_layers][n_kv_heads][max_ctx][head_dim]  f32
value_cache: [n_layers][n_kv_heads][max_ctx][head_dim]  f32

以 F32 格式存储为扁平 GPU 缓冲区。总大小：

key_cache_bytes = n_layers × n_kv_heads × max_ctx × head_dim × 4
total = key_cache_bytes × 2

TinyLlama @ 2048 上下文：22 × 4 × 2048 × 64 × 2 × 4 ≈ 88 MB

写入：预填充和每个解码步骤中，当前位置的新 K/V 向量被写入 cache[layer][head][position]。

读取：注意力计算时，读回到 current_position 为止的完整 K/V 切片。

重置：每次请求之间缓存会被重置（清零所有位置）。Shimmy 是无状态的——不维护会话级别的 KV 缓存。

采样链路 Copy Chapter

前向传播产生词表上的 logits（长度为 vocab_size 的浮点向量）后，采样按以下顺序处理：

logits[vocab_size]
  │
  1. 重复惩罚（如果 repeat_penalty > 1.0）：
     对过去 N 个 token 中出现过的每个 token t：
       logits[t] /= repeat_penalty（提高重复的"代价"）
  │
  2. 温度缩放：
     logits[i] = logits[i] / temperature
     （temperature = 0.0 → 贪心；temperature → ∞ → 均匀随机）
  │
  3. Softmax：
     probs[i] = exp(logits[i]) / Σ exp(logits[j])
  │
  4. Top-p（核采样）：
     按概率降序排列。
     累计到总和 ≥ top_p 阈值为止。
     对保留的 token 重新归一化。
     从此压缩分布中采样。
  │
  5. 采样 → token_id

确定性输出（贪心解码）：temperature=0.0, top_p=1.0

创意生成：temperature=0.8, top_p=0.95

重复惩罚与 API 的 frequency_penalty / presence_penalty 字段的映射关系：

raw = max(frequency_penalty, presence_penalty)
如果 raw > 0.0：repeat_penalty = 1.0 + raw × 0.5

采样器在 CPU 上运行（logits 在前向传播后从 GPU 显存读回）。对于词表规模 32K 以内的模型，每 token 添加的延迟 < 1ms。

着色器调度模式 Copy Chapter

每次矩阵乘法是一个独立的计算调度。以 TinyLlama（22 层）单个解码步骤为例：

每 token 的大致调度次数：
  22 × (Q投影 + K投影 + V投影 + 注意力输出 + FFN_gate/up + FFN_down)
  ≈ 132 次调度
  + 22 次 RMSNorm + 22 次 RoPE + 22 次注意力得分 + 22 次 Softmax
  ≈ 共 220 次 GPU 调度/token

每次调度使用 (16, 16, 1) 的工作组，最大计算调用数为 256/工作组。命令编码器将所有调度打包为每 token 一次命令缓冲区提交——而不是每次调度单独提交。这对性能至关重要：每 token 一次 GPU 往返，而非 220 次。

调试调度：设置 SHIMMY_DEBUG_RAW=1 可记录管线各点的原始激活值（极度冗长——仅用于着色器调试）。

输出头 Copy Chapter

最终层产生形状为 [1, n_embd] 的激活值（最后位置）。将其与输出嵌入权重（output.weight，形状 [vocab_size, n_embd]）相乘，得到 logits。

输出权重通常是显存占用最大的张量：以词表 32K、维度 4096 的模型为例，output.weight 全精度下为 32768 × 4096 × 4 ≈ 512 MB。GGUF 的 Q6_K 格式下约 210 MB。

WebGPU 2 GB 缓冲区限制：输出权重必须放入单个 GPU 缓冲区。词表规模超大的模型（如 Gemma-2 词表 256K）的输出嵌入超过 2 GB，无法加载。这是已知限制——参见故障排查指南。

性能参考 Copy Chapter

Token 生成速度主要取决于：

1. GPU 显存带宽（不是算力 FLOPS）——权重读取是主要开销
2. 模型大小——参数越多，每 token 读取的字节越多
3. 上下文长度——解码阶段注意力得分按 O(n) 规模增长

RTX 3060 12GB 实测基准：

预填充通常比解码快 3-10 倍（每 token），因为注意力和 FFN 可以在序列维度上并行处理多个 token。

延伸阅读 Copy Chapter

• 量化格式详解 — 着色器内部反量化
• 扩展上下文窗口 — YaRN RoPE 缩放实现
• 故障排查指南 — GPU 故障调试

💝 如果 Shimmy 对您有帮助，欢迎赞助支持——所有款项 100% 用于保持项目永久免费。

--- METRICS ---

• Files Extracted: 11
• Estimated Token Budget: ~20320 tokens
• Recency Window: Active (< 180 days)
• Canonical Reference: https://codewiki.google/github.com/Michael-A-Kuykendall/shimmy